LWN: Comments on "Testing in Go: philosophy and tools"

Testing in Go: philosophy and tools

jezuch — Mon, 01 Jun 2020 05:30:42 +0000

> the pedestrian approach feels often dull, but it gets the job done in a way that you will understand in 5 years from now

In the case of tests written as a not-very-structured mass-of-code, I'm pretty sure I won't understand the intent behind the test after 5 days :) The most important thing is not the code itself - it's the intent behind the code. That's something imperative programming is inherently bad at - and the language makes it even worse when it doesn't support concept X.

Testing in Go: philosophy and tools

cpitrat — Fri, 29 May 2020 14:36:28 +0000

The reality is that these are excuses to argue that not having generics is not a big deal. Writing a clean assert_equal method (and friends) is hard because of lack of generics. But rather than admit it and say it's a nice thing we can't have yet, the Go community developed some kind of Stockholm syndrome and says that they don't want it anyway for weird reasons, including that repeating yourself is great.

Testing in Go: philosophy and tools

Cyberax — Thu, 28 May 2020 21:40:56 +0000

> the lack of concepts makes it easy to learn
It also forces people to use reams of repetitive code and/or search for third-party libraries to do the most basic things. And due to limitations of the language, libraries often can't provide experience that is seamless in other languages.

Go has done a lot of things right, but testing is most definitely not one of them.

Testing in Go: philosophy and tools

kunitz — Thu, 28 May 2020 21:26:06 +0000

The article does contain only one classical Go test, but it uses the typical pattern. A list of test cases is defined that are each tested in a loop. So the functionality tested is at the start of the loop.

I'm always amused when people point out that Go doesn't support concept X. That is actually the strength of the language. The lack of concepts makes it easy to learn, the pedestrian approach feels often dull, but it gets the job done in a way that you will understand in 5 years from now. The strength of Go is the idea of construction of complex things out of little dull things.

Testing in Go: philosophy and tools

kunitz — Thu, 28 May 2020 20:45:13 +0000

The relevant Go Proverb is: "A little copying is better than a little dependency." But it doesn't mean that people should copy bugs.

Testing in Go: philosophy and tools

jezuch — Thu, 28 May 2020 05:52:32 +0000

I guess Go's designers are not very familiar with modern philosophy of unit testing. Nowadays you're supposed to divide each test into three parts, which make it clear that this is the set up of the environment, this is the OUT (object under test), and this is where we make sure that the test results are correct. Frameworks help a lot with this (see e.g. Spock). If you don't do that, like in the mass-of-regular-code examples in the article, I have no idea whatsoever what is actually being tested (I have to infer it from the code, which is always error prone). The least you could do is to add comments marking the three sections (usually called "given", "when" and "then", taking inspiration from - but not really being - Behavior Driven Development).

Testing in Go: philosophy and tools

cpitrat — Wed, 27 May 2020 17:11:03 +0000

By not having a "domain specific language" for testing (this is what we call functions and these are part of the language too, provided by a library) it means that go tests are full of copy-pasted code with repetition.

Among go tests, I've seen multiple times ettor messages that were invalid because copied from the previous function (e.g saying no error when it checked that there was an error). Or inconsistent order for messages saying what it got vs. what it wants, or worse printing got <want>, want <got>.

Duplication sucks because of that. Go community thinks that WET (write everything twice) is better than DRY (don't repeat yourself) ignoring years of computer science experience and wisdom. Proof-reading tests in a large Go code base is a good way to understand why they're wrong.

Testing in Go: philosophy and tools

kokkoro — Wed, 27 May 2020 06:03:11 +0000

Defer doesn't work properly with parallel tests. If you do:


func TestFoo(t *testing.T) {
  f := NewFoo()
  defer f.Close()
  for _, c := testCases {
    c := c
    t.Run(c.name, func(t *testing.T) {
      t.Parallel()
      c.func(t, f)
    })
  }

then f.Close() will be called immediately, concurrent to all of the parallel subtests. https://github.com/golang/go/issues/17791

Testing in Go: philosophy and tools

tymonx — Wed, 27 May 2020 05:00:10 +0000

Recently I have been working on a project that allow to work on Go projects out-of-box using a Docker image: https://gitlab.com/tymonx/docker-go

It contains preinstalled tools for developing, mocking, formatting, linting, building, testing and documenting Go projects.

For mocking Go interfaces, I'm using the standard and classic mockgen tool in reflect mode. One of annoying thing about that tool is command line invocations. You manually provide a module name/path with interface names separate with comma.

I have scripted that to automatically and recursively detect all Go interfaces. It excludes _test.go files, mocks directories and the package main.

Go module detection is based on the go env GOMOD command invocation. All mocks are generated under the mocks directory for given Go package.

There are more features that can help. Like:
* Automatically generating JUnit test XML after tests
* Automatically generating Cobertura coverage XML after tests
* Validating Go coverage value threshold with colorization
* Colorizing Go coverage results and format nicely with the column tool
* Automatically generating static HTML coverage report with the go tool cover tool after tests

Testing in Go: philosophy and tools

cyphar — Tue, 26 May 2020 22:57:23 +0000

While it isn't ideal, I did manage to implement coverage testing of integration tests for a moderately sized project in Go -- I even wrote a blog post about it[1]. Yes, you do have some boilerplate to put inside a TestXxx() function in order to get it to work, but that's a fairly fixed cost.

[1]: https://www.cyphar.com/blog/post/20170412-golang-integrat...

Testing in Go: philosophy and tools

phlogistonjohn — Tue, 26 May 2020 22:55:40 +0000

One pet peeve I have with the testing regime in Go is that the tests require '*testing.T' which is a concrete type rather than an interface. I think it's totally fine that the built-in testing tool wants to be simple. However, by building the test framework on '*testing.T' its more strongly coupled to the testing package than it needs to be. If an interface had been used instead, other frameworks could wrap tests that are written to work with the testing package by default and do some pretty interesting things (IMO, of course). It's also a slight failure of this part of the Go standard library to practice what Go often preaches in regards to interfaces.

On a different note, I didn't quite see the rationale for Cleanup, given that Go has defer. The issue that ended up getting the feature added seems to be the clearest explanation of the why behind Cleanup: https://github.com/golang/go/issues/32111
Personally, I don't see it as all that compelling but not bad or anything. :-)

Testing in Go: philosophy and tools

phlogistonjohn — Tue, 26 May 2020 22:33:35 +0000

Here's an example of wrapping the main function in a test to get coverage.

I have to second Cyberax's comment. Last time I looked into this it quickly became too complicated to deal with, and I didn't need it very much and gave up. I too peeked at the code to implement this and it was too complex for me to follow (having been doing Go mostly full time for a year or so).

It's rather unfortunate that this isn't a feature of the compiler and say a helper module you can import into your own main. I lean heavily normal on the coverage feature for tests in a few of the projects I work on. I would love to have a simple, noninvasive way to generate coverage from any binary.

Testing in Go: philosophy and tools

benhoyt — Tue, 26 May 2020 19:19:41 +0000

Interesting. I'd love to know more about this; any links? Our team at work has recently started running integration tests using the regular testing package and it seems to work well, but these are fairly lightweight "PostgreSQL database only" integration tests, not "run all the services" integration tests.

Testing in Go: philosophy and tools

Cyberax — Tue, 26 May 2020 18:50:09 +0000

> Go test's -cover mode produces code coverage profiles that can be viewed as HTML
Go coverage analysis SUCKS. Seriously.

For example, it's impossible to do code coverage for integration tests or for manual tests. There are workarounds like writing a fake test that just runs the project's true "func main", but it's not enough for many reasons.

I've looked into trying to add ability to instrument non-test code, but the underlying rewriter is a freaking mess that is not modular and can't be easily extracted without major surjery.