Testing in Go: philosophy and tools
The Go programming language comes with tools for writing and running tests: the standard library's testing package, and the go test command to run test suites. Like the language itself, Go's philosophy for writing tests is minimalist: use the lightweight testing package along with helper functions written in plain Go. The idea is that tests are just code, and since a Go developer already knows how to write Go using its abstractions and types, there's no need to learn a quirky domain-specific language for writing tests.
The Go Programming Language by Brian Kernighan and Alan Donovan summarizes this philosophy. From chapter 11 on testing:
Many newcomers to Go are surprised by the minimalism of Go's testing framework. Other languages' frameworks provide mechanisms for identifying test functions (often using reflection or metadata), hooks for performing "setup" and "teardown" operations before and after the tests run, and libraries of utility functions for asserting common predicates, comparing values, formatting error messages, and aborting a failed test (often using exceptions). Although these mechanisms can make tests very concise, the resulting tests often seem like they are written in a foreign language.
To see this in practice, here's a simple test of the absolute value function Abs() using the testing package and plain Go:
func TestAbs(t *testing.T) { got := Abs(-1) if got != 1 { t.Errorf("Abs(-1) = %d; want 1", got) } }
Contrast that with the following version, written using the popular (though I would argue non-idiomatic) Ginkgo library that provides a means to write RSpec-style tests for Go:
Describe("Abs", func() { It("returns correct abs value for -1", func() { got := Abs(-1) Expect(got).To(Equal(1)) }) })
The functions Describe, Expect, etc, make the test "read like English", but means that there is suddenly a whole new sub-language to learn. The thinking of Go contributors such as Donovan is that there are already tools like == and != built into the language, so why is To(Equal(x)) needed?
That said, Go doesn't stop developers from using such libraries, so developers coming from other languages often find using them more familiar than vanilla testing. One relatively lightweight library is testify/assert, which adds common assertion functions like assert.Equal(), and testify/suite, which adds test-suite utilities like setup and teardown. The "Awesome Go" website provides an extensive list of such third-party packages.
One useful testing tool that's not part of the testing package is reflect.DeepEqual(), which is a standard library function that uses reflection to determine "deep equality", that is, equality after following pointers and recursing into maps, arrays, and so on. This is helpful when tests compare things like JSON objects or structs with pointers in them. Two libraries that build on this are Google's go-cmp package and Daniel Nichter's deep, which are like DeepEqual but produce a human-readable diff of what's not equal rather just returning a boolean. For example, here's a (deliberately broken) test of a MakeUsers() function using go-cmp:
func TestMakeUser(t *testing.T) { got := MakeUser("Bob Smith", "bobby@example.com", 42) want := &User{ Name: "Bob Smith", Email: "bob@example.com", Age: 42, } if diff := cmp.Diff(want, got); diff != "" { t.Errorf("MakeUser() mismatch (-want +got):\n%s", diff) } }
And the human-readable output is:
user_test.go:16: MakeUser() mismatch (-want +got): &main.User{ Name: "Bob Smith", - Email: "bob@example.com", + Email: "bobby@example.com", Age: 42, }
Built-in testing features
The built-in testing package contains various functions to log information and report failures, skip tests at runtime, or only run tests in "short" mode. Short mode provides a way to skip tests that are long running or have a lot of setup, which can be helpful during development. It is enabled using the -test.short command line argument.
Go's test runner executes tests sequentially by default, but there's an opt-in Parallel() function to allow running explicitly-marked tests at the same time across multiple cores.
In Go 1.14, the testing package added a Cleanup() function that registers a function to be called when the test completes. This is a built-in way to simplify teardown, for example to delete database tables after a test finishes:
func createDatabase(t *testing.T) { // ... code to create a test database t.Cleanup(func() { // ... code to delete the test database // runs when the test finishes (success or failure) }) } func TestFetchUser(t *testing.T) { createDatabase(t) // creates database and registers cleanup user, err := FetchUser("bob@example.com") if err != nil { t.Fatalf("error fetching user: %v", err) } expected := &User{"Bob Smith", "bob@example.com", 42} if !reflect.DeepEqual(user, expected) { t.Fatalf("expected user %v, got %v", expected, user) } }
Go 1.15 is adding a test
helper, TempDir(),
that creates (and cleans up) a temporary directory for the current
test. There's a high bar for adding to the testing package, but
Russ Cox on the core Go team gave
his approval for this addition:
"It seems like temporary directories do come up in
a large enough variety of tests to be part of testing proper.
"
Table-driven tests
A common idiom in Go to avoid repetition when testing various edge cases is called "table-driven tests". This technique iterates over the test cases in a "slice" (Go's term for a view into a resizable array), reporting any failures for each iteration:
func TestAbs(t *testing.T) { tests := []struct { input int expected int }{ {1, 1}, {0, 0}, {-1, 1}, {-maxInt, maxInt}, {maxInt, maxInt}, } for _, test := range tests { actual := Abs(test.input) if actual != test.expected { t.Errorf("Abs(%d) = %d; want %d", test.input, actual, test.expected) } } }
The t.Errorf() calls report the failure but do not stop the execution of the test, so multiple failures can be reported. This style of table-driven test is common throughout the standard library tests (for example, the fmt tests). Subtests, a feature introduced in Go 1.7, gives the ability to run individual sub-tests from the command line, as well as better control over failures and parallelism.
Mocks and interfaces
One of Go's well-known language features is its structurally-typed
interfaces,
sometimes referred to as "compile-time
duck typing". "Interfaces in Go provide a way to specify the
behavior of an object: if something can do this, then it can be used
here.
" Interfaces are important whenever there is a need to vary
behavior at runtime, which of course includes testing. For example, as Go
core contributor Andrew Gerrand said in the slides for his 2014 "Testing
Techniques" talk, a file-format parser should not have a
concrete file type passed in like this:
func Parse(f *os.File) error { ... }
Instead, Parse() should simply take a small interface that only implements the functionality needed. In cases like this, the ubiquitous io.Reader is a good choice:
func Parse(r io.Reader) error { ... }
That way, the parser can be fed anything that implements io.Reader, which includes files, string buffers, and network connections. It also makes it much easier to test (probably using a strings.Reader).
If the tests only use a small part of a large interface, for example one method from a multi-method API, a new struct type can be created that embeds the interface to fulfill the API contract, and only overrides the method being called. A full example of this technique is shown in this Go Playground code.
There are various third party tools, such as GoMock and mockery, that autogenerate mock code from interface definitions. However, Gerrand prefers hand-written fakes:
[mocking libraries like gomock] are fine, but I find that on balance the hand-written fakes tend be easier to reason about and clearer to see what's going on, but I'm not an enterprise Go programmer so maybe people do need that so I don't know, but that's my advice.
Testable examples
Go's package documentation is generated from comments in the source code. Unlike Javadoc or C#'s documentation system, which make heavy use of markup in code comments, Go's approach is that comments in source code should still be readable in the source, and not sprinkled with markup.
It takes a similar approach with documentation examples: these are runnable code snippets that are automatically executed when the tests are run, and then included in the generated documentation. Much like Python's doctests, testable examples write to standard output, and the output is compared against the expected output, to avoid regressions in the documented examples. Here's a testable example of an Abs() function:
func ExampleAbs() { fmt.Println(Abs(5)) fmt.Println(Abs(-42)) // Output: // 5 // 42 }
Example functions need to be in a *_test.go file and prefixed with Example. When the test runner executes, the Output: comment is parsed and compared against the actual output, giving a test failure if they differ. These examples are included in the generated documentation as runnable Go Playground snippets, as shown in the strings package, for example.
Benchmarking
In addition to tests, the testing package allows you to run timed benchmarks. These are used heavily throughout the standard library to ensure there are not regressions in execution speed. Benchmarks can be run automatically using go test with the -bench= option. Popular Go author Dave Cheney has a good summary in his article "How to write benchmarks in Go".
As an example, here's the standard library's benchmark for the strings.TrimSpace() function (note the table-driven approach and the use of b.Run() to create sub-benchmarks):
func BenchmarkTrimSpace(b *testing.B) { tests := []struct{ name, input string }{ {"NoTrim", "typical"}, {"ASCII", " foo bar "}, {"SomeNonASCII", " \u2000\t\r\n x\t\t\r\r\ny\n \u3000 "}, {"JustNonASCII", "\u2000\u2000\u2000☺☺☺☺\u3000\u3000\u3000"}, } for _, test := range tests { b.Run(test.name, func(b *testing.B) { for i := 0; i < b.N; i++ { TrimSpace(test.input) } }) } }
The go test tool will report the numbers; a program like benchstat can be used to compare the before and after timings. Output from benchstat is commonly included in Go's commit messages showing the performance improvement. For example, from change 152917:
name old time/op new time/op delta TrimSpace/NoTrim-8 18.6ns ± 0% 3.8ns ± 0% -79.53% (p=0.000 n=5+4) TrimSpace/ASCII-8 33.5ns ± 2% 6.0ns ± 3% -82.05% (p=0.008 n=5+5) TrimSpace/SomeNonASCII-8 97.1ns ± 1% 88.6ns ± 1% -8.68% (p=0.008 n=5+5) TrimSpace/JustNonASCII-8 144ns ± 0% 143ns ± 0% ~ (p=0.079 n=4+5)
This shows that the ASCII fast path for TrimSpace made ASCII-only inputs about five times as fast, though the "SomeNonASCII" sub-test slowed down by about 9%.
To diagnose where something is running slowly, the built-in profiling tools can be used, such as the -cpuprofile option when running tests. The built-in go tool pprof displays profile output in a variety of formats, including flame graphs.
The go test command
Go is opinionated about where tests should reside (in files named *_test.go) and how test functions are named (they must be prefixed with Test). The advantage of being opinionated, however, is that the go test tool knows exactly where to look and how to run the tests. There's no need for a makefile or metadata describing where the tests live — if files and functions are named in the standard way, Go already knows where to look.
The go test command is simple on the surface, but it has a number of options for running and filtering tests and benchmarks. Here are some examples:
go test # run tests in current directory go test package # run tests for given package go test ./... # run tests for current dir and all sub-packages go test -run=foo # run tests matching regex "foo" go test -cover # run tests and output code coverage go test -bench=. # also run benchmarks go test -bench=. -cpuprofile cpu.out # run benchmarks, record profiling info
Go test's -cover mode produces code coverage profiles that can be viewed as HTML using go tool cover -html=coverage.out. When explaining how Go's code coverage tool works, Go co-creator Rob Pike said:
For the new test coverage tool for Go, we took a different approach [than instrumenting the binary] that avoids dynamic debugging. The idea is simple: Rewrite the package's source code before compilation to add instrumentation, compile and run the modified source, and dump the statistics. The rewriting is easy to arrange because the go command controls the flow from source to test to execution.
Summing up
Go's testing library is simple but extendable, and the go test runner is a good complement with its test execution, benchmarking, profiling, and code-coverage reporting. You can go a long way with the vanilla testing package — I find Go's minimalist approach to be a forcing function to think differently about testing and to get the most out of native language features, such as interfaces and struct composition. But if you need to pull in third party libraries, they're only a go get away.
Index entries for this article | |
---|---|
GuestArticles | Hoyt, Ben |
Posted May 26, 2020 18:50 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
For example, it's impossible to do code coverage for integration tests or for manual tests. There are workarounds like writing a fake test that just runs the project's true "func main", but it's not enough for many reasons.
I've looked into trying to add ability to instrument non-test code, but the underlying rewriter is a freaking mess that is not modular and can't be easily extracted without major surjery.
Posted May 26, 2020 19:19 UTC (Tue)
by benhoyt (subscriber, #138463)
[Link] (1 responses)
Posted May 26, 2020 22:33 UTC (Tue)
by phlogistonjohn (subscriber, #81085)
[Link]
Here's an example of wrapping the main function in a test to get coverage.
I have to second Cyberax's comment. Last time I looked into this it quickly became too complicated to deal with, and I didn't need it very much and gave up. I too peeked at the code to implement this and it was too complex for me to follow (having been doing Go mostly full time for a year or so).
It's rather unfortunate that this isn't a feature of the compiler and say a helper module you can import into your own main. I lean heavily normal on the coverage feature for tests in a few of the projects I work on. I would love to have a simple, noninvasive way to generate coverage from any binary.
Posted May 26, 2020 22:57 UTC (Tue)
by cyphar (subscriber, #110703)
[Link]
[1]: https://www.cyphar.com/blog/post/20170412-golang-integrat...
Posted May 26, 2020 22:55 UTC (Tue)
by phlogistonjohn (subscriber, #81085)
[Link] (1 responses)
On a different note, I didn't quite see the rationale for Cleanup, given that Go has defer. The issue that ended up getting the feature added seems to be the clearest explanation of the why behind Cleanup: https://github.com/golang/go/issues/32111
Posted May 27, 2020 6:03 UTC (Wed)
by kokkoro (guest, #139153)
[Link]
Posted May 27, 2020 5:00 UTC (Wed)
by tymonx (guest, #139151)
[Link]
It contains preinstalled tools for developing, mocking, formatting, linting, building, testing and documenting Go projects.
For mocking Go interfaces, I'm using the standard and classic mockgen tool in reflect mode. One of annoying thing about that tool is command line invocations. You manually provide a module name/path with interface names separate with comma.
I have scripted that to automatically and recursively detect all Go interfaces. It excludes _test.go files, mocks directories and the package main.
Go module detection is based on the go env GOMOD command invocation. All mocks are generated under the mocks directory for given Go package.
There are more features that can help. Like:
Posted May 27, 2020 17:11 UTC (Wed)
by cpitrat (subscriber, #116459)
[Link] (2 responses)
Among go tests, I've seen multiple times ettor messages that were invalid because copied from the previous function (e.g saying no error when it checked that there was an error). Or inconsistent order for messages saying what it got vs. what it wants, or worse printing got <want>, want <got>.
Duplication sucks because of that. Go community thinks that WET (write everything twice) is better than DRY (don't repeat yourself) ignoring years of computer science experience and wisdom. Proof-reading tests in a large Go code base is a good way to understand why they're wrong.
Posted May 28, 2020 20:45 UTC (Thu)
by kunitz (subscriber, #3965)
[Link] (1 responses)
Posted May 29, 2020 14:36 UTC (Fri)
by cpitrat (subscriber, #116459)
[Link]
Posted May 28, 2020 5:52 UTC (Thu)
by jezuch (subscriber, #52988)
[Link] (3 responses)
Posted May 28, 2020 21:26 UTC (Thu)
by kunitz (subscriber, #3965)
[Link] (2 responses)
I'm always amused when people point out that Go doesn't support concept X. That is actually the strength of the language. The lack of concepts makes it easy to learn, the pedestrian approach feels often dull, but it gets the job done in a way that you will understand in 5 years from now. The strength of Go is the idea of construction of complex things out of little dull things.
Posted May 28, 2020 21:40 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Go has done a lot of things right, but testing is most definitely not one of them.
Posted Jun 1, 2020 5:30 UTC (Mon)
by jezuch (subscriber, #52988)
[Link]
In the case of tests written as a not-very-structured mass-of-code, I'm pretty sure I won't understand the intent behind the test after 5 days :) The most important thing is not the code itself - it's the intent behind the code. That's something imperative programming is inherently bad at - and the language makes it even worse when it doesn't support concept X.
Testing in Go: philosophy and tools
Go coverage analysis SUCKS. Seriously.
Interesting. I'd love to know more about this; any links? Our team at work has recently started running integration tests using the regular testing package and it seems to work well, but these are fairly lightweight "PostgreSQL database only" integration tests, not "run all the services" integration tests.
Testing in Go: philosophy and tools
Testing in Go: philosophy and tools
Testing in Go: philosophy and tools
Testing in Go: philosophy and tools
Personally, I don't see it as all that compelling but not bad or anything. :-)
Defer doesn't work properly with parallel tests. If you do:
Testing in Go: philosophy and tools
then f.Close() will be called immediately, concurrent to all of the parallel subtests.
https://github.com/golang/go/issues/17791
func TestFoo(t *testing.T) {
f := NewFoo()
defer f.Close()
for _, c := testCases {
c := c
t.Run(c.name, func(t *testing.T) {
t.Parallel()
c.func(t, f)
})
}
Testing in Go: philosophy and tools
* Automatically generating JUnit test XML after tests
* Automatically generating Cobertura coverage XML after tests
* Validating Go coverage value threshold with colorization
* Colorizing Go coverage results and format nicely with the column tool
* Automatically generating static HTML coverage report with the go tool cover tool after tests
Testing in Go: philosophy and tools
Testing in Go: philosophy and tools
Testing in Go: philosophy and tools
Testing in Go: philosophy and tools
Testing in Go: philosophy and tools
Testing in Go: philosophy and tools
It also forces people to use reams of repetitive code and/or search for third-party libraries to do the most basic things. And due to limitations of the language, libraries often can't provide experience that is seamless in other languages.
Testing in Go: philosophy and tools