How to selectively export identifiers in a package for testing

TL;DR

Just add an export_test.go file and specify the package to be the same as the package under test (instead of the test code). E.g. export_test.go should have a package athena clause in the header, whereas the test code will use package athena_test.

Add any useful code to export (or wrap) package local identifiers in export_test.go which can then be used by the test code (in package athena_test). Note, because the file name ends with "_test.go", it is conditionally compiled when go test is run, but isn't part of the build or godoc.

The benefits:

Package and test code remain in distinct packages. (Test code uses the package as a regular user).

No need to compromise on exporting internal identifiers in the package to regular users or rewriting exported interfaces to aid testing at the detriment to user experience.

Dig right in and take a squizz at export_test.go

Testing AWS Athena

Example code base: https://github.com/KablamoOSS/exportexample.

The code base provides a limited use wrapper to AWS Athena. For the purposes of demonstrating this idiom, you don't really need to know what Athena does, but if you must know, it provides an SQL-like interface to big data (S3) stuff. I had to use it once for a very specific use case, so I wrote some tests for it, which are available in the example code base above.

The wrapper uses the Athena client from AWS Go SDK. The SDK also provides a full interface specification of the Athena client to aid mocking and unit testing. This means you can wrap the SDK provided Athena client using the interface specification instead of the actual client object.

So in my wrapper, I use the following code to create a real client.

package athena
 
import (
 "fmt"
 "github.com/aws/aws-sdk-go/aws/session"
 sdkathena "github.com/aws/aws-sdk-go/service/athena"
 "github.com/aws/aws-sdk-go/service/athena/athenaiface"
)
 
// Client is used to query AWS Athena.
type Client struct {
 // api is either an athena.Athena or a mock implementation for testing.
 api athenaiface.AthenaAPI
}
 
// NewClient creates and returns a new Athena client.
//
// Caller must provide a valid AWS session.
//
// This can be used for concurrent requests as the underlying Athena
// supports concurrent queries.
func NewClient(session *session.Session) (Client, error) {
 if session == nil {
  return Client{}, fmt.Errorf("nil session")
 }
 
 return Client{sdkathena.New(session)}, nil
}

Note: the above and following code are slight variations of what is contained in the example code base to help illustrate the idiom.

So how do we mock?

The SDK provides a full interface specification (in the above code it is athenaiface.AthenaAPI). This allows you to write a mock client that satisfies the interface and substitute a real client (created using athena.New(*session.Session) above).

The mock client is defined here. It implements athenaiface.AthenaAPI.

To facilitate creating a wrapper client using a mock client object we have the following function:

// NewCustomClient is used specifically for testing purposes;
// you want to create a mock client that satisfies athenaiface.AthenaAPI
// which then implements
func NewCustomClient(mockapi athenaiface.AthenaAPI) Client {
 return Client{api: mockapi}
}

Getting closer to the point

So a natural question would be where to put NewCustomClient?

NewCustomClient is returning a Client object and assigning to unexported field api. The unexported field can only be assigned for code residing in the package. Ergo NewCustomClient must live in the athena package.

This raises a dilemma. NewCustomClient is used exclusively to test. By placing it in the athena package, you're leaking testing code to outside consumers. This is not ideal.

Here are some possible alternatives:

Test code resides in the athena package. This way it can use non-exported identifiers. You can then place NewCustomClient in test code which is only built during go test. However, sharing package namespace with test code is undesirable. The biggest concern is that tests could have unintended side-effects by mutating internal package state which can mask issues during testing.

Export the Client.api field, i.e. rename the field to Client.API. This is a flipped version of the previous option, but now users can mutate package level internals they shouldn't have any business accessing in the first place.

Rewrite athena.NewClient to accept athenaiface.AthenaAPI instead of a Session object. This means test code can eschew NewCustomClient and instead supply a mock client to NewClient. However this is at the cost of sacrificing user experience, and also it's no longer really a "NewClient" but probably a "NewWrapper". This also violates the spirit of wrapping the SDK Athena client.

Fundamentally, all options lead to some kind of sacrifice, be it aesthetic or user experience. If I had to choose, it'd be option 1. Just employ strict discipline to not mess around with internal package state and minimally rely on internal identifiers in test code.

Thankfully there is another option which lets us have our cake and eat it too.

The export_test.go idiom

The export_test.go idiom is simple but subtle. To know about it, someone has to point it out, or you have to accidentally stumble upon it trawling a codebase that employs it (e.g. Go standard library).

Prerequisites:

• athena.NewClient and Client.api stay unchanged.

• Test code resides in a distinct package athena_test (with a slight exception).

Now, add a file named export_test.go and specify its package as "athena". This is where NewCustomClient will reside.

Hopefully it's now obvious. export_test.go is only compiled when go test runs, and test code can call athena.NewCustomClient.

This means no inappropriate internal identifiers or testing code is exported in the athena package to end users; the package and test code are separate. It just happens that export_test.go provides a bridge for test code to work with internal identifiers in the package.

Here's what export_test.go looks like:

package athena
 
import "github.com/aws/aws-sdk-go/service/athena/athenaiface"
 
// NewCustomClient creates and returns a custom Athena client.
//
// A mock implementation of the Athena Interface can be provided for testing.
func NewCustomClient(api athenaiface.AthenaAPI) Client {
 return Client{api}
}

When to use export_test.go

Apply the export_test.go idiom when faced with the following alternatives:

• Sharing package namespace with test code, or

• Rewriting the exported package interface to accommodate testing, or

• Exporting internal package identifiers to aid testing at the detriment of user experience.

Case in point: none of the above sacrifices are made in the Go standard library, which means you don't need to either.

But use it judiciously. You don't want to bridge the entire set of non-exported identifiers to test code, which in this case you're better off with choice (1) above. Put politely, if this is your situation, then you need to reconsider the design of the package and tests.

As an aside, I hope it is obvious that you don't have to name the exports file export_test.go. As long as the name ends with "_test.go" the idiom works (thanks to conditional compilation). But seriously, make your fellow developers' lives easier and just name it export_test.go.

Acknowledgements

I said earlier this idiom isn't obvious unless someone points it out, unless you (luckily) stumbled on it trawling a code base where it's used. I learned this from a regular on #go-nuts IRC channel on freenode. So thank you for pointing out this underrated idiom.

Also, I should mention the Go authors. The standard library is a solid example of writing Go, and I would encourage you to explore further by trawling the source in your friendly neighbourhood go env GOROOT.