Go filesystems and file embedding

The Go team has recently published several draft designs that propose changes to the language, standard library, and tooling: we covered the one on generics back in June. Last week, the Go team published two draft designs related to files: one for a new read-only filesystem interface, which specifies a minimal interface for filesystems, and a second design that proposes a standard way to embed files into Go binaries (by building on the filesystem interface). Embedding files into Go binaries is intended to simplify deployments by including all of a program's resources in a single binary; the filesystem interface design was drafted primarily as a building block for that. There has been a lot of discussion on the draft designs, which has been generally positive, but there are some significant concerns.

Russ Cox, technical lead of the Go team, and Rob Pike, one of the creators of Go, are the authors of the design for the filesystem interface. Cox is also an author of the design for file embedding along with longtime Go contributor Brad Fitzpatrick. Additionally, Cox created YouTube video presentations of each design for those who prefer that format (the filesystem interface video and the file-embedding video). Both designs are quick to note that they are not (yet) formal proposals:

Many smaller language and library changes are discussed on the GitHub issue tracker, but for these larger discussions the Go team is trying to use r/golang Reddit threads to scale the discussion — GitHub issues do not have any form of threading, so multiple conversations are hard to keep track of. There is a Reddit thread for each draft—the filesystem interface thread and the file-embedding thread—with quite a few comments on each. There is also a lengthy Hacker News thread that discusses the file-embedding design.

A filesystem interface

The crux of the filesystem interface design is a single-method interface named FS in a new io/fs standard library package:

    type FS interface {
        Open(name string) (File, error)
    }

This means that every filesystem implementation must at least implement the ability to open a file by name, returning a File as well as an error. The File interface is defined as follows:

    type File interface {
        Stat() (os.FileInfo, error)
        Read(buf []byte) (int, error)
        Close() error
    }

In other words, a file has the following characteristics: is able to provide file information like that returned from stat(), is able to be read, and can be closed. These are the bare minimum that a conforming filesystem needs to provide, but an implementation "may also provide other methods to optimize operations or add new functionality". The standard library's file type (os.File) already implements these three methods, so it is a conforming fs.File implementation.

If a File is actually a directory, the file information returned by Stat() will indicate that; in that case, the File returned from Open() must also implement the Readdir() method on top of the File interface. Readdir() returns a list of os.FileInfo objects representing the files inside the directory.

Filesystem implementations can expose additional functionality using what the design calls an "extension interface", which is an interface that "embeds a base interface and adds one or more extra methods, as a way of specifying optional functionality that may be provided by an instance of the base interface." For example, it is common to read a whole file at once, and for in-memory filesystem implementations, it may be inefficient to do this using Open(), multiple calls to Read(), and Close(). In cases like this, a developer could implement the ReadFile() method as defined in the ReadFileFS extension interface:

    type ReadFileFS interface {
        FS  // embed the filesystem interface (Open method)
        ReadFile(name string) ([]byte, error)
    }

Along with the extension interface, the design adds a ReadFile() helper function to the io/fs package that checks the filesystem for the ReadFileFS extension, and uses it if it exists, otherwise it falls back to performing the open/read/close sequence. There are various other extension interfaces defined in the draft proposal, including StatFS, ReadDirFS, and GlobFS. The design does not provide ways to rename or write files, but that could also be done using extensions.

In addition to the new io/fs types and helper functions, the design suggests changes to various standard library packages to make use of the new FS interface. For example, adding a ParseFS() method to the html/template package to allow parsing templates from an in-memory filesystem, or making the archive/zip package implement FS so that developers can treat a zip file as a filesystem and use it wherever FS is allowed.

Much of the feedback on the Reddit discussion has been positive, and it seems like an interface of this kind is something that developers want. However, one of the criticisms made by several people is about the drawbacks of extension interfaces. "Acln0" summarized the concerns:

Peter Bourgon, a well-known Go blogger and speaker, believes that this use of extension interfaces means that it "becomes infeasible to use the (extremely useful) decorator pattern. That's really unfortunate. To me that makes the proposal almost a non-starter; the decorator pattern is too useful to break in this way." The decorator pattern wraps an interface and adds some functionality. It is often used for logging or authentication middleware in web servers; in the context of filesystems it would likely be used to add a caching or transformation layer. If a middleware author does not take into account the various optional interfaces, the resulting wrapper will not support them. Nick Craig-Wood, author of Rclone, a cloud-storage tool written in Go, likes the proposal but expressed similar concerns: "Extension (or optional as I usually call them) interfaces are a big maintenance burden - wrapping them is really hard".

The design states that "enabling that kind of middleware is a key goal for this draft design", so it would seem wise for the design's authors to tackle this problem head on. Cox hasn't yet proposed a solution, but acknowledged the issue: "It's true - there's definitely a tension here between extensions and wrappers. I haven't seen any perfect solutions for that.".

Another concern came from "TheSwedeheart" regarding contexts (the standard way in Go to explicitly propagate timeouts, cancellation signals, and request-scoped values down a call chain): "One thing I'm missing to migrate [his virtual filesystem] over to this is support for propagating contexts to each operation, for cancellation.". Cox replied that a library author could "probably pass the context to a constructor that returns an FS with the context embedded in it, and then have that context apply to the calls being made with that specific FS." As "lobster_johnson" pointed out, this goes against the context package's guideline to explicitly pass context as the first function argument, not store a context inside a struct. However, Cox countered with an example of http.Request doing something similar: "Those are more guidelines than rules. [...] Sometimes it does make sense."

There are of course the usual bikeshedding threads that debate naming; "olegkovalov" said: "I'm somewhat scared about io/fs name, fs is a good variable name, it'll cause many troubles to the users when io/fs will appear". After some back-and-forth, Cox stressed the need for a short name to keep the focus on application developers rather than on the filesystem implementers:

Embedding files in binaries

The other draft design proposes a way to embed files (or "static assets") in Go binaries and read their contents at runtime. This simplifies releases and deployments, since developers can simply copy around a large binary with no external dependencies (for SQL snippets, HTML templates, CSS and JavaScript assets for a web application, and so on). As the document points out, there are already over a dozen third-party tools that can do this, but "adding direct support to the go command for the basic functionality of embedding will eliminate the need for some of these tools and at least simplify the implementation of others". Including embedding in the standard go tool will also mean there is no pre-build step to convert files to data in Go source code, and no need to commit those generated files to version control.

The authors of the design make it clear that this is a tooling change, not a Go language change:

The go tool already looks for special comments in Go source files for various things, including // +build tags to include certain files only on specific architectures, and //go:generate comments that tell go generate what commands to run for code-generation purposes. This file-embedding design proposes a new //go:embed comment directive that goes directly above a variable declaration and tells go build to include those files in the resulting binary associated with the variable. Here is a concrete example:

    // The "content" variable holds our static web server content.
    //go:embed image/* template/*
    //go:embed html/index.html
    var content embed.Files

This would make go build include all the files in the image and template directories, as well as the html/index.html file, and make them accessible via the content variable (which is of type embed.Files). The embed package is a new standard library package being proposed that contains the API for accessing the embedded files. In addition, the embed.Files type implements the fs.FS interface from the filesystem design discussed above, allowing the embedded files to be used directly with other standard library packages like net/http and html/template, as well as any third-party packages that support the new filesystem interface.

The design limits the scope of the proposal in an important way. There are many ways that the data in the files could be transformed before being included in the binary: data compression, TypeScript compilation, image resizing, and so on. This design takes a simple approach of just including the raw file data:

Again, the feedback on the Reddit thread was mostly positive, with comments like this one from "bojanz": "This looks like a great start. Thank you for tackling this." There are a few minor suggestions, such as a comment by "zikaeroh" in favor of adding a more powerful path-matching API that supports double star for recursive path matching, like glob('**/*.png', recursive=True) in Python. Kevin Burke, who is the maintainer of a file-embedding package, suggested also storing a cryptographic hash of each file's content so the developer does not have to hash the file at runtime: "This is useful for e.g. cache busting on a static file server".

One of the repeated critiques is from developers who don't like overloading source code comments with the special //go:embed syntax. "Saturn_vk" stated bluntly, "I really don't like the fact that comments are being abused for actual work", and Hacker News commenter "breakingcups" strongly advocated for the use of a project file instead of directives in comments:

Cox summed up his thinking about this with the following comment, which compares the syntax with #pragma for C:

Next up

There is a fair amount of community support for both draft designs, particularly the more user-facing proposal for file embedding. Many developers are already using third-party file-embedding libraries to simplify their deployments and these efforts will standardize that tooling. It seems likely that the designs will be refined and turned into full proposals. With Go 1.15 due out on August 1, it's possible that these proposals would be ready for Go 1.16 (scheduled for six months out), but if there needs to be another round of feedback — for example, regarding the problems with extension interfaces — it is more likely to be included in Go 1.17 in a year's time.