100% Reproducible Builds in Go

This post was featured in our July 2015 anthology of most popular SolarWinds Database Performance Monitor (DPM) blog posts of all time. One of the nice things about Go is statically compiled binaries, which don’t have external dependencies. There are lots of reasons this is a great thing. On the other hand, Go packages aren’t versioned. This results in a minor irritation that isn’t fixed natively in the Go-provided toolchain: how can I constrain which versions of external packages are imported into my application? And, given that this isn’t one of the things that is supplied with Go itself, how can I a) determine what source code and which versions of packages were used to build an app, and b) reproduce that exactly for troubleshooting and debugging? At SolarWinds, we’ve developed a couple of tools and a technique to solve this problem. We’ve open-sourced most of the tools already, and in this blog post I’ll explain how we tie these together into a fully functioning solution. First of all, we use a technique that we call Godeps-in-Git. This is shorthand slang for one of the two approaches mentioned in the Johnny Deps documentation. This means that we commit a full, explicit Godeps file into our applications’ Git repos. These contain the Git SHA of every non-core (i.e. not part of the Go standard library) package. To generate this list, we use the generate_deps tool provided in Johnny Deps. Then we use a build script instead of calling “go build” directly. In addition to executing johnny_deps, it has a pre-build step that fetches the application repo’s Git SHA, writes out a temp file, and then performs the build. The script looks like this: {% highlight bash %} echo “package main” > godeps.go echo “func init() {” >> godeps.go echo “ BuildSHA = \“# Built $(date) with $(go version) at $(git rev-parse HEAD)\“” >> godeps.go echo “}” >> godeps.go {% endhighlight %} What does this do? It writes out an init() function which sets a global variable called BuildSHA. Now, we add a commandline flag to our application, called –build-sha. If this flag is present, we’ll print out the value of BuildSHA: {% highlight go %} Config.BoolVar(&WantSHA, “build-sha”, WantSHA, “Print the packages and versions in this build”) // … if WantSHA { fmt.Println(BuildSHA) os.Exit(0) } {% endhighlight %} The result is that I can execute the app with the –build-sha flag and get output like the following: {% highlight bash %} $ ./app –build-sha # Built Mon Sep 9 20:53:06 EDT 2013 with go version go1.1 darwin/amd64 at 1f9398aebb652f7d494c783f83393b8cab7c54bc {% endhighlight %} If I want to reproduce that build, I can: {% highlight bash %} $ git checkout 1f9398aebb652f7d494c783f83393b8cab7c54bc $ johnny_deps {% endhighlight %} And I’m done. Now I have exactly the source code, in the app’s repository as well as every external package, that was used to build the app. This is pretty similar to “pip freeze,” if you’re familiar with that. There’s also a little helper in our build script, which we invoke manually. It runs generate_deps to find the app’s dependencies, pulls them from Git, and commits them to Godeps again. We use this to “upgrade” an app’s Godeps file, so we don’t stay pinned to old versions of dependencies. Hopefully you find these tools and practices helpful in creating reproducible builds for your Go applications. Let us know if you have any questions or suggestions!