In this post I’ll illustrate two ways I’ve accidentally caused slow but steady memory consumption in Go programs. The phrase “memory leak” isn’t really accurate, but I can’t think of a better one. It’s more like “using memory in a way that will never be garbage collected.”
First - what does such a problem look like? Here’s a screenshot of what happened when I introduced one of these bugs into our operating system metrics agent:
The ticks on the time-scale are in days. The memory growth is slow enough that even if you watch the agent over a few hours after a new release, you might not see it. The sharp jumps downwards are when this agent was released and restarted. You can see that at first its memory usage grows a bit, and then it settles down; this is what we’re used to, and the steady climb only becomes clear after most of a day has passed.
Leak 1: Deferred Code In A Loop
The first type of problem I introduced was by (ab)using Go’s idiomatic way to clean up resources, the
defer statement. If you’re not familiar with it, you usually use it thusly:
fh, err := os.Open("/path/to/file.txt") if err != nil { // } defer fh.Close()
Imagine this snippet of code in a larger program. The
defer allows you to place the closing/cleanup of the resource right next to where it’s created. The code is guaranteed to run when the function returns, even if there’s a panic. This helps make your code much cleaner. If you’ve ever written larger programs that open and need to clean up lots of such resources, you’ll know what I mean: it’s hard to cross-check and ensure that all the t’s have been crossed and the i’s have been dotted. This idiom avoids the need to cross-check parts of code against other parts.
But I made a subtle mistake: I placed my
defer inside a loop in a long-running function:
func somefunc() { for { // ... defer something.Cleanup() } }
The problem here is that the deferred code never executes. Any variable referenced in that code is never garbage-collected. Presto: slow-growing memory usage.
Leak 2: Blocked Goroutines In A Loop
The second type of leak I accidentally created actually looked pretty similar. Go has a
go keyword that runs a function in a new goroutine. A goroutine is kind of like a lightweight thread, and the
go keyword is a little bit like backgrounding a process in shell with a trailing
&.
It’s fine to create goroutines in a loop. Goroutines are cheap and they’re meant to be used this way:
for { go doSomething() }
The problem I created was because I created goroutines that blocked. They were meant to send a value to a list of channels, and I didn’t want them to block, so I “backgrounded” them in goroutines to make them asynchronous. This was the wrong way to do it, as I’ll discuss in a bit:
for i := range channels { go func() { channels[i] <- value }() }
And this worked – unless the goroutine that was receiving from the channel stopped receiving for any reason. Which they did. Some of them returned from their function and exited; others had errors and exited.
My sending loop, of course, continued merrily on its way, unblocked as intended. But little did I realize that every time it iterated, it created a new goroutine that tried to send to a channel, and blocked forever, thus never exiting. As a result, the program created goroutines forever, and its memory grew steadily.
The correct way to do that, by the way, is with a nonblocking
select statement. This was the fix for the bug:
for i := range channels { select { case channels[i] <- value: default: } }
This idiom is discussed at length in the Go documentation.
Conclusion
I hope this “tale of two memory leaks” helps illustrate subtleties in creating and cleaning up resources in Go. Go is pretty obsessed with memory, and exposes that to the programmer. I don’t expect that these two problems would be surprising to proficient Go programmers, but perhaps this is useful for those of us who are less experienced.
