Many of our agent programs have several independent worker routines that all need to be timed on clock ticks, sometimes with a specific time offset. We recently open-sourced
multitick, a library we built to make the code a little easier and more testable.
First, a description of the problem we needed to solve. Our agents often have routines that we want to run concurrently, doing their work at the same moment as much as possible. For example, some of the agents capture time-series metrics, and we’d like to capture the metrics at or near the same time, for consistency.
Some of the tasks need to be aligned to the clock-tick. We can get quite close to the clock-tick by sleeping the number of nanoseconds remaining in the current second and then starting one of Go’s builtin time.Ticker types on a one-second interval. In practice we end up within a few milliseconds of the clock-tick, which is good enough for our purposes.
Some other tasks need to be aligned halfway between clock-ticks. For example, we measure latency of certain processes by making one task update a value on the clock-tick. Another task reads the same value mid-tick and subtracts half a second from the result. This lets us avoid timing race conditions that would happen if we tried to make both the writer and reader operate at the same time: some of the time the reader would read just before the writer writes, and we’d read a value that’s 1 second old, getting a falsely high latency.
How we used to do it
We originally wrote our worker routines with time.Ticker objects inside them, and looped forever over the ticks coming from the time.Tickers. This meant that we had the following undesirable characteristics in our code:
- Untestable. Because a non-deterministic, uncontrollable process (clocks ticking) was happening inside the routine, we couldn’t inject it from tests.
- Repeated code. All of the code to sleep until the desired boundary (clock-tick, or midway between ticks) was duplicated and messy.
A later iteration of this code did something slightly better: we moved the ticker outside of the routine and sent ticks to a channel, which was an argument to the routine. We used a single ticker to feed all of the routines, and copied each tick into each channel so they all got the same ticks. Now the routines were more easily tested, but
we I created a subtle and serious bug in the process. See if you can spot it in the following simplified code:
ticks1 := make(chan time.Time) ticks2 := make(chan time.Time) go doWork1(ticks1) go doWork2(ticks2) for now := range time.Tick(time.Second) { go func() { ticks1 <- now }() go func() { ticks2 <- now }() }
The bug is what happens if one of the worker routines stops receiving ticks from the channel. This could happen if it returns for some reason, for example; or if it gets blocked and can’t continue doing work. If this happens, the sending code shown above will block. The anonymous goroutines that try to send now to one of the channels will never complete, and the for-loop will continue to iterate once a second. This will result in slow but steady memory bloat as goroutines accumulate. Another bug is that if a number of ticks piled up and then the worker reads them later, it’s not guaranteed it will get them in order.
How multitick solved our problems
Multitick solves all of the problems in the above code in one small package. You can create a multitick ticker at the specified interval, and at a specified offset after the interval. This takes care of all of the sleep-until-it-is-time-to-start code.
A multitick object supports multiple subscribers. This means we can create a single ticker and subscribe lots of workers to it. They’ll all get the same time.Time values. The values are sent to the channels one after the other, so they will be slightly (on the order of microseconds) delayed from each other. But that doesn’t matter for our purposes.
A multitick object is non-blocking. If a worker routine blocks or exits, its channel won’t be ready to receive. Multitick uses non-blocking “select” statements, instead of goroutines, to prevent this. Missed ticks will simply be dropped on the floor and no memory bloat or goroutine proliferation happens.
Multitick is simple to use; the documentation shows an example. Our agent’s clock-tick code was reduced to a few lines without bugs or messiness.
Conclusion
If you need multiple subscribers to the same clock ticker, you need all the subscribers to get the same values, and you need alignment to a desired time offset,
multitick might be what you’re looking for. We hope you find it helpful, and that you’ll contribute improvements to it.
