Quantifying Abnormal Behavior

June 25, 2013

Quantifying Abnormal Behavior

At Velocity last week, I spoke about how we quantify abnormality in a system’s time-series metrics cheaply, in realtime, at high frequency.

Note that this is not the same thing as our Adaptive Fault Detection algorithm. Our abnormality algorithm is one of the low-level building blocks of the adaptive fault detection algorithm. But as I pointed out in the talk, if you look at a system’s metrics in short time intervals, you will find abnormalities constantly. That’s why abnormality is a blunt instrument, not good enough to significantly reduce false alarms. If you alert on abnormalities, you’ll get a lot of spam, just like you will with thresholds on a metric.

Still, abnormality is at least a place to start, right? In the progression towards true fault detection, you can think of it this way: a fault is more specific than an abnormality, and an abnormality is more specific than a threshold being crossed. This assumes that you agree with our definition of a fault, which is defined in terms of the system not getting its assigned work done.

Baron is a performance and scalability expert who participates in various database, open-source, and distributed systems communities. He has helped build and scale many large, high-traffic services for Fortune 1000 clients. He has written several books, including O'Reilly's best-selling High Performance MySQL. Baron has a CS degree from the University of Virginia.