Log Analysis Tools Review: Syslog, Filtering the Most Relevant Events

Who’s passed more than a weekend going almost blind because something in your didn't work as it should, and it produced, hopefully, thousands of lines of error messages, almost all of which were the same? The key words here are, “almost the same." A syslog system could save you precious time, and your family will be thankful for it, but it's up to you to find a way to filter the lines that are "almost" like the others, the ones that are different save for a single cypher that makes your day. Syslog is a tool. Actually, it is the first logging tool. Try to put a screwdriver beside your PC, you won't be disappointed if it doesn't disassemble the machine. The tool leverages your skills, it doesn’t replace them. There are several tricks to filter the lines produced by a syslog. The key is to find the right pattern. Today there are so many GUI tools to perform these operations, but the core is the same: match a pattern to the bundle of lines, and filter the ones that will help to solve your problem. Taking a step back, during syslog configuration, you can set it up to write the logs in a manner that is closer to your pattern’s creation. We could ask to put the time-data at the beginning of the line, or maybe the severity, or, again, the service involved. That's not important. What is important is that you do it in a way that suits you. So, the pattern. This is the critical point, the difference between success or wasted nights. Similar as a Google research, you need to find the right keyword for the best result. There are several free tools to parse logs, and the same goes for patterns. There are many databases of them, but maybe your issue is more specific? Anyway, a good starting point is log parsing. Today there are many advanced utilities born from syslog, a number of which are open source: rsyslog, syslog-ng, logwatch, just to name a few. The main difference is that rsyslog apply filters on the logs produced by syslog to perform actions, for example, if present the word "localhost" sends an email to someone@domain.com, or if the source IP 10.10.10.10 is present it writes the line on file "10101010". Syslog-ng is a more complex utility that not only filters, but also correlates and classifies. Logwatch is interesting as well. All of the tools use a set of precompiled patterns, in many cases modifiable. Let me say that it's quite unusual to not find the right filter for your very specific requirements. Besides these tools, there's another subset to consider when talking of logs: SNMP - traps and polling. Usually they're used for monitoring purposes instead of analysis, but the core concept is the same: a tool that writes lines that are constantly filtered by another tool that sends traps - or waits to be read by a poller to raise an alert. The first part of the process is the same: logging - and the second is similar too: filtering. So, enjoy, try, install, reinstall, destroy, but above all, keep logging! It can save you a lot of precious free time, and it can help your peers as well, even those from other parts of the world.