On Log Browsers and Working Around a Log Message
In these modern times, with these modern services running online, dealing with logs is a huge task. These systems produce so much logging that it’s a wonder that we can get anything from them at all. Log indexing tools like Grafana and Kibana are therefore a necessarily when investigating issues in these sorts of services. Since they’re real text search engines, using them find log messages that match a particular search pattern is really helpful, and reduces the time it takes to comb through the mountain of log messages produce by these systems from hours to seconds.
But sometimes there is a need to comb through the logs just a bit, to get a sense as to what is happening around the time of a particular log message. The searching technology used by these tools can locate discrete instances of a log message, but in my experience, there is very little in way of searching around that message — sometimes called the context. It’s sort of using a metal detector to find a needle in a haystack: the metal detector can show you the general area that needle might be, but you still need digging tools to find the needle itself.
So here’s a list of features needed in log browsers to make this easier:
- When displaying the context of a log message, offer the ability to filter the surrounding log lines with a substring expression. Many systems interlace the log files of different threads. It’s possible to get the tread IDs within the log lines, but when a lot is going on around that time, it’s hard to see it in the context.
- Offer a quick way to set the time range to a specific log message, and “expand” the range relative to the particular time stamp. That is, I should be able to quickly click the time stamp in a log message to set the search time range. I should then be able to “expand” that time range slightly. Maybe by adding a second to each end, then maybe 2 seconds, and so on. This is like zooming out in the time space, slowing revealing more on what happened around that time. This is the same thing that context supports, but uses the full search query facilities.
- Finally, offer the ability to export contexts so that I can use tools like
awkagainst the search results. Getting back to the needle and haystack, this is like offering a shovel alongside the metal detector, so I could take a bit of hay and comb through it to find that needle. This is such a basic feature it’s really surprising to see so few of these browsers offer it.