This is my PERSONAL blog, as as of August 1, 2011, it focuses on personal matters and various things I find to be fun.

Tuesday, June 03, 2008

Logging Poll #8 Analysis: Needed Log Context

In my poll #8, I asked a question: what information is most important when analyzing a particular log record. Live results are here and final count is also below:

What can we conclude?

First, good documentation never hurts :-) - indeed, the most popular information to look for when facing a new log record is documentation on what it means. While some software vendors are great in this regard, many other don't bother documenting their logs or document them only when customers complain.

Second, I was not sure that the second popular choice would be "Other logs from about the same time (this and other systems)." This strongly points at huge value of cross-device log analysis (see this recent log entry on that), where all the logs are consolidated and analyzed together (it goes without saying that time is synchronized OR at least corrected across those logs). Indeed, if you are confused about a log and documentation is not available, reviewing "what else was/is going on?" is smart. Trusting log time stamps across many systems is also key for that.

Third, having IP addresses in logs is great, but human-readable names are better: IPs in logs needs to be mapped to DNS or Netbios names. Indeed, given that often such names reveal where the system is, who might own it, what its function is, etc this information is not just a mapping, but true log information enrichment.

Fourth, so, what's next? The above 3 top responses are indeed universally useful, but the next choice digs deeper: flows, packets, connections and other network information does complement logs and is often studied in combination with logs (e.g. see a strange log entry then go see who connected to the system at that time or where the system itself connected to).

Fifth, next comes a group of pretty much everything else: other logs from the same system, logs about the same system as well as loosely defined 'similar' log entries. These come handy, but are not top choices. In fact, from this I conclude that a lot of additional context information is needed to make sense of a confusing log entry.

Sixth, what was surprising? I thought that identity lookups (e.g. IP to real name or other user identity information) would score higher. I also suspect that people were confused by "logs ABOUT the same systems" (what I meant is, for example, use firewall logs that mention the system which log we are now analyzing) and this should score higher.

Seventh, anything fun in the "Other" category? Yes, there were a few insightful ones: first, results of a Google search (supposedly for the info from the log entry in question)! Very true indeed. Also named were logs from the same daemon/program (how can I miss it?), logs from previous incidents and information on the logging system owner. All very useful indeed. Thanks for good ideas!

Finally, a brief message to people that work for a certain log-related vendor of ill repute who keep polluting my polls: if I catch you, I will kick you in the butt :-) Or, better, I will hammer you with a big and heavy log (you know, the wooden kind) over your miniscule heads ...

About Me

He is a recognized security expert in the field of log management and PCI DSS compliance. He is an author of books "Security Warrior", "PCI Compliance", "Logging and Log Management" and a contributor to "Know Your Enemy II", "Information Security Management Handbook" and others. Anton has published dozens of papers on log management, correlation, data analysis, PCI DSS, security management, honeypots, etc . His blog securitywarrior.org was one of the most popular in the industry.

In addition, Anton teaches classes (including his own SANS class on log management) and presents at many security conferences across the world; he recently addressed audiences in United States, UK, Singapore, Spain, Russia and other countries. He worked on emerging security standards and served on the advisory boards of several security start-ups.

Before joining Gartner in 2011, Anton was running his own security consulting practice www.securitywarriorconsulting.com, focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging Evangelist, tasked with educating the world about the importance of logging for security, compliance and operations. Before LogLogic, Anton was employed by a security vendor in a strategic product management role. Anton earned his Ph.D. degree from Stony Brook University.