It hasn’t seen much of any production traffic, but my logging library for Ocaml made no distinction between log messages and instrumentation. They all had the same basic properties: a unique name in the code, time stamp, host name of origin, and service of origin. Then backends to the logger did differnt things depending on the situation. It’s rather elegant but I haven’t tested it enough to know if it sucks. I do think that, fundamentally, instrumentation and logs are the same thing, they just have different information. My library also really suggests against log messages but rather using a key-value of log information, making it easier to parse by automated tooling down stream.

At Heroku, we’re big fans of logfmt, and most of our internal services use it. There are of course issues with this. In lots of cases our real-time metrics are parsed and pushed to Librato from the logfmt interface as well. It’s convenient as hell, though it does come with some overhead.

I don’t agree at all with the suggestion that you shouldn’t log things like HTTP requests in a production system. At Joyent, we have found it immensely helpful to log all kinds of data about request processing, including latency breakdowns for different processing phases. We also throw in a per-request UUID so we can track work from the frontend through backend services and even into some database queries. In a distributed system you need all the help you can get, and logging well can be a huge source of useful diagnostic data.

Our logs are almost all in the bunyan format: linefeed separated JSON with some standard fields. Having a standard machine-readable format with rich, but easy to use, tooling for humans has been extremely valuable.