Data architecture for event log metrics?UPDATE over a year later we built a system that logged everything, and periodically iterated over the logs counting things, and then stored those counted numbers into a database (could/should have been a a time-series database, but MySQL sufficed). This was a few weeks of work but ended up being a surprisingly powerful/fast approach--when it's just your code iterating over logged JSON, it's easy to add a lot of metadata, and easy for your code to have flexible rules for exactly what it wants to count.

Data architecture for event log metrics?@hiwaylon We've ended up using a hybrid system: 1) MySQL where possible (low volume) (makes aggregation easy using SELECT...GROUP BY, can easily store the results of SELECTs), 2) using Graphite for simple large-scale aggregation and visualization, and 3) logging full events for reference, and for watching details of the data flow in in real time. Each has actually been valuable in different ways.