Rapid7 Blog

Dynamite Plots, Logs, &amp; the Joy In Knowing

POST STATS:

SHARE

I saw this online and chuckled.

I believe it was Mark Twain that said, “Humor is the good natured side of truth.” Well, in my humble opinion, humor can be used as the genesis for interesting blog posts. So, you may be wondering, what do Dynamite Plots, logs and the joy of knowledge have to do with each other?

Well, if you have a few minutes, I’ll try to make the link.

For me, the humor of the cartoon above is rooted in multiple areas, but most of all in the idea that the data inherently has variability; if you don’t know the whole picture and aren’t careful in your analysis, bad things can happen.

Log data provides the most granular view into what is happening across your systems, applications, and end users. Logs can show you where the issues are in real-time,* and* provide a historical trending view over time. Logs give you the whole picture.

Yesterday we announced a new Community Pack for our friends at Fastly. I’m a big fan of Fastly. Not only do they have a compelling mission of changing the way people experience the Internet, but I like their simple assertion that slow is unacceptable.

With the new Fastly Community Pack, users of Logentries and Fastly have an out-of-the-box way to easily consume their CDN log data, and analyze that data to assist in better decision making (aka, not being slow).

And, while CDNs are critical for quick content delivery, any DevOps person will tell you, modern applications are comprised of complex, interrelated, dependent systems and components. These modern applications often work well in a controlled, static environment, however the real-world is often full of little surprises and unexpected occurrences.

Thus the increasing importance and value, of log data.

If your application is your business, and it’s the basis through which you provide a value service or engage with your customers, neither your business nor your application can be slow.

However, like beauty, slowness can be in the eye of the beholder.

If a user complains your application or site is slow (or, if your application starts throwing off exceptions; the number of users starts to drop; other key business metrics start heading in the wrong direction), important questions need to be answered. Let’s assume you’re running a Java application, with a Nginx web server, on AWS EC2 with a few dozen Linux instances, with auto scaling setup to handle changes in demand. Let’s also assume you have a couple of Cassandra instances running, and a Postgres database on the backend. Now, let’s call this a fairly common scenario. In fact, here at Logentries, we’d call this an everyday customer environment. I think we’d all agree in this case, and as Fastly asserts, SLOW = BAD.

In their essence, logs allow you to answer the question of why?

Why is my application slow?

Are there requests by users in specific countries that aren’t receiving an appropriate response time (i.e. how’s my CDN performing)?

Why are there HTTP status codes being returned (the dreaded 400, 404, 413 or others)?

How’s my web server doing?

What’s going on with my AWS services?

Should the developers take a look at the code (i.e. let’s look at the application logs)?

Can you combine these all together, with your Linux OS logs, to easily aggregate, correlate and analyze what’s going on to find the needle in the haystack that tells you how to improve your application performance and appropriately tune your application (i.e. what’s my end-to-end application performance look like)?

Where do I focus my troubleshooting efforts?

This is why we built Logentries — to help users easily analyze and understand their application end-to-end, and top to bottom. All in real-time, regardless of the environment.

To help deliver more value out of the box for our 35,000+ users from around the world, and to help us open up logs (and Logentries) to virtually anyone, we’ve unveiled the Logentries Community and free, downloadable Community Packs.

As technology is often expensive to use, complex to setup and difficult to manage, and traditional log management and performance monitoring tools are equally as expensive, complex and difficult (if not more so), we’re on mission to Democratize the Power of Log Data.
And like any mission, we believe in a few fundamentals:

Sweet Data Harmony – Logs provide the ability to ask the tough questions. If you have visibility into the logs you can be confident the answers are there, or can be more easily discovered.

Better Insights, Better Decisions – If you have the data, and you have the freedom and flexibility to explore it, you increase your likelihood for better insights, and therefore better decision making.

Operational Insights for Everyone – Our friends at New Relic have coined the term Data Nerd (We’re not only a New Relic partner, but also believe in the Data Nerd cause). This notion that a healthy obsession for data can help improve outcomes. And as logs are the ultimate data source, used universally by CDNs, applications, mobile devices, servers, IoT systems and much more, the insights they provide can help almost anyone, from Developers, QA testers, IT operations teams, to product, marketing, and sales managers.

We are here to enable you to Revel in Knowing. There is a certain joy in knowing what’s happening and why, to be able to make better decisions, and as needed, take calculated risks.

In a world where the volume, variety and velocity of machine generated log data is only increasing, and as the world around us is increasingly connected, a new day in data is emerging. Whether you’re a data scientist, a data nerd, or just trying to figure out why your application is slow, there is joy in knowing. You could almost Revel in it…