Rapid7 Blog

POST STATS:

SHARE

Once you learn what the normal baseline on your network is, you should start to look for abnormal activities. This will help you understand any sudden changes that might affect the overall performance and help in easier troubleshooting. If you are already caching with Fastly, you might seek to pull more information from your log data to give you deeper visibility. There are a couple of formats in which you can send logs from Fastly to Logentries. You get a choice of either using Fastly’s default logging format, which is essentially an Apache Log Format, or you can jump into custom VCL (Varnish Configuration Language) and use the available VCL extensions to add more details to your log events. In this article, we take the second approach and present techniques that might be used by Fastly users in conjunction with the Logentries real-time log management analytics service.

equipped with a number of powerful “plug and play” Tags, Saved Queries and Visualizations to help analyze your Fastly log data. The Community Pack can be easily imported into your Logentries account so that you can get quick visibility into your Fastly logs.

What’s included in the Fastly Community Pack?

Searches:

Device Detection

Unique IP Count

Requests by: country, region, Fastly datacenter, country code, country name

Status Codes

Total Hits over time

Average Hits over time

Total Data downloaded

Tags & Alerts:

200-OK tag

404-NOT FOUND alert

20% increase in a download of files larger than 1MB alert

Application Not Accessed for the past 24h alert

iPad user tag

iPhone user tag

Android user tag

Desktop user tag

Graphs:

Device Detection pie chart

Unique IP counter

Requests by Country

Requests by Fastly Datacenter

Status Codes bar chart

Hits over time line chart

Average Request Size line chart

Total Data Downloaded

Fastly logs are fitted out with many interesting caching related variables by default. In addition to the above, the Pack allows you to directly configure caching using VCL extensions. VCL together with a powerful log analysis platform enables you to view data from different angles and makes log analytics more dynamic and valuable to monitoring your overall infrastructure. To get a better understanding of the process we will present a couple of interesting approaches you can introduce to the standard VCL file.

Simple Device Detection Mechanism

First, use one of the VCL extensions variable req.http.User-Agent in order to get started and throw in few simple if-statements at the top of the main VCL file.

Whichever type of machine or browser used to access your cached application via Fastly gets picked up by req.http.User-Agent. The above code example is simply a customization and shortening of already existing user agent standard format to make the log output more readable and to allow for easy identification of different device types. For example, you might be interested in a creation of a set of tags to identify distinct mobile users from desktop users. Using the above code, you can store the type of a device in req.http.X-Device variable which can then be used in your log output as follows:

Once you have everything set up and are successfully feeding Logentries with Fastly logs, it is time to make use of the data and perform some interesting searches. First, you are able to create a pie chart to outline the breakdown of devices used to access your cached application. This and many more searches are included in the free Community Pack.

You can customise your VCL to detect what type of machine accessed your application cached on Fastly.com

What about real-time tagging?

You are able to create custom tags using Logentries or simply import it straight from Fastly Community Pack (Please note that variable names in your custom VCL file need to be matching with variable names defined in JSON Community Pack File downloaded)

Custom tags are used to mark important events or actions

GeoIP Related Logs

One of the most exciting features of a VCL is the ability to feed your logs with approximate geographical location such as:

With current growth rate, Fastly is bringing online 1.5 POPs a month, and this will only increase as time goes on.

You can gain this data by adding your VCL extension to the main VCL file

User data like this has a great value and potential. The more you search, plot or analyze, the more you learn about your users.

Size Related Visualization

For improved analytics and more insight into what happened in a request, Fastly added size-related variables to VCL, such as:

Total size of a request

Total size of a header

Total size of a body

Total size of a response

Size of a header in a response

Size of a body in a response

Whether the response was successfully completed or not

Total data downloaded over a period of time in bytes.

You can add the variables above to your logs and use it to create Anomaly Alerts or graphs to visualize your network load and request characteristics.

Never enough tools!

Getting to know your users is crucial, but knowing your data is equally important. This is why the Fastly Community Pack was equipped with searches and graphs such as Unique IP counter, size related searches and requests information analytics. We encourage you to explore our sample anomaly and inactivity alerts included in this Pack to power up your first line of defense in application monitoring. The main reason why you would create alerts like this is to ensure that you always have up-to-the-second data about normal and abnormal activities on your network, enabling you to react more quickly and solve the issue before your users notice.