“Time keeps on slipin’ slipin’ slipin’, into the future” — “Fly Like An Eagle”, The Steve Miller Band

Your web server’s system time can keep on slippin’ far into the future if Network Time Protocol (NTP) is not properly configured. Having accurate system time is critical for application logic, scheduled jobs, and of course logging. With logging, if the system time is off, log forensics and log correlation of security events across systems become a nightmare, if not impossible. Many software components on your servers rely on accurate system time, including the Signal Sciences agent. The agent uses the local system clock to generate timestamps when logging detections and other events. As a result, ensuring accurate system time on all your web servers is a must for robust security — and this is especially true for virtual machine based deployments.

Why do I call out virtual machines? Our agent has been deployed in numerous environments and on thousands of machines (bare metal, virtual machines, containers, IaaS, PaaS, and whatever else you might imagine). In new deployments, virtual machines without NTP configured stick out like a sore thumb since their system time is way off. We’ve seen ranges from tens of thousands to hundreds of thousands of seconds off into the future! This drastic time skew has a severe impact on security related events since the timestamps can be so significantly skewed. One example of impact is when presenting events in a dashboard that is time-based, e.g. displaying all events within the last 6 hours. You won’t see the data if its timestamps are several hours in the future. Time skew can happen on any infrastructure, but if you are running virtualized infrastructure you’ll want to pay close attention.

In addition to NTP not being configured, there has been research published on attacks against the NTP protocol. Some of which result in shifting time on NTP clients. This research is published here and referenced in this Ars Technica article. Another threat consideration is a malicious insider, who could modify system time in attempts to hide events or manipulate time sensitive transactions.

“Watching” Time

How can we mitigate the risk of skewed timestamps, missing data, and possible attacks on system time? First, NTP is a beautiful thing. Make sure you have a consistent NTP implementation across all your servers. Second, periodically verify the system time on your servers is not skewed beyond an acceptable threshold. Fortunately for Signal Sciences customers, you already have visibility into your server time! Since time is such a crucial part of security, our agent provides visibility on your server clock skew. This is just one of the many metrics reported on by the agent, and it can be found via the UI by navigating to the agent’s graph page. On that page, scroll down towards the bottom to find the Agent Clock Skew graph.

The screenshot above is from a VM, where the clock skew is obviously drastically off until it was corrected.

The screen shot above is from an agent running in a popular PaaS environment. A one-second clock skew is relatively negligible, but depending on your circumstances perhaps it could be an issue.

In the next section, I’ve published two scripts, one is a proof-of-concept Bash shell script that can be used to help monitor system time across many servers.

Monitoring Clock Skew

Ok, visibility on clock skew — great! Pretty graph — great! But what if you are like many of our customers and have hundreds of agents deployed. That’s a lot of looking at pretty graphs to find hosts with clock skew — not great! Good news, Signal Sciences makes all data available via its API — great!!! This means you now have the ability to automate monitoring of clock skew across your web server farm.

With the API you can integrate with any monitoring system you may have, however, I’ve provided an example of checking clock skew with just two scripts. The first script, SigSciApiPy, handles authenticating and pulling agent metrics from the API. The second script is shown below, clock_skew.py, loops through the agent data looking for the value of host.clock_skew. Host.clock_skew is the median number of seconds that the particular agent’s clock is skewed compared to the time on our servers, over the past 5 minutes. So for example, if we have “host.clock_skew”: 30, that means that we saw a median skew of 30 seconds over the past 5 mins. The clock_skew.py script will print a warning or alert message to the console based on defined thresholds.

Note, if you do use clock_skew.py, you’ll want to set the WARN and ALERT threshold variables to values that make sense to you.

Below is a script that can achieve a similar result in monitoring for clock skew. If shell scripts are not your thing, there are many other tools that may be a better fit in your environment. For example, monitoring systems like Nagios or Zabbix, or configuration management tools such as Ansible, Chef, or Puppet. I’m sure there are other, and even better, ways to monitor for clock skew. If you have suggestions please add them in the comments section.

Monitoring for Time Skew Is a Security Must

I think we can all agree that ensuring you have accurate system time across your servers is critical for security. Also, having the ability to monitor for time skew is equally as important. I hope the example scripts I’ve provided help to jump start your clock skew monitoring efforts in a timely manner.