Synchronizing Networks with NTP

Is Your Network Running On Time?

Most system administrators don't consider the importance of
accurate network timing. Under Unix, log daemons such as
syslogd write their information to the central log files
in 00:00:00 (hour:minute:second) format.
tcpdump writes information in the
00:00:00.000000 format. Cron jobs depend on an accurate
system software clock to execute processes. Apache timestamps its
logs.

Why Do We Need Accuracy?

If your server doesn't keep accurate time, your log files are
useless in the event of an incident that requires log-dependent
information, including security breaches. E-mail servers and other
clients depend on accurate time to relay, send, and receive data. What
good is the date stamp contained in an e-mail if the server that
passed that information is inaccurate? These programs all must be
timed precisely to within 1/100 of a second.

Last year our
network suffered a power outage. Until I started reviewing the
data in the log files, the exact time wasn't clear. I was pleased to
find accurate information that pinpointed the exact time of failure on
each separate A/C line and UPS in the facility for each server.

Around the same time, another network I administered was
breached. In this case, each server reported to a remote log server
via syslogd that was timed to a Network Time
Server. After the system was attacked, the perpetrator attempted to
cover his tracks by altering the local logfile on the compromised
hosts. What he didn't realize was that another log existed -- "the
real McCoy" on the remote monitor, containing unaltered accurate
information and synchronized via NTP. The information I obtained
proved invaluable and helped to find and prosecute the intruder. When
presenting evidence to law enforcement, the first question they asked
was, "How accurate is the time on your logs?" I was quickly able to
prove the accuracy of the system's time.

Many organizations use a time standardazation protocol called NTP (Network Time Protocol). A team of
professionals at the University of Delaware maintains it.

The Time Protocols

NTP is based on its own unique protocol. Today, NTP has become the
de facto standard as it has evolved through various RFCs.

The most recent release is a complete implementation of version 4.
It retains compatibility with version 3, as defined by RFC 1305, and
versions 1 and 2, defined by RFC 1059 and RFC 1119,
respectively. NTP v.4 implements asymmetric encryption, to prevent
malicious users from spoofing time packets over port 123. There is no
formal RFC for version 4 yet, though there is a
draft for using public key cryptography with NTP.

The NTP Chain of Commands

An NTP primary server, or stratum 1, is connected to a
high precision reference clock and equipped with NTP software. Other
computers (stratum 2s), equipped with similar software automatically
query the primary server to synchronize their system clocks. The
stratum 2 computers can synchronize other computers, called stratum
3s, and so on, to 16 stratums. The further a computer is from stratum
1, the less accurate its timing information. In this example, each
computer can be a client of a server in a higher stratum and a server
for computers in a lower stratum. A single primary server can
indirectly serve an unlimited number of clients.

To increase reliability, each client may query multiple servers in the
upper stratum. In this case, the NTP software continuously monitors the
stability and accuracy of all configured servers, switching dynamically to the
server with the best figures.

Where does a stratum 1 get its initial data? A stratum 1 requires
a radio or satellite receiver. Modern systems retrieve their data from
the NIST radio broadcasts or GPS
data.

Implementation

NFS, the Network File System, has long been known to have remote
exploits and vulnerabilities. Even secure NFS has weaknesses. However,
NFS is also a very reliable means of copying data to a central backup
server. Several systems I work with implement NFS for that very
reason. Here's where the NTP comes in handy. Each server runs a cron
job at a specified time to execute its backup. The backup job has the
ability to enable and to disable the NFS server. Although I
incorporate NFS, I don't allow it to be on all the time - only during
automated backups.

Here's the order of operations. First, the cron job fires on the client.
It starts NFS. The server knows when this should happen, so it mounts the
client share. The server backs up the files on the share and unmounts it.
Finally, the client stops NFS.

Once the backup server and clients have synchronized, the script
executes on each machine. This is all based on extremely accurate,
split-second timing. After the backup is complete, the scripts
disable NFS. If one machine is out of sync, that machine won't be able
to perform a backup to the central server.

The Importance of Choosing an Accurate NTP

Three years ago, I had several servers timed to a single
NTP server located at a university. Unbeknownst to me, the
administrators were checking for the Unix 2038 time overflow bug.
During their test, they advanced their real time clock to the year
2038. Suddenly, each of my log files started writing the year 2038
instead of 1999. This caused massive problems for systems across the
network. The lesson learned was that most NTP clients have the ability
to query multiple stratums, writing the average of difference to the
software clock. Where the difference is significantly out of
sync, the client will either compensate to the clock closest to the
last known value or exit, leaving the local clock(s) unchanged.

Choosing an NTP Architecture

Arguments abound on how to implement NTP on a local network. Some say that
a (single) local server should be designated to poll the upstream stratums,
synchronize, then broadcast to the local network. Others believe the local area
hosts should (directly) poll the upstream stratums. Most modern systems use
the first approach.

The advantage of an internal stratum is so that all machines on your LAN
may be timed to the local network. Only one machine needs to be timed to the
outside. The disadvantage is that downstream systems may be affected if your
local stratum goes off sync.

We'll use a Unix host running ntpd to connect to a stratum 2
across the Internet. Our local machines will each synchronize to the local
stratum.

The ntpd Server

The ntpd daemon can operate as a server, a client, or a relay
from a set of servers to a dependent client population on a local net.

To configure ntpd, edit /etc/ntp.conf.
As the ntp.conf man page is very long, listing every
option is outside the scope of this article. However, useful
configuration commands include server, peer,
broadcast, manycastclient, and options
include autokey, burst, iburst,
minpoll and maxpoll.

Here is the basic ntp.conf for "timekeeper", a stratum
3 on our internal network. Note each server listed is an official
stratum 2.

In each example, you must touch the file /etc/ntp.drift, which
contains the offset differences of each upstream stratum. Before attempting to
retrieve data, wait until ntpd has had time to synchronize.

Start ntpd by running it from the command line
without any arguments. Be sure to add the daemon to your local start
script. Once the server has been started, check the offset by tailing
your messages log. You should see a line resembling:

Nov 13 03:16:52 striker ntpd[31962]: time set -0.096919 s

If you are running in a production environment, I suggest reading
the man section pertaining to restricted access and
encryption. ntpd runs on port 123. Be sure other
applications such as PortSentry are not
bound to this port.

Running NTP Clients

I recommend running NTP software hourly from your cron. Most
Unixes provide ntpdate. (Some Linux distributions
provide a program called netdate, but it does not support
version 4 of the protocol.) Both commands follow the same syntax:

$ ntpdate <primary time server> <secondary time server>

For a Windows server or workstation, you can download and install
a program called Dimension 4.
This program provides several options for checking and correcting the
time, and has an excellent, up-to-date list of time servers.

Recent versions of Mac OS X provide a simple checkbox under the
Network Time tab of Date & Time settings.

Summary

Time servers are a science in themselves. As we speak, scientists
are working on improvements. If you take time synchronization for
granted, then I suggest you read more about this fascinating
technology. For those of you worried about the 2038 bug, relax. In 36
years most Unixes as we know them will have been improved; many of us
will be retired on a beach in Bermuda or so we hope.