Resolve Leap Second Issues in Red Hat Enterprise Linux

Rate

Available Translations

Premium Content

This Article has been reviewed for technical accuracy, optimized for search, and integrated with Product Documentation and/or Red Hat Access Labs. Much like when a software package is accepted upstream, this content has moved from the general KCS editing workflow into the responsibility of Customer Content Services as maintainers.

Contents

Introduction

Leap seconds are a periodic one-second adjustment of Coordinated Universal Time(UTC) in order to keep a system's time of day close to the mean solar time. However, the Earth's rotation speed varies in response to climatic and geological events, and due to this, UTC leap seconds are irregularly spaced and unpredictable. This article seeks to provide information regarding the leap seconds themselves, along with addressing how Red Hat Enterprise Linux handles the insertion of one.

Upcoming Leap Second Events

Environment

Red Hat Enterprise Linux versions 4

Red Hat Enterprise Linux versions 5

Red Hat Enterprise Linux versions 6

Red Hat Enterprise Linux versions 7

Understanding the Leap Second

The basic time for mostly all of the world's local time zones is called Coordinated Universal Time, UTC, which is derived from a bunch of atomic clocks which are distributed in several countries all over the world. The rotation of the earth is not very constant and varies a bit over time, while decreasing the mean rotation speed slowly. This is the reason why so called leap seconds are inserted into the UTC time scale, they adjust the UTC time to the real earth rotation.

Why this extra second? It exists because the rotation of the Earth on its axis, which determines the passing of days and nights, slows down over a long period, mainly as a consequence of Moon-Sun attraction effects. In addition, the Earth is affected by its internal (core, mantle) and external (atmosphere, oceans) constituents. Nowadays, though, time is measured largely by 250 atomic clocks belonging to several countries, which function by measuring the transition of energy levels in an atom. Together these clocks are used to calculate UTC, and as this time measuring mechanism is independent of the Earth periodically the two must be brought into alignment with a leap second. In addition, we have to consider that the length of the day is nowadays 2 ms longer than in the year 1820. Not surprisingly then, the Earth's rotation slowly gets out of synchronization with UTC.

The International Earth Rotation Service, IERS, is measuring the true earth rotation and determines when a leap second has to be inserted. Insertion of a leap second is always scheduled for the end of the last day of a month, preferably at the end of June or December, at UTC midnight. In the past all leap seconds had been inserted at either one of those times(*). Announcements whether a leap second is scheduled or not are published by the IERS in their Bulletin C. The current Bulletin C is published about half a year before the next possible date for a leap second.

The IERS Bulletin C #30 from July 2005 announced a leap second to be inserted at the end of December 31, 2005, at UTC midnight. This was the first leap second that had been inserted since the end of 1998. This is why many applications which had been developed during the previous 7 years could not handle the leap second correctly.

Since, the leap second was inserted at the same moment all over the world, the local (civil) time of the insertion depends on the local time offset from UTC, e.g. if the time zone is UTC +3h then the leap second will be inserted when the wall clocks show 3 hours after midnight.

If the system clock is kept in TAI and a right/* timezone is used then 23:59:60 may be listed; however, as 23:59:60 does not exist in Unix's implementation of UTC then the linux kernel inserts the leap second by stepping the system clock back by one second on the first clock update after 0:00 UTC. In local time counting depends on the time zone offset, e.g. for UTC+3h you would observe the following:

Leap seconds are a discontinuity of civil time. The time does not continue to increase monotonically but it is stepped by one second. Let's have a look at the time stamps of a leap second which is inserted, and the second after that leap second:

2011-12-31 23.59.60 <-- leap second
2012-01-01 00.00.00

We can normalize the time and date of the leap second:
60 seconds are 1 minute, which lets the minutes increment from 59 to 60
60 minutes are 1 hour, which lets the hours increment from 23 to 24
24 hours are 1 day, which lets the date increment, and so on.
Finally we can say that both lines represent exactly the same time, or 2 consequent seconds have the same time stamp.

Several time dissemination services also propagate the announcement of a leap second after this has been determined by the IERS. This includes for example the German long wave transmitter DCF77 and the satellite based navigation system GPS, so receivers which decode the signals from those systems can also decode the leap second announcement. Applications which read the time from those receivers can also determine the leap second announcement if this information is included in the applied protocol, e.g the time string transmitted by the receiver.

Please note that time code receivers can only pass the announcement of a leap second to the application, and count the time correctly over that period. It's the task of the application and/or operating system to handle that leap second correctly.

(*): An additional leap second notification was erroneously issued by some NTP (Network Time Protocol) servers within the North America region for midnight UTC on August 31, 2012. Two stratum 1 servers -- truechimer.cites.illinois.edu and time-b.nist.gov -- were identified as erroneously issuing the leap second notification. Any higher stratum (stratum 2, 3, 4....) NTP server with either of these two servers in their hierarchy should be considered as having issued the leap second incorrectly. NTP clients using these servers will set the kernel's leap indicator flag, which cannot be cleared until the system is rebooted.

Handling of the Leap Second

The leap second will be handled in two primary ways, depending on how the system clock is configured.

The system clock is keeping time with UTC and synchronized via NTP or PTP

As mentioned previously, the leap second must be skipped when the system is using UTC, as this second does not exist. Therefore, to prevent the system clock from being one second ahead of UTC, the clock must be corrected after the leap second occurs. This correction may occur in a variety of ways, outlined below:

Notify the kernel, allowing it to step back the clock

The default approach by most daemons is to notify the kernel, and allow it to insert the leap second. For instance, during the last day before a leap second correction NTP servers should notify their clients that a leap second will occur, and at 23:59:59 UTC, the Linux kernel should add or remove an extra second by making the 60th second occur twice or removing it entirely. Thus, Red Hat Enterprise Linux systems running an NTP client, with a default configuration, during the last leap second correction should have counted time as follows:

When the leap second occurs, the kernel prints a message to the system log.

Have the daemon step back the clock

Instead of notifying the kernel, and allowing it to step the time back, the daemon may step the system clock backwards. Time will be counted similarly to the kernel inserting the leap second; however, as the daemon performs the actual insertion any bugs specific to the kernel will not be encountered.

Slew the leap second

RHEL 6.8 and RHEL 7.2 introduced an option to slew the leap second for systems using Chrony, allowing this second to be spread out over a period of time instead of an immediate step back in time. This slew is handled as a one-second offset accumulated at 00:00:00 UTC, for leap second insertions, or 23:59:59 UTC, for leap second deletions, that is smoothed out over time. This method may be preferred over stepping when applications running on the system are sensitive to jumps in system time and it is acceptable that the clock will be off for a longer time.

Ignore the leap second

Instead of addressing the leap second daemons may be configured to ignore it entirely, allowing time to be slowly corrected as any other unexpected change in time. The clock will continue with the incorrect time until it is addressed through normal operation. If multiple systems are configured with this method it is possible that their times will drift apart from each other, as the correction process is random to a certain extent. This method is not recommended for systems that require in-sync communication.

Leap smear on NTP Server

RHEL 6.8 and RHEL 7.2 introduced an option to smear the leap second when Chrony is running on the NTP server(s). Instead of having the clients slew the leap second, it may instead be suppressed on the server and slowly correct the served time by slewing the leap second instead of stepping it. Clients do not need any special configuration, as they do not know there is a leap second, and will continue to follow the server time until they are in sync with UTC. Smearing the leap second on the server is much slower than doing so on individual clients; however, it allows all clients to connect.

When configuring a NTP server to smear the leap second it is necessary to ensure that all clients using it are only pointing to servers that are performing the smear in an identical fashion. If clients are configured to receive updates from multiple servers, and these servers handle the leap second in a different fashion, inconsistent results will occur.

The following table demonstrates the effects of each configuration method:

Client kernel step

Client daemon step

Client slew

Client ignore (ntpd -x)

Server slew (smear)

Continuous time

No

No

Yes

Yes

Yes

Clock off for

1 second

1 second

Seconds or minutes

Hours

Hours

Frequency error

Zero*

Zero*

Large

Small

Small

Kernel bugs

Yes

No

No

No

No

Clients stay synchronized to one another

Yes*

Yes*

Yes

No

Yes

Ignoring the discontinuity in time

The system clock is keeping time in TAI-10 seconds, is not synchronized, and is using a right/* timezone

Systems may also be configured to report time corrected for leap seconds by updating the tzdata package to the latest version available, copying the appropriate file from the /usr/share/zoneinfo/right directory hierarchy to /etc/localtime, and resetting the clock to the correct local time. The files in /usr/share/zoneinfo/right contain local time information corrected for all leap seconds that have occurred since the beginning of the Epoch on 1970-01-01 00:00:00 UTC. The other time zone files in /usr/share/zoneinfo do not have leap second corrections added. After the 2008 leap second, there were 24 leap seconds added since the Epoch. Any application that expects the time to be in UTC will have issues if a right/* timezone is used.

As an example, if a system is in the America/Los_Angeles (US Pacific) time zone, you can reconfigure the system to report leap-second-corrected time by running the following and resetting the clock to Pacific Time:

cp /usr/share/zoneinfo/right/America/Los_Angeles /etc/localtime

After /etc/localtime has been changed glibc will reload this file automatically; in addition, note that adjusting timezones does not change the system time, only the conversion from the system time to local time that happens in applications via glibc functions, such as localtime(), ctime(), etc. A system restart is not required after this update; however, if any applications are caching results from these functions then these applications may need to be restarted after tzdata has been updated.

Note: Red Hat does not recommend running without a source of time synchronization enabled as there is no guarantee that systems will remain in sync with each other. Time drift may occur on each system as determined by its clock source. The decision to run without NTP/PTP/Chrony will need to be made on a per-system or per-cluster basis with the understanding that time drift may occur.

By default, Linux systems not using NTP or PTP to synchronize their timekeeping will not correct for leap seconds, and the time reported by these systems will have a one-second difference relative to UTC after the leap second correction. You should reset the clock manually after leap seconds occur.

For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 4 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el4, or a later version, to receive this fix.

For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 5 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el5, or a later version, to receive this fix.

For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 6 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el6, or a later version, to receive this fix.

For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 7 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el7, or a later version, to receive this fix.

In addition, Red Hat strongly recommends that customers test their builds and environments, and instructions on how to test, including a sample program to do so, are provided at Are we susceptible to a leap second event?.

Example Configurations

Stepping the leap second with Chrony

To step the leap second, add the following options to the /etc/chrony.conf:

leapsecmode step

These options will instruct chronyd to correct the system clock by stepping the leap second, bypassing any bugs with the kernel's insertion of the leap second.

Slewing the leap second with chrony

To slew the leap second, add the following options to the /etc/chrony.conf:

leapsecmode slew

These options will instruct chronyd to correct the system clock by slewing the leap second instead of stepping back immediately.

Smearing the leap second for clients on chrony NTP server

To smear the leap second, add the following options to the /etc/chrony.conf:

leapsecmode slew
maxslewrate 1000
smoothtime 400 0.001 leaponly

These options will instruct chronyd to correct the system clock by slewing the leap second and limit the slewing rate of the local clock to 1000 parts per million (ppm).

Ignore the leap second with NTPD

To ignore the leap second, allowing it to be corrected by normal slewing operations, ensure the following line is present in /etc/sysconfig/ntpd:

OPTIONS="-g -x"

With this flag added the kernel's leap second correction is disabled, and the leap second will be corrected over a period of time.

Additional Resources

Additional information on leap seconds and how they are handled in Linux and by NTP can be found at the following links:

All of my tomcat6-on-RHEL6 servers froze. You could still ssh in, tomcat will still give some responses, but a server that could normally handle 1000 req/s was unable to handle 100 req/s until after a reboot.

All of my RHEL6 machines are pegged at 100% CPU and very high load. Restarting services doesn't help and disabling NTP doesn't do anything. Splunk is freaking out and causing issues for me. All of the RHEL5 boxes are fine. I ran "uptime" against my RHEL6 boxes using mcollective, and all the servers are at a load of 5 or higher. What a mess.

"I have updated the RHEL server using steps "Systems not running NTP or PTP".
Manually updated the latest tzdata rpm to "tzdata-2015a-1.el5.i386.rpm" . But I am not sure how to test this fix as it working fine.

Now to check the vulnerability used the " Leap Second Vulnerability Detector" tool and the result says "Not vulnerable".
$ ./leap_vulnerability.sh
Installed kernel version: 2.6.18-371.1.2.el5PAE
Installed tzdata version: tzdata-2015a-1.el5
Not vulnerable

Now we are confused as zdump output says leapsecond is coming and the "Vulnerability Detector" tools outputs says we are safe.

As long as the kernel is updated you should not have to worry about a crash; the tzdata package is primarily for when the right/* timezones are in use. If a timezone from the right/* directory is not in use then you will not require the update to the tzdata package.

It won't cause any issues (and we typically recommend running the latest packages if possible), but it will not be necessary for the leap second if you're not using a timezone from the right/* directory.

Hi Christian. Is not big deal. Already talk to RedHat support. Forget about the flag. You turn off ntp in order to avoid a possible brutal desynchronization (+leap second) that some kernels will not manage. You turn it on a few hours later and that's it.

Hi All,
In our environment we are synchronized with a 3rd party Vendor NTP server using Solaris 9 NTP server.All our client server are RHEL 4/5/6 and is synced to the Solaris server. And as we confirmed with the 3rd party vendor they will insert a leap second at June 30th, 2015 at 23:59:60 UTC.
Can someone help to clarify below concerns.
1. To avoid any impact to all clients is it safe to run the NTP server within our premise in slew mode ?
2. If we enable slew mode in server side does all client also need to be changed to slew mode ? ( does change in NTP server side alone fix ?
3. As confirmed with Redhat in a separate case , if the kernel versions are updated to the recommended level then we shouldn't hit the OS crash issue when leap second insertion occur. However High CPU can occur for some applications irrespective of kernel version or ntp package version , Can slew mode can be used to avoid this situation ?
4. Any other tips to better handle this situation would be much appreciated.

I am under the impression from the redhat article that if running ntp client that I should not worry about a kernel crash. I have some servers that need a kernel update however my oracle DBA's do not want me to update the kernel because their ASM disks /apps will no longer work with a new version, they would need to rebuild and tweak their apps. But what I am reading in the posts is that if the kernel is not updated to the latest version, then when the leap second happens, this could cause a kernel crash. .stuck between a rock and a hard place

for the leap second my servers are running ntp client. the tzdata is up to date, but there are a couple of kernels that need to be updated but our oracle DBA's do not want the kernel updated. Do i need to stop ntp and restart after the leap second happens or run ntp in skew mode, or just keep ntp running and hope that no kernel crash happens.

when I run yum list tzdata, I got only tzdata 2015d-1.el5 available for update but in package change log this one showing related to Egyptian DST stuff, but when i check the same for tzdata 2015a the one showing leap second issue.

Now guys could someone correct do we need to go for 2015a or 2015d version and how to get only the version related to leap second issue (may be 2015a)?

For RHEL4, the latest ntp tzdata package is tzdata-2015d-1.el4. The latest package is shipped via RHEL 4 Extended life cycle channel. To access the channel, you need to have Extended Lifecycle support subscription

Thank you Ranjith for your response. We do have the Extended Life Cycle support subscription, I updated the base channels. however we have 4.7 and the channel that i had to choose from was 4.8 and i did not see any errata package for tzdata. Am i looking in the wrong place. Also we have some kernels that should be updated, however Oracle DBA's do not want to update because it will mess with their ASM configurations/applications. I have ntp client running. Should i be worried about a kernel crash?

Contact Red Hat Technical Support if you need assistance for subscribing the system to the appropriate ELS channels

Regarding kernel update

As mentioned in this article, For RHEL4 make sure kernel is above kernel-2.6.9-89.EL

If you are not able to update the kernel due to application dependency, then disable ntpd service on the system prior to the event (minimum 24-48 hours) and then start it post the leap second event.

Some NTP servers can notify the ntp clients about leap second even prior to 24/48 hours. using the command ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum" name1.test.com | grep -E "leap=([^0].|.[^0])" you can check whether the ntp client has already received the leap second notification. If you notice leap=01 in the above command output, then it has received the leap second event.

If the NTP client has received the notification, then you cannot clear the leapevent from the kernel side. You will have to reboot the node. Prior to reboot disable ntpd service from autostarting (chkconfig ntpd off) and then after reboot chkconfig ntpd on; followed by service ntpd start

So it would be better to verify whether the RHEL4 received the leap second notification if not then configure ntp service in slew mode

If you have any further questions, I would recommend to contact Technical support so that we can review our environment and answer your questions better

If the NTP client has received the notification, then you cannot clear the leapevent from the kernel side. You will have to reboot the node. Prior to reboot disable ntpd service from autostarting (chkconfig ntpd off) and then after reboot chkconfig ntpd on, Configure NTP in slew mode and then start ntpd service

"Systems running any version of Red Hat Enterprise Linux should automatically account for leap second corrections if they are using the NTP (Network Time Protocol) daemon to synchronize their local timekeeping with an NTP server."

This is not technically correct. If the NTP server that you point to does not have a leap file with leap second data accounted for, then the NTP client will not get the leap second. You will need to check for the leap second flag on the client if you do not own the NTP server.

I got below Workaround from Redhat. No kernal upgrade, no tzupdate. No reboot server.

Red Hat strongly recommends that customers update their systems to the most recent available Red Hat Enterprise kernels.

If it is not possible and you want to avoid the Leap Second insertion, you should disable ntpd daemon by running "service ntpd stop" before Jun 30th.

At some point in the future you might want to sync the systems again with the rest of the world.
These methods could be used:

stepping the clock: when running ntpd again ("service ntpd start"), the clock will be stepped

correct the clock slowly: ntpdate -B could be used, the correction takes about 2000 seconds. After that, ntpd can be started again and the error will be already corrected, so ntpd won't need to perform any steps.

yes, you can go for workaround listed here. https://access.redhat.com/articles/199563.
Note1:- On RHEL 5, stopping ntp must be done before last 24 hours. Once kernel receive that informations, it can't reset.

Abdul, in order for tzdata to correct for the leap second you have to copy over or soft link to the right/* timezones to /etc/localtime (rpm -ql tzdata | grep right).

But generally speaking we don't recommend that you do that (as noted above). As with a disabled ntp your systems can stray away from each other at different rates. So you'd probably be better off just using slew/-x option or just go with the default ntp configuration that will step.

The major issues last time around were do to hangs, these have all been fixed with kernel updates.

Usage of ntp configured with -x/slew will make your systems take longer to time correct when not going outside of the step threshold (600s)

Default configuration of ntp (without -x) will cause your system to step, but it should time correct much quicker than a typical slew correction.

We've documented two best approaches to workarounds for leap second insertion concerns.

How to clear the Leap Second Insertion flag after it has been received?
https://access.redhat.com/articles/199563

Now if you've been able to update your kernels, you're most likely going to be fine with ntp running in step (without -x) mode or in slew (-x) mode without having to worry about those two workarounds noted above.

Hi All,
Am I right saying that after the kernel is up-to-date being not vulnerable the server either needs to have a tzdata installed newer than tzdata-2015a-1.el5 if NTP is not running. I understand this does not need a system restart but is ntpd needs to be restarted?
Also in case of using NTP, can the leapsecond notification passed over multiple hops? Meaning that if my system's NTP server(not having up-to-date tzdata) but pointing to the RedHat pool then everything should be fine? Thanks

If i have a system running at a kernel level which could hang once it receives notification of the insertion of a leap second and i'm running the latest version of tzdata that handles the leap second what could happen? I believe the system could still hang if it's on a unpatched kernel as above?
Thanks