Implementing Frequency Synchronization over Ethernet

In today’s networks frequency and time synchronization over network elements is a key requirement for network service providers. Now, service providers widely use ethernet for data transmission due to low cost and high bandwidth. However frequency and time synchronization between network elements over Ethernet is a key challenge due to the asynchronous nature of ethernet.

Frequency synchronization over ethernet can be implemented in two ways:

Synchronous Ethernet (SyncE) - Synchronized Ethernet (SyncE) supports frequency synchronization over ethernet by leveraging the physical layer of ethernet. For more information on SyncE and supported line cards see Understanding Synchronous Ethernet

IEEE 1588v2 - IEEE 1588v2 is the IEEE standard used for frequency and time synchronization. While SyncE supports only frequency synchronization and each network element along the synchronization path needs to support SyncE, 1EEE 1588 v2 supports frequency synchronization over asynchronous networks. 1588v2 support requires 2-Port Gigabit Synchronous Ethernet SPA. For more information on IEEE 1588v2, see Understanding 1588v2.

Understanding Synchronous Ethernet

Synchronous Ethernet (SyncE) is a procedure where we use a physical layer interface to pass timing from node to node in the same way timing is passed in SONET or SDH. SyncE, defined by the ITU-T standards such as G.8261, G.8262, G.8264, and G.781, leverages the physical layer of Ethernet to transmit frequency to remote sites. SyncE over Ethernet provides a cost-effective alternative to the networks. For SyncE to work, each network element along the synchronization path must support SyncE.

SSM and ESMC

Synchronization Status Message

Network elements use Synchronization Status Messages (SSM) to inform the neighboring elements about the Quality Level (QL) of the clock. The non-ethernet interfaces such as optical interfaces and SONET/T1/E1 SPA framers uses SSM. The key benefits of the SSM functionality:

Prevents timing loops.

Provides fast recovery when a part of the network fails.

Ensures that a node derives timing from the most reliable clock source.

Ethernet Synchronization Messaging Channel

In order to maintain a logical communication channel in synchronous network connections, ethernet relies on a channel called Ethernet Synchronization Messaging Channel (ESMC) based on IEEE 802.3 Organization Specific Slow Protocol standards. ESMC relays the SSM code that represents the quality level of the Ethernet Equipment Clock (EEC) in a physical layer.

The ESMC packets are received only for those ports configured as clock sources and transmitted on all the SyncE interfaces in the system. These packets are then processed by the Clock selection algorithm on RP and are used to select the best clock. The Tx frame is generated based on the QL value of the selected clock source and sent to all the enabled SyncE ports.

Clock Selection Algorithm

Clock selection algorithm selects the best available synchronization source from the nominated sources. The clock selection algorithm has a non-revertive behavior among clock sources with same QL value and always selects the signal with the best QL value. For clock option 1, the default is revertive and for clock option 2, the default is non-revertive.

The clock selection process works in the QL enabled and QL disabled modes. When multiple selection processes are present in a network element, all processes work in the same mode.

QL-enabled mode

In QL-enabled mode, the following parameters contribute to the selection process:

Quality level

Signal fail via QL-FAILED

Priority

External commands.

If no external commands are active, the algorithm selects the reference (for clock selection) with the highest quality level that does not experience a signal fail condition. If multiple inputs have the same highest quality level, the input with the highest priority is selected. For multiple inputs having the same highest priority and quality level, the existing reference is maintained (if it belongs to this group), otherwise an arbitrary reference from this group is selected.

QL-disabled mode

In QL-disabled mode, the following parameters contribute to the selection process:

Signal failure

Priority

External commands

If no external commands are active, the algorithm selects the reference (for clock selection) with the highest priority that does not experience a signal fail condition. For multiple inputs having the same highest priority, the existing reference is maintained (if it belongs to this group), otherwise an arbitrary reference from this group is selected.

Restrictions and Usage Guidelines

Follow these restrictions and usage guidelines when configuring the SyncE on an ES40 line card:

If the network clock algorithm is enabled, all the ES+ cards on the router use the system clock as Tx clock (synchronous mode) for its ethernet interfaces. You cannot change the synchronous mode on a per interface basis for the line card. The whole line cards functions in the same mode.

On an ES+ card, you can have a maximum of 4 ports configured as clock source at a time.

For a 20x1 gigabit ES+ line card, you can select a maximum of two ports from each NPU.

For a 40x1 gigabit ES+ line card, you can select only one port from each NPU.

You can configure a maximum of 6 ports as a clock source for a Cisco 7600 router.

The line to external for clock clean up is supported only if the line interface and the external (BITS) interface are on the same ES+ line card.

SyncE feature is SSO co-existent, but not compliant. The clock selection algorithm is restarted on a switchover. During the switchover the router goes into hold-over mode.

The ES+ SyncE interfaces in WAN mode cannot be used for QL-enabled clock selection. You should either use them with the system in QL disabled mode or disable ESMC on the interfaces and use them as QL-disabled interfaces.

It is recommended that you do not configure multiple input sources with the same priority as this impacts the TSM switching delay.

You cannot implement the network-clock based clock selection algorithm and the new algorithm simultaneously. Both these algorithms are mutually exclusive.

Configuring Clock Recovery on the Cisco 7600 Router

This section describes how to configure clock recovery on the Cisco 7600 Router with ES+ Line Card. Clock recovery is implemented on Cisco 7600 router using four different configurations:

Clock Recovery from SyncE: System clock is recovered from the SyncE clocking source (gigabit and ten gigabit interfaces only). Router uses this clock as the Tx clock for other SyncE interfaces or ATM/CEoP interfaces. For configuration information, see “Configuring the Clock Recovery from SyncE” section.

Line to External: The clock received from an Ethernet is forwarded to an external Synchronization Supply Unit (SSU). During a synchronization chain, the received clock may have unacceptable wander and jitter. The router recovers the clock from the SyncE interface, converts it to the format required for the BITS interface, and sends to a SSU through the BITS port. The SSU performs the cleanup and sends it back to the BITs interface. This clock is used as Tx clock for the SyncE ports. For configuration information, see “Configuring the System to External” section.

System to External: The system clock is used as Tx clock for an external interface. By default the system clock is not transmitted on an external interface. For configuration information, see “Configuring the Line to External” section.

Configuring the Clock Recovery from SyncE

This section describes how to configure clock recovery over ES+ card on Cisco 7600 router using clock recovery from SyncE method.

Configures the equipment to work in synchronization network. The option_id value 1 refers to synchronization networks design for Europe. This is the default value. The option_id value 2 refers to synchronization networks design for US.

Configures the equipment to work in synchronization network. The option_id value 1 refers to synchronization networks design for Europe. This is the default value. The option_id value 2 refers to synchronization networks design for US.

Configures the equipment to work in synchronization network. The option_id value 1 refers to synchronization networks design for Europe. This is the default value. The option_id value 2 refers to synchronization networks design for US.

Configures the equipment to work in synchronization network. The option_id value 1 refers to synchronization networks design for Europe. This is the default value. The option_id value 2 refers to synchronization networks design for US.

Managing Synchronization on ES+ Card

Manage the synchronization on ES+ cards with these management commands:

Quality Level Enabled Clock Selection: Use the network-clock synchronization mode QL-enabled command in global configuration mode to configure the automatic selection process for QL-enabled mode. This succeeds only if the SyncE interfaces are capable of sending SSM. The following example shows how to configure network clock synchronization (QL-enabled mode) in global configuration mode:

Router(config)# network-clock synchronization mode QL-enabled

ESMC Process: Use the esmc process command in global configuration mode to enable the ESMC process at system level. The no form of the command disables the ESMC process. This command fails if there is no SyncE-capable interface installed in the platform. The following example shows how to enable ESMC in global configuration mode:

Router(config)# esmc process

ESMC Mode: Use the esmc mode [tx | rx |<cr>] command in interface configuration mode to enable ESMC process at interface level. The no form of the command disables the ESMC process. The following example shows how to enable ESMC in interface configuration mode:

Router(config-if)# esmc mode tx

Network Clock Source Quality level: Use the network-clock source quality-level command in interface configuration mode to configure the QL value for ESMC on gigabitethernet port. The value is based on global interworking options.

– If Option 1 is configured, the available values are QL-PRC, QL-SSU-A, QL-SSU-B, QL-SEC, and QL-DNU.

– If Option 2 is configured with GEN 2, the available values are QL-PRS, QL-STU, QL-ST2, QL-TNC, QL-ST3, QL-SMC, QL-ST4 and QL-DUS.

– If option 2 is configured with GEN1, the available values are QL-PRS, QL-STU, QL-ST2, QL-SMC, QL-ST4 and QL-DUS

Use the network-clock quality-level command in global configuration mode to configure the QL value for SSM on BITS port. The following example shows how to configure network-clock quality-level in global configuration mode:

The following example shows how to configure network-clock source quality-level in interface configuration mode:

Router(config-if)# network-clock source quality-level QL-PRC

Wait-to-Restore: Use the network-clock wait-to-restore timer global command to set wait-to-restore time. You can configure the wait-to-restore time between 0 to 86400 seconds. The default value is 300 seconds. The wait-to-restore timer can be set at global configuration mode and interface configuration mode. The following example shows how to configure wait-to-restore timer in global configuration mode:

Router(config)# network-clock wait-to-restore 10 global

The following example shows how to configure the wait-to-restore timer in interface configuration mode:

Router(config)# int ten 7/1

Router(config-if)# network-clock wait-to-restore 10

Hold-off Time: Use network-clock hold-off timer global command to configure hold-off time. You can configure the hold-off time to zero or any value between 50 to 10000 milliseconds. The default value is 300 milliseconds. The network-clock hold-off timer can be set at global configuration mode and interface configuration mode.The following example shows how to configure hold-off time:

Router(config)# network-clock hold-off 50 global

Force Switch: Use the network-clock switch force command to forcefully select a synchronization source irrespective of whether the source is available and within the range. The following example shows how to configure manual switch:

Manual Switch: Use network-clock switch manual command to manually select a synchronization source provided the source is available and within the range. The following example shows how to configure manual switch:

Clear Manual and Force Switch: Use the network-clock clear switch controller-id command to clear the manual or switch it by force. The following example shows how to clear a switch:

Router(config)# network-clock clear switch t0

Lock out a Source: Use the network-clock set lockout command to lock-out a clock source. A clock source flagged as lock-out is not selected for SyncE. To clear the lock-out on a source, use the network-clock clear lockout command. The following example shows how to lock out a clock source:

Reproduce the current issue and collect the logs using the debug network-clock errors, debug network-clock event, and debug network-clock sm RP commands.

Warning We suggest you do not use these debug commands without TAC supervision.

Contact Cisco technical support if the issue persists.

Incorrect quality level (QL) values when you use the show network-clock synchronization detail command.

Use the network clock synchronization SSM ( option 1 |option 2) command to confirm that there is no framing mismatch. Use the show run interface command to validate the framing for a specific interface. For the SSM option 1 framing should be SDH or E1 and for SSM option 2, it should be SONET or T1.

Interfaces with alarms or OOR cannot be the part of selection process even if it has higher queue limit or priority. Use the debug platform network-clock RP command to troubleshoot network clock issues.

Reproduce the issue using the debug platform network-clock command enabled in a route processor or enable the debug network-clock event and debug network-clock errors RP commands.

Warning We suggest you do not use these debug commands without TAC supervision.

Understanding 1588v2

IEEE 1588-2008 is a protocol specification standard. It is also known as Precision Time Protocol Version 2 (PTPv2). It is a specifically designed to provide precise timing and synchronization over packet-based ethernet infrastructures. While SyncE supports only frequency synchronization and each network element along the synchronization path needs to support SyncE, IEEE 1588v2 supports frequency synchronization over asynchronous networks and also timing synchronization. 1588v2 is supported only on 2-Port Gigabit Synchronous Ethernet SPA.

Components of a PTP Enabled Network

PTP employs a hierarchy of clock types to ensure that precise timing and synchronization is maintained between the timing and synchronization source and the numerous PTP clients that are distributed throughout the network.

The four PTP clock types are Master, Slave, Boundary Clock and Transparent clock.

PTP Master: A PTP Master has a precise clock from the primary reference clock (PRC) or GPS. This clock enables the timestamp engine to derive accurate timestamps.

PTP Slave: A PTP slave is a network element that recovers the frequency and phase clock, from the timestamps sent by the Master.

Boundary Clock: The Boundary clock functions as both PTP master and slave. It acts as the slave to a Grand Master and derive the reference from the Grand Master. Boundary clock starts its own PTP session with a number of downstream slaves. The boundary clock mitigates the number of network hops and results in packet delay variations in the packet network between the Grand Master and Slave.

Transparent clock: A Transparent clock is a device that calculates the time it requires to forward traffic and updates the PTP time correction field to account for the delay, making the device transparent in terms of time calculations.

Timing over Packet Interface

Timing over packet (ToP) works as a virtual interface on route processor which is the address for the 2-Port Gigabit Synchronous Ethernet SPA’s PTP stack to outside world. Other PTP entities send and receive packets from the interface’s IP address.

When a packet is received on the router destined to ToP’s IP address, the router’s hardware redirects to use the 2-Port Gigabit Synchronous Ethernet SPA and not the route processor. ToP is configured with 32 bit mask. ToP does not support QOS. CoPP is supported.

Basic Operation of 1588v2

This section describes how the 1588v2 works. Figure 68-1 shows the message exchange between the PTPv2 Master and Slave.

Figure 68-1 PTPv2 Message Exchange

The message exchange occurs in this sequence:

The master relays a SYNC message to the slave. The time at which this message is received is recorded by the hardware assist unit on the slave. In Figure 68-1, this is represented as t1.

The master records the actual time the SYNC message was sent (t0) from its own hardware assist unit and relays a follow-up message containing the time stamp of the previous SYNC message to the salve.

To calculate the network delay, the slave sends a “Delay Request” message (t2) to the master. The slave hardware assist unit records the time when the message is sent.

Upon receiving the delay request message, the master transmits a delay response message (t3), with the time stamp of t2, back to the slave.

The slave uses the timestamps, t0 through t3, to calculate the offset and propagation delay to correct its clock.

1588v2 Supported Models

These are the two 1588v2 supported PTP models:

Service SPA Model:

In service SPA model, packets originates and terminate on the 2-Port Gigabit Synchronous Ethernet SPA through SIP400. The service SPA model is simple, uses the existing infrastructure, and works with different encapsulations.

The 2-Port Gigabit Synchronous Ethernet SPA receives redirected PTP packets, processes and sends the reply packets to the central switching engine. These packets are forwarded based on the IP address of the client.

These are the restrictions for the service SPA model:

– The time is not stamped done at the exact packet entry or exit of the system.

– The PTP packet does not remain constant, leading to delays called the packet delay variations (PDV).

Direct SPA Model:

2-Port Gigabit Synchronous Ethernet SPA is capable of accurately timestamping the packet, on the receiver and transmitter for the existing line cards on 7600. So to meet the ideal requirements of 1588v2, the PTP packets are received and transmitted on the same 2-Port Gigabit Synchronous Ethernet SPA.

In the Direct SPA model, PTP packets are received or transmitted through the Ethernet port of the 2-Port Gigabit Synchronous Ethernet SPA. The PTP packets coming on a 2-Port Gigabit Synchronous Ethernet SPA Ethernet interface are diverted to the PTP stack on the SPA by the field-programmable gate array (FPGA). The PTP stack or the algorithm then takes necessary action based on the configuration (master or slave). The reply packets are sent out of the SPA’s Ethernet ports.

These are the restrictions for the direct SPA model:

– Only Limited encapsulations are supported.

– The PTP packets are received only on 2-Port Gigabit Synchronous Ethernet SPA ports.

Supported Transport Modes

These are the transport modes that 1588v2 supports:

Unicast Mode: In unicast mode, the 1588v2 master transmits the Sync or Delay_Resp messages to the slave on the unicast IP address of the slave and the slave in turn transmits the Delay_Req to the master on the unicast IP address of the master.

Unicast Negotiation Mode: In unicast negotiation mode, Master does not know of any slave at the outset. The slave sends a negotiation message to the Master. Unicast Negotiation mode is good for scalability purpose as one master can have multiple slaves.

Mix-multicast model: In Mix-multicast model, the master transmits messages in a multicast packet, to the IP address 224.0.1.129 (defined by the 1588v2 standard). The slave learns the IP address of the master in this process and transmits a delay request message. The master then transmits back a delay response message to the slave in unicast mode.

To send messages in multicast mode, the master needs to explicitly specify the multicast egress interface. This enables the intermediate network to route the IP address 224.0.1.129 to the slave.

Time of Day (TOD)

The physical interfaces used to retrieve Time of Day(ToD) and estimated phase are:

1PPS interface

RJ45 interface

Figure 68-2 shows the Time of Day(ToD) and 1 PPS Synchronization using 1588v2:

Figure 68-2 Block Diagram for Time of Day(ToD) and 1 PPS Synchronization using 1588v2

Time of Day on the 1588v2 Master

In 1588v2 master mode, Time of Day (TOD) enables 2-port Gigabit synchronous Ethernet SPA to receive the time from the GPS receiver through RJ45 interface and synchronizes with the SPA's current time. The 1588V2 master requires 1PPS input from the GPS device to read ToD correctly.

Verifying ToD Configuration on the 1588v2 Slave

This example helps you verify the ToD configuration on the1588v2 slave.

Router# show ptp clock runn dom 0

PTP Ordinary Clock [Domain 0]

State Ports Pkts sent Pkts rcvd

ACQUIRING 1 5308 27185

PORT SUMMARY

Name Tx Mode Role Transport State Sessions

SLAVE unicast slave To3/1/2 - 1

SESSION INFORMATION

SLAVE [To3/1/2] [Sessions 1]

Peer addr Pkts in Pkts out In Errs Out Errs

3.3.3.3 27185 5308 0 0

Use the show platform ptp tod all command to display the sample output.

Router# show ptp clock runn dom 0

PTP Ordinary Clock [Domain 0]

State Ports Pkts sent Pkts rcvd

PHASE_ALIGNED 1 21428 109772

PORT SUMMARY

Name Tx Mode Role Transport State Sessions

SLAVE unicast slave To3/1/2 - 1

SESSION INFORMATION

SLAVE [To3/1/2] [Sessions 1]

Peer addr Pkts in Pkts out In Errs Out Errs

Router# show platform ptp tod all

--------------------------------

ToD/1PPS Info for SPA 3/1

--------------------------------

ToD CONFIGURED : YES

ToD FORMAT : CISCO

ToD DELAY : 0

1PPS MODE : OUTPUT

OFFSET : 0

PULSE WIDTH : 0

ToD CLOCK : Mon Aug 30 09:52:08 UTC 2010

--------------------------------

Network Clocking

The network clocking support for 76-ES+XT-2TG3CXL and 76-ES+XT-4TG3CXL line cards is built on top of the existing network clocking feature with SIP-200 and SIP-400 line cards. All the original network clock sources provided by SPA interfaces on SIP-200 and SIP-400 line cards operate the same way as before. Additionally, you can use network clocking support for the 76-ES+XT-2TG3CXL and 76-ES+XT-4TG3CXL to configure:

BITS clock source

10GE interface clock source

These enhancements provide Synchronous Ethernet (SyncE) feature support for service provider applications making the 76-ES+XT-2TG3CXL and 76-ES+XT-4TG3CXL line cards the preferred choices for carrier Ethernet environments.

Note This feature is applicable only for Cisco IOS releases 12.2 (33) SRD and SRE. Effective with Cisco IOS release15.0(1)S, you can use SyncE for frequency synchronization over ethernet.

The 76-ES+XT-2TG3CXL or 76-ES+XT-4TG3CXL line cards operate in three different modes for clock synchronization depending on the configuration and the current source state.

Free-running—A line card that is not participating in network clocking or a line card that is actively sourcing the clock operates in free-running mode. In this mode, the line card internal oscillator generates the reference clock to the backplane.

Note In a nonpartcipating mode or a disabled mode, the line card distributes a Stratum 3-quality timing signal to an external reference clock. Other interfaces on different line cards receive either the backplane reference clock or the external reference clock depending on their configurations.

Note Line card operation is in free-running mode only if it is not participating in the system clocking, is configured as the active source using on-board oscillator, or does not currently have a valid clock source before the first clock synchronization; otherwise the line cards operate in normal mode.

Normal—In normal mode, the module synchronizes with an externally supplied network timing reference, sourced from one of the chassis BITS inputs or recovered from a network interface. In this mode, the accuracy and stability of the output signal is determined by the accuracy and stability of the input reference.

Holdover—In holdover mode, the network timing module generates a timing signal based on the stored timing reference used when operating in normal mode. Holdover mode is automatically selected when the recovered reference is lost or has drifted excessively.

Note You cannot configure the drift range; it is set internally on the line card to +/–9.2~12 ppm (parts per million) by default. This ppm setting is typical for applications that requires a clock quality level of Stratum 3/3E, ITU-T G.813 option 1.

Note All line cards operate in the free-running mode until the network clock is configured.

For network clocking information for SIP 200, see the following links:

How to Configure Network Clocking

Configuring BITS Clock Support

You can select and configure the BITS port on the 76-ES+XT-2TG3CXL or 76-ES+XT-4TG3CXL line card as the system clock source. This will synchronize the system backplane clock with the corresponding BITS port input clock and distribute the BITS port input clock across the chassis as the transmit clock reference for all other interfaces that support network clocking.

Usage Guidelines

Use the following guidelines:

When the network clocking configuration is present in the startup configuration, the clocking configuration is not applied until five minutes after the configuration has been parsed. This prevents clocking instability on the backplane when the interfaces and controllers come up out of order.

Network clocking is enabled by default for the 76-ES+XT-2TG3CXL and 76-ES+XT-4TG3CXL.

If there is a BITS clock source flap because of Loss of Signal (LOS), Loss of Frame (LOF), T1 Blue Alarm, or E1 Alarm Indication Signal (AIS), there is an interval of 150 seconds before the source becomes valid and active.

In the event of an Out-of-Range (OOR) switchover (revertive mode), the source switchover occurs when the clock offset crosses the +/–12 ppm threshold. If this occurs, you must reconfigure the source.

The default signal type is T1 with ESF framing and a Line Build-Out Select value of 133 feet.

Step 4

network-clock select priority slot slot bits number

Router(config)# network-clock select 1 slot 1 bits 0

Names a source to provide timing for the network clock and specifies the selection priority for this clock source.

Step 5

exit

Router(config)# exit

Exits global configuration mode and returns to privileged EXEC mode.

Example

The following example shows how to configure BITS clock support for the Cisco 76-ES+XT-2TG3CXL and 76-ES+XT-4TG3CXL.

Router# enable

Router# configure terminal

Router(config)# network-clock slot 1 bits 0 ?

2m 2.048MHz square wave signal type

e1 E1 signal type

j1 Japan J1 signal type

t1 T1 signal type

Router(config)# network-clock slot 1 bits 0 t1 ?

d4 T1 D4 framing mode

esf T1 ESF framing mode

Router(config)# network-clock slot 1 bits 0 t1 d4 ?

133ft Line Build-Out Select 0 to 133 feet

266ft Line Build-Out Select 133 to 266 feet

399ft Line Build-Out Select 266 to 399 feet

533ft Line Build-Out Select 399 to 533 feet

655ft Line Build-Out Select 533 to 655 feet

Router(config)# network-clock slot 1 bits 0 t1 d4 266ft

Router(config)# network-clock select 1 slot 1 bits 0

Router(config)# exit

Configuring 10GE Interface as Clock Source

This will set up the line card to extract the received clock from the 10GE interface, either the LAN PHY or the WANPHY, and have the system backplane clock synchronized to it. Then the system will use it as the transmission clock reference for all other interfaces in the chassis that support the network clocking feature.

Usage Guidelines

Use the following guidelines:

When the network clocking configuration is present in the startup configuration, the clocking configuration is not applied until five minutes after the configuration has been parsed. This prevents clocking instability on the backplane when the interfaces/controllers come up out of order.

Network clocking is enabled by default for the 76-ES+XT-2TG3CXL and 76-ES+XT-4TG3CXL.

Cisco IOS Release 12.2(33)SRD1 does not support Ethernet Synchronization Message Channel (ESMC) on LAN PHY and SSM received from SONET/SDH frames for WANPHY.

If there is a clock source flap because of interface up and down events, there is an interval of 150 seconds before the source becomes valid and active.

In the event of an Out-of-Range (OOR) switchover (revertive mode), but the interface stays up, the source switchover occurs when the clock offset crosses the +/–12 ppm threshold. If this occurs, you must reconfigure the source.

For both 10GE port clock recovery and BITS port clock recovery, when the clock source is recovered, the line card will send notification to the RP. Then after a 150-second debounce period, the RP sends a control message to every participant to synchronize with the valid clock source again.