Newsletter 2015 12

White Rabbit Newsletter - December 2015

CERN (including Alessandro Rubini)

White Rabbit Switch

During 2015 our WR Switch developments were focused mainly in two areas:
improving reliability under a heavy load of traffic and ensuring ways
for remote configuration and diagnostics. The former included various
tests performed with a professional IT network tester and fixes to the
WR Switch gateware. For the latter, we now have a set of SNMP exports
that let operators determine the overall state of a device as well as
provide detailed information for expert users. We have also added a
possibility of fetching the main configuration file from a central
TFTP/HTTP/FTP server on boot time and generating a WR-synchronized 10MHz
clock on the latest v3.4 switch hardware. All these and other features
and bugfixes were included in the new stable v4.2 firmware which was
released on the 28th of August.

WR Switch Grandmaster phase noise analysis

Currently release 3.4 of the WR switch operating in G.M. mode (locked to
a Cesium clock) achieves 8-9 ps of RMS jitter (1Hz-100kHz). Most of the
noise lies in the 1Hz-2kHz bandwidth (SoftPLL Nyquist region). Mattia
Rizzi found that a large amount of jitter was related to the Xilinx DCM
used to multiply the 10MHz input reference clock rather than the phase
detector (DDMTD). The Xilinx DCM outputs a clock with noise
contributions above 20kHz. Most of the high-frequency phase noise is not
filtered by the DDMTD and appears as white noise in the SoftPLL
bandwidth due to aliasing. A clock with a better phase noise profile
above 2 kHz may improve the G.M. jitter to less than 4ps (1Hz-100kHz).

White Rabbit PTP Core

We were also working on the WR PTP Core. Among others, main new features
include the possibility of storing calibration data and init script in
the Flash memory on the carrier, various bugfixes to withstand heavy
Ethernet traffic and LM32 software cleanup. These and other features and
bugfixes, as well as external contributions (like Kintex-7 support done
by Peter Jansweijer, of Nikhef) were part of the new v3.0 WR PTP Core
released on the 16th of December.

Applications

Regarding new CERN WR deployments, this year we have provided a small WR
network consisting of one WR Switch and two VME nodes to synchronize
vibration measurement equipment on the surface and underground in the
LHC Point 1 [1]. We also developed a fixed-latency trigger
distribution system to study beam instabilities in the LHC [2] and
tested a proof-of-concept system for RF distribution using WR [2].

WR Reliability

A proof-of-concept implementation of switch-over between synchronization
paths was developed and tested. The implementation supports multiple
backup paths, it provides short term holdover and hardware-assisted
notification that a BC lost connection to the Grandmaster (clockClass
degradation). The results show a phase-jump of up to 800ps when
switching to a single available backup after unplugging the active link,
a phase-jump during switchover resulting from fiber attenuation is up to
100ps. When multiple backup links are provided, majority voting is used
in failure pre-detection and the 800ps phase-jump can be reduced to
around 100ps.

WR Standardization

The work on WR standardization is slowly converging. WR will likely find
its way into IEEE1588 standard in the following parts:

JINR

The Data Acquisition Group in the Laboratory of High Energy Physics
(LHEP) of the Joint Institute for Nuclear
Research (Dubna, Russia) develops and produces
readout electronics and software for data taking. This year WR-enabled
devices took part in a real data taking run. The BMN [1] technical run
was carried out at the extracted beam of the Nuclotron in February -
March 2015.

Readout electronics included waveform digitizers [2] for Zero-Degree
and Electromagnetic Calorimeters readout and time-stamping TDC for
Time-of-Flight and Drift Chamber detectors. Multiple TDC boards [3]
[4], Trigger Master [5] and Readout Controller were built in the VME
standard. Trigger and clock are distributed within the crate over a
private bus driven by the Trigger Master. Both the waveform digitizer
and the trigger master run the White Rabbit Node Core for time
synchronization. They have a hardware IP stack for communication.

In the waveform digitizers we had to synchronize a 62.5 MHz clock across
devices. It is derived directly from the 125 MHz WR clock and its phase
is aligned with timestamp. The TDC system uses a 41.667 MHz clock that
is exactly 1/3 of the WR clock. A hardware divider-by-3 is reset every
time WR acquires sync and timestamp is a multiple of 3. This method is
not ideal and requires a complete system reset after each divider reset.

White Rabbit technology provided a stable and precise time base. The
Gigabit data path has been used for readout and board control. The use
of an 18-port WR switch for bulk data transfer is limited by the 1 Gb/s
port bandwidth. This may be improved by either link aggregation
(etherchannel) or implementing 10G interfaces.

WR technology is planned to be used for the control and monitoring
systems of NICA, Nuclotron-based Ion Collider Facility in JINR. WR is
considered as a timing system in the Multi-Purpose Detector (MPD) of
NICA.

Activities in the Netherlands

Work is in progress to create and validate VHDL sources for a
deterministic PHY for the Xilinx Artix-7 family (using a GTP). This will
supplement the already validated 7-series GTX (used in Kintex-7 and
Virtex-7).

Some WR users want the freedom to use different SFPs (or other
converters), for example for long haul applications. One would like to
be able to field-exchange SFPs without the need to do re-calibration. To
be able to do so, each network component should be individually
calibrated and contain as set of calibration parameters. System level
aspects in WR implementations call for remote (and possibly dynamic)
update procedures of calibration parameters from network management
entities.

Currently there is a single set of calibration parameters for a WR port
(combining FPGA, PCB, FMC and SFP propagation delays). We propose to
split these parameters by defining an electrical phase plane for each WR
port and calibrate the SFPs and FMCs individually. Having the electrical
phase plane as a reference implies calibration of the electrical/optical
and optical/electrical converters. This domain crossing calibration is
under development and first results are promising.

Work is in progress to enable absolute delay calibrations of WR gear.
This would enable independent developers or vendors to exchange their WR
gear. The latter is an important feature for standardisation. Absolute
calibrated devices can be used as "golden standards" for the existing
relative calibration procedure.

Uniformly defined measurement definitions, procedures and tooling are
under under development. Plans for system level calibration aspects
still need to be
detailed.

A long-haul test link of 2 x 137km between the Dutch National Metrology
institute (VSL) in Delft and Nikhef in Amsterdam is used to disseminate
UTC (VSL). The link includes two quasi-bidirectional semiconductor
optical amplifiers. After a standard WR-calibration, first time transfer
tests result in 2(8) ns time transfer accuracy on the round trip over
the fibre. The accuracy and uncertainty on this link are almost
completely determined by the unknown chromatic dispersion encountered.
Calibration efforts aiming at sub-ns time transfer accuracy are in
progress. Schemes to determine chromatic dispersion without modifying
the fibre infrastructure between the sites will be tested. If
successful, this will lift a barrier for implementation of WR on
existing fibre infrastructure.

CTA (UVA, Nikhef):

The Cherenkov Telescope Array (CTA:http://cta-observatory.org) is the
next generation very high energy gamma-ray experiment surpassing current
instruments by at least an order of magnitude in sensitivity. The
University of Amsterdam with the help of Nikhef in Amsterdam, the APC
(http://www.apc.univ-paris7.fr/APC_CS) in Paris and DESY in Zeuthen
(http://www.desy.de) are working on a prototype of the timing system for
CTA using White Rabbit as the baseline technology. For Amsterdam and
DESY, these developments are also integrated into the ASTERICS
infrastructure cluster which is a part of the european Horizon 2020
program (http://asterics2020.eu/).

Within ASTERICS, as part of the CLEOPATRA work package, there is a joint
timing effort to enhance White Rabbit to deliver higher precision over
very large distances for radio telescopes for VLBI, to facilitate the
precision calibration for large scale projects, to operate up to
thousands of timing nodes in harsh environments, and to port White
Rabbit to 10 Gb/s Ethernet. This ASTERICS timing project is a
cooperation of the University of Granada in Spain, DESY in Zeuthen in
Germany, JIVE, ASTRON, Vrije Universiteit Amsterdam and the University
of Amsterdam, Nikhef, and SURFnet in the Netherlands.

The CTA focus is on using White Rabbit in harsh environments and
improvements in White Rabbit calibration, and synergies between KM3NeT
and CTA are being explored. The VLBI subtask aims for a
proof-of-principle demonstration of VLBI between telescope sites at
Dwingeloo and Westerbork, synchronised through a 200 km DWDM fiber link
with live data traffic.

University of Granada (UGR) and Seven Solutions

UGR is contributing to the White-Rabbit community with different R&D
projects involving the WR Switch, WR nodes and, together with Seven
Solutions, the development of new stand-alone versions of WR nodes based
on the Xilinx Zynq architecture [1]. At the same time, UGR is
collaborating with other WR partners in the development of a reliable WR
node with redundancy capabilities to avoid single point of failure
issues.

WR-ZEN developments

The WR-ZEN [2] is an open hardware board designed by Seven Solutions.
It uses the Xilinx Zynq technology that contains an ARM processor and a
FPGA device in the same System on a Chip (SoC) taking the advantages of
a better HW/SW partitioning and interoperability between the FPGA and
the ARM processor.

UGR with the collaboration of Seven Solutions have implemented a WR node
for the WR-ZEN board with two fully functional SFP ports implementing
Fine Delay FMC features.

The WR-ZEN gateware has been designed by UGR and Seven Solutions
including some additional IP cores such as a dual-port WRPC, an AXI-WB
Bridge core and an I2C Arbiter. In addition, LM32 firmware has been
updated to work properly with the new gateware and board architecture
taking into account its new features and peculiarities.

In terms of software, the board runs a SMP Linux kernel [3] where
several drivers have been adapted from the SPEC project to control the
NIC and the Fine Delay cores. On the userspace, some tools have been
modified to work with this new architecture. Some examples of these
tools are the zen-vuart, zenmem and those related with the Fine Delay
driver such as fmc-fdelay-input and fmc-fdelay-pulse.

White Rabbit High-availability Seamless Redundancy (WR-HSR)

UGR is working on increasing availability and robustness of the White
Rabbit technology to extend its utilization to industrial networks such
as Smart Grid. Applications related to this field demand high
availability and very short switch over time. In order to meet these
requirements, the electrical substation automation standard IEC 61850
[4] suggests the High-availability Seamless Redundancy (HSR, IEC
62439-3) [5] as one of the redundancy protocols. HSR guarantees
zero-time recovery in case of failure in ring and mesh network
topologies for both time and data frames and it is based on the
duplication of frames in the network.
UGR is currently implementing the HSR protocol for WR devices, in
particular the WR-Switch to increase availability and avoid single point
of failure for time and data distribution. This work involves the
development of the Peer-2-Peer mechanism [6] for clock propagation,
PeerDelay [6] for the delay measurement, the utilization of HSR tags
[5] in frames, and the SwitchOver mechanism developed by CERN to
recover from a node failure. Furthermore, additional gateware must be
developed to duplicate and forward frames. Current progress and more
information can be found at the OHWR WR-HSR webpage:
https://www.ohwr.org/project/wr-hsr/wiki

DESY (CTA, HiSCORE)

DESY has continued its WR-based timing support for HiSCORE, the Gamma
and Cosmic Ray detector in the Tunka-valley/Siberia.

The success of this pioneering WR-timing application yielded critical
input to propose for the Cherenkov Telescope Array (CTA; see
http://cta-observatory.org) - the next generation very high energy
gamma-ray experiment - a centralized, WR-based clock-distribution and
trigger-time stamping system with sub-nsec precision and sufficient
redundancy to operate in harsh conditions.
The system is now in its prototyping phase (activity in collaboration
with APC, Paris and UvA, Amsterdam).

HiSCORE

The 9-station HiSCORE setup has been successfully operating since the
2013/2014 observation period. The Array has been enlarged to 28 stations
covering 0.3 km^2 in 2014.

The 9-station WR-setup is based on a SPEC with DIO5Ch, which is operated
as a 1-nsec time-stamping unit, with the analog PMT signals directly fed
into the DIO5Ch, thus acting as a discriminator with adjustable
threshold and digital pulse selection capability. Additional DAQ
functionality for waveform sampling readout (DRS4-EB; trigger and busy)
turned the SPEC/DIO5Ch into a complete front-end board. See
contributions to ECRS2014 and ICRC2015 conferences.

The new stations (2014) have a custom-made front-end board (DRS4-based,
MSU/Moscow) connected as Mezzanine to a WR-SPEC. Detailed long-term test
of WR performance with a custom made time-distribution system shows
excellent WR-perfomance.

DESY has worked in close collaboration with Humboldt-University Berlin
(Martin Brueckner, now PSI).

Tsinghua University

WR development

1:Fixed delay constrain for WR node
The location and inter-connection of WRPC logic and the PPS signal path
are all decided by the P&R engine without any constrains, thus the
fixed delay varies each time the project is re-compiled. Since in our
application the WRPC is integrated in the same FPGA with customer logic
which can be remotely upgraded from time to time, the requirement for
fixed delay re-calibration is not convenient and even not possible.
Besides, the WR node suffers more temperature variation from the
un-constrained long inter connection delay.
By adding additional constrains to lock the location of key blocks of
the WRPC, the inter-connection delay of the WRPC would be reduced and
relatively fixed. This makes the integration of WRPC with other logic
much easier and releases the necessity of fixed delay temperature
correction.

2:Dual-Port WR node development
The PCB design of Dual-Port WR node in FMC form (Cute-WR-DP) has been
delivered for fabrication. The XC6SLX45T-3CSG324C is used to fit within
the FMC form together with two SFP sockets. The two ports work in
parallel to provide redundancy for high-reliability applications. More
flexible topologies like daisy chain are still under consideration.

WR applications

The prototype array of 60 WR nodes and 4 WRS deployed in Tibet has
operated for almost 18 months. Preliminary physics analysis from both
telescope and cosmic sources confirmed that all normal detector units
are aligned within nano-second precision which include the contributions
from crystal, PMT, digitization electronics and WR network. Some nodes
still suffered from link connection failures; nodes lost synchronization
during the run which can be explained by a known WRPC bug. Further
analysis is ongoing.

GSI

During 2015 the Timing Team at GSI has been heavily involved in the
development and commissioning of the General Machine Timing (GMT) system
[1] for the CRYRING [2] project. CRYRING is an ion storage ring,
formerly located at Stockholm University. Besides it serves as a test
case for developments relevant for the FAIR [3] project. So far, the
injector of CRYRING is routinely operated using FAIR-style solutions
like the GMT since a few months. Injection into the ring itself is
planned for spring 2016.

In addition to the "production" WR network for CRYRING, two other
networks have been set up at GSI: The first one is a "user" network,
where other developers use WR to synchronize their systems under
development (e.g. RF System). The second one is a "test-bench" network
used by the Timing Team for testing and debugging the latest version of
the control system and WR firmware. Monitoring tools (e.g.
Nagios/Icinga) are being used in the GSI WR networks for monitoring
status, stability and synchronization quality of the WR Switches as well
as Altera based TRs. Moreover, the monitoring tools serve to detect and
troubleshoot networking problems such as broken links or
unregistered/broken nodes.

Together with the related WR activities, other essential features for
the GMT have been developed during this year:

Simplified API for Timing ("SAFTlib") [4]

Upgrade of the Data Master [1]

I/O Control and clock generator VHDL modules

Integration of the timing system with higher layers of the control
System (e.g FESA)

Along with the CRYRING project, the Timing Team has been developing
WR-based TRs in form factors AMC and PMC in collaboration with Cosylab
(FAIR in-kind contribution) based on the in-house developed WR TR
(Pexaria, Vetar and Exploder [5]). The designs will be published in
OHWR in due time.

Creotech

Creotech Instruments SA in cooperation with N.A.T develops a WR-enabled
MCH module with dual SFP, dual WR D3S and TDC/DTC. This module can be
installed on top of a standard N.A.T MCH in both single width and double
width modules. Optionally, the WR-MCH gets access to the MLVDS ports via
a modified PENTAIR (SCHROFF) MTCA.4 backplane (coming soon).
Creotech also ported the WR node to the AMC-FMC Carrier with Kintex 7
(AFCK) [1].
Currently, in the frame of the WR-SYNTEF project, a feasibility study is
carried out on the use of WR to synchronize the work of ESA Ground
Stations Network. This solution would provide optimal use of ground
station resources and ground stations network synchronization.
We are also working in cooperation with Warsaw University of Technology
on the next generation of WR-enabled AMC-FMC carriers based on ZynQ
UltraScale+ SoC devices. This module will be optimized for recent
JESD204B ADC/DAC GS/S converters.