We are going back 100G links to RBX.
For now, we find a saturation between RBX and AMS.

Comment by OVH - Wednesday, 06 December 2017, 22:15PM

We are shutting AMSIX / Worldstream temporarily.

Comment by OVH - Wednesday, 06 December 2017, 22:16PM

Good evening,
On November 9, we encountered a big problem
on our optical network at RBX. The problem was related
to a software bug on the equipment we use
which caused the deletion of the configuration.

Since then we have updated the equipment on everything
our network. Also to prevent this type of bug from
never again causes a worry about our DCs, we have
decided to divide equipment clusters into 3
on the RBX website. So, if we ever have again
this bug, the configuration would only impact 30%
traffic.

During the preparation of the maintenance that was to
start at 23:00, the configuration disappeared again
at 8:20 pm and all the links were down again. !!!!!
The database has been deleted while we are using
the latest software version. So there is another bug!http://travaux.ovh.net/?do=details&id=28835

We quickly returned the configuration. Some
links have returned, but not yet all. Currently
traffic passes Paris> GRA> AMS> RBX instead of
directly Paris> RBX. Hence the important latencies.

We look with Cisco to understand why all
the links are not UP while the configuration was
delivery to RBX.

We are not going to do the intervention tonight.
I want to understand why the configuration fades
and how can we do this maintenance without
again have the least concern about production!

Sincerely
Octave

Comment by OVH - Wednesday, 06 December 2017, 22:17PM

The configuration is in place but it does not work.
We erase each configuration and rebuild them to
by hand.

RBX <> LDN: UP

Comment by OVH - Wednesday, 06 December 2017, 22:17PM

Traffic isolation between Amsterdam and Warsaw to reduce saturation

Comment by OVH - Wednesday, 06 December 2017, 22:25PM

Isolation of VAC Roubaix to reduce saturation

Comment by OVH - Wednesday, 06 December 2017, 22:37PM

We rebuild the circuits by hand Paris <> RBX.
In a few minutes we should have 400G of UP capability which will reduce the latency in the network.

Comment by OVH - Wednesday, 06 December 2017, 22:48PM

4 GSW <> RBX circuits are UP. The others are reactivated.

Comment by OVH - Wednesday, 06 December 2017, 22:56PM

4 other circuits are UP.
We rebuild the other circuits that are still down.

Comment by OVH - Wednesday, 06 December 2017, 23:42PM

RBX<>FRA 400G UP

Comment by OVH - Thursday, 07 December 2017, 00:08AM

RBX<>LDN 600G UP

Comment by OVH - Thursday, 07 December 2017, 00:09AM

RBX<>BHS UP

Comment by OVH - Thursday, 07 December 2017, 00:22AM

We are working to bring up the links between RBX and SBG. 200G are actually UP

Comment by OVH - Thursday, 07 December 2017, 00:55AM

we brought a 100G link between RBX and AMS

Comment by OVH - Thursday, 07 December 2017, 01:05AM

600G between RBX and AMS

Comment by OVH - Thursday, 07 December 2017, 01:05AM

600G between RBX and AMS

Comment by OVH - Thursday, 07 December 2017, 01:06AM

Links to BRU POP are now UP

Comment by OVH - Thursday, 07 December 2017, 01:27AM

Everything is working now.

We are so sorry for this breakdown, we will working on this to avoid any other issue like this one