Tuesday 5th May 2020

Global Backbone Fiber cut between DC2 and DC3 (south cable)

Our teams have noticed a network incident between DC2 and DC3.
Thanks to our redundancy capacity, we are not noticing any major impact as of now.

However, we initiated preventive actions by rerouting part of our traffic to avoid network congestion.

For our Colocation & Datacenter customers, we are still investigating the precise impact it has on all services.
DC2/DC3 offsets (south path) are cut, but it seems like some other cables may be impacted between Paris and Courbevoie.

We will keep updating this status as we have additional details to share.

===================

06/05/20 2020Z (2220LT)

All our fibers have now been repaired, the last link came back up around 7PM (local time).
The situation is now back to normal.

06/05/20 1650Z (1850LT)

Almost everything is back online, we are redundant again. We will keep monitoring the situation and the last links should be fixed tomorrow.
Our providers are fully operational again.

Please open a ticket if you still have performance or latency issues.

06/05/20 1500Z (1700LT)

Soldering is still ongoing but many links are now up and back in production, the infrastructure stress is going down, our supplier is announcing a complete recovery of the cable before 1AM LT tonight.

06/05/20 1400Z (1600LT)

Peering issues with Google are now fixed. We are turning back up a lot of links now.

06/05/20 1300Z (1500LT)

We have some issues with our peering with Google. We have some trouble finding a clean path toward them (no saturation on our side, but probably on the other side).

06/05/20 0820Z (1020LT)

During night , 2 new cuts of the cable have been found, soldering teams have been deployed on all 3 points and have started soldering fibers, the first impact point was harder to fix, new cable has been deployed to replace a broken part.
As of this morning, first full path fibers should come up, we expect service to slowly come up as the teams are soldering on the 3 impacts.

05/05/20 1700Z (1900LT)

Repairs are starting on the cut fibers. The situation should start to improve tomorrow around midday.
We continue to monitor our links and reroute traffic dynamically in order to prevent as much saturation as possible, especially during the evening usage peak.

We have noticed saturation on our network (RPN and public, see our netmap).

This incident is global and affects us, as well as multiple of our providers and peers.

05/05/20 1130Z (1330LT)

Fiber cut on our cables has been found, damage is extensive and repairs will take a while.

05/05/20 0745Z (0945LT)

Issue under investigation.