Saturday 12th October 2019

DC5-A Network RPN saturation between DC5-A and other locations

Due to a failure on our primary RPN link between DC5-A and other datacenters, RPN traffic may be saturated from/to this DC as all traffic has been rerouted to our backup link.
We will update this status as soon as we have additional details to share.

===================

10.28.19 RFO

Impact: degradation of service RPN between DC3 and DC5-A

Timeline: On Saturday 12th October, at 10h50 LT, our primary RPN link with DC5-A went down. Traffic shifted to our backup link immediately, however the backup link capacity was not sufficient enough to handle all the traffic (works are underway to increase this capacity but were not completed before this incident due to technical difficulties), this lead to low traffic performance and packet loss Our team received first alarms at 11h LT and started investigations. First diagnostics suggested it could be broken optics, so we sent technicians on site to try to replace optics. As it did not work, they started to investigate fiber links on both datacenters and try to fix bad optical levels. At 17h40 LT, our dark fiber provider was involved to help identify the issue. At 18h45 LT, our provider confirmed a cut on their path and sent technicians to the area On Sunday 13th, at 1h00 LT, our provider fixed the fiber cut, but we still had not a good enough signal to bring up the link. While our technicians were working on cleaning links and swapping optics again, we initiated 2 backups plans to restore full capacity on other paths (these were not used in the end as primary link was fixed) At 7h LT, link was fixed and traffic was shifted back on primary link

28th October : received supplier RFO for primary link cut : cable was partially cut at both extremeties on some part of the cable in the sewers, reasons still unclear as of today how these cuts were caused. Supplier switched our fibers to a working pair of fibers during the night of intervention, a plan is being prepared to replace the whole cable in this area

10.13.19 0540Z (0740LT)

Optical level are stable, issue fixed.

10.13.19 0500Z (0700LT)

Link is now up since 25min, we still monitor it

10.12.19 2256Z (0056LT)

Fiber has been fixed but we have attenuation issues that prevent the link from getting back up. Still working on the issue with on-site teams

10.12.19 1840Z (2040LT)

Physical cable cut has been localized, teams are now on working on fixing it and restoring network.

10.12.19 1645Z (1845LT)

Our network provider is suspecting a cut on their end, they are still investigating the issue with a team on-site.
No ETA yet, we will keep you updated.

10.12.19 1445Z (1645LT)

Our teams are still focused on the issue.
We have both Network engineers and datacenter technicians doing cross-tests between datacenter.
Status will be updated as soon as we have additional information.

10.12.19 1000Z (1200LT)

Issue has been escalated to the datacenter.
Awaiting further updates.