Friday 8th February 2019

DC2 [DC2] Partial network outage

A network issue was detected by our team, we are investigating the root cause and are working on reestablishing full connectivity as soon as possible. Your machines and data aren't impacted, only disconnected.

08/02/19 1400Z (1500LT) Our team are still investigating

08/02/19 1410Z (1510LT) A large fiber incident on our network backbone has been identified causing partial connectivity loss to several servers in PAR1. Our Network Team is on it.

08/02/19 1420Z (1520LT) Our fiber connection provider is going to the area where the fiber has been cut. Our teams have restablished most of the traffic by rerouting the traffic to alternate routes.

08/02/19 1440Z (1540LT) Connectivity is back for DC2 dedicated servers. Connectivity is coming back progressively for Scaleway instances. Our teams are currently working to distribute better the traffic and to fix room DC2/101 and the link from DC2 to AMS.

08/02/19 1500Z (1600LT) We managed to distribute the bandwidth between our providers. The link between DC2 and AMS is still unstable.

We are also setting up additional cables between DC2 rooms, to fix the remaining local issues.

08/02/19 1530Z (1630LT) Situation is stabilized and traffic is nearly normal. We'll stay working with our fiber provider to ensure definitive fix.

08/02/19 1800Z (1900LT) Our provider team is still working to repair the cable. The return to normal is progressive. We expect some delay before the come back to 100% availability. Network traffic is operational being routed on alternates path.

11/02/19 0930Z (1030LT) About 90 percent of fibers are back and traffic is now normal.

13/02/19 0830Z (0930LT) Still missing few fibers. Traffic is nominal.

18/02/19 1600Z (1700LT) All is fixed. Fiber is fully back and traffic restored to nominal routes. Incident closed.

Update 08/02:19 15:20 : Les équipes de notre fournisseur convergent vers la zone de coupure. Nous avons rétabli une grande partie des serveurs depuis un chemin alternatif.

Update 08/02:19 15:40 : Retour de la connectivité pour les serveurs isolés de DC2 et retour à la normale sur les instances Scaleway. Nos équipes travaillent sur les saturations en salle DC2/101 et vers Amsterdam.

Update 08/02:19 16:00 : Nous avons résorbé la saturation de certains peerings et transitaires, les opérations se concentrent sur le Paris/Amsterdam. Coté DC2/101 nous sommes en train de re-câbler physiquement certains routeurs pour limiter la saturation.

Update 08/02:19 16:30 : La situation est maitrisée. Le traffic est sous controle. Nous continuons à travailler avec notre fournisseur pour le fix définitif.

Update 08/02:19 19:00 : Notre fournisseur travaille à la réparation du cable. Nous nous attendons à un délai avant retour à la normale. Mais le traffic réseau est opérationel grace au routage sur des chemins alternatifs.

Update 11/02:19 10:30 : Environ 90 pourcent des fibres sont à nouveau opérationnelles. Le traffic est normal.

Update 13/02:19 09:30 : Il manque encoere quelques fibres. Le traffic est normal.

Update 18/01/19 17:00 : Tout a été réparé. Le traffic a été remis en place sur les routes nominales. Incident fermé