Sunday 18th August 2019

Global Backbone Fiber provider incident on 6x100G links

Our teams have been alerted that one of our Fiber provider is currently facing an outage on 6 different 100G links we are using between AMS1 and PAR1.
We have isolated traffic to avoid saturation as much as possible.
Some customers might observe lower bandwidth or increased latency for certain destinations.

We will update this status as soon as we have additional details from them.

===================

18/08/19 1400Z (1600LT)

Backups WDM links over another Fiber path are up for 6x100G since - Aug 18 12:47:21.637 UTC -
During rollback on traffic offload, we had an issue in a procedure that lead to some flaps between AMS1 and PAR1.
These flaps could be seen by some drops / latencies between - Aug 18 13:42 UTC - and - Aug 18 13:46 UTC -
We are waiting the RFO from the provider and we will rollback to previous fiber path when it will be fixed.

18/08/19 1100Z (1300LT)

Things got better now.
By this moment no major saturation anymore but we are still monitoring and ready to act in case of instability.

18/08/19 0900Z (1100LT)

Network team acknowledged the issue.
We have contacted our network provider and awaiting updates.