Following a core network maintenance operation, a bug within the BGP daemon used by our VPC services was triggered. This led to a connectivity loss and instabilities on Kapsule clusters in FR-PAR-2 and in between FR-PAR-2 and other AZ.
07h02 - Beginning of core network maintenance 07h37 - VPC incident 09h02 - VPC services are back on track with degraded connectivity 10h05 - Root cause is identified 10h11 - Root cause is isolated, but connectivity remains degraded 10h22 - Root cause is repaired, VPC services are now functional 10h27 - Connectivity is restored 16h00 - Core network maintenance complete VPC services are at this point functional without redundancy
10h44 - Redundancy is restored after patch update 11h04 - Redundancy is lost during the patch update on the second half of the infrastructure 11h15 - 11h20 - Attempt at restoring redundancy 13h26 - Restoring redundancy on the entire infrastructure with patch application. VPC services are at this point functional with redundancy.
We will provide a more in depth analysis in a few days after a more thorough analysis of the collected data.