Status impact color legend
- Black impact: None
- Yellow impact: Minor
- Red impact: Major
- Blue impact: Maintenance
FR-PAR-2 connectivity loss
Postmortem
Correction to Error Report dated June 3, 2026
FR PAR 2 Connectivity Loss
Incident Overview
On 22 May 2026, power maintenance in the OpCore PAR5 datacenter led to a total loss of power in the 2 operator rooms. This shutdown caused a global connectivity loss for the FR-PAR-2 availability zone from 11:08 to 11:10 UTC+2. Service restoration continued until 12:30 UTC+2.
Incident status links: https://status.scaleway.com/incidents/qkj49g6g3ykp
Regions/AZ Impacted: FR-PAR-2
Initial Duration: 1h50
Duration by impacted products :
- Control-Plane: mainly 20 minutes.
- Data-Plane: mainly 20 minutes, though some effects continued after connectivity was restored.
- Instance: 1h50 for instances with block volumes.
- Kapsule: 1h50, but actions were required on the customer side. (please check on your end for faulty sockets or connections on your controller or workload. For example, broken SFS sockets).
- Database: 1h50, to verify that all databases had returned to a healthy state.
- Object Storage: TBD (pending investigation). Latencies persisted after connectivity was restored.
- File Storage: 1h50, but actions were required on the customer side.
- Serverless Containers and Functions: 3h30, due to a Cilium bug affecting recovery after the incident
- Dedibox / Elastic Metal: 20 minutes
Primary Impact: Connectivity Loss
Secondary Impact: Products with a presence in FR-PAR-2. Mainly Instance, Kapsule, Databases, Object Storage, File Storage, and Serverless Containers & Functions.
Impact on Customers
During the incident, all services hosted in FR-PAR-2 were unreachable
Root Cause Analysis
5 whys
- Why did all networks go down in FR-PAR-2 ?
The datacenter hosting FR-PAR-2 is connected to the backbone through two distinct network paths and equipment, hosted in two distinct rooms in the datacenter. Each of these rooms is powered by two electrical paths. Each electrical path hosts an electrical workshop that converts power (230 V / 48 V) to feed the equipment in the room. For a reason that is still unknown, all two electrical workshops encountered an issue triggering network equipment reboot.
- Why did power have an issue ?
Maintenance was planned by the datacenter operator to test the power generators. This is a recurring test, carried out every few weeks to ensure the power generators, batteries, and power switching systems are working correctly. Rectifier batteries failed to support load during switch phase from main to generators leading to 10 seconds power loss.
- Why were software services affected by the power/network issue?
When the availability zone became disconnected from the other AZs, it caused a disconnection of redundant systems (failover, DBs, data synchronisation, etc.). When the AZ reconnected, we needed to unlock DBs, reconnect master/slaves, and resync data from the other AZs, which caused some delay in fully restoring services. In addition, many customer requests (automated or not) were queued and had to be processed in order to ensure the system remained consistent, causing increased latencies and delays.
- Why were other Availability Zones affected ?
Other availability zones in FR-PAR were partially impacted during the disconnection event. This was mostly caused by failover systems handling the large surge of requests needed to fail over every redundant part of each software service, causing delays and latencies in processing everything.
Conclusion We will continue to analyse the various events that slowed down some failover events with the other AZs. A series of batteries were confirmed as faulty and have been replaced in the following 24 hours by the provider. The batteries were guaranteed for 12 years and were only at 8 years of service. Investigation is ongoing with the battery provider to understand the underlying cause of early faults.
RemediationShort term
Datacenter provider:
- Battery have been replaced
Mid term
Scaleway:
- Working on improving our recovery process.
Datacenter provider:
- Pending investigation.
Contact If you have any further questions or need assistance, please contact our support team.
Impacted services state.
| Service | Mono AZ impact duration | Multi AZ impact duration | Comments |
|---|---|---|---|
| Instance | 1 hour and 50 minutes | N/A | Instances on FR-PAR-2 impacted. |
| Block storage | 1 hour and 50 minutes | N/A | Availability for some elements stored in FR-PAR-2 (20 minutes)After power restarted, some latencies (10 minutes) |
| Kapsule (K8S) | 1 hour and 50 minutes | 10 minutes | Time to converge and respawn assets. |
| Object storage | 1 hour and 50 minutes | 0 minutes | Multi AZ not impacted |
| One zone object storage: if the bucket is on FR-PAR-2, this one was unavailable. | |||
| Managed Databases | 1 hour and 50 minutes (only multi AZ replica impacted) | N/A | It was not possible to create a multi AZ HA database, or to connect/promote a read replica. |
| api.scaleway.com | N/A | 0 minutes | api is regional no outage. |
| console.scaleway.com | N/A | 1 hour and 50 minutes | failover not working as expected. |
| file storage | 1 hour and 50 minutes | N/A | manual intervention was required from customers to recover. |
| Serverless Container and Functions | 3 hours and 30 minutes | Network problems on recovery led to an extended outage. |
Resolved
Update
Update
Removed batteries are under tests and investigations to confirm their involvement in the initial incident.
No more action on production expected for now.
We are still on power generators for the time being.
Update
Datacenter teams will proceed with the next paths.
Update
This operation will be done live and should have no impact.
Once replaced, old batteries will be inspected and tested to confirm they were the cause of the incident.
Power will be kept on generators during this operation and after until the root cause is fully confirmed
Update
- Object Storage: Everything back to normal
- Serveless containers and functions: Everything is back to normal.
- File: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected
As the failover malfunction has not been diagnosed at this time, our datacenter manager will keep the affected elements under a generator.
Update
- Object Storage: still some latency issues and timeout on connecting some buckets
- Serveless containers and functions: Everything is back to normal. Situation recovered and stable since 02:40 PM
- File Storage: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected
Update
- Kapsule: everything back to normal
- Databases: everything back to normal
- Object Storage: still some latency issues and timeout on connecting some buckets
- Serveless containers and functions: might face connection timeout issues when contacting their applications
- File Storage: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected
Update
Monitoring
- Instances: everything back to normal
- Kapsule: same status
- Databases: same status
- Object Storage: same status
- Serveless containers and functions: might face connection timeout issues when contacting their applications
Update
Identified
Update
- Instances: you may encounter issues with l_ssd/block snapshots or volumes
- Kapsule: Control planes are reachable and in nominal state, you may need to verify that your controllers reconnected properly on the apiservers. You may have experienced node replacement due to the autoheal or autoscaling process, this may delay volume re-attachments.
- Databases: snapshots may be blocked, some failovers have been started for HA
- Object storage: you may encounter some latencies
Update
Both operator rooms (network connectivity) were powerless for a few minutes leading to a complete isolation of the availability zone during the issue. We are investigating with the datacenter provider to understand the root cause.”
Monitoring
Other service on par1 & par3 were impacted during the service transition.