Incident Overview

On July 21, a block storage cluster incident affected the Instance product on FR-PAR-1 between 20:14 and 03:45 UTC. This incident caused a regional service degradation (FR-PAR), resulting in the following impacts:

• Kapsule operations were unavailable on a regional level between 21:30 and 01:30 UTC.
• Instances on FR-PAR-1 were non-functional.
• All managed products relying on Instance and Kapsule experienced disruptions.

Root Cause Analysis 

5 whys

• Why were Instance and its dependent products unavailable?
  Because disk I/O was impossible.
• Why was disk I/O impossible?
  Because cluster doesn't accept I/O operations.
• Why cluster doesn’t accept I/O operations?
  Due to a fault in a cluster component (crash).
• Why was a cluster component faulty?
  It suffered an OOM (Out of Memory) kill.
• Why did an OOM kill occur?
  The memory limit was incorrectly configured, causing the component to consume excessive memory and triggering an OS-level OOM kill.
• Why was the memory limit incorrectly configured?
  The cluster hardware is heterogeneous; nodes have varying memory capacities. A global setting was applied without accounting for nodes with lower memory, leading to node-level OOM kills.
• Why was the configuration error not detected?
  We lack sufficient safeguards for this specific configuration.

Impact on Kapsule Product

‌

• Why did an incident in a single Availability Zone (AZ) impact the entire FR-PAR region for Kapsule?The Kapsule API was proactively disabled to prevent a cascading failure ("snowball effect") caused by auto-healing mechanisms reacting to widespread instance unavailability.
• Why is a multi-AZ product affected by a single API endpoint failure?The Kapsule API architecture is currently regional and lacks the granularity to isolate or disable specific Availability Zones. Consequently, disabling the regional API was the necessary precautionary measure to protect block storage convergence.

Summary of Events

Incident Timeline (UTC)

Resolution and Improvements

Short-term Actions

• During the incident timelapse, we identified a configuration issue and corrected the memory limits across all cluster nodes.

Mid-term Actions

• Enhance cluster configuration management to prevent environment-specific mismatches (e.g., node-level hardware heterogeneity).
• Implement isolation mechanisms to ensure that individual component failures do not impact the overall cluster stability.
• We also notice network saturation during the recovery leading to latencies increase, we may need to rework this part.
• Optimize network performance during recovery phases to prevent latency spikes caused by saturation.
• Improve the incident response to avoid logical bias

Contact

If you have any further questions or need assistance, please contact our support team.