Status impact color legend
- Black impact: None
- Yellow impact: Minor
- Red impact: Major
- Blue impact: Maintenance
[COCKPIT] - [fr-par/pl-waw] - Scaleway Metrics/Logs ingestion down
Incident
Report for Scaleway
Resolved
This incident has been resolved.
Update
We applied and monitored today the last fix for high RAM consumption. All seems fine now on Cockpit performances for all regions.
Update
Root cause was finally identified yesterday evening, we make an evolution on staging environments and monitored its behavior today. We plan to deploy this fix next monday morning.
Metrics & Logs acquisition are fine since yesterday with a scaling workaround, we keep this solution for the week-end to not impact production on a friday afternoon.
Metrics & Logs acquisition are fine since yesterday with a scaling workaround, we keep this solution for the week-end to not impact production on a friday afternoon.
Monitoring
Cockpit impacted this night betwen 1am30 and 8am30 with data loss for all Products metrics and logs on FR-PAR & PL-WAW Regions
Root cause : based on last week Cockpit ingestion solution, a high RAM consumption saturated nodes and corrupted replay solution.
Status : Cockpit fixed at 8am30 and fix in development to avoid replay corruption on OOM events
Root cause : based on last week Cockpit ingestion solution, a high RAM consumption saturated nodes and corrupted replay solution.
Status : Cockpit fixed at 8am30 and fix in development to avoid replay corruption on OOM events
This incident affected: Elements - Products (Cockpit).