Status impact color legend
- Black impact: None
- Yellow impact: Minor
- Orange impact: Major
- Blue impact: Maintenance
[Storage]-Object Storage instability
Incident
Report for Scaleway
Resolved
With this update finally completed, we have achieved an even better level of stability and performance than before the incident. This incident is now closed, and we still have a number of improvements planned, but these are no longer part of the incident correction.
Update
On October, 21st, we successfully released a new version in pl-waw region, which significantly improved the availability and performance of the service. The deployment of this update is currently ongoing in nl-ams region and is planned tomorrow morning on October, 23rd for fr-par region. Please note that no interruption of service is expected.
Monitoring
The actions of mitigation are completed and both performance and availability are almost back to normal. We will keep the incident open for monitoring until the long term fix is released within a couple of weeks.
Update
Out of the monitoring, we confirm that the latency due to the incident has significantly decreased in Paris Region and is almost back to normal (only the 99,9 percentiles are still above usual values). We are planning a hardware update today by the end of the day to remove most services that still generate 5xx errors. A more long term fix is planned in the coming days to fix the incident root cause. As always, we will keep you updated.
Update
After monitoring the impact of our intervention, we do see that the globale response time is better (even if there is still some cases, on 99.9 percentil where we have to much latency). We will pursue on this Monday, and remove the faulty services that still respond 5XX errors.
During the week end, we added ressources to our teams on-call to handle trouble that may rise in this situation
On the Bug Fix side, we do have a more long term fix in review today, and will test it next week before we decide when to push it in production.
As allways, we will keep you update.
During the week end, we added ressources to our teams on-call to handle trouble that may rise in this situation
On the Bug Fix side, we do have a more long term fix in review today, and will test it next week before we decide when to push it in production.
As allways, we will keep you update.
Update
A maintenance will happen on some servers between 10/03/2024 3PM CEST and 10/04/2024 6PM CEST to upgrade the resources. There should be no service interruption.
Investigating
Even after the fixes, we're still experiencing time-outs and the error "Reduce your request rate".
Our team is actively working towards a lasting resolution, thank you for your patience and understanding.
Our team is actively working towards a lasting resolution, thank you for your patience and understanding.
Monitoring
Following a faulty fix deployed Friday 27 Sept at 10PM CEST, the object storage solution experienced instability. At 14:30PM CEST on the 28 Sept, we began rolling out a downgrade of that fix. Everything was deployed at 6PM CEST and the service is stable again.
Investigating
We are experiencing instability on the Object Storage solution in fr-par region since 8PM UTC on friday 27/09, resulting in HTTP 503 errors for customers.
We are currently investigating this issue.
We are currently investigating this issue.
This incident affected: Elements - Products (Object Storage).