Update - After monitoring the impact of our intervention, we do see that the globale response time is better (even if there is still some cases, on 99.9 percentil where we have to much latency). We will pursue on this Monday, and remove the faulty services that still respond 5XX errors.
During the week end, we added ressources to our teams on-call to handle trouble that may rise in this situation
On the Bug Fix side, we do have a more long term fix in review today, and will test it next week before we decide when to push it in production.
As allways, we will keep you update.
Oct 04, 2024 - 17:06 CEST
Update - A maintenance will happen on some servers between 10/03/2024 3PM CEST and 10/04/2024 6PM CEST to upgrade the resources. There should be no service interruption.
Oct 03, 2024 - 15:02 CEST
Investigating - Even after the fixes, we're still experiencing time-outs and the error "Reduce your request rate".
Our team is actively working towards a lasting resolution, thank you for your patience and understanding.
Oct 01, 2024 - 12:36 CEST
Monitoring - Following a faulty fix deployed Friday 27 Sept at 10PM CEST, the object storage solution experienced instability. At 14:30PM CEST on the 28 Sept, we began rolling out a downgrade of that fix. Everything was deployed at 6PM CEST and the service is stable again.
Sep 28, 2024 - 18:12 CEST
Investigating - We are experiencing instability on the Object Storage solution in fr-par region since 8PM UTC on friday 27/09, resulting in HTTP 503 errors for customers.
We are currently investigating this issue.
Sep 28, 2024 - 13:40 CEST