Update - On October, 21st, we successfully released a new version in pl-waw region, which significantly improved the availability and performance of the service. The deployment of this update is currently ongoing in nl-ams region and is planned tomorrow morning on October, 23rd for fr-par region. Please note that no interruption of service is expected.
Oct 22, 2024 - 11:27 CEST
Monitoring - The actions of mitigation are completed and both performance and availability are almost back to normal. We will keep the incident open for monitoring until the long term fix is released within a couple of weeks.
Oct 10, 2024 - 15:51 CEST
Update - Out of the monitoring, we confirm that the latency due to the incident has significantly decreased in Paris Region and is almost back to normal (only the 99,9 percentiles are still above usual values). We are planning a hardware update today by the end of the day to remove most services that still generate 5xx errors. A more long term fix is planned in the coming days to fix the incident root cause. As always, we will keep you updated.
Oct 07, 2024 - 10:33 CEST
Update - After monitoring the impact of our intervention, we do see that the globale response time is better (even if there is still some cases, on 99.9 percentil where we have to much latency). We will pursue on this Monday, and remove the faulty services that still respond 5XX errors.
During the week end, we added ressources to our teams on-call to handle trouble that may rise in this situation
On the Bug Fix side, we do have a more long term fix in review today, and will test it next week before we decide when to push it in production.
As allways, we will keep you update.
Oct 04, 2024 - 17:06 CEST
Update - A maintenance will happen on some servers between 10/03/2024 3PM CEST and 10/04/2024 6PM CEST to upgrade the resources. There should be no service interruption.
Oct 03, 2024 - 15:02 CEST
Investigating - Even after the fixes, we're still experiencing time-outs and the error "Reduce your request rate".
Our team is actively working towards a lasting resolution, thank you for your patience and understanding.
Oct 01, 2024 - 12:36 CEST
Monitoring - Following a faulty fix deployed Friday 27 Sept at 10PM CEST, the object storage solution experienced instability. At 14:30PM CEST on the 28 Sept, we began rolling out a downgrade of that fix. Everything was deployed at 6PM CEST and the service is stable again.
Sep 28, 2024 - 18:12 CEST
Investigating - We are experiencing instability on the Object Storage solution in fr-par region since 8PM UTC on friday 27/09, resulting in HTTP 503 errors for customers.
We are currently investigating this issue.
Sep 28, 2024 - 13:40 CEST