[Elements] Performance issues on Multi-AZ Object Storage buckets

Incident Report for Scaleway

Resolved

Our supervision confirm that everything is ok, performance is good and no latency has been seen

Posted Jul 26, 2022 - 17:18 CEST

Update

Thanks to all the updates on the cluster, the situation is back to normal, we remain monitoring the cluster.

Posted Jul 11, 2022 - 14:59 CEST

Update

We have received, racked and deployed new hardware dedicated to our Object Storage product.
We will be closely monitoring the situation during the next few days while the cluster is spreading the load to the new servers.
If you are still experiencing performance issues on your end, please contact our support team directly via ticket.

Posted Jul 08, 2022 - 17:09 CEST

Monitoring

We've added 25% more compute capacity last week on our fr-par cluster, that leads to significant improvements on tail latency, resulting on more stable performances. To provide an even better service, we plan to double the compute capacity by 10 days. The team are monitoring carefully in the mean time

Posted Jun 22, 2022 - 15:45 CEST

Identified

Services are partially degraded.
Our servers are showing some highs and low in terms of workload, this might lead to potential slowness / timeouts for the user.

Our team is still working on a durable fix to those issues, your data remains safe in the meantime.

Posted May 25, 2022 - 13:58 CEST

Monitoring

A fix has been deployed and customers seem to have gained back their original performances.
We are still monitoring the situation at this time

Posted May 24, 2022 - 13:35 CEST

Update

The issue also impacts our Container Registry service, you may encounter some timeouts while pulling images.

Posted May 24, 2022 - 11:42 CEST

Investigating

We have noticed some issues with multi-AZ buckets.
You may experience an increase in latency with your buckets, as well as some 5xx errors.

Posted May 24, 2022 - 11:26 CEST

This incident affected: Elements - Products (Object Storage, Container Registry).