Wednesday 2nd October 2019

Object Storage [Object Storage] AMS - Buckets unavailability

We have detected an issue with our Object Storage service in AMS.

Buckets may be unavailable for some customers, and may fail to show both in the console or through the API.

Our engineers are working on fixing this as soon as possible.

===================

03/10/2019 1330Z (1530LT)

Dear Scaleway customer,
Yesterday, October the 2nd, at 11AM (Paris time), we launched a maintenance on our clusters located in Amsterdam. This maintenance was meant to enhance the size of the clusters, adding new storage pods to host your ever-growing data.
Even though this maintenance underwent successful testing in our staging environment, we faced an availability outage that impacted your buckets. Your data remain safe and are not impacted. However, you might have noticed 40x errors, or no bucket listed on the console making your data unreachable during 12 to 30 hours. The team spent the whole afternoon fixing the issue and your data are now available again. We fully understand that this situation is not acceptable and we truly apologize for the inconvenience it caused. In case your bucket(s) remain(s) unavailable, please contact our support team directly.

03/10/2019 1530Z (1730LT)

We are still restoring pending buckets. No data loss to expect, but process is quite long since each bucket has to be processed one by one.
We will continue updating this status in the process.

2019-10-02 1920Z (2120LT)

The issue has been fixed for almost all buckets. We are still looking to resolve the matter for a few buckets.

2019-10-02 1530Z (1730LT)

Unavailable buckets are coming back up, but the process will be long.

2019-10-02 1400Z (1600LT)

The issue is taking longer than expected to fix.
The product team is still working on permanently fixing the issue.

2019-10-02 1000Z (1200LT)

The product team has identified the root cause and working on a fix

2019-10-02 0945Z (1145LT)

Issue has been escalated to local team