[FR-PAR / NL-AMS / PL-WAW] Intermittent DNS failures in Serverless Containers cluster

Incident Report for Scaleway

Resolved

The incident is now closed.
More details regarding the incident: following an unattended upgrade on some nodes in our infrastructure, some services have encountered networking problems. We have patched the whole infrastructure to prevent this specific issue from happening again.
This affected around ~50% of nodes in all regions, between around 2025-02-07 06:30 AM UTC and 2025-02-07 12:15 PM. During that time, functions/containers might have encountered timeouts when doing DNS resolutions (for example, this is the case when calling external services).
Sorry again for any inconvenience.

Posted Feb 07, 2025 - 16:49 CET

Monitoring

Faulty nodes on AMS and PAR have been fixed. We are going to monitor, but incident should not have impact anymore.
Sorry about the inconvenience.

Posted Feb 07, 2025 - 12:15 CET

Update

Faulty nodes on WAW have been fixed. Issue still remains on AMS and PAR, we are on it.

Posted Feb 07, 2025 - 11:45 CET

Update

We have also noticed this kind of errors on other regions (nl-ams, pl-waw). We are also doing operations on nodes there. Sorry about any inconvenience.

Posted Feb 07, 2025 - 11:36 CET

Identified

We've identified the issue and are in the process of replacing faulty hosts running Serverless Containers and Functions.
This may cause addition disruptions while we conduct those operations.
Sorry for the inconvenience, we're doing everything we can to resolve as swiftly as possible.

Posted Feb 07, 2025 - 11:14 CET

Investigating

We are currently investigating this issue.

Posted Feb 07, 2025 - 10:29 CET

This incident affected: Elements - Products (Serverless Functions and Containers).