Status impact color legend
- Black impact: None
- Yellow impact: Minor
- Red impact: Major
- Blue impact: Maintenance
[Serverless Containers] - [nl-ams] - some containers are restarting because of aggressive probing
Incident
Report for Scaleway
Resolved
We have finished monitoring the situation.
Container instances are now running without a liveness probe, preventing them from restarting. Now, restarts are only triggered by users applicative errors (e.g. if the container crashes).
Note that after our fix made earlier, users might have seen their container waken up, due to the deployed change.
Sorry again for the inconvenience.
Container instances are now running without a liveness probe, preventing them from restarting. Now, restarts are only triggered by users applicative errors (e.g. if the container crashes).
Note that after our fix made earlier, users might have seen their container waken up, due to the deployed change.
Sorry again for the inconvenience.
Monitoring
We have deployed a fix rolling back the probe change done during the migration.
Now, health checks are used as before as only readiness probes, and are not used to restart the containers.
We are still monitoring, but containers shouldn't restart anymore because of aggressive probing.
We will reintroduce the "liveness probe" feature later (maybe optional), but we will make sure to communicate clearly and measure more thoroughly the impacts.
Sorry about any inconvenience.
Now, health checks are used as before as only readiness probes, and are not used to restart the containers.
We are still monitoring, but containers shouldn't restart anymore because of aggressive probing.
We will reintroduce the "liveness probe" feature later (maybe optional), but we will make sure to communicate clearly and measure more thoroughly the impacts.
Sorry about any inconvenience.
Investigating
Since 03/02 and the start of this maintenance: https://status.scaleway.com/incidents/p2cj27y80n9w, an understated breaking change was deployed.
Now, health checks configured by users are not just used as readiness probes, but liveness probes.
As a result, some users containers with failing probes are restarted in loop, even before they have a chance to start correctly.
We will disable temporarily this new behavior until we measure the impacts and find a remediation.
Sorry about any inconvenience.
Now, health checks configured by users are not just used as readiness probes, but liveness probes.
As a result, some users containers with failing probes are restarted in loop, even before they have a chance to start correctly.
We will disable temporarily this new behavior until we measure the impacts and find a remediation.
Sorry about any inconvenience.
This incident affected: Elements - AZ (nl-ams-1, nl-ams-2, nl-ams-3) and Elements - Products (Serverless Functions and Containers).