Investigating - Since around 2024-04-18 06:20 UTC, we are experiencing some network issues on our nl-ams infrastructure. Users with functions/containers in nl-ams region might experience the following:
- 5xx errors (or timeouts) when calling their function/container - high latency when calling their function/container - sporadic network issues (e.g. DNS not resolving) for processes running in their function/container
We are investigating. Sorry about the inconvenience.
Apr 18, 2024 - 10:30 CEST
Investigating - The reconciliation between Grafana's datasources and new regionalized datasources take more time than attended. Some users can't see their data from scaleway products on nl-ams and pl-waw.
Apr 17, 2024 - 20:18 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 15, 2024 - 09:00 CEST
Update - Update: Scheduled for Apr 15, 2024 - Apr 19, 2024
Apr 15, 202409:00 - Apr 19, 202421:00 CEST
Scheduled - Kubernetes Kapsule clusters in the NL-AMS region with public-only endpoints will be migrated to Private Networks.
Network downtime: this migration will result in a temporary network loss of 1 to 10 minutes.
With the new default isolation configuration, worker nodes still have their public IPs to access the Internet. After migrating, existing security groups configuration won’t be overridden and RR wildcard DNS still point to public IPs.
Update - You can still manage datacenter intervention from your dedibox console, in Housing
Dec 19, 2023 - 17:47 CET
Update - We are continuing to investigate this issue.
Dec 19, 2023 - 16:48 CET
Investigating - Ticketing directed to Opcore datacenters is currently unavailable to our dedirack clients. Our team is currently investigating.
Dec 19, 2023 - 16:48 CET
Investigating - We have noticed that problems with connecting to the dedibackup service can occur. We will get back to you as soon as we have more information on the situation.
Apr 06, 2023 - 12:23 CEST
Elements - AZ
Operational
90 days ago
99.88
% uptime
Today
fr-par-1
Operational
90 days ago
99.78
% uptime
Today
fr-par-2
Operational
90 days ago
99.9
% uptime
Today
fr-par-3
Operational
90 days ago
99.9
% uptime
Today
nl-ams-1
Operational
90 days ago
99.78
% uptime
Today
pl-waw-1
Operational
90 days ago
99.98
% uptime
Today
nl-ams-2
Operational
90 days ago
99.78
% uptime
Today
pl-waw-2
Operational
90 days ago
100.0
% uptime
Today
nl-ams-3
Operational
90 days ago
99.78
% uptime
Today
pl-waw-3
Operational
90 days ago
100.0
% uptime
Today
Elements - Products
Degraded Performance
90 days ago
98.86
% uptime
Today
Instances
Operational
90 days ago
94.91
% uptime
Today
BMaaS
Operational
90 days ago
100.0
% uptime
Today
Object Storage
Degraded Performance
90 days ago
99.96
% uptime
Today
C14 Cold Storage
Operational
90 days ago
100.0
% uptime
Today
Kapsule
Operational
90 days ago
96.33
% uptime
Today
DBaaS
Operational
90 days ago
94.39
% uptime
Today
LBaaS
Operational
90 days ago
94.91
% uptime
Today
Container Registry
Operational
90 days ago
98.05
% uptime
Today
Domains
Operational
90 days ago
100.0
% uptime
Today
Elements Console
Operational
90 days ago
100.0
% uptime
Today
IoT Hub
Operational
90 days ago
100.0
% uptime
Today
Account API
Operational
90 days ago
99.98
% uptime
Today
Billing API
Operational
90 days ago
100.0
% uptime
Today
Functions and Containers
Degraded Performance
90 days ago
99.93
% uptime
Today
Block Storage
Operational
90 days ago
99.98
% uptime
Today
Elastic Metal
Operational
90 days ago
100.0
% uptime
Today
Apple Silicon M1
Operational
90 days ago
100.0
% uptime
Today
Private Network
Operational
90 days ago
99.66
% uptime
Today
Hosting
?
Operational
90 days ago
100.0
% uptime
Today
Observability
Operational
90 days ago
99.07
% uptime
Today
Transactional Email
Operational
90 days ago
100.0
% uptime
Today
Dedibox - Datacenters
Degraded Performance
90 days ago
99.08
% uptime
Today
DC2
Operational
90 days ago
99.65
% uptime
Today
DC3
Operational
90 days ago
97.23
% uptime
Today
DC5
Operational
90 days ago
99.74
% uptime
Today
AMS
Degraded Performance
90 days ago
99.7
% uptime
Today
Dedibox - Products
Operational
90 days ago
99.11
% uptime
Today
Dedibox
Operational
90 days ago
92.89
% uptime
Today
Hosting
Operational
90 days ago
100.0
% uptime
Today
SAN
Operational
90 days ago
100.0
% uptime
Today
Dedirack
Operational
90 days ago
100.0
% uptime
Today
Dedibackup
Operational
90 days ago
100.0
% uptime
Today
Dedibox Console
Operational
90 days ago
100.0
% uptime
Today
Domains
Operational
90 days ago
100.0
% uptime
Today
RPN
Operational
90 days ago
100.0
% uptime
Today
Miscellaneous
Operational
90 days ago
100.0
% uptime
Today
Excellence
Operational
90 days ago
100.0
% uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Related
No incidents or maintenance related to this downtime.
Update -
Update: Scheduled for Apr 29, 2024 - May 10, 2024
Feb 28, 2024 - 11:04 CET
Scheduled -
Kubernetes Kapsule clusters in the FR-PAR region with public-only endpoints will be migrated to Private Networks.
Network downtime: this migration will result in a temporary network loss of 1 to 10 minutes.
With the new default isolation configuration, worker nodes still have their public IPs to access the Internet. After migrating, existing security groups configuration won’t be overridden and RR wildcard DNS still point to public IPs.
Resolved -
This incident has been resolved.
Apr 17, 20:47 CEST
Monitoring -
Reconciliation after maintenance take more time than attended
The reconciliation between Grafana's datasources and new regionalized datasources take more time than attended. Some users can't see their data from scaleway products on nl-ams and pl-waw.
Apr 17, 20:16 CEST
Investigating -
Metrics and Logs generated by Scaleway on nl-ams and pl-waw regions were not send to Cockpit.
Apr 15, 13:26 CEST
Completed -
The scheduled maintenance has been completed.
Apr 17, 13:00 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 17, 10:00 CEST
Scheduled -
This maintenance will result in a temporary freeze on orders for Elastic metal and Domain, coupon activation and budget alerts creation and update.
Purchase of domains / monhtly elastic metal server, activation of coupons, budget alerts creation and modification are unavailable.
Apr 16, 11:53 CEST
Completed -
The scheduled maintenance has been completed.
Apr 16, 16:00 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 16, 10:00 CEST
Update -
We will be undergoing scheduled maintenance during this time.
Apr 5, 17:15 CEST
Update -
We will be undergoing scheduled maintenance during this time.
Apr 16, 17:11 CEST
Update -
We will be undergoing scheduled maintenance during this time.
Apr 16, 17:03 CEST
Scheduled -
Cockpit product is being regionalized, scaleway products metrics and logs will be stored in the region they are generated, your Grafana product dashboards will be updated to allow you to have a list of the regional datasource.
Apr 16, 15:00 CEST
Completed -
Maintenance is postponed until tomorrow. We apologize for the inconvenience.
Apr 16, 11:51 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 16, 10:00 CEST
Scheduled -
This maintenance will result in a temporary freeze on orders for Elastic metal and Domain, coupon activation and budget alerts creation and update.
Purchase of domains / monhtly elastic metal server, activation of coupons, budget alerts creation and modification are unavailable
Apr 11, 11:26 CEST
Resolved -
An issue with some control planes has been found, this resulted in some cluster to fail to create/upgrade properly. The issue has been fixed.
Start time: Apr 12, 2024 12:04 UTC End time: Apr 14, 2024 13:05 UTC
Apr 15, 12:41 CEST
Investigating -
We are currently investigating this issue.
Apr 15, 09:13 CEST
Resolved -
This incident has been resolved.
Apr 12, 12:11 CEST
Update -
We are continuing to investigate this issue.
Apr 12, 12:11 CEST
Investigating -
Connectivity between fr-par-3 and the rest of fr-par was half lost in VPC. Kapsule users with nodes in fr-par-3 may have been impacted. It happened between 10:38 and 10:56 CET.
Apr 10, 11:12 CEST
Resolved -
The IP concerned is now delisted and the services are back to normal. If you encounter any issue, you can open a ticket to contact the support so we investigate.
Apr 12, 10:49 CEST
Monitoring -
The IP concerned has been delisted. Please allow up to 48 hours for mitigation.
Apr 9, 16:37 CEST
Investigating -
Our service is currently experiencing disruption due to blacklisting by Microsoft. We are actively working with Microsoft to resolve this issue as soon as possible.
Mar 25, 12:51 CET
Resolved -
All domains returning 404 errors this morning are now working again. Sorry about the inconvenience.
Apr 11, 15:18 CEST
Update -
We are continuing to work on a fix for this issue.
Apr 11, 13:18 CEST
Update -
This might also affect some domains in "ready" status, we are investigating.
Apr 11, 13:18 CEST
Identified -
This affects users having domains configured long ago (> 2 months) with Cloudflare (or similar proxy system), and in "error" status.
Despite the domains in "error" status, these still continued to serve traffic correctly. But, due to a change in our infrastructure, these domains are now unreachable (returning 404 errors). Users with domains in "error" status have to recreate their custom domain.
Note that the function/containers under the custom domain are still reachable through their default endpoint: only the custom domain is returning 404 errors.
Apr 11, 11:42 CEST
Completed -
The scheduled maintenance has been completed.
Apr 11, 12:51 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 9, 15:13 CEST
Scheduled -
The default IP creation value for an instance api will shift from NAT to Routed IP. If you have any script calling the API directly to create instances, please check that it will continue working after the default switch. If you are using our CLI or Terraform adapter, please update to the latest version to support these changes.
Apr 9, 15:12 CEST
Completed -
The scheduled maintenance has been completed.
Apr 11, 11:00 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 8, 11:00 CEST
Scheduled -
From the 8th of April and until the 11th, all Instances using bootscripts will be switched to local boot. If your Instances are using bootscript, they will be restarted and could be unable to boot again. Find our documentation at: https://www.scaleway.com/en/docs/compute/instances/troubleshooting/bootscript-eol/
Apr 3, 17:03 CEST
Resolved -
This incident has been resolved.
Apr 11, 10:29 CEST
Identified -
In normal cases, if a SQS message can't be processed by a SQS trigger because the underlying function/container returns a non-OK status code (4xx, 5xx), our system will retry 3 times before dropping the message. Today, this retry mechanism is broken due to incompatibilities with the SQS protocol, so messages will be replayed infinitely if the function/container return a non-OK status code.
As a workaround, users can always return a 200 status code in their function/container, even if the processing of the message returns an applicative error. This is just a workaround though, as we have identified the root cause, and a fix is coming.
Completed -
The scheduled maintenance has been completed.
Apr 9, 12:00 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 9, 09:00 CEST
Scheduled -
The default IP creation value for an instance api will shift from NAT to Routed IP. If you have any script calling the API directly to create instances, please check that it will continue working after the default switch. If you are using our CLI or Terraform adapter, please update to the latest version to support these changes.
Mar 13, 11:16 CET
Resolved -
Everything is back to nominal state. We are closely monitoring the situation.
Apr 8, 15:04 CEST
Identified -
5xx error rate is back to normal. Around 5% of the PUT request are still slowed down, we are working to fix the issue.
Apr 8, 14:50 CEST
Investigating -
Following a maintenance of the s3 backend, the s3 endpoints in nl-ams are returning an increased rate of 5xx errors. We have found the issue and are on it. Impacts: all interactions with s3 in nl-ams might result in a 5xx error. Sorry for the inconvenience.
Apr 8, 14:20 CEST
Completed -
The scheduled maintenance has been completed.
Apr 8, 12:00 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 8, 11:00 CEST
Scheduled -
We need to replace the power supplies of the dedirack router in DC2 room 103. No downtime is expected, but the router might reboot when the PSU are swapped, which would make dediracks unavailble for a few minutes.
Apr 4, 14:55 CEST
Resolved -
A network device rebooted today. Public service on some servers located at DC5 Room 1 Rack 38 was unavailable between 14:58 and 15:00 UTC. Our investigation did not permit to understand the root cause.
Apr 5, 17:41 CEST
Completed -
The scheduled maintenance has been completed.
Apr 5, 17:00 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 1, 09:00 CEST
Update -
Update: Scheduled for April 1, 2024 to Apr 5, 2024
Feb 28, 10:55 CET
Scheduled -
Kubernetes Kapsule clusters in the PL-WAW region with public-only endpoints will be migrated to Private Networks.
Network downtime: this migration will result in a temporary network loss of 1 to 10 minutes.
With the new default isolation configuration, worker nodes still have their public IPs to access the Internet. After migrating, existing security groups configuration won’t be overridden and RR wildcard DNS still point to public IPs.
Resolved -
This incident has been resolved.
Apr 5, 10:23 CEST
Investigating -
The public device located in DC2 room 202A rack A11 have rebooted today. The public service for the associated servers was down for 5 minutes between 10:12 and 10:17 UTC. Service is now up and running. We are investigating on the root cause of this reboot.
Apr 4, 13:03 CEST
Completed -
The scheduled maintenance has been completed.
Apr 4, 18:23 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 4, 15:53 CEST
Scheduled -
Due to hardware failure, we needed to reload a router that carry PAR2 Instances services. Maintenance is scheduled between 14:30 and 17:00 CET
Apr 4, 15:52 CEST
Resolved -
This incident has been resolved.
Apr 4, 14:39 CEST
Monitoring -
A fix has been implemented and we are monitoring the results.
Mar 12, 18:17 CET
Investigating -
We are currently investigating on an issue on the webmail interface that is unreachable currently https://webmail.online.net/
Mar 12, 11:21 CET
Resolved -
This incident has been resolved. If the issue remains, do not hesitate to contact the support team.
Apr 4, 14:33 CEST
Investigating -
The issue is more widespread and there seems to be a global issue with the reachability between Scaleway and many Internet network operator located in Egypt. The same issues of reaching Egyptian ressources are also observed from other (than Scaleway) internet operators. We are still investigating what is causing this issue, and trying to escalate the problem to the Egyptian network operators, but at this point data we have indicated that the issue is global and out of our control
Jan 25, 15:35 CET
Update -
Our investigation is suggesting that the issue is located on Orange Egypt side, and we are trying to get in touch with them to solve the issue.
Jan 24, 10:45 CET
Identified -
The issue has been identified, and is located on the provider side, Orange Egypt.
Jan 19, 11:21 CET
Investigating -
Our network is unreachable from one ( or more ) Egypt internet provider. We are currently investigating this issue.
Jan 19, 09:03 CET
Resolved -
This incident has been resolved.
Apr 4, 13:05 CEST
Update -
The root cause has been identified, we are working on a fix
Apr 4, 10:44 CEST
Investigating -
We are currently experiencing problems with devices using MQTT to connect to their Hub. The devices may fail to connect or get disconnected after a short period of time.
Apr 4, 10:17 CEST
Resolved -
This incident has been resolved.
Apr 4, 11:51 CEST
Identified -
Our API in pl-waw for serverless functions and containers are returning sporadic 5xx errors. We have found the issue, and are on it. Impacts: all interactions with the API in pl-waw might result in a 5xx error. Sorry about the inconvenience.
Apr 4, 11:19 CEST