Some systems are experiencing issues

About This Site

Maintenances and incidents regarding our services, networks and datacenters can be followed here.

Scheduled Maintenance
Scheduled border leaf maintenance

Location : DC2

We'll replace one of our border leaf by a bigger shelf.
Replace shelf will be isolated and links shifted to new one.

No impact expected as our shelves are redundant.

Start : 5-Aug-2020 0700Z 0900LT

Duration : 180 minutes

[INSTANCE] Scheduled maintenance on FR-PAR instance API

Our Instance team is scheduling a maintenance on the instance API.
The maintenance will be done in a 1-hour window.

Impact :
The response time can slightly increase and some 500 errors may occur.

[DC3] Semi-annual maintenance of chillers

[DC3] Semi-annual maintenance of chillers

We will do the semi-annual maintenance of the chillers, according to the manufacturer’s maintenance range.
Operations will be performed by the manufacturer and supervised by our maintainer, according to the following previsional schedule:

  • Monday 17/08: Chiller n°1, Substation A
  • Tuesday 18/08: Chiller n°2, Substation A
  • Wednesday 19/08: Chiller n°3, Substation A
  • Thursday 20/08: Chiller n°4, Substation B
  • Friday 21/08: Chiller n°5, Substation B
  • Monday 24/08: Chiller n°6, Substation B
  • Monday 31/08: Chiller n°7, Substation C
  • Tuesday 01/09: Chiller n°8, Substation C

Substations A and B supply the chilled water network of the computer halls.
Substation C supplies the chilled water network of the operators halls and the air handling units.

Impact : None expected

Start: 17-Aug-2020 0630Z (0830LT)
End: 01-Sep-2020 0330Z (0530LT)
[DC3] Load tests of generators and PLC software updates

[DC3] Load tests of generators and PLC software updates

As part of the preventive maintenance of the Datacenter, we will proceed to a redundancy test of the Datacenter infrastructure by carrying out generator tests chronologically on the following electrical chains : A / B / C / D / F1 and F2.

During the load tests on the electric chains A / B / C and D, we will proceed to F1 line up backup test.
Description of operations :

  • 25/08/2020:
    • Morning: Load test on the electric chain A with F1 line up backup test.
    • Afternoon: Load test on the electric chain B with F1 line up backup test.
  • 26/08/2020:
    • Morning: Load test on the electric chain C with F1 line up backup test.
    • Afternoon: Load test on the electric chain D with F1 line up backup test.
  • 27/08/2020: As part of the mobile chillers installation, we will update PLC software on chains F1 and F2. After we will proceed to load test on the chains F1 andF2.

During the F1 line up backup test, we will simulate a failure of the generators in order to validate the F1 lineup backup scenario.Our teams and usual maintenance companies will be mobilized for the perfect workflow.

Impact : None expected

Start: 25-Aug-2020 0600Z (0800LT)
End: 27-Aug-2020 1600Z (1800LT)

Past Incidents

Tuesday 10th December 2019

[DC3] Maintenance on high-voltage delivery station, scheduled 7 months ago

[DC3] Maintenance on high-voltage delivery station DATA1 and sub-station A/C/F1

Location : DC3

What is done : We will proceed to a cells maintenance to “DATA 1”, sub-station “A/C/F1” and electrical protection tests. The electrical load of DC3 has to be supplied from delivery station “DATA 2” before the maintenance. After the maintenance of “DATA1” the load of sub-stations “A/C/F1” and “B/D/F2” will be switched back to “DATA1”. The electrical load will be supplied from generators during the HV operations.

Impact : None

Start : 10-Dec-2019 0700Z 0800LT

Duration : 10 hours

Monday 9th December 2019

[DC3] Transfer of the electrical supply of sub-station A/C/F1 and B/D/F2 from DATA1 to DATA2, scheduled 7 months ago

[DC3] Transfer of the electrical supply of sub-station A/C/F1 and B/D/F2 from DATA1 to DATA2

Location : DC3

What is done : The intervention consist to supply both sub-stations A/C/F1 and B/D/F2 to DATA2 delivery station to prepare for the preventive maintenance of DATA1 the 10/12/2019. The electrical load will be supplied from generators during the operations on high-voltage cells.

Impact : None

Start : 9-Dec-2019 0700Z 0800LT

Duration : 10 hours

Wednesday 27th November 2019

Web Cloud webhosting

One of our Cloud Web Hosting’s load balancer is failing. Some websites are not reachable for the moment. We are working on a quick solution.

==================

11.27.2019 0500Z (1600LT)

Issue has been fixed. Thank you for your patience. If you notice any issue on your Cloud hosting service, please contact our support.

11.27.2019 0800Z (0900LT)

Issue has been escalated again to the local team

11.26.2019 0845Z (0945LT)

The issue has been fixed. If you are still encountering an issue, please contact our assistance.

11.26.2019 0815Z (0915LT)

Issue has been escalated to the local team

Tuesday 26th November 2019

Console Internal issue with ticketing system

Our teams noticed an internal issue regarding tickets' display on our internal tools.
Our support team is currently unable to process pending request while we are investigating the issue.

Should you have any urgent issue, please get in touch with our support team directly through Slack :
http://scaleway-community.slack.com
Channels :

  • #community (english)
  • #community-fr (french)

===================

11.26.2019 1830Z (1930LT)

Issue has now been fixed.
Creating ticket is available again and we are able to process pending requests accordingly.
Our team will do its best to process all pending tickets as fast as possible.

11.26.2019 1720Z (1820LT)

Ticket creation has been disabled on customer's console, since we aren't able to retrieve these tickets on our backend.
Please use Slack should you have any urgent request.
Our teams are still investigating the issue.

AMS1 Network RPN between AMS and DC3 down

Link between AMS1 and DC3 for the RPN network is down.

We will update this status as soon as we have additional details to share.

===================

11.27.2019 1430Z (1530LT)

All services are back online since last night (around 1am CEST).
Following further monitoring, it appears traffic is now normally routed between AMS and PAR.

11.26.2019 1930Z (2030LT)

Part of the services are back online and the remaining ones should be back up during the evening.

11.26.2019 1000Z (1200LT)

Issue has been identified, it is currently being worked on by our fiber provider.
However, we do not have any ETA yet.

11.26.2019 1000Z (1200LT)

Impacted operator is still working on the precise diagnosis of the current incident

11.26.2019 0800Z (1000LT)

Issue has been escalated to local team

Monday 25th November 2019

Object Storage [SCW] Object Storage - Disconnection and Low Performances

[SCW] Object Storage - Disconnection and Low Performances

PAR and AMS region

We are currently investigating the source of major slowdowns and some disconnections regarding Object Storage

===================

Post-mortem

On Friday 11.22 @ 5 PM GMT, a RAM related incident occured on different servers. This incident slowly cascaded into database unavailability and thus the unability to access some buckets. The incident continued to spread over the weekend despite our efforts and was finally adressed on Sunday 11. 24 @ 9.30 PM GMT.

27.11.19 1630Z (17:30LT)

Issue has been fixed. Thank you for your patience. If you notice any issue with the service please contact our technical support.

25.11.19 0830 (0930LT)

Issue has been escalated to local team

Sunday 24th November 2019

No incidents reported

Saturday 23rd November 2019

No incidents reported

Friday 22nd November 2019

[Scaleway AMS clusters] Scheduled maintenance on hypervisors' network process, scheduled 8 months ago

Following an issue identified in our hypervisors' network configuration, we are going to restart the network process of multiple hypervisors during the day.
You may experience a very short outage while the process is restarted (a few seconds).

Our support team remains at your disposal should you have any question.

Start : 22-Nov-2019, Between 1300Z 1400LT and 1630Z 1730LT

Duration : A few seconds

Thursday 21st November 2019

Dedibackup [DC3] One Dedibackup server unavailable

One of our dedibackup server on DC3 is currently not reachable.
If you are not able to connect on your dedibackup, it means you are impacted by this outage.

Next update as soon as we have more information

===================

11.21.19 0845Z (0945LT)

Issue has been escalated to local team