Some systems are experiencing issues

About This Site

Maintenances and incidents regarding our services, networks and datacenters can be followed here.

Scheduled Maintenance
[Amsterdam core router bb2.ams1] Router upgrade

Location : Amsterdam - Core router bb2.ams1
What is done : Upgrade Software and Reload bb2.ams1 to fix FlowSpec issues. Rollback : rollback to previous release
Expected Impacts : None, traffic should be rerouted elsewhere.

Preparation - duration 2h

-start time : 26-Feb-2020 0500Z (0600LT)
-end time : 26-Feb-2020 0700Z (0800LT)

Full upgrade - duration 4H

-start time : 27-Feb-2020 0300Z (0400LT)
-end time : 27-Feb-2020 0700Z (0800LT)

[DC3] Scheduled maintenance in Room 3-4 / Racks : C1>C22 - D1>D21

A scheduled maintenance is planned in part of the room 3-4, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C22 - D1>D21

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 3rd, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 205 / Racks : C1>C8 - D1>D8 - E1>E7

A scheduled maintenance is planned in part of the room 205, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C8 - D1>D8 - E1>E7

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 3rd, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC3] Scheduled maintenance in Room 4-6 / Racks : A1>A23 - B1>B23

A scheduled maintenance is planned in part of the room 4-6, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : A1>A23 - B1>B23

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 4th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

UPDATE:

2/12/2020 - 0410Z (0510LT)

Racks A1>A23 are upgraded and functional

[DC3] Scheduled maintenance in Room 4-6 / Racks : C1>C22 - D1>D22

A scheduled maintenance is planned in part of the room 4-6, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C22 - D1>D22

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 4th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 101 / Racks : A20>A28 - B20>B26 - L31>L37 - K37

A scheduled maintenance is planned in part of the room 101, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : A20>A28 - B20>B26 - L31>L37 - K37

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 5th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 203A / Racks : M1>M14 - N1>N6

A scheduled maintenance is planned in part of the room 203A, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : M1>M14 - N1>N6

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 5th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 205 / Racks : A1>A8 - B1>B8 - G2>G8 - H2

A scheduled maintenance is planned in part of the room 205, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : A1>A8 - B1>B8 - G2>G8 - H2

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 10th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC3] Scheduled maintenance in Room 3-5 / Racks : E1>E21 - F1>F21

A scheduled maintenance is planned in part of the room 3-5, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : E1>E21 - F1>F21

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 10th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC3] PLC and HMI software update on chains F1 and F2for mobile chillers power supply supervision

[DC3] PLC and HMI software update on chains F1 and F2for mobile chillers power supply supervision

As part of the mobile chillers installation work, we will update PLC and HMI softwares on chains F1 and F2. The modifications affect the supervision of mobile chillers breakers (display, alarms).

All the changes that will be made will only be one electrical chain at a time. Under no circumstances will two electrical chains be modified simultaneously.

Impact : none for customers

Start : 12-Mar-2020 0700Z 0800LT

Duration : 1 day

[DC3] Preventive maintenance of CRAC units

We will carry out the semi-annual preventive maintenance of CRAC units according to the schedule below. CRAC units will be shut down one at a time, so as not to impact the cooling of halls. Each IT zone has CRAC unit called « backup » : each CRAC unit is doubled. Each MVLB hall has a main air hanling unit (AHU).

The maintenance will be done according to the following schedule :

  • MLVBs 27/03/2020 and 30/03/2020
  • Operators halls : 20/03/2020
  • IT halls n°4 : 16 and 17/03/2020
  • IT halls n°3 : 18 and 19/03/2020
  • IT halls n°2 : 23 to 25/03/2020
  • IT halls n°1 : 25 and 26/03/2020

Impacts on services : Loss of cooling redundancy during the preventive maintenance operation of a CRAC unit. No impact on the cooling of facilities. The maximum cut-off time per CRAC unit is otherwise defined at 1 hour.

[DC3] Maintenance of electric generators

Location : DC3

What is done : We’re going to realize the preventive electrical maintenance of generators, the maintenance will be performed electrical chain by electrical chain :

Chain A : 23/03/30
Chain B : 24/03
Chain C : 25/03
Chain D : 26/03
Chain F1 : 30/03
Chain F2 : 31/03

Start : 23-Mars-2020 0730Z (0830LT) End : 31-Mars-2020 0430Z(0530LT)

Impact : Degradation of electrical redundancy Duration 6 days

[DC3] Pressurization maintenance

[DC3] Pressurization maintenance

We will realized a pressurization maintenance on substation SSA, SSB and SSC

Impact : none

Start : 31-Mar-2020 0830Z 0930LT
End : 2-Apr-2020 1600Z 1700LT

Duration : 3 days

Past Incidents

Tuesday 10th December 2019

[DC3] Maintenance on high-voltage delivery station, scheduled 2 months ago

[DC3] Maintenance on high-voltage delivery station DATA1 and sub-station A/C/F1

Location : DC3

What is done : We will proceed to a cells maintenance to “DATA 1”, sub-station “A/C/F1” and electrical protection tests. The electrical load of DC3 has to be supplied from delivery station “DATA 2” before the maintenance. After the maintenance of “DATA1” the load of sub-stations “A/C/F1” and “B/D/F2” will be switched back to “DATA1”. The electrical load will be supplied from generators during the HV operations.

Impact : None

Start : 10-Dec-2019 0700Z 0800LT

Duration : 10 hours

Monday 9th December 2019

[DC3] Transfer of the electrical supply of sub-station A/C/F1 and B/D/F2 from DATA1 to DATA2, scheduled 2 months ago

[DC3] Transfer of the electrical supply of sub-station A/C/F1 and B/D/F2 from DATA1 to DATA2

Location : DC3

What is done : The intervention consist to supply both sub-stations A/C/F1 and B/D/F2 to DATA2 delivery station to prepare for the preventive maintenance of DATA1 the 10/12/2019. The electrical load will be supplied from generators during the operations on high-voltage cells.

Impact : None

Start : 9-Dec-2019 0700Z 0800LT

Duration : 10 hours

Wednesday 27th November 2019

Web Cloud webhosting

One of our Cloud Web Hosting’s load balancer is failing. Some websites are not reachable for the moment. We are working on a quick solution.

==================

11.27.2019 0500Z (1600LT)

Issue has been fixed. Thank you for your patience. If you notice any issue on your Cloud hosting service, please contact our support.

11.27.2019 0800Z (0900LT)

Issue has been escalated again to the local team

11.26.2019 0845Z (0945LT)

The issue has been fixed. If you are still encountering an issue, please contact our assistance.

11.26.2019 0815Z (0915LT)

Issue has been escalated to the local team

Tuesday 26th November 2019

Console Internal issue with ticketing system

Our teams noticed an internal issue regarding tickets' display on our internal tools.
Our support team is currently unable to process pending request while we are investigating the issue.

Should you have any urgent issue, please get in touch with our support team directly through Slack :
http://scaleway-community.slack.com
Channels :

  • #community (english)
  • #community-fr (french)

===================

11.26.2019 1830Z (1930LT)

Issue has now been fixed.
Creating ticket is available again and we are able to process pending requests accordingly.
Our team will do its best to process all pending tickets as fast as possible.

11.26.2019 1720Z (1820LT)

Ticket creation has been disabled on customer's console, since we aren't able to retrieve these tickets on our backend.
Please use Slack should you have any urgent request.
Our teams are still investigating the issue.

AMS1 Network RPN between AMS and DC3 down

Link between AMS1 and DC3 for the RPN network is down.

We will update this status as soon as we have additional details to share.

===================

11.27.2019 1430Z (1530LT)

All services are back online since last night (around 1am CEST).
Following further monitoring, it appears traffic is now normally routed between AMS and PAR.

11.26.2019 1930Z (2030LT)

Part of the services are back online and the remaining ones should be back up during the evening.

11.26.2019 1000Z (1200LT)

Issue has been identified, it is currently being worked on by our fiber provider.
However, we do not have any ETA yet.

11.26.2019 1000Z (1200LT)

Impacted operator is still working on the precise diagnosis of the current incident

11.26.2019 0800Z (1000LT)

Issue has been escalated to local team

Monday 25th November 2019

Object Storage [SCW] Object Storage - Disconnection and Low Performances

[SCW] Object Storage - Disconnection and Low Performances

PAR and AMS region

We are currently investigating the source of major slowdowns and some disconnections regarding Object Storage

===================

Post-mortem

On Friday 11.22 @ 5 PM GMT, a RAM related incident occured on different servers. This incident slowly cascaded into database unavailability and thus the unability to access some buckets. The incident continued to spread over the weekend despite our efforts and was finally adressed on Sunday 11. 24 @ 9.30 PM GMT.

27.11.19 1630Z (17:30LT)

Issue has been fixed. Thank you for your patience. If you notice any issue with the service please contact our technical support.

25.11.19 0830 (0930LT)

Issue has been escalated to local team

Sunday 24th November 2019

No incidents reported

Saturday 23rd November 2019

No incidents reported

Friday 22nd November 2019

[Scaleway AMS clusters] Scheduled maintenance on hypervisors' network process, scheduled 3 months ago

Following an issue identified in our hypervisors' network configuration, we are going to restart the network process of multiple hypervisors during the day.
You may experience a very short outage while the process is restarted (a few seconds).

Our support team remains at your disposal should you have any question.

Start : 22-Nov-2019, Between 1300Z 1400LT and 1630Z 1730LT

Duration : A few seconds

Thursday 21st November 2019

Dedibackup [DC3] One Dedibackup server unavailable

One of our dedibackup server on DC3 is currently not reachable.
If you are not able to connect on your dedibackup, it means you are impacted by this outage.

Next update as soon as we have more information

===================

11.21.19 0845Z (0945LT)

Issue has been escalated to local team