Some systems are experiencing issues

About This Site

Maintenances and incidents regarding our services, networks and datacenters can be followed here.

Scheduled Maintenance
[Amsterdam core router bb2.ams1] Router upgrade

Location : Amsterdam - Core router bb2.ams1
What is done : Upgrade Software and Reload bb2.ams1 to fix FlowSpec issues. Rollback : rollback to previous release
Expected Impacts : None, traffic should be rerouted elsewhere.

Preparation - duration 2h

-start time : 26-Feb-2020 0500Z (0600LT)
-end time : 26-Feb-2020 0700Z (0800LT)

Full upgrade - duration 4H

-start time : 27-Feb-2020 0300Z (0400LT)
-end time : 27-Feb-2020 0700Z (0800LT)

[DC3] Scheduled maintenance in Room 3-4 / Racks : C1>C22 - D1>D21

A scheduled maintenance is planned in part of the room 3-4, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C22 - D1>D21

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 3rd, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 205 / Racks : C1>C8 - D1>D8 - E1>E7

A scheduled maintenance is planned in part of the room 205, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C8 - D1>D8 - E1>E7

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 3rd, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC3] Scheduled maintenance in Room 4-6 / Racks : A1>A23 - B1>B23

A scheduled maintenance is planned in part of the room 4-6, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : A1>A23 - B1>B23

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 4th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

UPDATE:

2/12/2020 - 0410Z (0510LT)

Racks A1>A23 are upgraded and functional

[DC3] Scheduled maintenance in Room 4-6 / Racks : C1>C22 - D1>D22

A scheduled maintenance is planned in part of the room 4-6, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C22 - D1>D22

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 4th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 101 / Racks : A20>A28 - B20>B26 - L31>L37 - K37

A scheduled maintenance is planned in part of the room 101, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : A20>A28 - B20>B26 - L31>L37 - K37

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 5th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 203A / Racks : M1>M14 - N1>N6

A scheduled maintenance is planned in part of the room 203A, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : M1>M14 - N1>N6

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 5th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 205 / Racks : A1>A8 - B1>B8 - G2>G8 - H2

A scheduled maintenance is planned in part of the room 205, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : A1>A8 - B1>B8 - G2>G8 - H2

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 10th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC3] Scheduled maintenance in Room 3-5 / Racks : E1>E21 - F1>F21

A scheduled maintenance is planned in part of the room 3-5, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : E1>E21 - F1>F21

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 10th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC3] PLC and HMI software update on chains F1 and F2for mobile chillers power supply supervision

[DC3] PLC and HMI software update on chains F1 and F2for mobile chillers power supply supervision

As part of the mobile chillers installation work, we will update PLC and HMI softwares on chains F1 and F2. The modifications affect the supervision of mobile chillers breakers (display, alarms).

All the changes that will be made will only be one electrical chain at a time. Under no circumstances will two electrical chains be modified simultaneously.

Impact : none for customers

Start : 12-Mar-2020 0700Z 0800LT

Duration : 1 day

[DC3] Preventive maintenance of CRAC units

We will carry out the semi-annual preventive maintenance of CRAC units according to the schedule below. CRAC units will be shut down one at a time, so as not to impact the cooling of halls. Each IT zone has CRAC unit called « backup » : each CRAC unit is doubled. Each MVLB hall has a main air hanling unit (AHU).

The maintenance will be done according to the following schedule :

  • MLVBs 27/03/2020 and 30/03/2020
  • Operators halls : 20/03/2020
  • IT halls n°4 : 16 and 17/03/2020
  • IT halls n°3 : 18 and 19/03/2020
  • IT halls n°2 : 23 to 25/03/2020
  • IT halls n°1 : 25 and 26/03/2020

Impacts on services : Loss of cooling redundancy during the preventive maintenance operation of a CRAC unit. No impact on the cooling of facilities. The maximum cut-off time per CRAC unit is otherwise defined at 1 hour.

[DC3] Maintenance of electric generators

Location : DC3

What is done : We’re going to realize the preventive electrical maintenance of generators, the maintenance will be performed electrical chain by electrical chain :

Chain A : 23/03/30
Chain B : 24/03
Chain C : 25/03
Chain D : 26/03
Chain F1 : 30/03
Chain F2 : 31/03

Start : 23-Mars-2020 0730Z (0830LT) End : 31-Mars-2020 0430Z(0530LT)

Impact : Degradation of electrical redundancy Duration 6 days

[DC3] Pressurization maintenance

[DC3] Pressurization maintenance

We will realized a pressurization maintenance on substation SSA, SSB and SSC

Impact : none

Start : 31-Mar-2020 0830Z 0930LT
End : 2-Apr-2020 1600Z 1700LT

Duration : 3 days

Past Incidents

Thursday 21st November 2019

[DC3] Maintenance of HV stations DATA 1, scheduled 3 months ago

[DC3] Maintenance of HV stations DATA 1

The planned preventive maintenance consists of the maintenance of HV cells and electrical protection tests on delivery station “DATA1”. The electric chains will be supplied from generators during this maintenance.

Impact : This intervention does not present any particular risk of service interruption and will be processed during working hours

Start : 21-Nov-2019 0700Z 0800LT

Duration : 10 hours

Monday 18th November 2019

[RPN] DC3 >< AMS1 - SIPARTECH Network update, scheduled 3 months ago

SIPARTECH is doing a maintenance on the network.

Location : DC3/AMS1

What is done : preventive network maintenance

Impact : RPN Service will be unavailable

Start : 18-Nov-2019 2259Z 2359LT

Duration : 6 hours.

Thursday 7th November 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B17-B20, scheduled 3 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B17-B20

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 07-Nov-2019 0800Z 0900LT

Duration : 60 minutes

Wednesday 6th November 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B9-B17, scheduled 3 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B9-B17

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 06-11-2019 0800Z 0900LT

Duration : 60 minutes

Tuesday 5th November 2019

[PAR1, AMS1] BIOS and kernel upgrade on DEV1 and GP1 hypervisors, scheduled 3 months ago

[PAR1, AMS1] BIOS and kernel upgrade on DEV1 and GP1 hypervisors

Location : PAR1, AMS1

What is done : We are about to upgrade our DEV1 & GP1 hypervisors, so as to:

  • implement kernel upstream changes committed by Scaleway contributors for new upcoming features,
  • implement vendor BIOS upgrade to fix major availability issues and improve performances of our DBaaS product,
  • implement Ubuntu bugfix to the snapshot functionality to improve reliability of snapshots.

Impact : 20 minutes downtime (per hypervisor)

We are canceling this maintenance and will give a new date as soon as possible.

Duration : 7 days on Paris region & 5 days on Amsterdam region

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B3-B9, scheduled 3 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B3-B9

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 05-Nov-2019 0800Z 0900LT

Duration : 60 minutes

Sunday 3rd November 2019

No incidents reported

Saturday 2nd November 2019

Console [CONSOLE] online.net and console.online.net unavailability

Website and console (online.net) are currently unavailable. Our team has already been warned and is currently working on it in order to fix it as quickly as possible.

Our apologies for the inconvenience.

===================

02.11.2019 1215Z (1315LT)

Console and website are back, we will keep monitoring it to make sure that everything is fully resolved

Webhosting issue

We have detected a problem on some hosting that appear in maintenance or error 500. Our teams are working on the resolution.

===================

02/11/19 1300Z (1400LT)

Services are now back online and websites should be reachable again. Do not hesitate to contact our support team if you experience any other issue.

Friday 1st November 2019

Object Storage Registry SSL incident

We are currently facing a SSL certificat issue on Registry Container product.

It is not possible to pull images from registry product.

On Kapsule product, cluster creations/upgrade or scaling are unavailable since 5:14 AM.

Regarding Registry, It is not possible to pull images from registry product from midnight.

Next update by 11.01.19 1600Z (1700LT)

===================

11.01.19 1000Z (1100LT)

Issue has been escalated to local team

11.01.19 1120Z (1220LT)

Issue has been fixed

Thursday 31st October 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/A15-B3, scheduled 3 months ago

We are also upgrading another TOR uplinks from 1G to 4G for racks S301/A15-B3

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 31-Oct-2019 0800Z 0900LT

Duration : 60 minutes

DC3 Oncurrent battery maintenance

[Curative maintenance on battery of UPS]

We will do the curative maintenance of the inverters of electrical chains A and B. The inverters of these electrical chains supply racks of IT halls. Operations will be performed by our maintainer under our permanent supervision. UPS impacted :

ASI A1: 6 batterys

ASI B2 : 1 battery

This maintenance requires isolating each UPS in order to intervene on the device. The downtime of an UPS is estimated to take 30 minutes.

The operation will be done UPS by UPS and will have no impact on your equipments housed in our Datacenter : the others UPS of the electrical chain are sized to recover the load from the offline UPS.

Location : DC3 DATACENTER

Impact : None

Start : 31-Oct-2019 0730Z 0930LT
End : 31-Oct-2019 1530Z 1730LT

Duration : 1 day

Wednesday 30th October 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/A9-A15, scheduled 3 months ago

We are upgrading TOR uplinks from 1G to 4G for racks S301/A9-A15

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 30-Oct-2019 0800Z 0900LT

Duration : 60 minutes

DC5-A Network [DC5-A] Short network outage

We encountering a network downtime for less than a minute on DC5-A today at 04:30PM (Local Time).
The issue has been fixed. Network traffic is now working properly.

===================

10.30.19 1530Z (1630LT)

Issue has been escalated to local team and fixed.

[DC2] Room 103 - Rack A6 - Network issues

We detected an outage on DC2 - Room 103 - Rack A6.
Our network team is working on a fix.
Next update in 2 hours

===================

10.30.2019 1530Z (1630LT)

The switch has been replaced. Everything is back to normal.

10.30.2019 1400Z (1500LT)

The switch is dead. We need to replace it. It might take a while.

10.30.2019 1320Z (1420LT)

Issue has been escalated to local team

Console Licence Windows issue

We have noticed that some Microsoft Windows licenses are marked as terminated when they should not be. We are working on a solution.

===================

10.30.19 0730Z (0830LT)

Issue has been escalated to local team

10.30.19 1015Z (1115LT)

Licenses expirations are now corrected. Don’t hesitate to open a support ticket if you wish a manual verification.

DC5-A Network RPN saturation between DC5-A and other locations

Due to a failure on our primary RPN link between DC5-A and other datacenters, RPN traffic may be saturated from/to this DC as all traffic has been rerouted to our backup link.
We will update this status as soon as we have additional details to share.

===================

10.30.19 0540Z (0640LT)

Issue has been escalated to local team

10.30.19 0740Z (0830LT)

The issue is now fixed.

Tuesday 29th October 2019

Console [Dedibox] Installation issue

We detected an issue with our install system.
Your installation might not go through. We are currently investigating on this matter.
Next update as soon as we have more information.

===================

##### 10.29.19 2000Z (2200LT) Issue has been escalated to local team

30/10/19 0845Z (0945LT)

Issue is now fixed

[SCW] Container Registry unavailable

We are currently encountering an internal issue which impacts our Container Registry service.
The service may not be reachable.

Next update in 2 hours

===================

10.29.19 2030Z (2130LT)

The API issue linked to our backend has been fixed. Everything is back to normal.

10.29.19 1730Z (1830LT)

The issue has been found and it will be fixed ASAP.

10.29.19 1530Z (1630LT)

Issue has been escalated to local team

Monday 28th October 2019

No incidents reported