Some systems are experiencing issues

About This Site

Maintenances and incidents regarding our services, networks and datacenters can be followed here.

Scheduled Maintenance
[DC3] Maintenance of chillers

[DC3] Maintenance of chillers

Location: DC3

What is done: We will do the semi-annual maintenance of the chillers, according to the following schedule :

Chiller n°8: 27/01
Chiller n°7: 28/01
Chiller n°1: 29/01
Chiller n°2: 30/01
Chiller n°3: 31/01
Chiller n°4: 03/02
Chiller n°5: 04/02
Chiller n°6: 05/02

Impact: No impact is to be expected.

Start: 27-Jan-2020 0730Z 0830LT
End: 05-Feb-2020 1630Z 1730LT

Duration : 8 days

[DC5] Maintenance on Generator set

[DC5] Preventive mechanical maintenance of generating sets

Location : DC5 What is done : We will carry out generator maintenance according to this planing:

GE BAT 1: 01/29 GE LT1.2: 01/30 GE LT1.1: 01/31

Impact : No impact is to be expected.

Start : 29-Jan-2020 0700Z 0800LT

Duration : 3 days

[DC3] Load tests of generators

As part of the preventive maintenance of the Datacenter, we will proceed to a redundancy test of the Datacenter infrastructure. The operation consists of simulating a power failure on the MVLBs and validating the electrical load recovery and working on generator for 4 hours. Our teams and usual maintenance companies will be mobilized for the perfect workflow.

This intervention does not present any particular risk of service interruption and will be processed on worked hours.

Start: 2020-01-30 0800Z (0900LT)
End: 2020-01-30 1500Z (1600LT)

Duration: 7 hours

[DC5-A] Scheduled maintenance Room 120 - Racks G1, F11

A scheduled maintenance is planned in DC5-A (Room 120 - Racks G1, F11) in order to optimize and upgrade our infrastructure.

It will be done gradually on the racks, in the same order as in the object of the maintenance.

As a result servers will lose their public network connectivity as our switches are being upgraded, but private network connectivity will still be functional. It will come back gradually following the scheduled upgrades.

Updates will be posted below as the maintenance is going on.

Start : February 4th, 2020 - 0600Z (0700LT)

Duration : 2 hours

[DC5-A] Scheduled maintenance Room 120 - Racks G15, G16, B17

A scheduled maintenance is planned in DC5-A (Room 120 - Racks G15, G16, B17) in order to optimize and upgrade our infrastructure.

It will be done gradually on the racks, in the same order as in the object of the maintenance.

As a result servers will lose their public network connectivity as our switches are being upgraded, but private network connectivity will still be functional. It will come back gradually following the scheduled upgrades.

Updates will be posted below as the maintenance is going on.

Start : February 5th, 2020 - 0600Z (0700LT)

Duration : 2 hours

[DC5-A] Scheduled maintenance Room 120 - Racks B18, B19, F1

A scheduled maintenance is planned in DC5-A (Room 120 - Racks B18, B19, F1) in order to optimize and upgrade our infrastructure.

It will be done gradually on the racks, in the same order as in the object of the maintenance.

As a result servers will lose their public network connectivity as our switches are being upgraded, but private network connectivity will still be functional. It will come back gradually following the scheduled upgrades.

Updates will be posted below as the maintenance is going on.

Start : February 6th, 2020 - 0600Z (0700LT)

Duration : 2 hours

[Amsterdam core router bb1.ams1] Router upgrade

Location : Amsterdam - core router bb1.ams1

What is done : Upgrade Software and Reload bb1.ams1 to fix FlowSpec issues. Rollback to previous release.

Impact : None expected, traffic will be rerouted elsewhere.

Preparation - duration 2h

  • start time : 10-Feb-2020 0500Z (0600LT)
  • end time : 10-Feb-2020 0700Z (0800LT)

Full upgrade - duration 4h

  • start time : 11-Feb-2020 0300Z (0400LT)
  • end time : 11-Feb-2020 0700Z (0800LT)

Past Incidents

Thursday 21st November 2019

[DC3] Maintenance of HV stations DATA 1, scheduled 2 months ago

[DC3] Maintenance of HV stations DATA 1

The planned preventive maintenance consists of the maintenance of HV cells and electrical protection tests on delivery station “DATA1”. The electric chains will be supplied from generators during this maintenance.

Impact : This intervention does not present any particular risk of service interruption and will be processed during working hours

Start : 21-Nov-2019 0700Z 0800LT

Duration : 10 hours

Monday 18th November 2019

[RPN] DC3 >< AMS1 - SIPARTECH Network update, scheduled 2 months ago

SIPARTECH is doing a maintenance on the network.

Location : DC3/AMS1

What is done : preventive network maintenance

Impact : RPN Service will be unavailable

Start : 18-Nov-2019 2259Z 2359LT

Duration : 6 hours.

Thursday 7th November 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B17-B20, scheduled 2 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B17-B20

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 07-Nov-2019 0800Z 0900LT

Duration : 60 minutes

Wednesday 6th November 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B9-B17, scheduled 2 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B9-B17

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 06-11-2019 0800Z 0900LT

Duration : 60 minutes

Tuesday 5th November 2019

[PAR1, AMS1] BIOS and kernel upgrade on DEV1 and GP1 hypervisors, scheduled 2 months ago

[PAR1, AMS1] BIOS and kernel upgrade on DEV1 and GP1 hypervisors

Location : PAR1, AMS1

What is done : We are about to upgrade our DEV1 & GP1 hypervisors, so as to:

  • implement kernel upstream changes committed by Scaleway contributors for new upcoming features,
  • implement vendor BIOS upgrade to fix major availability issues and improve performances of our DBaaS product,
  • implement Ubuntu bugfix to the snapshot functionality to improve reliability of snapshots.

Impact : 20 minutes downtime (per hypervisor)

We are canceling this maintenance and will give a new date as soon as possible.

Duration : 7 days on Paris region & 5 days on Amsterdam region

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B3-B9, scheduled 2 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B3-B9

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 05-Nov-2019 0800Z 0900LT

Duration : 60 minutes

Sunday 3rd November 2019

No incidents reported

Saturday 2nd November 2019

Console [CONSOLE] online.net and console.online.net unavailability

Website and console (online.net) are currently unavailable. Our team has already been warned and is currently working on it in order to fix it as quickly as possible.

Our apologies for the inconvenience.

===================

02.11.2019 1215Z (1315LT)

Console and website are back, we will keep monitoring it to make sure that everything is fully resolved

Webhosting issue

We have detected a problem on some hosting that appear in maintenance or error 500. Our teams are working on the resolution.

===================

02/11/19 1300Z (1400LT)

Services are now back online and websites should be reachable again. Do not hesitate to contact our support team if you experience any other issue.

Friday 1st November 2019

Object Storage Registry SSL incident

We are currently facing a SSL certificat issue on Registry Container product.

It is not possible to pull images from registry product.

On Kapsule product, cluster creations/upgrade or scaling are unavailable since 5:14 AM.

Regarding Registry, It is not possible to pull images from registry product from midnight.

Next update by 11.01.19 1600Z (1700LT)

===================

11.01.19 1000Z (1100LT)

Issue has been escalated to local team

11.01.19 1120Z (1220LT)

Issue has been fixed

Thursday 31st October 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/A15-B3, scheduled 2 months ago

We are also upgrading another TOR uplinks from 1G to 4G for racks S301/A15-B3

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 31-Oct-2019 0800Z 0900LT

Duration : 60 minutes

DC3 Oncurrent battery maintenance

[Curative maintenance on battery of UPS]

We will do the curative maintenance of the inverters of electrical chains A and B. The inverters of these electrical chains supply racks of IT halls. Operations will be performed by our maintainer under our permanent supervision. UPS impacted :

ASI A1: 6 batterys

ASI B2 : 1 battery

This maintenance requires isolating each UPS in order to intervene on the device. The downtime of an UPS is estimated to take 30 minutes.

The operation will be done UPS by UPS and will have no impact on your equipments housed in our Datacenter : the others UPS of the electrical chain are sized to recover the load from the offline UPS.

Location : DC3 DATACENTER

Impact : None

Start : 31-Oct-2019 0730Z 0930LT
End : 31-Oct-2019 1530Z 1730LT

Duration : 1 day

Wednesday 30th October 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/A9-A15, scheduled 2 months ago

We are upgrading TOR uplinks from 1G to 4G for racks S301/A9-A15

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 30-Oct-2019 0800Z 0900LT

Duration : 60 minutes

DC5-A Network [DC5-A] Short network outage

We encountering a network downtime for less than a minute on DC5-A today at 04:30PM (Local Time).
The issue has been fixed. Network traffic is now working properly.

===================

10.30.19 1530Z (1630LT)

Issue has been escalated to local team and fixed.

[DC2] Room 103 - Rack A6 - Network issues

We detected an outage on DC2 - Room 103 - Rack A6.
Our network team is working on a fix.
Next update in 2 hours

===================

10.30.2019 1530Z (1630LT)

The switch has been replaced. Everything is back to normal.

10.30.2019 1400Z (1500LT)

The switch is dead. We need to replace it. It might take a while.

10.30.2019 1320Z (1420LT)

Issue has been escalated to local team

Console Licence Windows issue

We have noticed that some Microsoft Windows licenses are marked as terminated when they should not be. We are working on a solution.

===================

10.30.19 0730Z (0830LT)

Issue has been escalated to local team

10.30.19 1015Z (1115LT)

Licenses expirations are now corrected. Don’t hesitate to open a support ticket if you wish a manual verification.

DC5-A Network RPN saturation between DC5-A and other locations

Due to a failure on our primary RPN link between DC5-A and other datacenters, RPN traffic may be saturated from/to this DC as all traffic has been rerouted to our backup link.
We will update this status as soon as we have additional details to share.

===================

10.30.19 0540Z (0640LT)

Issue has been escalated to local team

10.30.19 0740Z (0830LT)

The issue is now fixed.

Tuesday 29th October 2019

Console [Dedibox] Installation issue

We detected an issue with our install system.
Your installation might not go through. We are currently investigating on this matter.
Next update as soon as we have more information.

===================

##### 10.29.19 2000Z (2200LT) Issue has been escalated to local team

30/10/19 0845Z (0945LT)

Issue is now fixed

[SCW] Container Registry unavailable

We are currently encountering an internal issue which impacts our Container Registry service.
The service may not be reachable.

Next update in 2 hours

===================

10.29.19 2030Z (2130LT)

The API issue linked to our backend has been fixed. Everything is back to normal.

10.29.19 1730Z (1830LT)

The issue has been found and it will be fixed ASAP.

10.29.19 1530Z (1630LT)

Issue has been escalated to local team

Monday 28th October 2019

No incidents reported