Some systems are experiencing issues

About This Site

Maintenances and incidents regarding our services, networks and datacenters can be followed here.

Scheduled Maintenance
Scheduled border leaf maintenance

Location : DC2

We'll replace one of our border leaf by a bigger shelf.
Replace shelf will be isolated and links shifted to new one.

No impact expected as our shelves are redundant.

Start : 5-Aug-2020 0700Z 0900LT

Duration : 180 minutes

[DC3] Semi-annual maintenance of chillers

[DC3] Semi-annual maintenance of chillers

We will do the semi-annual maintenance of the chillers, according to the manufacturer’s maintenance range.
Operations will be performed by the manufacturer and supervised by our maintainer, according to the following previsional schedule:

  • Monday 17/08: Chiller n°1, Substation A
  • Tuesday 18/08: Chiller n°2, Substation A
  • Wednesday 19/08: Chiller n°3, Substation A
  • Thursday 20/08: Chiller n°4, Substation B
  • Friday 21/08: Chiller n°5, Substation B
  • Monday 24/08: Chiller n°6, Substation B
  • Monday 31/08: Chiller n°7, Substation C
  • Tuesday 01/09: Chiller n°8, Substation C

Substations A and B supply the chilled water network of the computer halls.
Substation C supplies the chilled water network of the operators halls and the air handling units.

Impact : None expected

Start: 17-Aug-2020 0630Z (0830LT)
End: 01-Sep-2020 0330Z (0530LT)
[DC3] Load tests of generators and PLC software updates

[DC3] Load tests of generators and PLC software updates

As part of the preventive maintenance of the Datacenter, we will proceed to a redundancy test of the Datacenter infrastructure by carrying out generator tests chronologically on the following electrical chains : A / B / C / D / F1 and F2.

During the load tests on the electric chains A / B / C and D, we will proceed to F1 line up backup test.
Description of operations :

  • 25/08/2020:
    • Morning: Load test on the electric chain A with F1 line up backup test.
    • Afternoon: Load test on the electric chain B with F1 line up backup test.
  • 26/08/2020:
    • Morning: Load test on the electric chain C with F1 line up backup test.
    • Afternoon: Load test on the electric chain D with F1 line up backup test.
  • 27/08/2020: As part of the mobile chillers installation, we will update PLC software on chains F1 and F2. After we will proceed to load test on the chains F1 andF2.

During the F1 line up backup test, we will simulate a failure of the generators in order to validate the F1 lineup backup scenario.Our teams and usual maintenance companies will be mobilized for the perfect workflow.

Impact : None expected

Start: 25-Aug-2020 0600Z (0800LT)
End: 27-Aug-2020 1600Z (1800LT)

Past Incidents

Thursday 21st November 2019

[DC3] Maintenance of HV stations DATA 1, scheduled 8 months ago

[DC3] Maintenance of HV stations DATA 1

The planned preventive maintenance consists of the maintenance of HV cells and electrical protection tests on delivery station “DATA1”. The electric chains will be supplied from generators during this maintenance.

Impact : This intervention does not present any particular risk of service interruption and will be processed during working hours

Start : 21-Nov-2019 0700Z 0800LT

Duration : 10 hours

Monday 18th November 2019

[RPN] DC3 >< AMS1 - SIPARTECH Network update, scheduled 8 months ago

SIPARTECH is doing a maintenance on the network.

Location : DC3/AMS1

What is done : preventive network maintenance

Impact : RPN Service will be unavailable

Start : 18-Nov-2019 2259Z 2359LT

Duration : 6 hours.

Thursday 7th November 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B17-B20, scheduled 8 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B17-B20

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 07-Nov-2019 0800Z 0900LT

Duration : 60 minutes

Wednesday 6th November 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B9-B17, scheduled 8 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B9-B17

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 06-11-2019 0800Z 0900LT

Duration : 60 minutes

Tuesday 5th November 2019

[PAR1, AMS1] BIOS and kernel upgrade on DEV1 and GP1 hypervisors, scheduled 8 months ago

[PAR1, AMS1] BIOS and kernel upgrade on DEV1 and GP1 hypervisors

Location : PAR1, AMS1

What is done : We are about to upgrade our DEV1 & GP1 hypervisors, so as to:

  • implement kernel upstream changes committed by Scaleway contributors for new upcoming features,
  • implement vendor BIOS upgrade to fix major availability issues and improve performances of our DBaaS product,
  • implement Ubuntu bugfix to the snapshot functionality to improve reliability of snapshots.

Impact : 20 minutes downtime (per hypervisor)

We are canceling this maintenance and will give a new date as soon as possible.

Duration : 7 days on Paris region & 5 days on Amsterdam region

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/B3-B9, scheduled 8 months ago

Upgrade TOR uplinks from 1G to 4G for racks S301/B3-B9

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 05-Nov-2019 0800Z 0900LT

Duration : 60 minutes

Sunday 3rd November 2019

No incidents reported

Saturday 2nd November 2019

Console [CONSOLE] online.net and console.online.net unavailability

Website and console (online.net) are currently unavailable. Our team has already been warned and is currently working on it in order to fix it as quickly as possible.

Our apologies for the inconvenience.

===================

02.11.2019 1215Z (1315LT)

Console and website are back, we will keep monitoring it to make sure that everything is fully resolved

Webhosting issue

We have detected a problem on some hosting that appear in maintenance or error 500. Our teams are working on the resolution.

===================

02/11/19 1300Z (1400LT)

Services are now back online and websites should be reachable again. Do not hesitate to contact our support team if you experience any other issue.

Friday 1st November 2019

Object Storage Registry SSL incident

We are currently facing a SSL certificat issue on Registry Container product.

It is not possible to pull images from registry product.

On Kapsule product, cluster creations/upgrade or scaling are unavailable since 5:14 AM.

Regarding Registry, It is not possible to pull images from registry product from midnight.

Next update by 11.01.19 1600Z (1700LT)

===================

11.01.19 1000Z (1100LT)

Issue has been escalated to local team

11.01.19 1120Z (1220LT)

Issue has been fixed

Thursday 31st October 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/A15-B3, scheduled 9 months ago

We are also upgrading another TOR uplinks from 1G to 4G for racks S301/A15-B3

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 31-Oct-2019 0800Z 0900LT

Duration : 60 minutes

DC3 Oncurrent battery maintenance

[Curative maintenance on battery of UPS]

We will do the curative maintenance of the inverters of electrical chains A and B. The inverters of these electrical chains supply racks of IT halls. Operations will be performed by our maintainer under our permanent supervision. UPS impacted :

ASI A1: 6 batterys

ASI B2 : 1 battery

This maintenance requires isolating each UPS in order to intervene on the device. The downtime of an UPS is estimated to take 30 minutes.

The operation will be done UPS by UPS and will have no impact on your equipments housed in our Datacenter : the others UPS of the electrical chain are sized to recover the load from the offline UPS.

Location : DC3 DATACENTER

Impact : None

Start : 31-Oct-2019 0730Z 0930LT
End : 31-Oct-2019 1530Z 1730LT

Duration : 1 day

Wednesday 30th October 2019

[DC2] Upgrade TOR uplinks from 1G to 4G for racks S301/A9-A15, scheduled 9 months ago

We are upgrading TOR uplinks from 1G to 4G for racks S301/A9-A15

Location : DC2

Impact : Short unreachabilities for Public & IPMI networks (< 5min)

Start : 30-Oct-2019 0800Z 0900LT

Duration : 60 minutes

DC5-A Network [DC5-A] Short network outage

We encountering a network downtime for less than a minute on DC5-A today at 04:30PM (Local Time).
The issue has been fixed. Network traffic is now working properly.

===================

10.30.19 1530Z (1630LT)

Issue has been escalated to local team and fixed.

[DC2] Room 103 - Rack A6 - Network issues

We detected an outage on DC2 - Room 103 - Rack A6.
Our network team is working on a fix.
Next update in 2 hours

===================

10.30.2019 1530Z (1630LT)

The switch has been replaced. Everything is back to normal.

10.30.2019 1400Z (1500LT)

The switch is dead. We need to replace it. It might take a while.

10.30.2019 1320Z (1420LT)

Issue has been escalated to local team

Console Licence Windows issue

We have noticed that some Microsoft Windows licenses are marked as terminated when they should not be. We are working on a solution.

===================

10.30.19 0730Z (0830LT)

Issue has been escalated to local team

10.30.19 1015Z (1115LT)

Licenses expirations are now corrected. Don’t hesitate to open a support ticket if you wish a manual verification.

DC5-A Network RPN saturation between DC5-A and other locations

Due to a failure on our primary RPN link between DC5-A and other datacenters, RPN traffic may be saturated from/to this DC as all traffic has been rerouted to our backup link.
We will update this status as soon as we have additional details to share.

===================

10.30.19 0540Z (0640LT)

Issue has been escalated to local team

10.30.19 0740Z (0830LT)

The issue is now fixed.

Tuesday 29th October 2019

Console [Dedibox] Installation issue

We detected an issue with our install system.
Your installation might not go through. We are currently investigating on this matter.
Next update as soon as we have more information.

===================

##### 10.29.19 2000Z (2200LT) Issue has been escalated to local team

30/10/19 0845Z (0945LT)

Issue is now fixed

[SCW] Container Registry unavailable

We are currently encountering an internal issue which impacts our Container Registry service.
The service may not be reachable.

Next update in 2 hours

===================

10.29.19 2030Z (2130LT)

The API issue linked to our backend has been fixed. Everything is back to normal.

10.29.19 1730Z (1830LT)

The issue has been found and it will be fixed ASAP.

10.29.19 1530Z (1630LT)

Issue has been escalated to local team

Monday 28th October 2019

No incidents reported