Some systems are experiencing issues

About This Site

Maintenances and incidents regarding our services, networks and datacenters can be followed here.

Scheduled Maintenance
[MAINTENANCE DC5-A] Racks D14;D15;D16;D17;D18;D19;C1;C2;C3;C4;C7;C8

A scheduled maintenance is planned in DC5-A with a physical displacement of your servers listed above in order to optimize and upgrade our infrastructure. Servers will be unavailable for the duration of the maintenance detailed below.

This status will be updated as the operations are going through to keep you informed about the events.

Start : July 6th, 2020 - 2000UTC (2200LT)

End : July 7th, 2020 - 0300UTC (0500LT)

[MAINTENANCE DC5-A] Racks H1;H2:H3;H4;H5;H6;H7;H8;H9;H10;H11;H12;H13;H14;H15;H16;H17

A scheduled maintenance is planned in DC5-A with a physical displacement of your servers listed above in order to optimize and upgrade our infrastructure. Servers will be unavailable for the duration of the maintenance detailed below.

This status will be updated as the operations are going through to keep you informed about the events.

Start : July 15th, 2020 - 2000UTC (2200LT)

End : July 16th, 2020 - 0300UTC (0500LT)

[DC5] Load test of generators

[DC5] Load tests of generators

As part of the preventive maintenance of the Datacenter, we will proceed to a redundancy test of the Datacenter infrastructure by carrying out generator tests.

The operation consists in simulating a cut of electrical TGBT and validate load recovery as well as function through generators. Running time on the generators will be reduced in view of the low load on the site.

Impact : None expected

Start: 16-Jul-2020 0600Z (0800LT)
End: 16-Jul-2020 1200Z (1400LT)
[DEDIBOX] API and Console session handler migration

[DEDIBOX] API and Console session handler migration

We'll migrate our Console and API session handler.

Console and API will be unavailable during the migration process.

Start : 25-Jun-2020 0300Z 0500LT

Duration : 30 min

[DC3] Semi-annual maintenance of chillers

[DC3] Semi-annual maintenance of chillers

We will do the semi-annual maintenance of the chillers, according to the manufacturer’s maintenance range.
Operations will be performed by the manufacturer and supervised by our maintainer, according to the following previsional schedule:

  • Monday 17/08: Chiller n°1, Substation A
  • Tuesday 18/08: Chiller n°2, Substation A
  • Wednesday 19/08: Chiller n°3, Substation A
  • Thursday 20/08: Chiller n°4, Substation B
  • Friday 21/08: Chiller n°5, Substation B
  • Monday 24/08: Chiller n°6, Substation B
  • Monday 31/08: Chiller n°7, Substation C
  • Tuesday 01/09: Chiller n°8, Substation C

Substations A and B supply the chilled water network of the computer halls.
Substation C supplies the chilled water network of the operators halls and the air handling units.

Impact : None expected

Start: 17-Aug-2020 0630Z (0830LT)
End: 01-Sep-2020 0330Z (0530LT)
[DC3] Load tests of generators and PLC software updates

[DC3] Load tests of generators and PLC software updates

As part of the preventive maintenance of the Datacenter, we will proceed to a redundancy test of the Datacenter infrastructure by carrying out generator tests chronologically on the following electrical chains : A / B / C / D / F1 and F2.

During the load tests on the electric chains A / B / C and D, we will proceed to F1 line up backup test.
Description of operations :

  • 25/08/2020:
    • Morning: Load test on the electric chain A with F1 line up backup test.
    • Afternoon: Load test on the electric chain B with F1 line up backup test.
  • 26/08/2020:
    • Morning: Load test on the electric chain C with F1 line up backup test.
    • Afternoon: Load test on the electric chain D with F1 line up backup test.
  • 27/08/2020: As part of the mobile chillers installation, we will update PLC software on chains F1 and F2. After we will proceed to load test on the chains F1 andF2.

During the F1 line up backup test, we will simulate a failure of the generators in order to validate the F1 lineup backup scenario.Our teams and usual maintenance companies will be mobilized for the perfect workflow.

Impact : None expected

Start: 25-Aug-2020 0600Z (0800LT)
End: 27-Aug-2020 1600Z (1800LT)

Past Incidents

Friday 7th June 2019

Global Backbone [NETWORK] worldstream

We have detected an issue with bandwidth and connectivity between our network and worldstream.

Our network team is investigating for the moment.

===================

07/06/2019 1600Z (1800LT)

WorldStream has changed their optic, issue is considered as fixed but we will keep monitoring the situation.
If you are still experiencing network instabilities through WorldStream, but directly contact our support team about your issue.

Thursday 6th June 2019

No incidents reported

Wednesday 5th June 2019

Compute Nodes [Scaleway] AMS - Provisioning instabilities

We encountered an outage with our provisioning system.
If you are currently trying to create an instance, it might be stuck in 'provisioning state'.
Our team is working on a fix.

===================

05/06/2019 1720Z (1820LT)

The issue has been fixed. You may now create an instance without any problem.

05/06/2019 1650Z (1750LT)

The product team has identified the root cause and working on a fix

05/06/2019 1640Z (1740LT)

Issue has been escalated to the local team

Global Backbone [NETWORK] DC5 >> DC3

We have detected a fiber break on a span between TH2 and DC5.
Span between DC5 and DC3 is still up and traffic still flows through this optical path with 400G capacity.
We are investigating with our dark fiber provider.

This concerns both public and private (RPN) network.

===================

05/06/19 1510Z (1710LT)

Provider's maintenance still ongoing.
We don't have any specific ETA yet, we will keep monitoring the situation.

06/06/19 1000Z (1200LT)

Issue has been resolved

Tuesday 4th June 2019

Console Scaleway console and APIs unavailability

We are currently facing APIs unavailability on both AMS and PAR1.
Issue has been identified, we are working on a fix and will update this status as soon as we have additional details.

Please note that there is no impact on running services.
Issue is only affecting console's management and API calls (startup, stop, etc).

===================

04/06/19 1415Z (1615LT)

Issue is now closed, services are up and stable.
Thank you for your patience.

04/06/19 1400Z (1600LT)

A fix has been applied and deployed.
Services are slowly coming back to life, we are still monitoring the situation.

04/06/19 1345Z (1545LT)

issue has been identified by the product's teams.
We are working on an urgent fix.

Monday 3rd June 2019

DC5 [DC5] DC5 into degraded mode - No local technical support - No physical operations on site

The wind conditions have changed and there are some toxic gaz moving to the DC5.
The local team has been evacuated and we do not have people on site for now.
Services are running. In case of hardware failure on one of the servers, customer will have to migrate services on other servers.
We'll keep this status updated to say when we go back to normal.

UPDATE: DC5 is now back to normal. All operations can now be done and all technicians are on site

DC5 Network Router reboot impacting racks C36-C42 and D65-66

Our team noticed an unexpected reboot on a DC5 router managing trafic on the following racks :

  • C36 to C42
  • D65 and D66

As we currently have no technical team on site, we are trying to identify the root cause from remote access and logs.
Services are already back online, but we are still investigating.

Sunday 2nd June 2019

DC5 [DC5] Fire threat

This morning, a fire has started on the building next to our DC5 datacenter. Authorities are on site and currently fighting against the fire. We are heavily mobilized to monitor the situation and keeping our datacenter safe.

Authorities have warned us about chemical products being stored in this neighbor building. There is an explosion threat for the moment.

We will keep this status updated as soon as we get informations. For the moment, all services are reachable.

Scaleway compute instances are not concerned.

Concerned services: Dedibox Dedicated Servers hosted in DC5 and Scaleway Object Storage in Paris region

UPDATE 1: Fire is still ongoing, authorities are working hard on containing it.
UPDATE 2: There is a slight improvement over the fire. We are still on alert.
UPDATE 3: The fire is now under control.
UPDATE 4: Fire is now extinguished. We are still heavily watching this.
UPDATE 5: Incident is now over. No downtime or damages happened on our datacenter due to this.

Saturday 1st June 2019

No incidents reported