Some systems are experiencing issues

About This Site

Maintenances and incidents regarding our services, networks and datacenters can be followed here.

Scheduled Maintenance
[Amsterdam core router bb2.ams1] Router upgrade

Location : Amsterdam - Core router bb2.ams1
What is done : Upgrade Software and Reload bb2.ams1 to fix FlowSpec issues. Rollback : rollback to previous release
Expected Impacts : None, traffic should be rerouted elsewhere.

Preparation - duration 2h

-start time : 26-Feb-2020 0500Z (0600LT)
-end time : 26-Feb-2020 0700Z (0800LT)

Full upgrade - duration 4H

-start time : 27-Feb-2020 0300Z (0400LT)
-end time : 27-Feb-2020 0700Z (0800LT)

[DC3] Scheduled maintenance in Room 3-4 / Racks : C1>C22 - D1>D21

A scheduled maintenance is planned in part of the room 3-4, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C22 - D1>D21

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 3rd, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 205 / Racks : C1>C8 - D1>D8 - E1>E7

A scheduled maintenance is planned in part of the room 205, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C8 - D1>D8 - E1>E7

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 3rd, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC3] Scheduled maintenance in Room 4-6 / Racks : A1>A23 - B1>B23

A scheduled maintenance is planned in part of the room 4-6, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : A1>A23 - B1>B23

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 4th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

UPDATE:

2/12/2020 - 0410Z (0510LT)

Racks A1>A23 are upgraded and functional

[DC3] Scheduled maintenance in Room 4-6 / Racks : C1>C22 - D1>D22

A scheduled maintenance is planned in part of the room 4-6, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : C1>C22 - D1>D22

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 4th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 101 / Racks : A20>A28 - B20>B26 - L31>L37 - K37

A scheduled maintenance is planned in part of the room 101, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : A20>A28 - B20>B26 - L31>L37 - K37

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 5th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 203A / Racks : M1>M14 - N1>N6

A scheduled maintenance is planned in part of the room 203A, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : M1>M14 - N1>N6

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 5th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC2] Scheduled maintenance in Room 205 / Racks : A1>A8 - B1>B8 - G2>G8 - H2

A scheduled maintenance is planned in part of the room 205, DC2, in order to optimize and upgrade our infrastructure. The impacted racks are : A1>A8 - B1>B8 - G2>G8 - H2

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 10th, 2020 - 0400Z (0500LT)

Duration : 90 minutes

[DC3] Scheduled maintenance in Room 3-5 / Racks : E1>E21 - F1>F21

A scheduled maintenance is planned in part of the room 3-5, DC3, in order to optimize and upgrade our infrastructure. The impacted racks are : E1>E21 - F1>F21

As a result servers will lose their public network connectivity as our router is being upgraded, but private network connectivity will still be functional. Public network will be back online once the update is complete.

If your server is in the said racks, you will receive a ticket on your account.

Start : March 10th, 2020 - 0530Z (0630LT)

Duration : 90 minutes

[DC3] PLC and HMI software update on chains F1 and F2for mobile chillers power supply supervision

[DC3] PLC and HMI software update on chains F1 and F2for mobile chillers power supply supervision

As part of the mobile chillers installation work, we will update PLC and HMI softwares on chains F1 and F2. The modifications affect the supervision of mobile chillers breakers (display, alarms).

All the changes that will be made will only be one electrical chain at a time. Under no circumstances will two electrical chains be modified simultaneously.

Impact : none for customers

Start : 12-Mar-2020 0700Z 0800LT

Duration : 1 day

[DC3] Preventive maintenance of CRAC units

We will carry out the semi-annual preventive maintenance of CRAC units according to the schedule below. CRAC units will be shut down one at a time, so as not to impact the cooling of halls. Each IT zone has CRAC unit called « backup » : each CRAC unit is doubled. Each MVLB hall has a main air hanling unit (AHU).

The maintenance will be done according to the following schedule :

  • MLVBs 27/03/2020 and 30/03/2020
  • Operators halls : 20/03/2020
  • IT halls n°4 : 16 and 17/03/2020
  • IT halls n°3 : 18 and 19/03/2020
  • IT halls n°2 : 23 to 25/03/2020
  • IT halls n°1 : 25 and 26/03/2020

Impacts on services : Loss of cooling redundancy during the preventive maintenance operation of a CRAC unit. No impact on the cooling of facilities. The maximum cut-off time per CRAC unit is otherwise defined at 1 hour.

[DC3] Maintenance of electric generators

Location : DC3

What is done : We’re going to realize the preventive electrical maintenance of generators, the maintenance will be performed electrical chain by electrical chain :

Chain A : 23/03/30
Chain B : 24/03
Chain C : 25/03
Chain D : 26/03
Chain F1 : 30/03
Chain F2 : 31/03

Start : 23-Mars-2020 0730Z (0830LT) End : 31-Mars-2020 0430Z(0530LT)

Impact : Degradation of electrical redundancy Duration 6 days

[DC3] Pressurization maintenance

[DC3] Pressurization maintenance

We will realized a pressurization maintenance on substation SSA, SSB and SSC

Impact : none

Start : 31-Mar-2020 0830Z 0930LT
End : 2-Apr-2020 1600Z 1700LT

Duration : 3 days

Past Incidents

Tuesday 12th November 2019

DC3 - Preventive maintenance of air-handling units, scheduled 3 months ago

DC3 - Preventive maintenance of air handling units

Location : DC3

What is done : preventive maintenance of air handling units will be performed on the whole site.

Thursday November 12th: Room 01 and 02

Wednesday November 13th: Room 03 and 04

Thursday November 14th: TGBT A and B

Friday November 15th: TGBT C and D

Monday November 18th: TGBT F1 and F2

Impact : loss of redundancy during the preventive maintenance of a TGBT air handling unit. No impact on TGBT cooling.

Start : 12-Nov-2019 0700Z 0900LT

Duration : 5 Days

Wednesday 6th November 2019

DC3 - Semi-annual maintenance of distribution pumps, scheduled 3 months ago

DC3 - Maintenance of distribution pumps

Location : DC3 Substation-A, Substation-B, Substation-C

What is done : We will perform the semi-annual maintenance of the distribution pumps. Operations will be performed by our maintainer under our permanent supervision, according to the following previsional schedule:

Wednesday November 6th: Sub-station A

Thursday November 7th Sub-station B

Friday November 8th Sub-station C

Substations A and B supply the chilled water network of the computer halls. Substation C supplies the chilled water network of the operators halls and the air handing units.

Impact : no impact for customers

Start : 6-Nov-2019 0700Z 0900LT

Duration : 3 Days

Thursday 10th October 2019

Refrigeration units preventive maintenance, scheduled 4 months ago

We will undertake a preventive maintenance on our refrigeration units on DC3.

Impact: Minor loss of redundancy, no customer impact.

START: 10-Oct-2019 0630Z 0830LT

END: 11-Nov-2019 0330Z 0530LT

Wednesday 9th October 2019

DC3 - Redundancy test of DC3 electrical generators, scheduled 4 months ago

DC3 electrical generators - Redundancy test of DC3 electrical generators

Location : DC3 What is done : As part of the preventive maintenance of the Datacenter, we will proceed to a redundancy test of the Datacenter infrastructure. The operation consists of simulating a power failure on the MVLBs and validating the electrical load recovery and working on generator for 4 hours. Impact : no impact on client servers

Start : 9-Oct-2019 0700-1400Z 900-1600LT

Duration : 4 hours

Thursday 12th September 2019

Console [FAILOVER] Attribution of Failover IP stuck in update state.

We encountered issue with attibution of Failover IP to servers using console. They stuck and remain in update state (orange)

Next update by 12.09.19 0800Z (1000LT)

===================

11.09.19 2230Z (12.09.2019 0030LT)*

Issue has been escalated to local team

===================

12.09.19 0830Z (12.09.2019 1030LT)*

The issue has been fixed.

Wednesday 11th September 2019

Public AMS Network Update, scheduled 5 months ago

In order to continuously improve performance, we will be updating part of our public network in Amsterdam.

Operation completed successfully.

Impact: Server reinstallation can be slowed down

START: 11-Sept-2019 0700Z 0900LT

END: 11-Sept-2019 0900Z 1100LT Operation completed successfully.

Tuesday 10th September 2019

Mail Mail Connection Issue

Our teams identified an issue regarding webmail connection.

We are currently investigating the issue and will update this status as soon as we have more details to share.

===================

10/09/19 1210 (1410LT)

Issue has been escalated to local team

10/09/19 1310 (1510LT)

The issue has been fixed. If you still experience webmail connection difficulties please contact support via ticket.

Monday 9th September 2019

No incidents reported

Sunday 8th September 2019

DC5 Network Performance issues on one of our links

Performance issues were detected on one of DC5's links starting at 4pm utc on the 8th of September. You may have experienced latency or packet loss.

The issue is being mitigated by our network team since 8:30am utc today, the situation is now being monitored.

Saturday 7th September 2019

ODS ODS Instance unavailability

We have detected issues with several ODS Instances.

Our teams are currently working on solving the problem.

Impact: Some ODS Instances may be unavailable.

===================

09.09.19 - 1010Z (1210LT)

The issue has been identified and fixed.

09.07.19 - 0840Z (1040LT)

Issue has been escalated to local team

Friday 6th September 2019

No incidents reported

Thursday 5th September 2019

AMS1 [AMS] Room Hall 2 E-E Rack E14 - Power delivery issues

A rack in AMS1 is having power delivery issues, all servers in the rack are unreachable.

===================

09.05.19 1620Z (1820LT)

Issue has been escalated to local team

09.05.19 1815 (2015LT)

The product team has identified the root cause and working on a fix

09.05.19 1830 (2030LT)

Incident has been solved and servers are coming back up progressively.

Mail [HOSTING] Mail Connexion Issue

Our teams identified an issue regarding webmail connection. We are currently investigating the issue and will update this status as soon as we have more details to share.

===================

05/09/19 1230 (1030LT)

The product team has identified the root cause and working on a fix

05/09/19 1730 (1530LT)

Issue has been fixed now, if you are still experiencing connection issues please contact our support team via ticket.

Monday 2nd September 2019

AMS1/DC3 Public network, scheduled 5 months ago

Preventive maintenance over the network between AMS1 and DC3

Network supplier must perform maintenance on his Network.

Maintenance window is from September 7th, 2019 at 07:00 am UTC to September 7th, 2019 at 7:00 pm UTC.

The expected Impact length is 12 hours at the maximum, during the maintenance window

Impact : Possible downtime, flaps or degraded bandwidth on the network between AMS1 and DC3