Monday 22nd July 2019

Preventive maintenance due to current heatwave, scheduled 1 year ago

On 24th & 25th July we will reach record highs, with 42°C being the highest recorded temperature in the last 20 years.
This will be a challenge for our equipment and teams, but everyone is mobilized, and exceptional means have been put in place to keep everything up and running.

This status has been created preemptively and will be updated on July 24th as needed.

Risk evaluation per DC :
DC2 High
DC3 Very High
DC4 No risk
DC5 Low risk

Start : 26-Jul-2019 1015Z 1215LT

Situation is back to normal, status is now closed.
A blogpost is currently being drafted and will be published in the upcoming days.

Start : 25-Jul-2019 2045Z 2245LT

We are almost out of the crisis situation as temperature is slowly coming back to "normal" values (still more than 30°C).
Our duty team will keep monitoring situation for the night.
Expect a final update by tomorrow, and a more thoughtful statement in the upcoming days.

Start : 25-Jul-2019 1630Z 1830LT

Equipments in DC3 are still running at their maximum level regarding the temperature without any impact.
Temperature have been at its peak for 3 consecutive hours.
According to weather forecasts, we expect a sensible temperature decline within the next 45 minutes.

Start : 25-Jul-2019 1515Z 1715LT

DC3 :
One of our refrigeration equipment (GF3) did not resist the extreme temperatures and its performance were degraded.
We have manually switched to a rescue one (GF2).
No further consequence were observed, but we will keep monitoring it.

Start : 25-Jul-2019 1440Z 1640LT

All services and equipments are still being closely monitored.
Temperatures are still very high and challenging our equipments, but no further alert as of now.

Start : 25-Jul-2019 1300Z 1500LT

We are reaching new record high right now, temperatures have peaked and we expect the situation to stay at this level for approx. 2 hours.
Our teams are fully mobilized, both on-site and remotely to monitor, communicate and act as needed.
On site, we are regularly readjusting air conditioning settings and spare equipments have been started to help sustaining the load.

Start : 25-Jul-2019 1155Z 1355LT

Outdoor temperature has reached 42°C. We are now going through the temperature peak. All of our teams are currently heavily watching the situation. All of our services are running correctly at the moment.

Start : 25-Jul-2019 1120Z 1320LT

Peak temperature is planned f 2PM/CEST. It should reach 42°C outside.

Start : 25-Jul-2019 0845Z 1045LT

The heatwave is still ongoing and we expect 42°C temperature peak for today. Updates will be published on regular basis from 3pmCEST.

Start : 24-Jul-2019 1700Z 1900LT

As a last update for today, we can quite safely announce that today's heat peak did not have any critic impact on our services.
Temperatures are slowly getting lower, but our teams will keep monitoring all sensors and ensure that everything stays under control until tomorrow morning.
As weather forecasts are still anticipating very high temperature peaks for tomorrow, we will keep updating this status from tomorrow morning (CEST).

Start : 24-Jul-2019 1530Z 1730LT

Peak temperatures have been reached, our monitoring is not highlighting any out-of-control value but temperatures are still high.
Our teams are still focused and ready to act if anything happen.

Start : 24-Jul-2019 1500Z 1700LT

We are approaching peak temperatures for the day.
Monitoring has been increased across all our services, there is no cause for concern as of now.
Our teams remain mobilised and ready to act shall there be any issue.

Start : 24-Jul-2019 1420Z 1620LT

No notable evolution of the situation right now. We keep monitoring closely the infrastructure. Stay tunned.

Start : 24-Jul-2019 1330Z 1530LT

The situation remains stable at the moment.
The heatwave is expected to peak around 4.30pm CEST.
Updates will be published every 30 minutes as of 4.00pm CEST.

Start : 23-Jul-2019 1645Z 1845LT

Based on the latest weather data, manufacturer specifications, and our infrastructure simulations we have identified the following high risk periods for DC2 and DC3 on Wednesday and Thursday :
• Wednesday 24/07 from 17:00 to 21:00 CET
• Thursday 25/07 from 14:00 to 17:00 CET
Rest assured we are deploying all available countermeasures to keep the situation under control.
We would like to recommend all customers :

  • double check their backup systems, monitoring tools, etc.
  • study/run a disaster recovery plan outside the impacted region.
  • ensure the BMC automatic server shutdown temperature is set to at least 40 ° C (ILO, iDrac, IPMI...).
  • electrically turn off all unused or non-vital services.
  • Due to curent heatwave in PAR, we are switching the RPN-SAN-HA to DC2 site, HA will be restored when temperatures fall

Next update Wednesday at 16:00 CET.

Start : 23-Jul-2019 1300Z 1500LT

Weather forecast have been revised upwards for thursday. As a preventive measure, our teams are mobilized with important technical and human resources on our datacenters.