Tuesday 30th July 2019

[Paris core router bb1.dc3] Router upgrade, scheduled 2 months ago

Location : Paris - core router bb1.dc3

We are planning to upgrade the router firmware to the latest recommended by Cisco. The maintenance will be done in 2 steps to prepare the upgrade and then to apply.

Impact : This might lead to rerouting / latencies during the maintenance window. All Paris DCs are impacted.

Preparation : 30-Jul-2019 0400Z (0600LT) - duration 2h
Full upgrade : 31-Jul-2019 0100Z (0300LT) - duration 6h

UPDATE: Maintenance was planned for the 23th and 24th, due to heatwave, we are changing the dates for the 30th and 31st.

Start : 30-Jul-2019 0400Z 0600LT

We are now starting the preparations. bb1.dc3 will now be isolated from our network.

Start : 30-Jul-2019 0600Z 0800LT

We are now done for today, our tests were good and we are ready for tomorrow's firmware update.

Start : 31-Jul-2019 0100Z 0300LT

We have started the upgrade process

Start : 31-Jul-2019 0100Z 0400LT

We have lost some incoming traffic towards AMS datacenter (a9k-pni-1.ams1) for around 10min. Issue has been fixed.

Start : 31-Jul-2019 0345Z 0545LT

We noticed DHCP issues (Dedibox only) in AMS datacenter. It is now fixed

Start : 31-Jul-2019 0515Z 0715LT

Half of our outgoing traffic NL-IX was dropped during about 35min. This is now fixed

Start : 31-Jul-2019 0645Z 0845LT

Upgrade is now done