Opération #17

Titre Global slowdowns
État terminé
Date de début 14 déc. 2011 18:37
Date de fin 15 déc. 2011 22:44
Serveurs concernés


14 déc. 2011 18:37

We're investigating.

14 déc. 2011 19:02

A network issue is most certainly the cause of these slowdowns. Things are looking better right now, we're still monitoring the situation.

14 déc. 2011 19:09

Everything is now working as normal. We're still monitoring.

14 déc. 2011 23:03

Slowdowns are back.

15 déc. 2011 01:56

The issue is due to intermittent packet loss (~3%). It seems resolved for now.

15 déc. 2011 10:29

The intermittent packet loss is back again.

15 déc. 2011 11:14

We're doing everything we can to resolve this issue.

15 déc. 2011 11:37

No more packet loss for now, but we're still in contact with our provider to have this issue solved once and for all.

15 déc. 2011 16:03

Packet loss is back again.

15 déc. 2011 16:20

We're still in contact with our provider about the issues. We're also trying something that may help mitigate the packet loss.

15 déc. 2011 17:05

No more packet loss for now. Our provider has trouble identifying the issue; in the meantime, we'll deploy something new by the end of the day. We hope it will solve 90% of the problems.

Sorry again for all the trouble, we realize how painful it is, especially in this Christmas period.

15 déc. 2011 18:31

Several servers will be rebooted in order to deploy the fix we discussed earlier. Expected downtime is a few minutes - usually less than 2.

15 déc. 2011 20:47

The packet loss is back, we're started rebooting a few servers. The fix is not deployed yet.

15 déc. 2011 21:04

Good news: we've started deploying the fix, and it works well. The issue is NOT resolved yet (slowdowns are still here), but it should hopefully be in the next couple of hours.

15 déc. 2011 21:28

The fix has been deployed on most servers.

15 déc. 2011 22:42

All HTTP servers have the fix applied, all websites are now responsive as usual.