status

Title	Global slowdowns
ID	Operation #17
State	completed
Beginning date	12/14/2011 6:37 p.m.
End date	12/15/2011 10:44 p.m.
Affected servers

Messages

12/14/2011 6:37 p.m.	We’re investigating.
12/14/2011 7:02 p.m.	A network issue is most certainly the cause of these slowdowns. Things are looking better right now, we’re still monitoring the situation.
12/14/2011 7:09 p.m.	Everything is now working as normal. We’re still monitoring.
12/14/2011 11:03 p.m.	Slowdowns are back.
12/15/2011 1:56 a.m.	The issue is due to intermittent packet loss (~3%). It seems resolved for now.
12/15/2011 10:29 a.m.	The intermittent packet loss is back again.
12/15/2011 11:14 a.m.	We’re doing everything we can to resolve this issue.
12/15/2011 11:37 a.m.	No more packet loss for now, but we’re still in contact with our provider to have this issue solved once and for all.
12/15/2011 4:03 p.m.	Packet loss is back again.
12/15/2011 4:20 p.m.	We’re still in contact with our provider about the issues. We’re also trying something that may help mitigate the packet loss.
12/15/2011 5:05 p.m.	No more packet loss for now. Our provider has trouble identifying the issue; in the meantime, we’ll deploy something new by the end of the day. We hope it will solve 90% of the problems. Sorry again for all the trouble, we realize how painful it is, especially in this Christmas period.
12/15/2011 6:31 p.m.	Several servers will be rebooted in order to deploy the fix we discussed earlier. Expected downtime is a few minutes - usually less than 2.
12/15/2011 8:47 p.m.	The packet loss is back, we’re started rebooting a few servers. The fix is not deployed yet.
12/15/2011 9:04 p.m.	Good news: we’ve started deploying the fix, and it works well. The issue is NOT resolved yet (slowdowns are still here), but it should hopefully be in the next couple of hours.
12/15/2011 9:28 p.m.	The fix has been deployed on most servers.
12/15/2011 10:42 p.m.	All HTTP servers have the fix applied, all websites are now responsive as usual.

Messages