Help

Status

Check our services status at a glance

Title Global slowdowns
ID Operation #17
State completed
Beginning date 12/14/2011 6:37 p.m.
End date 12/15/2011 10:44 p.m.
Concerned servers

Messages

12/14/2011 6:37 p.m.

We're investigating.

12/14/2011 7:02 p.m.

A network issue is most certainly the cause of these slowdowns. Things are looking better right now, we're still monitoring the situation.

12/14/2011 7:09 p.m.

Everything is now working as normal. We're still monitoring.

12/14/2011 11:03 p.m.

Slowdowns are back.

12/15/2011 1:56 a.m.

The issue is due to intermittent packet loss (~3%). It seems resolved for now.

12/15/2011 10:29 a.m.

The intermittent packet loss is back again.

12/15/2011 11:14 a.m.

We're doing everything we can to resolve this issue.

12/15/2011 11:37 a.m.

No more packet loss for now, but we're still in contact with our provider to have this issue solved once and for all.

12/15/2011 4:03 p.m.

Packet loss is back again.

12/15/2011 4:20 p.m.

We're still in contact with our provider about the issues. We're also trying something that may help mitigate the packet loss.

12/15/2011 5:05 p.m.

No more packet loss for now. Our provider has trouble identifying the issue; in the meantime, we'll deploy something new by the end of the day. We hope it will solve 90% of the problems.

Sorry again for all the trouble, we realize how painful it is, especially in this Christmas period.

12/15/2011 6:31 p.m.

Several servers will be rebooted in order to deploy the fix we discussed earlier. Expected downtime is a few minutes - usually less than 2.

12/15/2011 8:47 p.m.

The packet loss is back, we're started rebooting a few servers. The fix is not deployed yet.

12/15/2011 9:04 p.m.

Good news: we've started deploying the fix, and it works well. The issue is NOT resolved yet (slowdowns are still here), but it should hopefully be in the next couple of hours.

12/15/2011 9:28 p.m.

The fix has been deployed on most servers.

12/15/2011 10:42 p.m.

All HTTP servers have the fix applied, all websites are now responsive as usual.