Status

Check our services status at a glance

Title Incoming mails delayed
ID Operation #109
State completed
Beginning date 06/13/2013 10:49 a.m.
End date 06/13/2013 11:19 a.m.
Affected servers
  • smtpin1

Messages

06/13/2013 10:49 a.m.

We’re investigating.

06/13/2013 11 a.m.

We’ve found the issue, it should be resolved in a few minutes.

06/13/2013 11:14 a.m.

Issue resolved. More details will follow shortly.

06/13/2013 11:38 a.m.

All emails delayed will be received later, as the sender hosts will retry to deliver their messages.

Here’s what happened: our main database server has been automatically protected by a DDoS protection device by our provider due to a small scale attack. This mitigation device unexpectedly blocked the PostgreSQL traffic between some of our servers. Unable to contact the main database, our mail servers refused all incoming mails.

We have a hot standby database server, in case the main database server is unreachable. Unfortunately, due to a misconfiguration in our firewalls, that server was not accessible. This was fixed.

The traffic from our MX servers to our main database server has been moved inside our internal VLAN to avoid issues with the mitigation device. We’re also working on radical changes to avoid such issues from our provider to happen again. We’ll communicate on this in a few weeks.