Status

Check our services status at a glance

Title mysql1 data corruption
ID Operation #116
State completed
Beginning date 07/16/2013 11:51 p.m.
End date 07/17/2013 2:11 p.m.
Concerned servers
  • mysql1

Messages

07/16/2013 11:13 p.m.

One of the disks is showing errors. We will have to restart the server. Expected downtime: 5 minutes.

07/16/2013 11:52 p.m.

The server is being rebooted.

07/16/2013 11:56 p.m.

We're seeing weird disk issues.

07/17/2013 12:02 a.m.

It looks like we have a bad disk corruption issue. Still investigating.

07/17/2013 12:13 a.m.

The data are most certainly unaccessible and we will have to load the backups. More details will follow later.

07/17/2013 12:20 a.m.

It will most certainly take several hours to recover from the backups. We're very sorry about this.

07/17/2013 1:49 a.m.

We're still working on restoring data.

07/17/2013 3:17 a.m.

Several database files seem to be readable and healthy. We will try to recover as much as we can from the old disk (with up-to-date data).

What cannot be recovered this way will have to be restored from the backups. We're still several hours away from having a working mysql1 server.

07/17/2013 4:51 a.m.

It looks like a large proportion of databases could be recovered without any loss. We're still not sure at this point.

07/17/2013 5:26 a.m.

We hope to have the first databases recovered and accessible in 2 hours. Loading all databases will take a few more hours.

07/17/2013 7:07 a.m.

We've started to import recovered databases. We don't have an ETA yet; at least a couple of hours.

07/17/2013 8:03 a.m.

The import process is still ongoing and will unfortunately take more time than we initially hoped.

Note that all customers hosting databases on mysql1 can ask us for a full refund for this month, as specified in our terms of services.

We would like to apologize again for this very long service interruption.

07/17/2013 8:39 a.m.

We're working on speeding up the import process and evaluating our options.

07/17/2013 8:59 a.m.

About 30% of all databases have been recovered and are now accessible.

07/17/2013 10:35 a.m.

About 80% of all databases have now been recovered.

07/17/2013 11:16 a.m.

99% of non-corrupted databases have been restored. We're now restoring the corrupted databases from the last backups.

07/17/2013 12:46 p.m.

Still working to restore corrupted databases. We manually check each of them.

07/17/2013 2:11 p.m.

All databases have now been imported. If something is still wrong with your database, please open a support ticket and we'll help you.

More than 96% of the databases on mysql1 didn't suffer any corruption, only an extended outage. The remaining 4% were partially or totally corrupted and had to be restored from the backups.

We're very sorry about this very serious issue. We'll write a blog post to explain in details what happened.

07/17/2013 2:34 p.m.

Note that among the 4%, a few databases may STILL be partially corrupted. For those, there are basically 2 options: either recover from a backup, or try to fix the current database with our help. Contact us if you fall in this category.