Check our services status at a glance
|Title||mysql1 data corruption|
|Beginning date||07/16/2013 11:51 p.m.|
|End date||07/17/2013 2:11 p.m.|
|07/16/2013 11:13 p.m.||
One of the disks is showing errors. We will have to restart the server. Expected downtime: 5 minutes.
|07/16/2013 11:52 p.m.||
The server is being rebooted.
|07/16/2013 11:56 p.m.||
We're seeing weird disk issues.
|07/17/2013 12:02 a.m.||
It looks like we have a bad disk corruption issue. Still investigating.
|07/17/2013 12:13 a.m.||
The data are most certainly unaccessible and we will have to load the backups. More details will follow later.
|07/17/2013 12:20 a.m.||
It will most certainly take several hours to recover from the backups. We're very sorry about this.
|07/17/2013 1:49 a.m.||
We're still working on restoring data.
|07/17/2013 3:17 a.m.||
Several database files seem to be readable and healthy. We will try to recover as much as we can from the old disk (with up-to-date data).
What cannot be recovered this way will have to be restored from the backups. We're still several hours away from having a working mysql1 server.
|07/17/2013 4:51 a.m.||
It looks like a large proportion of databases could be recovered without any loss. We're still not sure at this point.
|07/17/2013 5:26 a.m.||
We hope to have the first databases recovered and accessible in 2 hours. Loading all databases will take a few more hours.
|07/17/2013 7:07 a.m.||
We've started to import recovered databases. We don't have an ETA yet; at least a couple of hours.
|07/17/2013 8:03 a.m.||
The import process is still ongoing and will unfortunately take more time than we initially hoped.
Note that all customers hosting databases on mysql1 can ask us for a full refund for this month, as specified in our terms of services.
We would like to apologize again for this very long service interruption.
|07/17/2013 8:39 a.m.||
We're working on speeding up the import process and evaluating our options.
|07/17/2013 8:59 a.m.||
About 30% of all databases have been recovered and are now accessible.
|07/17/2013 10:35 a.m.||
About 80% of all databases have now been recovered.
|07/17/2013 11:16 a.m.||
99% of non-corrupted databases have been restored. We're now restoring the corrupted databases from the last backups.
|07/17/2013 12:46 p.m.||
Still working to restore corrupted databases. We manually check each of them.
|07/17/2013 2:11 p.m.||
All databases have now been imported. If something is still wrong with your database, please open a support ticket and we'll help you.
More than 96% of the databases on mysql1 didn't suffer any corruption, only an extended outage. The remaining 4% were partially or totally corrupted and had to be restored from the backups.
We're very sorry about this very serious issue. We'll write a blog post to explain in details what happened.
|07/17/2013 2:34 p.m.||
Note that among the 4%, a few databases may STILL be partially corrupted. For those, there are basically 2 options: either recover from a backup, or try to fix the current database with our help. Contact us if you fall in this category.