Status

One of the disks is showing errors. We will have to restart the server. Expected downtime: 5 minutes.

The server is being rebooted.

We’re seeing weird disk issues.

It looks like we have a bad disk corruption issue. Still investigating.

The data are most certainly unaccessible and we will have to load the backups. More details will follow later.

It will most certainly take several hours to recover from the backups. We’re very sorry about this.

We’re still working on restoring data.

Several database files seem to be readable and healthy. We will try to recover as much as we can from the old disk (with up-to-date data).

What cannot be recovered this way will have to be restored from the backups. We’re still several hours away from having a working mysql1 server.

It looks like a large proportion of databases could be recovered without any loss. We’re still not sure at this point.

We hope to have the first databases recovered and accessible in 2 hours. Loading all databases will take a few more hours.

We’ve started to import recovered databases. We don’t have an ETA yet; at least a couple of hours.

The import process is still ongoing and will unfortunately take more time than we initially hoped.

Note that all customers hosting databases on mysql1 can ask us for a full refund for this month, as specified in our terms of services.

We would like to apologize again for this very long service interruption.

We’re working on speeding up the import process and evaluating our options.

About 30% of all databases have been recovered and are now accessible.

About 80% of all databases have now been recovered.

99% of non-corrupted databases have been restored. We’re now restoring the corrupted databases from the last backups.

Still working to restore corrupted databases. We manually check each of them.

All databases have now been imported. If something is still wrong with your database, please open a support ticket and we’ll help you.

More than 96% of the databases on mysql1 didn’t suffer any corruption, only an extended outage. The remaining 4% were partially or totally corrupted and had to be restored from the backups.

We’re very sorry about this very serious issue. We’ll write a blog post to explain in details what happened.

Note that among the 4%, a few databases may STILL be partially corrupted. For those, there are basically 2 options: either recover from a backup, or try to fix the current database with our help. Contact us if you fall in this category.