Statuts des serveurs

Vérifiez l'état de nos services d'un seul coup d'œil

ID :
116
Titre :
mysql1 data corruption
Statut :
terminé
Date de début :
16 juil. 2013 23:51
Date de fin :
17 juil. 2013 14:11
Serveurs concernés :
mysql1

Mises à jour

16 juil. 2013
23:13

One of the disks is showing errors. We will have to restart the server. Expected downtime: 5 minutes.

16 juil. 2013
23:52

The server is being rebooted.

16 juil. 2013
23:56

We’re seeing weird disk issues.

17 juil. 2013
0:02

It looks like we have a bad disk corruption issue. Still investigating.

17 juil. 2013
0:13

The data are most certainly unaccessible and we will have to load the backups. More details will follow later.

17 juil. 2013
0:20

It will most certainly take several hours to recover from the backups. We’re very sorry about this.

17 juil. 2013
1:49

We’re still working on restoring data.

17 juil. 2013
3:17

Several database files seem to be readable and healthy. We will try to recover as much as we can from the old disk (with up-to-date data).

What cannot be recovered this way will have to be restored from the backups. We’re still several hours away from having a working mysql1 server.

17 juil. 2013
4:51

It looks like a large proportion of databases could be recovered without any loss. We’re still not sure at this point.

17 juil. 2013
5:26

We hope to have the first databases recovered and accessible in 2 hours. Loading all databases will take a few more hours.

17 juil. 2013
7:07

We’ve started to import recovered databases. We don’t have an ETA yet; at least a couple of hours.

17 juil. 2013
8:03

The import process is still ongoing and will unfortunately take more time than we initially hoped.

Note that all customers hosting databases on mysql1 can ask us for a full refund for this month, as specified in our terms of services.

We would like to apologize again for this very long service interruption.

17 juil. 2013
8:39

We’re working on speeding up the import process and evaluating our options.

17 juil. 2013
8:59

About 30% of all databases have been recovered and are now accessible.

17 juil. 2013
10:35

About 80% of all databases have now been recovered.

17 juil. 2013
11:16

99% of non-corrupted databases have been restored. We’re now restoring the corrupted databases from the last backups.

17 juil. 2013
12:46

Still working to restore corrupted databases. We manually check each of them.

17 juil. 2013
14:11

All databases have now been imported. If something is still wrong with your database, please open a support ticket and we’ll help you.

More than 96% of the databases on mysql1 didn’t suffer any corruption, only an extended outage. The remaining 4% were partially or totally corrupted and had to be restored from the backups.

We’re very sorry about this very serious issue. We’ll write a blog post to explain in details what happened.

17 juil. 2013
14:34

Note that among the 4%, a few databases may STILL be partially corrupted. For those, there are basically 2 options: either recover from a backup, or try to fix the current database with our help. Contact us if you fall in this category.