Statuts des serveurs
Vérifiez l'état de nos services d'un seul coup d'œil
- ID :
- 116
- Titre :
- mysql1 data corruption
- Statut :
- terminé
- Date de début :
- 16 juil. 2013 23:51
- Date de fin :
- 17 juil. 2013 14:11
- Serveurs concernés :
- mysql1
Mises à jour
23:13
One of the disks is showing errors. We will have to restart the server. Expected downtime: 5 minutes.
23:52
The server is being rebooted.
23:56
We’re seeing weird disk issues.
0:02
It looks like we have a bad disk corruption issue. Still investigating.
0:13
The data are most certainly unaccessible and we will have to load the backups. More details will follow later.
0:20
It will most certainly take several hours to recover from the backups. We’re very sorry about this.
1:49
We’re still working on restoring data.
3:17
Several database files seem to be readable and healthy. We will try to recover as much as we can from the old disk (with up-to-date data).
What cannot be recovered this way will have to be restored from the backups. We’re still several hours away from having a working mysql1 server.
4:51
It looks like a large proportion of databases could be recovered without any loss. We’re still not sure at this point.
5:26
We hope to have the first databases recovered and accessible in 2 hours. Loading all databases will take a few more hours.
7:07
We’ve started to import recovered databases. We don’t have an ETA yet; at least a couple of hours.
8:03
The import process is still ongoing and will unfortunately take more time than we initially hoped.
Note that all customers hosting databases on mysql1 can ask us for a full refund for this month, as specified in our terms of services.
We would like to apologize again for this very long service interruption.
8:39
We’re working on speeding up the import process and evaluating our options.
8:59
About 30% of all databases have been recovered and are now accessible.
10:35
About 80% of all databases have now been recovered.
11:16
99% of non-corrupted databases have been restored. We’re now restoring the corrupted databases from the last backups.
12:46
Still working to restore corrupted databases. We manually check each of them.
14:11
All databases have now been imported. If something is still wrong with your database, please open a support ticket and we’ll help you.
More than 96% of the databases on mysql1 didn’t suffer any corruption, only an extended outage. The remaining 4% were partially or totally corrupted and had to be restored from the backups.
We’re very sorry about this very serious issue. We’ll write a blog post to explain in details what happened.
14:34
Note that among the 4%, a few databases may STILL be partially corrupted. For those, there are basically 2 options: either recover from a backup, or try to fix the current database with our help. Contact us if you fall in this category.