Status

Check our services status at a glance

Title http7 is down
ID Operation #69
State completed
Beginning date 11/04/2012 4:38 a.m.
End date 11/04/2012 6:54 a.m.
Affected servers
  • http7

Messages

11/04/2012 5:02 a.m.

Server down, we are investigating.

11/04/2012 5:05 a.m.

XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1503 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff8125fb28

11/04/2012 5:11 a.m.

Filesystem recovery ongoing.

11/04/2012 5:18 a.m.

Still recovering. We have no ETA yet.

11/04/2012 5:41 a.m.

The recovery mount failed again with the same error. We’re now running xfs_repair -L.

11/04/2012 5:57 a.m.

Still repairing (phase 3, agno = 21).

11/04/2012 6:10 a.m.

xfs_repair has finished successfully. We’re now mounting the filesystem again; it’s checking quotas. It will probably take a few more minutes.

11/04/2012 6:26 a.m.

Still calculating quotas.

11/04/2012 6:39 a.m.

The quota calculation has frozen. We’re rebooting the server and will try to mount the filesystem with quota disabled.

11/04/2012 6:49 a.m.

The filesystem has mounted successfully. Since the filesystem log had to be zeroed, the latest changes may have been lost, although that will probably impact few files.

We’re obviously monitoring the situation very closely.

The root cause of this corruption is, again, these disk issues we’re trying to resolve. We’ve started replacing some Seagate disks by Hitachi disks, and we will continue over the next few days. Hopefully that will solve all these issues.

All impacted customers can ask for a full refund for this month. We apologize for this situation.