Opération #317

Titre /home is unavailable
État terminé
Date de début 16 sep. 2015 12:29
Date de fin 16 sep. 2015 12:50
Serveurs concernés


16 sep. 2015 12:29

We're investigating.

16 sep. 2015 12:39

The issue is fixed. We're still investigating what happened.

16 sep. 2015 13:53

Post-mortem: we have an internal tool that regularly checks that the permissions of a predefined list of system files and directories are correctly set and if not, changes them.

Earlier today, we added a new rule to that tool on one of our internal servers. That rule says that /home should be set to 751. That specific server, which hosts our administration panel, must be able to access all accounts, on both dedicated and shared servers.

The directory for that rule should have been set to '/home' in order to change the permissions on that directory only. Instead, we made a mistake and typed '/home/' (with a final slash). Ending the directory with a slash means that we want the tool to apply the permissions recursively, i.e. including all subdirectories.

The tool was automatically run at 12:17 and started changing permissions on user home directories at 12:25, in random order. It ran for several minutes until we noticed the error. At 12:37, all user directories had been manually reset to their original permissions.

During that 12 minutes window, a little less than half of accounts had wrong permissions. Those permissions prevented user applications from opening files under /home, which resulted in websites showing errors such as "Forbidden". Not all websites were impacted though, mostly those using PHP (which reopens the application files on each new request).

Other services (emails, databases) were not affected.

We apologize for that downtime impacting an exceptionally large number of our clients.