Statut

We’re investigating.

The issue is fixed. We’re still investigating what happened.

Post-mortem: we have an internal tool that regularly checks that the permissions of a predefined list of system files and directories are correctly set and if not, changes them.

Earlier today, we added a new rule to that tool on one of our internal servers. That rule says that /home should be set to 751. That specific server, which hosts our administration panel, must be able to access all accounts, on both dedicated and shared servers.

The directory for that rule should have been set to ‘/home’ in order to change the permissions on that directory only. Instead, we made a mistake and typed ‘/home/’ (with a final slash). Ending the directory with a slash means that we want the tool to apply the permissions recursively, i.e. including all subdirectories.

The tool was automatically run at 12:17 and started changing permissions on user home directories at 12:25, in random order. It ran for several minutes until we noticed the error. At 12:37, all user directories had been manually reset to their original permissions.

During that 12 minutes window, a little less than half of accounts had wrong permissions. Those permissions prevented user applications from opening files under /home, which resulted in websites showing errors such as “Forbidden”. Not all websites were impacted though, mostly those using PHP (which reopens the application files on each new request).

Other services (emails, databases) were not affected.

We apologize for that downtime impacting an exceptionally large number of our clients.