http://status.alwaysdata.comalwaysdata | statusenMon, 12 Aug 2024 15:28:41 +0000http://status.alwaysdata.com/operation/17/detail/Good news: we've started deploying the fix, and it works well. The issue is NOT resolved yet (slowdowns are still here), but it should hopefully be in the next couple of hours.Thu, 15 Dec 2011 20:04:10 +0000http://status.alwaysdata.com/operation/17/detail/The fix has been deployed on most servers.Thu, 15 Dec 2011 20:28:58 +0000http://status.alwaysdata.com/operation/17/detail/All HTTP servers have the fix applied, all websites are now responsive as usual.Thu, 15 Dec 2011 21:42:06 +0000http://status.alwaysdata.com/operation/18/detail/We're investigating.Fri, 23 Dec 2011 22:24:42 +0000http://status.alwaysdata.com/operation/18/detail/More than 50% packet loss on the main IP of these 2 servers. An administrator is investigating on the routers.Fri, 23 Dec 2011 22:43:21 +0000http://status.alwaysdata.com/operation/18/detail/Our DNS has been updated to temporarily point to different - fully functional - IP addresses.Fri, 23 Dec 2011 23:06:35 +0000http://status.alwaysdata.com/operation/18/detail/Although the issue seems resolved, we're still waiting for more details from our provider. We're still on the temporary IP addresses.Sat, 24 Dec 2011 00:38:27 +0000http://status.alwaysdata.com/operation/18/detail/The faulty router has been isolated, replaced and is being investigated to understand what happened. Our DNS are being updated to point back to the original IP addresses. This issue is a follow-up of the previous one (Global slowdowns) which was lingering for 10 days, even if our fixes suppressed the side effects. We're really sorry for these outages. A full postmortem on both issues will be posted in the next few days on our blog. Conclusions will obviously be drawn from what happened.Sat, 24 Dec 2011 01:13:47 +0000http://status.alwaysdata.com/operation/18/detail/The culprit was a defective optical link.Mon, 02 Jan 2012 19:02:38 +0000http://status.alwaysdata.com/operation/19/detail/http7 suffered a hardware error, causing the filesystem to be remounted read-only. We'll schedule a hardware change on this server. Several *.alwaysdata.com websites were down as well because the NFS link with http7 was frozen. We'll have to investigate this issue to prevent it from happening again.Fri, 20 Jan 2012 15:03:25 +0000http://status.alwaysdata.com/operation/20/detail/http7 disk cables and /dev/sda will be replaced tonight. The server will be unavailable during the operation.Sat, 21 Jan 2012 14:55:58 +0000http://status.alwaysdata.com/operation/20/detail/The operation has started.Sat, 21 Jan 2012 23:03:57 +0000http://status.alwaysdata.com/operation/20/detail/The operation completed successfully.Sat, 21 Jan 2012 23:09:48 +0000http://status.alwaysdata.com/operation/20/detail/The underlying issue (complete disks lockup despite RAID1) is still present. /dev/sdb will be replaced tonight from 00:00 (CET). Expected downtime: less than 10 minutes.Thu, 26 Jan 2012 10:36:49 +0000http://status.alwaysdata.com/operation/20/detail/The operation has started.Thu, 26 Jan 2012 23:07:47 +0000http://status.alwaysdata.com/operation/20/detail/The operation completed successfully. Total downtime: 5 minutes.Thu, 26 Jan 2012 23:11:56 +0000http://status.alwaysdata.com/operation/21/detail/We're investigating.Mon, 12 Mar 2012 08:46:58 +0000http://status.alwaysdata.com/operation/21/detail/The server stopped working for no apparent reason. It's being restarted.Mon, 12 Mar 2012 08:50:39 +0000http://status.alwaysdata.com/operation/22/detail/Some servers (mail, FTP) are partially inaccessible from some networks. We're investigating.Thu, 15 Mar 2012 16:55:57 +0000http://status.alwaysdata.com/operation/22/detail/The routing issue is resolved.Thu, 15 Mar 2012 17:04:49 +0000http://status.alwaysdata.com/operation/23/detail/Software issue, investigation in progress.Tue, 20 Mar 2012 17:21:10 +0000http://status.alwaysdata.com/operation/23/detail/A specifically crafted DoS attack hurt our HTTP frontend. The attacking IP was banned, protections will be added to make sure it won't happen again.Tue, 20 Mar 2012 17:39:13 +0000http://status.alwaysdata.com/operation/24/detail/We're currently having intermittent network issues.Wed, 28 Mar 2012 02:58:46 +0000http://status.alwaysdata.com/operation/24/detail/Our provider is still working on the issue.Wed, 28 Mar 2012 03:40:41 +0000http://status.alwaysdata.com/operation/24/detail/The issue is mostly resolved.Wed, 28 Mar 2012 04:11:56 +0000http://status.alwaysdata.com/operation/24/detail/The issue is solved. An unidentified bug crashed 24x10G cards in 2 core routers simultaneously (which should obviously never happened: those routers are redundant). Cisco will investigate the bug.Wed, 28 Mar 2012 04:51:34 +0000http://status.alwaysdata.com/operation/24/detail/Cisco released a software bug fix. It will be deployed on the core routers during this night.Wed, 28 Mar 2012 10:47:43 +0000http://status.alwaysdata.com/operation/25/detail/Looks like a disk hardware issue. We're rebooting the server.Wed, 28 Mar 2012 18:05:44 +0000http://status.alwaysdata.com/operation/25/detail/The server is up again.Wed, 28 Mar 2012 18:16:40 +0000http://status.alwaysdata.com/operation/26/detail/We're investigating.Wed, 04 Apr 2012 12:30:00 +0000http://status.alwaysdata.com/operation/26/detail/A IO spike caused the slowdowns, it's now getting better.Wed, 04 Apr 2012 12:48:02 +0000http://status.alwaysdata.com/operation/27/detail/A IO surge is slowing down the server.Thu, 26 Apr 2012 17:19:20 +0000http://status.alwaysdata.com/operation/28/detail/We're investigating.Mon, 30 Apr 2012 21:58:31 +0000http://status.alwaysdata.com/operation/28/detail/A disk IO error caused the downtime. Everything is now up again.Mon, 30 Apr 2012 22:30:34 +0000http://status.alwaysdata.com/operation/29/detail/Some network issues impacts email delivery, we're working on it.Thu, 03 May 2012 13:20:42 +0000http://status.alwaysdata.com/operation/29/detail/The network issue has been circumvented, mails are being delivered again. Mails accumulated in queue will be delivered over the next couple of hours.Thu, 03 May 2012 13:38:06 +0000http://status.alwaysdata.com/operation/30/detail/There's an issue with our provider.Thu, 03 May 2012 15:59:07 +0000http://status.alwaysdata.com/operation/31/detail/We got issues on MySQL server, working on it.Tue, 08 May 2012 09:33:21 +0000http://status.alwaysdata.com/operation/31/detail/The problem is resolved. More details later.Tue, 08 May 2012 09:58:50 +0000http://status.alwaysdata.com/operation/31/detail/A disk error happened on this server at 8:52, which caused MySQL to malfunction with write operations (INSERT, UPDATE, etc.). The read operations (SELECT) continued to work properly in the meantime. Our monitoring system failed to detect this particular and unusual error, and today being a day off in France, we were slower than usual to intervene, hence the long downtime. That simply should not have happened, we're very sorry. We're currently rebuilding a brand new monitoring system, which will be much more powerful and would have detected this type of failure. We apologize for this issue and will keep you informed soon about that new monitoring system on our blog.Tue, 08 May 2012 10:37:50 +0000http://status.alwaysdata.com/operation/32/detail/postgresql1, mongodb1 and couchdb1 will be migrated to a new physical server. Expected downtime: less than 5 minutes per service.Tue, 08 May 2012 15:13:33 +0000http://status.alwaysdata.com/operation/32/detail/The operation is starting.Tue, 08 May 2012 22:00:41 +0000http://status.alwaysdata.com/operation/32/detail/Everything has been migrated successfully. Downtime was less than 1 minute for MongoDB and CouchDB, less than 2 minutes for PostgreSQL.Tue, 08 May 2012 22:34:29 +0000http://status.alwaysdata.com/operation/33/detail/We're investigating.Wed, 16 May 2012 14:50:04 +0000http://status.alwaysdata.com/operation/33/detail/Everything is back to normal.Wed, 16 May 2012 14:58:01 +0000http://status.alwaysdata.com/operation/34/detail/This server leaks kernel memory since the end of April. It's located in the NFS code, and will eat up all the memory after ~10 days. Even the latest kernel version (3.4), released today, has this issue. We'll work with the kernel developers to try to isolate the culprit and have the bug fixed. In the meantime, we'll monitor this server closely and update this operation when we have more details.Mon, 21 May 2012 14:41:06 +0000http://status.alwaysdata.com/operation/35/detail/IMAP access (including webmail) is slowed down. An unusual IO spike seems to be the culprit, we're investigating.Wed, 23 May 2012 10:47:28 +0000http://status.alwaysdata.com/operation/35/detail/We were probably wrong: the IMAP connection (which queries our internal databases) seems to be longer than usual. Since the webmail reconnects on each page view, everything feels slower. Normal IMAP accesses are not much impacted, since you usually connect only once for the entire session.Wed, 23 May 2012 11:18:13 +0000http://status.alwaysdata.com/operation/35/detail/The speed is now almost back to normal. We'll still have to make some internal changes to make sure it won't happen again.Wed, 23 May 2012 11:32:28 +0000http://status.alwaysdata.com/operation/35/detail/Caching has been enabled to speed up authentication. We still work on our internal database queries to speed that up too.Wed, 23 May 2012 14:21:13 +0000http://status.alwaysdata.com/operation/35/detail/We've further optimized our queries. The webmail is now faster than it was even before this issue.Tue, 29 May 2012 11:01:13 +0000http://status.alwaysdata.com/operation/36/detail/We're investigating.Tue, 29 May 2012 13:15:43 +0000http://status.alwaysdata.com/operation/36/detail/The server didn't reboot properly. More investigation ongoing.Tue, 29 May 2012 13:20:03 +0000http://status.alwaysdata.com/operation/36/detail/The server is back online.Tue, 29 May 2012 13:30:51 +0000http://status.alwaysdata.com/operation/34/detail/We're now seeing leaks of xfs_inode. We'll have to roll back to an old kernel version and see if it stops.Wed, 30 May 2012 13:04:09 +0000http://status.alwaysdata.com/operation/34/detail/We've made a few changes on the kernel (while still running): it now seems to flush its cache permanently, which slows down the server due to high iowait. But the leak is gone. That's obviously not a solution, we'll try other things.Wed, 30 May 2012 17:25:35 +0000http://status.alwaysdata.com/operation/34/detail/We've managed to stop the permanent flush, which means the server is now operating normally. We've tweaked a few parameters again, we're now waiting to see if the leak is under control (xfs_inode is still increasing, but nothing anormal at this point).Wed, 30 May 2012 19:23:34 +0000http://status.alwaysdata.com/operation/34/detail/Everything seems stable and under control.Wed, 30 May 2012 22:28:11 +0000http://status.alwaysdata.com/operation/34/detail/We've finally found the root cause of the XFS leak (rsync on a particular account with more than 7 million files). The NFS leak is still being investigated.Thu, 31 May 2012 10:23:58 +0000http://status.alwaysdata.com/operation/37/detail/An unexplained memory peak triggered a cache flush.Mon, 04 Jun 2012 13:17:54 +0000http://status.alwaysdata.com/operation/37/detail/We suspect a SQL server was slowed down for a couple of minutes, which triggered the global memory usage increase and then, the cache flush on http servers. There's no easy way to prevent this snowball effect from happening with our current architecture.Mon, 04 Jun 2012 13:44:56 +0000http://status.alwaysdata.com/operation/37/detail/This problem happened again on http6. We're still unsure about the root cause.Mon, 04 Jun 2012 15:21:23 +0000http://status.alwaysdata.com/operation/37/detail/We've probably found the culprit.Mon, 04 Jun 2012 16:11:41 +0000http://status.alwaysdata.com/operation/38/detail/Internal changes had an unexpected side effect. Everything should be back soon.Thu, 14 Jun 2012 17:46:50 +0000http://status.alwaysdata.com/operation/38/detail/FTP is now working as usual.Thu, 14 Jun 2012 17:53:28 +0000http://status.alwaysdata.com/operation/39/detail/http7 first became unresponsive - that happens. However, it caused NFS freezes on the server that handles *.alwaysdata.com as well as our main internal database. Those freezes also froze our *.alwaysdata.com Web applications, which kept their PostgreSQL connections open (status "idle in transaction"). Which eventually consumed all PostgreSQL connections. The http7 server has been restarted, but it failed to connect to our internal database to fetch its configuration (as it was down). Which explains why all sites remained down on this server for much longer than it should have. This internal database is mirrored in real time, though, and the secondary server was ready to accept connections and serve requests. Unfortunately, this server has been overloaded very quickly, and stopped responding too. This is because its hardware hasn't been upgraded to match the primary's. This server also host status.alwaysdata.com, which is why it was down too. We still have to figure out why several protections didn't work out as they should. Of course, the secondary server will also be upgraded to be able to handle the same load as the primary. http7's hardware will also be replaced, something is definitely wrong with this server. We're really sorry for this unexpectely long downtime. All other http servers as well as other services weren't affected by this issue.Mon, 25 Jun 2012 16:26:51 +0000http://status.alwaysdata.com/operation/40/detail/Our provider has a network issue, several servers are unaccessible from various locations.Sat, 30 Jun 2012 14:43:36 +0000http://status.alwaysdata.com/operation/40/detail/Issue resolved.Sat, 30 Jun 2012 14:46:41 +0000http://status.alwaysdata.com/operation/41/detail/MongoDB will be upgraded to 2.0 from 1.8 tonight. No backward incompatibilities are listed. The service should not be down more than a couple of seconds.Wed, 04 Jul 2012 16:23:10 +0000http://status.alwaysdata.com/operation/41/detail/MongoDB has been upgraded. The service has been down for less than one second.Wed, 04 Jul 2012 21:03:16 +0000http://status.alwaysdata.com/operation/42/detail/We're investigating.Mon, 09 Jul 2012 05:27:48 +0000http://status.alwaysdata.com/operation/42/detail/The server has been suspended by our provider for abuse. We hope to have the situation resolved in the next hour.Mon, 09 Jul 2012 05:47:40 +0000http://status.alwaysdata.com/operation/42/detail/The suspension should be lifted in the next hour (confirmed by a telephone call). In the meantime, we're preparing another server in case the issue lingers, which may explain the temporary SSH key change if you try to connect to ssh.alwaysdata.com.Mon, 09 Jul 2012 06:13:14 +0000http://status.alwaysdata.com/operation/42/detail/ssh.alwaysdata.com is now up again, although using a temporary server.Mon, 09 Jul 2012 06:24:39 +0000http://status.alwaysdata.com/operation/42/detail/Note: cron jobs and services (which is in beta) are still down.Mon, 09 Jul 2012 06:29:09 +0000http://status.alwaysdata.com/operation/42/detail/The old SSH key and the crontabs have been reinstalled on the temporary server, so everything is pretty much back to normal. Now that we've gained access to our regular ssh.alaysdata.com, we'll investigate to prevent this issue from happening again. Once this is done, we'll switch back to the regular server.Mon, 09 Jul 2012 08:07:46 +0000http://status.alwaysdata.com/operation/42/detail/We have a clear understanding of what happened, as well as what needs to be improved on our internal firewall so that it doesn't happen again. We've started working on it, it should hopefully be done by the end of the day.Mon, 09 Jul 2012 08:20:06 +0000http://status.alwaysdata.com/operation/42/detail/We've switched back to the primary server. We've detected and delete the account responsible for the abuse. We're still working on our internal firewall to prevent this issue from happening again.Mon, 09 Jul 2012 13:02:13 +0000http://status.alwaysdata.com/operation/43/detail/We're investigating.Mon, 16 Jul 2012 13:23:52 +0000http://status.alwaysdata.com/operation/43/detail/The server is stuck after a reboot.Mon, 16 Jul 2012 13:30:36 +0000http://status.alwaysdata.com/operation/43/detail/A technician will go inspect the machine.Mon, 16 Jul 2012 13:34:18 +0000http://status.alwaysdata.com/operation/43/detail/The machine has booted. HTTP applications will be up in a couple of minutes.Mon, 16 Jul 2012 13:35:38 +0000http://status.alwaysdata.com/operation/43/detail/Service is up again.Mon, 16 Jul 2012 13:39:49 +0000http://status.alwaysdata.com/operation/44/detail/We're investigating.Sat, 21 Jul 2012 05:33:01 +0000http://status.alwaysdata.com/operation/44/detail/The server can't be reached, maybe a hardware issue. A technician will go inspect the machine.Sat, 21 Jul 2012 05:36:25 +0000http://status.alwaysdata.com/operation/44/detail/The server is up again. The original cause of the issue is still unknown (kernel panic?), but we highly suspect a hardware error. This machine will be replaced over the next few days. Sorry for the repeated downtime on http7. All affected customers can ask for a complete refund for this month by opening a support ticket.Sat, 21 Jul 2012 06:01:12 +0000http://status.alwaysdata.com/operation/45/detail/We are investigating.Sat, 21 Jul 2012 12:23:47 +0000http://status.alwaysdata.com/operation/45/detail/Everything is up again. A hardware error seems to be the cause of this freeze: Jul 21 14:01:09 http6 kernel: [11312172.619526] [Hardware Error]: Machine check events loggedSat, 21 Jul 2012 12:56:20 +0000http://status.alwaysdata.com/operation/46/detail/Working on it.Tue, 31 Jul 2012 13:46:24 +0000http://status.alwaysdata.com/operation/46/detail/Everything is up again. The server has had a IO failure again. The complete hardware will be changed in the next few days.Tue, 31 Jul 2012 13:53:12 +0000http://status.alwaysdata.com/operation/47/detail/The machine will be replaced. All accounts will be smoothly transferred over the new machine. Expected downtime: a few seconds per account. A final reboot of the new machine will then be done to make sure everything is OK, with an additional ~60 seconds downtime.Fri, 03 Aug 2012 10:01:14 +0000http://status.alwaysdata.com/operation/47/detail/We're starting the operation. Note: accounts will be migrated over the next few hours. A tiny HTTP downtime is expected (a few seconds), but SSH, FTP and WebDAV accesses will be unavailable for longer (until the whole server is migrated).Fri, 03 Aug 2012 17:01:19 +0000http://status.alwaysdata.com/operation/47/detail/The migration is almost complete. We'll soon transfer the IP addresses over the new machine and finally reboot it.Fri, 03 Aug 2012 20:33:57 +0000http://status.alwaysdata.com/operation/47/detail/A kernel crash during the reboot caused an unexpected HTTP downtime.Fri, 03 Aug 2012 20:44:41 +0000http://status.alwaysdata.com/operation/47/detail/The hard reboot causes XFS to recover the disk.Fri, 03 Aug 2012 20:52:32 +0000http://status.alwaysdata.com/operation/47/detail/Taking a lot of time, we're investigating.Fri, 03 Aug 2012 21:02:00 +0000http://status.alwaysdata.com/operation/47/detail/Unfortunately, XFS is taking ages to mount (recover) the filesystem. It could take hours. We're still considering our options there.Fri, 03 Aug 2012 21:20:31 +0000http://status.alwaysdata.com/operation/47/detail/The filesystem has mounted. We're making sure everything is OK.Fri, 03 Aug 2012 21:26:58 +0000http://status.alwaysdata.com/operation/47/detail/SSH, FTP and WebDAV accesses are back. Everything seems to be working properly. We're still reviewing everything.Fri, 03 Aug 2012 21:37:58 +0000http://status.alwaysdata.com/operation/47/detail/The migration is completed. Needless to say, it didn't happen as expected, due to that final kernel crash. The HTTP downtime should not have exceeded a couple of minutes. We're very sorry about the inconvenience. If your account is not working properly since the migration, please open a support ticket. We'll still need to schedule a reboot tomorrow night. The frequent hardware issues with http7 over the last few months should now be history.Fri, 03 Aug 2012 21:50:31 +0000http://status.alwaysdata.com/operation/48/detail/We're investigating.Mon, 06 Aug 2012 14:16:08 +0000http://status.alwaysdata.com/operation/48/detail/A DDoS is underway.Mon, 06 Aug 2012 14:22:50 +0000http://status.alwaysdata.com/operation/48/detail/Everything seems under control.Mon, 06 Aug 2012 14:44:06 +0000http://status.alwaysdata.com/operation/48/detail/The DDoS has stopped. We'll keep an eye in case it attacks us again.Mon, 06 Aug 2012 15:01:28 +0000http://status.alwaysdata.com/operation/49/detail/We're investigating.Tue, 14 Aug 2012 19:10:32 +0000http://status.alwaysdata.com/operation/49/detail/Everything is back. A network attack was launched by a fraudulent customer. Our firewall was supposed to block it automatically, but it didnd't. We'll investigate more to find out why.Tue, 14 Aug 2012 19:18:56 +0000http://status.alwaysdata.com/operation/49/detail/There was a bug in our firewall where attacks could fail to be blocked. They were detected just fine, but right before blocking it, a message was logged where we tried to resolve the destination IP address. If that resolve failed badly (e.g. DNS server unavailable, which can happen during an UDP flood), the whole process was cancelled, including the ban. The fix has now been deployed on all our servers.Tue, 14 Aug 2012 22:37:47 +0000http://status.alwaysdata.com/operation/50/detail/A DDoS is underway.Sat, 25 Aug 2012 15:35:08 +0000http://status.alwaysdata.com/operation/50/detail/The server is still under a strong attack, but it's responding again to HTTP requests, although slower than usual.Sat, 25 Aug 2012 15:48:19 +0000http://status.alwaysdata.com/operation/50/detail/Down again.Sat, 25 Aug 2012 15:56:10 +0000http://status.alwaysdata.com/operation/50/detail/Still fighting against the attack.Sat, 25 Aug 2012 16:08:03 +0000http://status.alwaysdata.com/operation/50/detail/The attack has ceased.Sat, 25 Aug 2012 16:40:31 +0000http://status.alwaysdata.com/operation/34/detail/A fix has been released. We'll deploy a new kernel version on all HTTP servers very soon.Fri, 31 Aug 2012 11:28:39 +0000http://status.alwaysdata.com/operation/34/detail/http4, http8 and http9 servers have been upgraded.Fri, 31 Aug 2012 22:38:21 +0000http://status.alwaysdata.com/operation/51/detail/Kernel is being upgraded.Sat, 01 Sep 2012 23:05:41 +0000http://status.alwaysdata.com/operation/51/detail/Filesystem quota is being rechecked. It may take some time.Sat, 01 Sep 2012 23:06:20 +0000http://status.alwaysdata.com/operation/51/detail/ETA: 15 minutes.Sat, 01 Sep 2012 23:10:33 +0000http://status.alwaysdata.com/operation/51/detail/Server is up again with the new kernel.Sat, 01 Sep 2012 23:14:44 +0000http://status.alwaysdata.com/operation/52/detail/We're investigating.Tue, 04 Sep 2012 01:57:40 +0000http://status.alwaysdata.com/operation/52/detail/It's a DDoS attack.Tue, 04 Sep 2012 02:03:00 +0000http://status.alwaysdata.com/operation/52/detail/Server is up again.Tue, 04 Sep 2012 02:13:14 +0000http://status.alwaysdata.com/operation/52/detail/Service is slow/disturbed.Tue, 04 Sep 2012 02:26:31 +0000http://status.alwaysdata.com/operation/52/detail/Everything is pretty much returned to normal. Let's stay vigilant.Tue, 04 Sep 2012 02:36:56 +0000http://status.alwaysdata.com/operation/52/detail/Everything is pretty much returned to normal. Let's stay vigilant.Tue, 04 Sep 2012 03:12:51 +0000http://status.alwaysdata.com/operation/52/detail/Down again.Tue, 04 Sep 2012 03:37:05 +0000http://status.alwaysdata.com/operation/52/detail/Service continues to be disturbed.Tue, 04 Sep 2012 04:17:04 +0000http://status.alwaysdata.com/operation/52/detail/Everything has been pretty normal during the past hour. We'll make changes later today which should help us deal with these attacks.Tue, 04 Sep 2012 05:34:55 +0000http://status.alwaysdata.com/operation/52/detail/The attack is back.Tue, 04 Sep 2012 16:34:26 +0000http://status.alwaysdata.com/operation/52/detail/The changes we've just put in production helped a lot. We're now seeing a high packet loss though, we investigate.Tue, 04 Sep 2012 16:56:36 +0000http://status.alwaysdata.com/operation/52/detail/Only ICMP suffers from packet loss; TCP/UDP works normally. Of course, we remain vigilant.Tue, 04 Sep 2012 17:39:53 +0000http://status.alwaysdata.com/operation/53/detail/One of the drives is dying; we will replace it tonight. Estimated downtime: 20 minutes.Thu, 06 Sep 2012 08:07:43 +0000http://status.alwaysdata.com/operation/54/detail/We are investigating.Thu, 06 Sep 2012 14:51:52 +0000http://status.alwaysdata.com/operation/54/detail/The server doesn't ping, the KVM isn't reachable. A technician will go check it.Thu, 06 Sep 2012 14:55:03 +0000http://status.alwaysdata.com/operation/54/detail/The server is restarting.Thu, 06 Sep 2012 15:08:50 +0000http://status.alwaysdata.com/operation/54/detail/Server is up again. Most likely culprit: we were running on Linux 3.6-rc4 since the latest DDoS attacks. It may be a kernel bug, although the screen remained black.Thu, 06 Sep 2012 15:11:12 +0000http://status.alwaysdata.com/operation/54/detail/Actually, the disk about to be replaced made the system hang. That's not a very nice disk.Thu, 06 Sep 2012 15:14:20 +0000http://status.alwaysdata.com/operation/54/detail/Server is down again. Human issue.Thu, 06 Sep 2012 15:45:04 +0000http://status.alwaysdata.com/operation/54/detail/Our provider has detected the issue and tried to fix it, although the disk was supposed to be changed tonight. The disk will thus be changed during the next hour.Thu, 06 Sep 2012 15:55:55 +0000http://status.alwaysdata.com/operation/54/detail/The disk has actually already been changed. Good call from our provider.Thu, 06 Sep 2012 16:06:49 +0000http://status.alwaysdata.com/operation/54/detail/Everything is now working normally. The RAID is being resynchronized.Thu, 06 Sep 2012 16:28:11 +0000http://status.alwaysdata.com/operation/55/detail/Some networks can't reach our main IP address. We're investigating.Tue, 11 Sep 2012 09:32:04 +0000http://status.alwaysdata.com/operation/55/detail/The IP is now accessible again. It's obviously a routing issue. We're still waiting for more details from our provider.Tue, 11 Sep 2012 09:43:20 +0000http://status.alwaysdata.com/operation/55/detail/It was an isolated switch issue.Tue, 11 Sep 2012 10:43:40 +0000http://status.alwaysdata.com/operation/55/detail/The IP address is inaccessible again.Tue, 11 Sep 2012 13:30:20 +0000http://status.alwaysdata.com/operation/55/detail/Solved. We're still monitoring the situation.Tue, 11 Sep 2012 14:16:36 +0000http://status.alwaysdata.com/operation/55/detail/We've tried to move the IP address to another server to proxy the requests. It didn't work right away (hence the "too many recoveries" error message), but it seems stabilized.Tue, 11 Sep 2012 14:43:58 +0000http://status.alwaysdata.com/operation/56/detail/A SYN flood again. We'll just deploy the protection that worked beautifully on http4. The server will be rebooted soon.Wed, 12 Sep 2012 14:49:57 +0000http://status.alwaysdata.com/operation/56/detail/The server has trouble rebooting. We're investigating.Wed, 12 Sep 2012 14:58:32 +0000http://status.alwaysdata.com/operation/56/detail/A technician will have to see what's wrong.Wed, 12 Sep 2012 15:06:20 +0000http://status.alwaysdata.com/operation/56/detail/It's due to the kernel (not surprising with a rc). We're trying another one.Wed, 12 Sep 2012 15:18:22 +0000http://status.alwaysdata.com/operation/56/detail/It booted. Everything should be back in a few seconds.Wed, 12 Sep 2012 15:20:16 +0000http://status.alwaysdata.com/operation/56/detail/Working normally, although the DDoS has stopped for now. The protection _should_ be effective though.Wed, 12 Sep 2012 15:25:06 +0000http://status.alwaysdata.com/operation/57/detail/Mails have been delayed. This is due to a bug we've been tracking for the past 3 days.Wed, 03 Oct 2012 08:15:54 +0000http://status.alwaysdata.com/operation/57/detail/The bug has been identified. When a LOT of different mails are being scanned at the same time, we may run out of file handles. Increasing the maximum file handles is easy, but that wouldn't really solve the underlying issue. We have to find a smart way to limit the scanning rate of clients. The sending rate is already limited - scanning is trickier.Wed, 03 Oct 2012 09:22:32 +0000http://status.alwaysdata.com/operation/57/detail/Scanning messages is now rate limited per client. That should solve the issue.Wed, 03 Oct 2012 14:46:02 +0000http://status.alwaysdata.com/operation/58/detail/We'll upgrade the hard drives firmware on this server. The server will be unavailable for about 10 minutes.Fri, 05 Oct 2012 22:53:04 +0000http://status.alwaysdata.com/operation/58/detail/The operation is starting.Sat, 06 Oct 2012 22:29:01 +0000http://status.alwaysdata.com/operation/58/detail/The firmware has been successfully upgraded. The operation lasted a bit longer than expected due to a human error.Sat, 06 Oct 2012 22:52:45 +0000http://status.alwaysdata.com/operation/59/detail/We're investigating.Thu, 11 Oct 2012 17:47:03 +0000http://status.alwaysdata.com/operation/59/detail/There was an error with a disk, it shouldn't have crashed the machine. The filesystem is being checked, everything should be up soon.Thu, 11 Oct 2012 17:50:24 +0000http://status.alwaysdata.com/operation/59/detail/The server is up again, albeit it will be slow for a few minutes.Thu, 11 Oct 2012 17:53:24 +0000http://status.alwaysdata.com/operation/59/detail/We'll check with our provider regarding this disk issue, although we're on the latest firmware.Thu, 11 Oct 2012 18:08:15 +0000http://status.alwaysdata.com/operation/60/detail/We've noticed that avast blacklisted one of our shared servers IP address (http9). If you use avast and go visit a website hosted on this server, you're likely to get errors such as "connection reset". avast never tried to contact us before. We've tried to telephone them earlier today, to no avail. We've sent them an email asking for more details, and their response was: this seems to be suspicious: "http://www.scumware.org/report/178.32.28.119" Considering that: * all websites listed on this page have been suspended or removed at least several weeks ago * we're always quick to delete fraudulent accounts * any shared hosting provider is bound to have such malware - some of them have many, many more than us and are not blacklisted we don't think they did a good job. They could have contacted us, banned subdomains or, at worst, the alwaysdata.net domain. Other security companies did not ban this server: * http://www.ipvoid.com/scan/178.32.28.119 * http://sitecheck.sucuri.net/results/178.32.28.119 We're encouraging our customers to let them know that they're not happy about the situation.Fri, 12 Oct 2012 14:48:03 +0000http://status.alwaysdata.com/operation/60/detail/The IP has been unblocked, reportedly on Saturday.Mon, 15 Oct 2012 08:11:43 +0000http://status.alwaysdata.com/operation/61/detail/Our internal VLAN is having issues, we'll check with our provider. The connection between HTTP/SSH and SQL databases was impacted for a few minutes. We have rerouted the connections to go through the external interfaces.Mon, 15 Oct 2012 09:09:00 +0000http://status.alwaysdata.com/operation/61/detail/SSH access is still mostly down (NFS won't remount with the new IP address easily).Mon, 15 Oct 2012 09:19:51 +0000http://status.alwaysdata.com/operation/61/detail/Internal routing is back.Mon, 15 Oct 2012 09:35:25 +0000http://status.alwaysdata.com/operation/62/detail/We're investigating.Fri, 19 Oct 2012 18:58:41 +0000http://status.alwaysdata.com/operation/62/detail/Disk issue again. The server has been rebooted, the filesystem is being checked.Fri, 19 Oct 2012 19:04:55 +0000http://status.alwaysdata.com/operation/62/detail/Up again.Fri, 19 Oct 2012 19:09:37 +0000http://status.alwaysdata.com/operation/62/detail/We're still fighting to find the root cause of these regular disk issues (on http7, http8, http9). We're currently testing a new firmware on http9, to be deployed soon on http7. Since these errors are not reproductible and somewhat infrequent, solving the issue is tough. We're sorry about this.Fri, 19 Oct 2012 19:12:21 +0000http://status.alwaysdata.com/operation/63/detail/We'll upgrade the hard drives firmware on this server. The server will be unavailable for about 10 minutes.Sat, 20 Oct 2012 10:28:35 +0000http://status.alwaysdata.com/operation/64/detail/We're investigating.Sat, 20 Oct 2012 15:05:12 +0000http://status.alwaysdata.com/operation/64/detail/Back again. The machine was found frozen and has to be rebooted. Probably a kernel issue, we'll have to update it soon.Sat, 20 Oct 2012 15:10:07 +0000http://status.alwaysdata.com/operation/64/detail/It happened again. We're forcing the kernel upgrade immediately.Sat, 20 Oct 2012 20:01:58 +0000http://status.alwaysdata.com/operation/64/detail/The kernel has been upgraded. We stay vigilant as these freezes may be caused by a hardware issue.Sat, 20 Oct 2012 20:15:06 +0000http://status.alwaysdata.com/operation/64/detail/We've found evidence of a probable hardware issue in the logs after the second freeze. We'll schedule a motherboard replacement.Sat, 20 Oct 2012 21:47:16 +0000http://status.alwaysdata.com/operation/64/detail/The motherboard will be replaced at 1:00 (in 35 minutes).Sat, 20 Oct 2012 22:25:27 +0000http://status.alwaysdata.com/operation/63/detail/The upgrade is about to start.Sat, 20 Oct 2012 22:30:03 +0000http://status.alwaysdata.com/operation/63/detail/The firmware has been upgraded.Sat, 20 Oct 2012 22:41:43 +0000http://status.alwaysdata.com/operation/64/detail/The operation is starting.Sat, 20 Oct 2012 23:02:35 +0000http://status.alwaysdata.com/operation/64/detail/The hardware is still being replaced, it takes longer than usual.Sat, 20 Oct 2012 23:47:24 +0000http://status.alwaysdata.com/operation/64/detail/We're stil waiting to hear from our provider. They're clearly not doing a good job right now.Sun, 21 Oct 2012 01:16:37 +0000http://status.alwaysdata.com/operation/64/detail/It seems like the new motherboard was not the exact same model as before, and it has incompatibility issues with the kernel. We'll know more when the operation has completed.Sun, 21 Oct 2012 02:11:28 +0000http://status.alwaysdata.com/operation/64/detail/Apparently, they didn't get the new motherboard to work. The old one is being put back into the server.Sun, 21 Oct 2012 03:08:09 +0000http://status.alwaysdata.com/operation/64/detail/And it still doesn't work (network down, as before). They must have misconfigured something during the operation. We're very, very sorry about all of this. We'll keep you informed as soon as we get more details.Sun, 21 Oct 2012 03:48:23 +0000http://status.alwaysdata.com/operation/64/detail/They're still trying to figure this out. Here is the error message that gets printed every 3 seconds, to be specific: ixgbe 0000:04:00.4 eth0: reset adapterSun, 21 Oct 2012 04:05:21 +0000http://status.alwaysdata.com/operation/64/detail/Nothing new since the last message. Just to make things clear: your data is fine, it's "just" a network issue. The machine is fully accessible by KVM.Sun, 21 Oct 2012 04:36:39 +0000http://status.alwaysdata.com/operation/64/detail/They still didn't make it work. A senior technician will arrive at 10:00. We cannot give an ETA right now, I'm sorry.Sun, 21 Oct 2012 05:49:45 +0000http://status.alwaysdata.com/operation/64/detail/They're preparing a spare server where our disks will be inserted. Hopefully that will solve the issue.Sun, 21 Oct 2012 06:03:46 +0000http://status.alwaysdata.com/operation/64/detail/The senior technician is now investigating on this issue. They're looking for another network card model, as it seems to be the cause of all this.Sun, 21 Oct 2012 08:04:26 +0000http://status.alwaysdata.com/operation/64/detail/The server is up again. It will be slower than usual for several minutes.Sun, 21 Oct 2012 08:17:21 +0000http://status.alwaysdata.com/operation/64/detail/The network card has been replaced. Why the previous one, which worked normally for 3 years, stopped working when the motherboard was replaced is still a mystery. We will have a chat with our provider next week to understand how such an extended downtime could have happened, especially on a high-end server such as http4. That's the worse downtime we've experienced with a server, by far. We will obviously take actions to avoid this to happen again. All customers on http4 can ask for a full refund for October by opening a support ticket. We are very, very sorry. This is clearly not the quality of service you should expect from us.Sun, 21 Oct 2012 08:40:55 +0000http://status.alwaysdata.com/operation/65/detail/We're investigating.Mon, 22 Oct 2012 23:07:42 +0000http://status.alwaysdata.com/operation/65/detail/Up again after a hard reboot. The kernel started to have very weird issues before a complete freeze: INFO: rcu_sched self-detected stall on CPU Never seen that before. We'll keep a close eye on that.Mon, 22 Oct 2012 23:15:19 +0000http://status.alwaysdata.com/operation/65/detail/Same message at 5:12 AM, and then every 3 minutes. Server was down from 5:45 AM to 5:53 AM.Tue, 23 Oct 2012 04:17:57 +0000http://status.alwaysdata.com/operation/65/detail/It happened again at 10:22 AM. We'll keep an eye on the server until the problem has been solved, this is our priority for now.Tue, 23 Oct 2012 08:55:10 +0000http://status.alwaysdata.com/operation/65/detail/It's a kernel issue. We'll downgrade the kernel.Tue, 23 Oct 2012 09:20:17 +0000http://status.alwaysdata.com/operation/65/detail/It crashed a last time before we had the chance to downgrade.Tue, 23 Oct 2012 10:42:11 +0000http://status.alwaysdata.com/operation/65/detail/The kernel has been downgraded to 3.5.7 from 3.6.2. We'll file a bug report.Tue, 23 Oct 2012 11:07:48 +0000http://status.alwaysdata.com/operation/66/detail/We are investigating.Wed, 24 Oct 2012 14:10:40 +0000http://status.alwaysdata.com/operation/66/detail/Resolved. A client abuse caused a massive slowdown.Wed, 24 Oct 2012 14:14:04 +0000http://status.alwaysdata.com/operation/67/detail/Similar issue as the one we had with http7 recently. We're downgrading the kernel.Sun, 28 Oct 2012 04:40:43 +0000http://status.alwaysdata.com/operation/68/detail/We are investigating.Fri, 02 Nov 2012 16:09:20 +0000http://status.alwaysdata.com/operation/68/detail/Some IO spike froze the server, everything is now up and we'll keep an eye on it.Fri, 02 Nov 2012 16:38:54 +0000http://status.alwaysdata.com/operation/69/detail/Server down, we are investigating.Sun, 04 Nov 2012 04:02:48 +0000http://status.alwaysdata.com/operation/69/detail/XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1503 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff8125fb28Sun, 04 Nov 2012 04:05:43 +0000http://status.alwaysdata.com/operation/69/detail/Filesystem recovery ongoing.Sun, 04 Nov 2012 04:11:27 +0000http://status.alwaysdata.com/operation/69/detail/Still recovering. We have no ETA yet.Sun, 04 Nov 2012 04:18:52 +0000http://status.alwaysdata.com/operation/69/detail/The recovery mount failed again with the same error. We're now running xfs_repair -L.Sun, 04 Nov 2012 04:41:37 +0000http://status.alwaysdata.com/operation/69/detail/Still repairing (phase 3, agno = 21).Sun, 04 Nov 2012 04:57:35 +0000http://status.alwaysdata.com/operation/69/detail/xfs_repair has finished successfully. We're now mounting the filesystem again; it's checking quotas. It will probably take a few more minutes.Sun, 04 Nov 2012 05:10:39 +0000http://status.alwaysdata.com/operation/69/detail/Still calculating quotas.Sun, 04 Nov 2012 05:26:13 +0000http://status.alwaysdata.com/operation/69/detail/The quota calculation has frozen. We're rebooting the server and will try to mount the filesystem with quota disabled.Sun, 04 Nov 2012 05:39:43 +0000http://status.alwaysdata.com/operation/69/detail/The filesystem has mounted successfully. Since the filesystem log had to be zeroed, the latest changes may have been lost, although that will probably impact few files. We're obviously monitoring the situation very closely. The root cause of this corruption is, again, these disk issues we're trying to resolve. We've started replacing some Seagate disks by Hitachi disks, and we will continue over the next few days. Hopefully that will solve all these issues. All impacted customers can ask for a full refund for this month. We apologize for this situation.Sun, 04 Nov 2012 05:49:04 +0000http://status.alwaysdata.com/operation/70/detail/DDoS ongoing.Sun, 04 Nov 2012 19:25:22 +0000http://status.alwaysdata.com/operation/70/detail/An updated kernel has been installed. The attacked site was suspended as it does not comply with our terms of service anyway.Sun, 04 Nov 2012 19:37:10 +0000http://status.alwaysdata.com/operation/71/detail/One disk will be changed, from Seagate to Hitachi. We're still struggling with disk issues on http7, http8 and http9. The disks sometimes timeout for no real reason. This issue started appearing nearly 18 months ago, and we've already tried many things (changing the disks, the SATA cables, the kernel, the whole motherboard, disabling smartd, disabling NCQ, etc.). We now suspect that Seagate hard drives are simply not reliable. We've alerted our provider, which alerted Seagate, but we don't hold our breath. We also need to be sure that these disks really are the source of these timeouts, which is why we want to test a Hitachi disk. Expected downtime: about 20 minutes.Tue, 06 Nov 2012 16:20:09 +0000http://status.alwaysdata.com/operation/71/detail/The operation has been cancelled and will be rescheduled, due to a communication error with our provider.Tue, 06 Nov 2012 23:14:03 +0000http://status.alwaysdata.com/operation/71/detail/The operation will actually start in the next few minutes.Tue, 06 Nov 2012 23:26:42 +0000http://status.alwaysdata.com/operation/71/detail/The operation has started.Tue, 06 Nov 2012 23:31:14 +0000http://status.alwaysdata.com/operation/71/detail/The operation has completed. The disk has been successfully changed.Tue, 06 Nov 2012 23:37:59 +0000http://status.alwaysdata.com/operation/72/detail/We're investigating.Thu, 08 Nov 2012 16:22:35 +0000http://status.alwaysdata.com/operation/72/detail/Once again, the disks have crashed the machine. We're rebooting it.Thu, 08 Nov 2012 16:28:39 +0000http://status.alwaysdata.com/operation/72/detail/Up again, although it will be slow for a few minutes.Thu, 08 Nov 2012 16:36:09 +0000http://status.alwaysdata.com/operation/73/detail/Kernel issue, we're rebooting and will later downgrade.Fri, 09 Nov 2012 13:19:50 +0000http://status.alwaysdata.com/operation/73/detail/Up again.Fri, 09 Nov 2012 13:22:53 +0000http://status.alwaysdata.com/operation/74/detail/We are investigating.Tue, 20 Nov 2012 02:23:49 +0000http://status.alwaysdata.com/operation/74/detail/We're getting errors: ixgbe 0000:04:00.0: eth0: Reset adapter ixgbe 0000:04:00.0: eth0: detected SFP+: 0 which is the exact same message we had on a previous operation. Our provider is working on the switches, which must explain these errors. We'll let you know when we have more details.Tue, 20 Nov 2012 02:40:27 +0000http://status.alwaysdata.com/operation/74/detail/Server is up again.Tue, 20 Nov 2012 03:06:56 +0000http://status.alwaysdata.com/operation/75/detail/We are expecting FTP issues for accounts located on http6, probably due to last operation. We are investigating.Tue, 20 Nov 2012 10:36:58 +0000http://status.alwaysdata.com/operation/75/detail/Fixed. For some reason, nfsd had to be restarted on http6 as it refused to serve our FTP server.Tue, 20 Nov 2012 10:51:45 +0000http://status.alwaysdata.com/operation/76/detail/One of the Seagate disks will be replaced by a Hitachi disk.Fri, 23 Nov 2012 13:16:43 +0000http://status.alwaysdata.com/operation/76/detail/The operation will start shortly.Fri, 23 Nov 2012 23:11:59 +0000http://status.alwaysdata.com/operation/76/detail/The operation has started.Fri, 23 Nov 2012 23:16:16 +0000http://status.alwaysdata.com/operation/76/detail/The server is up again.Fri, 23 Nov 2012 23:22:27 +0000http://status.alwaysdata.com/operation/77/detail/The second disk will be changed (from Seagate to Hitachi).Tue, 27 Nov 2012 09:33:33 +0000http://status.alwaysdata.com/operation/78/detail/One of the Seagate disks will be replaced with a Hitachi.Tue, 27 Nov 2012 13:36:54 +0000http://status.alwaysdata.com/operation/79/detail/We are investigating.Tue, 27 Nov 2012 16:41:55 +0000http://status.alwaysdata.com/operation/79/detail/Our provider had blocked the shared IP address for this server (accounts with a dedicated IP address weren't affected). We're in contact with our provider to have more details about their recent change in policy with regards to abuse (e.g. phishing). Blocking an IP address for a simple phishing page, with no previous alerts, is just crazy.Tue, 27 Nov 2012 17:04:06 +0000http://status.alwaysdata.com/operation/77/detail/The operation will start shortly.Tue, 27 Nov 2012 23:13:09 +0000http://status.alwaysdata.com/operation/77/detail/The operation has begun.Tue, 27 Nov 2012 23:15:00 +0000http://status.alwaysdata.com/operation/77/detail/The operation has completed successfully.Tue, 27 Nov 2012 23:22:50 +0000http://status.alwaysdata.com/operation/78/detail/The operation will start shortly.Wed, 28 Nov 2012 23:20:44 +0000http://status.alwaysdata.com/operation/78/detail/The operation has begun.Wed, 28 Nov 2012 23:24:09 +0000http://status.alwaysdata.com/operation/78/detail/The operation has completed successfully.Wed, 28 Nov 2012 23:29:30 +0000http://status.alwaysdata.com/operation/80/detail/The second hard drive will be changed from a Seagate to a Hitachi model.Sun, 09 Dec 2012 11:14:29 +0000http://status.alwaysdata.com/operation/80/detail/The operation is starting.Mon, 10 Dec 2012 23:10:26 +0000http://status.alwaysdata.com/operation/80/detail/The operation has completed successfully. The server will be slow for a few minutes.Mon, 10 Dec 2012 23:14:53 +0000http://status.alwaysdata.com/operation/81/detail/A short network outage has impacted several servers.Wed, 19 Dec 2012 16:46:40 +0000http://status.alwaysdata.com/operation/82/detail/We're investigating.Thu, 20 Dec 2012 10:59:54 +0000http://status.alwaysdata.com/operation/82/detail/Server is up again. It had gone out of memory and had to be rebooted; we're still investigating why.Thu, 20 Dec 2012 11:06:12 +0000http://status.alwaysdata.com/operation/83/detail/An IO spike slowed down the server.Wed, 02 Jan 2013 11:23:44 +0000http://status.alwaysdata.com/operation/84/detail/Due to an IO spike.Sun, 06 Jan 2013 16:57:47 +0000http://status.alwaysdata.com/operation/84/detail/It's still very slow.Sun, 06 Jan 2013 17:39:43 +0000http://status.alwaysdata.com/operation/84/detail/We're rebooting the server.Sun, 06 Jan 2013 17:45:35 +0000http://status.alwaysdata.com/operation/85/detail/Due to IO. We're investigating.Tue, 08 Jan 2013 18:20:50 +0000http://status.alwaysdata.com/operation/86/detail/Due to IO. We're investigating.Wed, 16 Jan 2013 15:29:15 +0000http://status.alwaysdata.com/operation/87/detail/We're investigating.Tue, 22 Jan 2013 10:43:55 +0000http://status.alwaysdata.com/operation/87/detail/Some users are consuming too much memory. We killed their processes and contact them to not restart them until they've fixed the problem.Tue, 22 Jan 2013 10:55:55 +0000http://status.alwaysdata.com/operation/88/detail/Server has been slowed down again, we are investigating.Tue, 22 Jan 2013 17:30:36 +0000http://status.alwaysdata.com/operation/88/detail/We found the processes using lot of memory and they have been isolated. Meanwhile, the users have been contacted.Tue, 22 Jan 2013 17:55:36 +0000http://status.alwaysdata.com/operation/89/detail/This server will be upgraded to better hardware. All accounts will be transparently migrated to the new machine over the next 36 hours. Once this is completed, the final switch will be made: a couple of minutes of downtime is then expected.Wed, 13 Mar 2013 16:22:26 +0000http://status.alwaysdata.com/operation/89/detail/All accounts have been successfully been migrated. The final switch will be done in about 4 hours.Thu, 14 Mar 2013 18:55:12 +0000http://status.alwaysdata.com/operation/89/detail/The server is about to be switched. Downtime expected for a few minutes.Thu, 14 Mar 2013 23:19:08 +0000http://status.alwaysdata.com/operation/89/detail/The migration has completed successfully.Thu, 14 Mar 2013 23:53:39 +0000http://status.alwaysdata.com/operation/90/detail/Caused by a motherboard issue. The server will be replaced by the end of the week.Thu, 21 Mar 2013 11:01:49 +0000http://status.alwaysdata.com/operation/91/detail/This server will be upgraded to better hardware. All accounts will be transparently migrated to the new machine over the next 12 hours. Once this is completed, the final switch will be made: a couple of minutes of downtime is then expected.Sun, 24 Mar 2013 09:29:32 +0000http://status.alwaysdata.com/operation/91/detail/The migration is postponed a later date. It had not even begun yet.Sun, 24 Mar 2013 14:33:15 +0000http://status.alwaysdata.com/operation/91/detail/The migration will start in a couple of hours.Tue, 26 Mar 2013 14:03:21 +0000http://status.alwaysdata.com/operation/91/detail/The migration has started.Tue, 26 Mar 2013 15:07:54 +0000http://status.alwaysdata.com/operation/91/detail/The final switch will be done very soon.Tue, 26 Mar 2013 22:48:33 +0000http://status.alwaysdata.com/operation/91/detail/The operation has completed.Tue, 26 Mar 2013 23:24:53 +0000http://status.alwaysdata.com/operation/92/detail/We're investigating.Wed, 27 Mar 2013 00:12:11 +0000http://status.alwaysdata.com/operation/92/detail/Up again. A network equipment upgrade triggered a well known bug with the network card. Solved by rebooting the server. The server will be changed soon anyway, which will definitely solve this specific issue. This is the only remaining server with this network card model (Intel 10 Gbps).Wed, 27 Mar 2013 00:18:36 +0000http://status.alwaysdata.com/operation/93/detail/We're investigating.Thu, 28 Mar 2013 17:58:53 +0000http://status.alwaysdata.com/operation/93/detail/The issue was initially between http8 and postgresql1. It escalated and http8 is partially unreachable.Thu, 28 Mar 2013 18:14:10 +0000http://status.alwaysdata.com/operation/93/detail/Seems resolved. It was a router issue.Thu, 28 Mar 2013 18:24:18 +0000http://status.alwaysdata.com/operation/94/detail/IP down due to a client abuse. We'll investigate as it should not happen.Sun, 31 Mar 2013 02:07:45 +0000http://status.alwaysdata.com/operation/94/detail/The reaction time of our firewall (up to a couple of seconds) was too high; malicious packets could be sent during this period. We'll work on that.Sun, 31 Mar 2013 02:27:17 +0000http://status.alwaysdata.com/operation/94/detail/We've actually found a bug in our firewall, so the fix will be even easier.Sun, 31 Mar 2013 02:36:46 +0000http://status.alwaysdata.com/operation/94/detail/The bug has been fixed. This specific scenario cannot happen anymore.Sun, 31 Mar 2013 10:10:48 +0000http://status.alwaysdata.com/operation/95/detail/The server is suffering from a DDoS attack.Tue, 02 Apr 2013 00:01:41 +0000http://status.alwaysdata.com/operation/95/detail/The attack seems to have stopped for now.Tue, 02 Apr 2013 00:05:04 +0000http://status.alwaysdata.com/operation/95/detail/The attack is back.Tue, 02 Apr 2013 05:10:42 +0000http://status.alwaysdata.com/operation/95/detail/Since the attack is coming and going, the server can be temporarily unreachable.Tue, 02 Apr 2013 05:25:59 +0000http://status.alwaysdata.com/operation/95/detail/No new attack since the last message. There were 3 attacks during the night, for a total downtime of 23 minutes.Tue, 02 Apr 2013 14:41:06 +0000http://status.alwaysdata.com/operation/95/detail/We've had another attack. We're testing mitigation solutions.Tue, 02 Apr 2013 19:21:15 +0000http://status.alwaysdata.com/operation/95/detail/We're quite confident that we're now able to mitigate this attack.Tue, 02 Apr 2013 19:34:22 +0000http://status.alwaysdata.com/operation/96/detail/All PostgreSQL daemons will be upgraded in the next few minutes to fix a serious vulnerability issue (CVE-2013-1899).Thu, 04 Apr 2013 13:32:40 +0000http://status.alwaysdata.com/operation/96/detail/All daemons have been upgraded. Downtime lasted less than 10 seconds.Thu, 04 Apr 2013 14:34:53 +0000http://status.alwaysdata.com/operation/97/detail/This server will be upgraded to better hardware. All accounts will be transparently migrated to the new machine. Once this is completed, the final switch will be made: a couple of minutes of downtime is then expected. Temporary connection issues with FTP and SSH may happen during the migration, although this is quite rare. Retrying a few minutes later usually solves the problem.Sat, 06 Apr 2013 22:32:18 +0000http://status.alwaysdata.com/operation/97/detail/The migration has started.Sun, 07 Apr 2013 09:12:12 +0000http://status.alwaysdata.com/operation/97/detail/The first step has completed successfully (all files have been migrated to the new server). The final switch is scheduled later tonight.Sun, 07 Apr 2013 14:24:04 +0000http://status.alwaysdata.com/operation/97/detail/The migration has successfully completed.Sun, 07 Apr 2013 22:03:53 +0000http://status.alwaysdata.com/operation/98/detail/A specific table is making MySQL crash on this server. It has already happened twice.Tue, 16 Apr 2013 08:56:12 +0000http://status.alwaysdata.com/operation/98/detail/The table has been manually repaired, the crash should not happen anymore.Tue, 16 Apr 2013 09:13:04 +0000http://status.alwaysdata.com/operation/99/detail/Due to a filesystem corruption, the backup6 server is now in read-only mode. No data was lost, but daily backups will be suspended until further notice.Tue, 23 Apr 2013 15:28:43 +0000http://status.alwaysdata.com/operation/99/detail/Backups are being transferred to a new server. According to our estimates, the daily backups should be run again starting from April 26th.Wed, 24 Apr 2013 16:26:51 +0000http://status.alwaysdata.com/operation/99/detail/The transfer is still not done yet.Thu, 25 Apr 2013 22:13:49 +0000http://status.alwaysdata.com/operation/99/detail/Backups have resumed.Sat, 27 Apr 2013 10:12:27 +0000http://status.alwaysdata.com/operation/100/detail/A filesystem issue made the server much slower than it should. We'll split this server in two and apply a fix that should prevent the issue to occur again. In the meantime, backups on this server are still done everyday, but the files and mails will not be stored for more than one day. Database backups are not affected.Thu, 02 May 2013 14:58:45 +0000http://status.alwaysdata.com/operation/101/detail/This is most certainly a hardware issue.Sun, 05 May 2013 20:46:11 +0000http://status.alwaysdata.com/operation/101/detail/Server is up again. The motherboard seems to be failing, we'll replace it soon.Sun, 05 May 2013 20:49:04 +0000http://status.alwaysdata.com/operation/102/detail/The motherboard seems to have caused the 2 latest downtimes on http10. It will be replaced.Mon, 06 May 2013 08:05:26 +0000http://status.alwaysdata.com/operation/102/detail/The operation has started.Mon, 06 May 2013 22:10:47 +0000http://status.alwaysdata.com/operation/102/detail/The operation has completed successfully.Mon, 06 May 2013 22:43:17 +0000http://status.alwaysdata.com/operation/103/detail/We're investigating.Tue, 07 May 2013 08:26:03 +0000http://status.alwaysdata.com/operation/103/detail/It looks like a hardware problem. A technician will go inspect the machine.Tue, 07 May 2013 08:28:15 +0000http://status.alwaysdata.com/operation/103/detail/The server is back online, MySQL is recovering.Tue, 07 May 2013 08:34:36 +0000http://status.alwaysdata.com/operation/103/detail/Up again.Tue, 07 May 2013 08:37:49 +0000http://status.alwaysdata.com/operation/100/detail/Several accounts are being migrated from backup6 to backup9.Tue, 07 May 2013 15:48:36 +0000http://status.alwaysdata.com/operation/100/detail/More than half of all accounts initially on backup6 have been migrated on backup9.Sun, 12 May 2013 12:49:55 +0000http://status.alwaysdata.com/operation/100/detail/Everything is now working as normal on all backup servers, including backup6.Mon, 13 May 2013 08:35:36 +0000http://status.alwaysdata.com/operation/104/detail/These servers will be rebooted over the next few minutes.Wed, 15 May 2013 22:53:51 +0000http://status.alwaysdata.com/operation/105/detail/We are investigating.Sat, 18 May 2013 21:14:08 +0000http://status.alwaysdata.com/operation/105/detail/Server is up again. Hardware issue, we will communicate on this problem as soon as we've got more details.Sat, 18 May 2013 21:44:14 +0000http://status.alwaysdata.com/operation/106/detail/We're investigating.Mon, 20 May 2013 21:05:56 +0000http://status.alwaysdata.com/operation/106/detail/This is mostly a false positive due to network changes made by our provider.Mon, 20 May 2013 21:16:21 +0000http://status.alwaysdata.com/operation/107/detail/A new DDoS protection device is causing connection issues on a small proportion of our clients. We're working on solving these problems.Tue, 21 May 2013 10:51:52 +0000http://status.alwaysdata.com/operation/107/detail/We're redirecting the traffic to a non-protected IP address until all remaining issues are resolved.Tue, 21 May 2013 11:01:00 +0000http://status.alwaysdata.com/operation/107/detail/A change has been applied to the protection device. Clients that had issues no longer have trouble connecting to the server.Tue, 21 May 2013 11:37:36 +0000http://status.alwaysdata.com/operation/108/detail/http4 has issues with our internal network. We're using the external network for now until we have more details.Wed, 22 May 2013 11:54:47 +0000http://status.alwaysdata.com/operation/108/detail/The internal network is up again. The issue was due to a faulty optical link.Wed, 22 May 2013 13:41:19 +0000http://status.alwaysdata.com/operation/109/detail/We're investigating.Thu, 13 Jun 2013 08:49:36 +0000http://status.alwaysdata.com/operation/109/detail/We've found the issue, it should be resolved in a few minutes.Thu, 13 Jun 2013 09:00:19 +0000http://status.alwaysdata.com/operation/109/detail/Issue resolved. More details will follow shortly.Thu, 13 Jun 2013 09:14:34 +0000http://status.alwaysdata.com/operation/109/detail/All emails delayed will be received later, as the sender hosts will retry to deliver their messages. Here's what happened: our main database server has been automatically protected by a DDoS protection device by our provider due to a small scale attack. This mitigation device unexpectedly blocked the PostgreSQL traffic between some of our servers. Unable to contact the main database, our mail servers refused all incoming mails. We have a hot standby database server, in case the main database server is unreachable. Unfortunately, due to a misconfiguration in our firewalls, that server was not accessible. This was fixed. The traffic from our MX servers to our main database server has been moved inside our internal VLAN to avoid issues with the mitigation device. We're also working on radical changes to avoid such issues from our provider to happen again. We'll communicate on this in a few weeks.Thu, 13 Jun 2013 09:38:13 +0000http://status.alwaysdata.com/operation/110/detail/Network issue, we're investigating.Thu, 13 Jun 2013 09:52:36 +0000http://status.alwaysdata.com/operation/110/detail/Only accounts on imap1 are unreachable. There's an issue on a router from our provider.Thu, 13 Jun 2013 10:00:36 +0000http://status.alwaysdata.com/operation/110/detail/The issue is still ongoing.Thu, 13 Jun 2013 10:21:26 +0000http://status.alwaysdata.com/operation/110/detail/The issue is fixed.Thu, 13 Jun 2013 10:59:50 +0000http://status.alwaysdata.com/operation/111/detail/Kernel panic. We've rebooted the server.Mon, 17 Jun 2013 22:15:29 +0000http://status.alwaysdata.com/operation/111/detail/Once again. We're investigating.Mon, 17 Jun 2013 22:32:25 +0000http://status.alwaysdata.com/operation/111/detail/MySQL is suddently using more than the available RAM, which causes the issue. We're still investigating why.Mon, 17 Jun 2013 22:38:07 +0000http://status.alwaysdata.com/operation/111/detail/MySQL has been upgraded, to no avail. We're still investigating and trying solutions.Mon, 17 Jun 2013 23:02:25 +0000http://status.alwaysdata.com/operation/111/detail/We've blocked SQL requests from http10 to mysql2 as a specific account on this server is probably triggering a MySQL bug.Mon, 17 Jun 2013 23:13:19 +0000http://status.alwaysdata.com/operation/111/detail/We may have isolated the account sending the requests that trigger the bug.Mon, 17 Jun 2013 23:25:39 +0000http://status.alwaysdata.com/operation/111/detail/The account has been confirmed. The customer has been contacted and we will investigate more together tomorrow.Mon, 17 Jun 2013 23:41:43 +0000http://status.alwaysdata.com/operation/111/detail/After more investigation, we've isolated the query triggering the bug. It's actually rather simple, but an index was recently created and caused the bug. This is a known issue with MySQL. Note that PostgreSQL cannot suffer from such issues (memory leak) thanks to its multi-process model.Tue, 18 Jun 2013 11:00:18 +0000http://status.alwaysdata.com/operation/112/detail/We're investigating.Sat, 22 Jun 2013 03:51:47 +0000http://status.alwaysdata.com/operation/112/detail/The IP 178.32.28.120 is not reachable. All other IP address still work fine. We're temporarily updating the DNS zones to point to another IP address.Sat, 22 Jun 2013 04:03:29 +0000http://status.alwaysdata.com/operation/112/detail/The IP is accessible again. We're still waiting from our provider to get more details.Sat, 22 Jun 2013 10:08:47 +0000http://status.alwaysdata.com/operation/113/detail/Due to a DDoS attack.Tue, 25 Jun 2013 20:22:55 +0000http://status.alwaysdata.com/operation/113/detail/The attack is now under control.Tue, 25 Jun 2013 20:55:18 +0000http://status.alwaysdata.com/operation/114/detail/The hardware will be upgraded. The server will be in read-only mode for about 30 minutes.Thu, 04 Jul 2013 15:32:58 +0000http://status.alwaysdata.com/operation/114/detail/The migration is starting.Sun, 07 Jul 2013 22:16:15 +0000http://status.alwaysdata.com/operation/114/detail/The server will actually need to be put offline rather than in read-only mode.Sun, 07 Jul 2013 22:18:17 +0000http://status.alwaysdata.com/operation/114/detail/The server is accessible again, but the migration is not complete yet.Sun, 07 Jul 2013 22:41:51 +0000http://status.alwaysdata.com/operation/114/detail/The migration has completed. Backups from databases on mysql1 will exceptionnally not be stored in your ~/admin/backup directory for today, although we do keep an internal backup if necessary.Sun, 07 Jul 2013 23:19:41 +0000http://status.alwaysdata.com/operation/115/detail/The kernel will be upgraded.Thu, 11 Jul 2013 21:21:57 +0000http://status.alwaysdata.com/operation/115/detail/The kernel has been successfully upgraded.Thu, 11 Jul 2013 21:27:17 +0000http://status.alwaysdata.com/operation/116/detail/One of the disks is showing errors. We will have to restart the server. Expected downtime: 5 minutes.Tue, 16 Jul 2013 21:13:50 +0000http://status.alwaysdata.com/operation/116/detail/The server is being rebooted.Tue, 16 Jul 2013 21:52:01 +0000http://status.alwaysdata.com/operation/116/detail/We're seeing weird disk issues.Tue, 16 Jul 2013 21:56:39 +0000http://status.alwaysdata.com/operation/116/detail/It looks like we have a bad disk corruption issue. Still investigating.Tue, 16 Jul 2013 22:02:08 +0000http://status.alwaysdata.com/operation/116/detail/The data are most certainly unaccessible and we will have to load the backups. More details will follow later.Tue, 16 Jul 2013 22:13:25 +0000http://status.alwaysdata.com/operation/116/detail/It will most certainly take several hours to recover from the backups. We're very sorry about this.Tue, 16 Jul 2013 22:20:12 +0000http://status.alwaysdata.com/operation/116/detail/We're still working on restoring data.Tue, 16 Jul 2013 23:49:51 +0000http://status.alwaysdata.com/operation/116/detail/Several database files seem to be readable and healthy. We will try to recover as much as we can from the old disk (with up-to-date data). What cannot be recovered this way will have to be restored from the backups. We're still several hours away from having a working mysql1 server.Wed, 17 Jul 2013 01:17:24 +0000http://status.alwaysdata.com/operation/116/detail/It looks like a large proportion of databases could be recovered without any loss. We're still not sure at this point.Wed, 17 Jul 2013 02:51:35 +0000http://status.alwaysdata.com/operation/116/detail/We hope to have the first databases recovered and accessible in 2 hours. Loading all databases will take a few more hours.Wed, 17 Jul 2013 03:26:07 +0000http://status.alwaysdata.com/operation/116/detail/We've started to import recovered databases. We don't have an ETA yet; at least a couple of hours.Wed, 17 Jul 2013 05:07:08 +0000http://status.alwaysdata.com/operation/116/detail/The import process is still ongoing and will unfortunately take more time than we initially hoped. Note that all customers hosting databases on mysql1 can ask us for a full refund for this month, as specified in our terms of services. We would like to apologize again for this very long service interruption.Wed, 17 Jul 2013 06:03:21 +0000http://status.alwaysdata.com/operation/116/detail/We're working on speeding up the import process and evaluating our options.Wed, 17 Jul 2013 06:39:06 +0000http://status.alwaysdata.com/operation/116/detail/About 30% of all databases have been recovered and are now accessible.Wed, 17 Jul 2013 06:59:18 +0000http://status.alwaysdata.com/operation/116/detail/About 80% of all databases have now been recovered.Wed, 17 Jul 2013 08:35:11 +0000http://status.alwaysdata.com/operation/116/detail/99% of non-corrupted databases have been restored. We're now restoring the corrupted databases from the last backups.Wed, 17 Jul 2013 09:16:35 +0000http://status.alwaysdata.com/operation/116/detail/Still working to restore corrupted databases. We manually check each of them.Wed, 17 Jul 2013 10:46:09 +0000http://status.alwaysdata.com/operation/116/detail/All databases have now been imported. If something is still wrong with your database, please open a support ticket and we'll help you. More than 96% of the databases on mysql1 didn't suffer any corruption, only an extended outage. The remaining 4% were partially or totally corrupted and had to be restored from the backups. We're very sorry about this very serious issue. We'll write a blog post to explain in details what happened.Wed, 17 Jul 2013 12:11:05 +0000http://status.alwaysdata.com/operation/116/detail/Note that among the 4%, a few databases may STILL be partially corrupted. For those, there are basically 2 options: either recover from a backup, or try to fix the current database with our help. Contact us if you fall in this category.Wed, 17 Jul 2013 12:34:58 +0000http://status.alwaysdata.com/operation/117/detail/We're having authentication issues to a repeated crash. We're investigating. In the meantime, if you get authentication errors, please try again.Wed, 31 Jul 2013 11:50:08 +0000http://status.alwaysdata.com/operation/117/detail/We're still trying to figure out why Dovecot's auth process is repeatedly crashing, although nothing has changed for months.Wed, 31 Jul 2013 12:23:42 +0000http://status.alwaysdata.com/operation/117/detail/We think we've found the issue. Still not 100% sure.Wed, 31 Jul 2013 12:51:02 +0000http://status.alwaysdata.com/operation/117/detail/Issue solved. Our internal authentication database was flooded by requests due to a client sending spams (account hacked), which increased the response time up to Dovecot's internal timeout, which eventually triggered the bug. Isolating our internal database is in our TODO.Wed, 31 Jul 2013 13:12:33 +0000http://status.alwaysdata.com/operation/118/detail/We're investigating.Thu, 01 Aug 2013 11:52:41 +0000http://status.alwaysdata.com/operation/118/detail/One disk had temporary issues, which caused the performance hit.Thu, 01 Aug 2013 12:05:07 +0000http://status.alwaysdata.com/operation/119/detail/The mysql2 server had a temporary network issue for 5 minutes.Sat, 03 Aug 2013 02:25:41 +0000http://status.alwaysdata.com/operation/120/detail/This seems to be caused by a kernel issue. We're investigating.Tue, 06 Aug 2013 05:25:44 +0000http://status.alwaysdata.com/operation/120/detail/The server is back online, but we're still seeing issues related to NFS.Tue, 06 Aug 2013 05:48:01 +0000http://status.alwaysdata.com/operation/120/detail/We're rebooting http6, which seems to have a NFS problem.Tue, 06 Aug 2013 05:57:23 +0000http://status.alwaysdata.com/operation/120/detail/Rebooting http6 solved the underlying issue, due to a NFS kernel bug.Tue, 06 Aug 2013 06:11:28 +0000http://status.alwaysdata.com/operation/121/detail/nfsd has crashed on this server, we'll have to restart it.Fri, 09 Aug 2013 08:32:03 +0000http://status.alwaysdata.com/operation/121/detail/The server has been restarted.Fri, 09 Aug 2013 08:40:02 +0000http://status.alwaysdata.com/operation/122/detail/We are investigating.Mon, 19 Aug 2013 20:11:40 +0000http://status.alwaysdata.com/operation/122/detail/Rebooting http11 solved the issue, it was probably a kernel one.Mon, 19 Aug 2013 20:16:31 +0000http://status.alwaysdata.com/operation/122/detail/A kernel upgrade will be done in the next few minutes.Mon, 19 Aug 2013 21:55:49 +0000http://status.alwaysdata.com/operation/122/detail/The kernel upgrade is done.Mon, 19 Aug 2013 22:07:58 +0000http://status.alwaysdata.com/operation/123/detail/We're investigating.Wed, 21 Aug 2013 11:48:44 +0000http://status.alwaysdata.com/operation/123/detail/Kernel crash, the server is rebooting.Wed, 21 Aug 2013 11:49:38 +0000http://status.alwaysdata.com/operation/123/detail/INFO: rcu_sched self-detected stall on CPU The server is back online.Wed, 21 Aug 2013 11:53:24 +0000http://status.alwaysdata.com/operation/124/detail/We're investigating.Sat, 24 Aug 2013 06:32:14 +0000http://status.alwaysdata.com/operation/124/detail/There's an issue in the rack, probably network related.Sat, 24 Aug 2013 06:35:31 +0000http://status.alwaysdata.com/operation/124/detail/We're still waiting for more details at this point. A couple of other racks seem to have issues as well.Sat, 24 Aug 2013 06:51:05 +0000http://status.alwaysdata.com/operation/124/detail/Racks are still down, we don't have any ETA yet. We're sorry for this.Sat, 24 Aug 2013 07:06:33 +0000http://status.alwaysdata.com/operation/124/detail/The server is up again. We'll update this operation as soon as we know what happened. We're very sorry for this rather long and unexpected downtime. We will soon make an announcement regarding future changes that will definitely reduce such downtimes.Sat, 24 Aug 2013 07:44:52 +0000http://status.alwaysdata.com/operation/125/detail/The power supply is defective and needs to be replaced. It was responsible for the outage last week.Wed, 28 Aug 2013 13:55:39 +0000http://status.alwaysdata.com/operation/126/detail/There are network issues in the datacenter.Thu, 29 Aug 2013 18:08:34 +0000http://status.alwaysdata.com/operation/126/detail/The technicians are still trying to fix the routing issues. There are apparently facing multiple problems.Thu, 29 Aug 2013 18:20:25 +0000http://status.alwaysdata.com/operation/126/detail/Email accounts on imap1 are still unavailable. No ETA yet.Thu, 29 Aug 2013 19:18:47 +0000http://status.alwaysdata.com/operation/126/detail/imap1 is accessible again.Thu, 29 Aug 2013 20:01:08 +0000http://status.alwaysdata.com/operation/125/detail/The operation has started.Fri, 30 Aug 2013 04:05:47 +0000http://status.alwaysdata.com/operation/125/detail/The server is up again, the operation terminated successfully.Fri, 30 Aug 2013 04:20:41 +0000http://status.alwaysdata.com/operation/127/detail/Kernel issue. We're rebooting the server.Sat, 31 Aug 2013 23:16:41 +0000http://status.alwaysdata.com/operation/128/detail/This server has filesystem issues. No file was lost or corrupted, but we'll have to transfer the files over to another server. Backups are suspended on this server until further notice.Fri, 06 Sep 2013 18:03:57 +0000http://status.alwaysdata.com/operation/128/detail/80% of accounts have been migrated.Sun, 08 Sep 2013 20:55:42 +0000http://status.alwaysdata.com/operation/129/detail/Caused by a kernel issue.Mon, 09 Sep 2013 15:46:21 +0000http://status.alwaysdata.com/operation/128/detail/All accounts have been migrated.Mon, 09 Sep 2013 21:38:14 +0000http://status.alwaysdata.com/operation/130/detail/We're investigating.Fri, 13 Sep 2013 12:10:51 +0000http://status.alwaysdata.com/operation/130/detail/The server is up again. It was due to client abuse. We will take the necessary measures. The server will still slow down for a few minutes.Fri, 13 Sep 2013 12:40:01 +0000http://status.alwaysdata.com/operation/131/detail/We're investigating.Wed, 25 Sep 2013 11:57:02 +0000http://status.alwaysdata.com/operation/131/detail/Most likely a disk issue. We're rebooting the server.Wed, 25 Sep 2013 11:59:32 +0000http://status.alwaysdata.com/operation/131/detail/The server has restarted successfully. We'll schedule a disk replacement.Wed, 25 Sep 2013 12:05:23 +0000http://status.alwaysdata.com/operation/132/detail/A faulty hard drive will be replaced. The server will be offline during the operation.Sat, 28 Sep 2013 21:36:11 +0000http://status.alwaysdata.com/operation/132/detail/The operation has started.Sat, 28 Sep 2013 22:31:25 +0000http://status.alwaysdata.com/operation/132/detail/The disk has been replaced, the server has restarted.Sat, 28 Sep 2013 22:38:44 +0000http://status.alwaysdata.com/operation/134/detail/We're investigating.Tue, 08 Oct 2013 09:50:56 +0000http://status.alwaysdata.com/operation/134/detail/Up again. A memory usage spike caused slowdowns.Tue, 08 Oct 2013 09:56:00 +0000http://status.alwaysdata.com/operation/135/detail/Nous enquêtons.Wed, 16 Oct 2013 23:14:52 +0000http://status.alwaysdata.com/operation/135/detail/Mise à jour des switchs du réseau. Votre serveur est de nouveau disponible. Wed, 16 Oct 2013 23:20:07 +0000http://status.alwaysdata.com/operation/137/detail/Our internal VLAN has issues, which caused mostly SQL errors. We've switched to the external network until the underlying issue is fixed.Thu, 17 Oct 2013 15:34:29 +0000http://status.alwaysdata.com/operation/138/detail/We're investigating.Sat, 19 Oct 2013 14:01:05 +0000http://status.alwaysdata.com/operation/138/detail/The server is frozen. We're rebooting it.Sat, 19 Oct 2013 14:05:48 +0000http://status.alwaysdata.com/operation/138/detail/The server has restarted, MySQL is recovering.Sat, 19 Oct 2013 14:08:31 +0000http://status.alwaysdata.com/operation/138/detail/The recovery is done, MySQL is up again.Sat, 19 Oct 2013 14:18:08 +0000http://status.alwaysdata.com/operation/139/detail/In consequence, SSH and FTP is malfunctioning for http7 accounts.Tue, 22 Oct 2013 13:21:26 +0000http://status.alwaysdata.com/operation/139/detail/http7 will be rebooted.Tue, 22 Oct 2013 13:34:16 +0000http://status.alwaysdata.com/operation/139/detail/The reboot fixed the (kernel) issue.Tue, 22 Oct 2013 14:04:44 +0000http://status.alwaysdata.com/operation/140/detail/We're investigating.Wed, 23 Oct 2013 09:49:38 +0000http://status.alwaysdata.com/operation/140/detail/Looks like a kernel or hardware issue. We're rebooting the server.Wed, 23 Oct 2013 09:53:52 +0000http://status.alwaysdata.com/operation/140/detail/The server has rebooted successfully.Wed, 23 Oct 2013 09:58:32 +0000http://status.alwaysdata.com/operation/141/detail/Caused by a kernel issue.Thu, 24 Oct 2013 06:45:37 +0000http://status.alwaysdata.com/operation/141/detail/There's an issue at the reboot. We're investigating.Thu, 24 Oct 2013 06:48:04 +0000http://status.alwaysdata.com/operation/141/detail/We're facing a known kernel issue with Intel 10Gb cards. We're working on solving it.Thu, 24 Oct 2013 07:07:33 +0000http://status.alwaysdata.com/operation/141/detail/Should be OK in the next few minutes.Thu, 24 Oct 2013 07:24:31 +0000http://status.alwaysdata.com/operation/141/detail/The server is up again.Thu, 24 Oct 2013 07:30:18 +0000http://status.alwaysdata.com/operation/143/detail/The main IP address was down due to a client abuse. It's now fixed.Wed, 30 Oct 2013 12:44:38 +0000http://status.alwaysdata.com/operation/144/detail/We're investigating.Sat, 02 Nov 2013 01:17:04 +0000http://status.alwaysdata.com/operation/144/detail/The kernel is frozen. We're rebooting the server.Sat, 02 Nov 2013 01:20:51 +0000http://status.alwaysdata.com/operation/147/detail/We're investigating.Tue, 12 Nov 2013 14:09:26 +0000http://status.alwaysdata.com/operation/147/detail/There was a pretty big IO spike that caused slowdowns. It's getting better.Tue, 12 Nov 2013 14:22:22 +0000http://status.alwaysdata.com/operation/148/detail/We're investigating.Thu, 21 Nov 2013 16:49:34 +0000http://status.alwaysdata.com/operation/148/detail/The server is frozen. We're restarting it.Thu, 21 Nov 2013 16:52:37 +0000http://status.alwaysdata.com/operation/148/detail/The server has restarted, MySQL is now recovering.Thu, 21 Nov 2013 16:59:37 +0000http://status.alwaysdata.com/operation/148/detail/It's still recovering. As there are many databases, it takes some time.Thu, 21 Nov 2013 17:08:53 +0000http://status.alwaysdata.com/operation/148/detail/The recovery is done, the kernel has been upgraded. MySQL is (slowly) starting responding requests again.Thu, 21 Nov 2013 17:12:33 +0000http://status.alwaysdata.com/operation/148/detail/The RAID reconstruction causes an increased latency on the server. It is expected to take 4 more days.Fri, 22 Nov 2013 11:43:57 +0000http://status.alwaysdata.com/operation/148/detail/The server has frozen again.Fri, 22 Nov 2013 21:34:12 +0000http://status.alwaysdata.com/operation/148/detail/We'll install another kernel, but we suspect a hardware failure. MySQL is recovering.Fri, 22 Nov 2013 21:46:40 +0000http://status.alwaysdata.com/operation/148/detail/We've stopped the RAID reconstruction for now. We'll need to know more about the freezes.Fri, 22 Nov 2013 21:54:03 +0000http://status.alwaysdata.com/operation/149/detail/We're investigating.Tue, 03 Dec 2013 23:06:17 +0000http://status.alwaysdata.com/operation/149/detail/Up again. A network issue caused the downtime.Tue, 03 Dec 2013 23:11:27 +0000http://status.alwaysdata.com/operation/150/detail/Due to a customer abuse, the server is experiencing slowdowns. We've suspended the customer, everything will be back to normal in the next few minutes.Tue, 10 Dec 2013 10:36:57 +0000http://status.alwaysdata.com/operation/151/detail/We're investigating.Tue, 17 Dec 2013 17:10:45 +0000http://status.alwaysdata.com/operation/151/detail/Looks like a kernel semi-freeze, we're rebooting the server.Tue, 17 Dec 2013 17:12:20 +0000http://status.alwaysdata.com/operation/151/detail/The server has restarted, the services are up again. It will remain slow for a few minutes.Tue, 17 Dec 2013 17:14:45 +0000http://status.alwaysdata.com/operation/152/detail/NFS is down, most likely due to a kernel bug. SSH and FTP accesses are down for now. We'll have to restart the server and upgrade the kernel.Thu, 26 Dec 2013 17:24:39 +0000http://status.alwaysdata.com/operation/152/detail/The server has been restarted.Thu, 26 Dec 2013 17:31:53 +0000http://status.alwaysdata.com/operation/155/detail/We're investigating.Thu, 09 Jan 2014 15:26:47 +0000http://status.alwaysdata.com/operation/155/detail/The server became unresponsive after an IO spike. We've had to reboot it.Thu, 09 Jan 2014 15:37:22 +0000http://status.alwaysdata.com/operation/156/detail/We're investigating.Mon, 13 Jan 2014 21:47:06 +0000http://status.alwaysdata.com/operation/156/detail/The server didn't reboot correctly. We're still investigating.Mon, 13 Jan 2014 21:54:42 +0000http://status.alwaysdata.com/operation/156/detail/The server is up again.Mon, 13 Jan 2014 22:16:26 +0000http://status.alwaysdata.com/operation/157/detail/We will reboot the server.Tue, 14 Jan 2014 10:14:27 +0000http://status.alwaysdata.com/operation/158/detail/We're investigating.Fri, 17 Jan 2014 22:03:14 +0000http://status.alwaysdata.com/operation/158/detail/We have rebooted the server, it will slow down a few minutes.Fri, 17 Jan 2014 22:12:33 +0000http://status.alwaysdata.com/operation/158/detail/It was a memory issue, we have taken measures against the responsible.Fri, 17 Jan 2014 22:34:47 +0000http://status.alwaysdata.com/operation/159/detail/One hard drive will be replaced tonight.Sun, 19 Jan 2014 13:53:22 +0000http://status.alwaysdata.com/operation/159/detail/The operation has started.Sun, 19 Jan 2014 18:06:45 +0000http://status.alwaysdata.com/operation/159/detail/The server fails to boot properly. We're working on it.Sun, 19 Jan 2014 18:31:49 +0000http://status.alwaysdata.com/operation/159/detail/The operation has completed.Sun, 19 Jan 2014 18:38:38 +0000http://status.alwaysdata.com/operation/160/detail/Due to a kernel lockup. We're rebooting it.Mon, 20 Jan 2014 11:55:42 +0000http://status.alwaysdata.com/operation/160/detail/The server is up again.Mon, 20 Jan 2014 12:01:04 +0000http://status.alwaysdata.com/operation/161/detail/We're investigating.Tue, 21 Jan 2014 10:48:40 +0000http://status.alwaysdata.com/operation/161/detail/It looks like a kernel crash. We're restarting the server.Tue, 21 Jan 2014 10:51:19 +0000http://status.alwaysdata.com/operation/161/detail/The server is up again, although it will be slow for several minutes.Tue, 21 Jan 2014 10:57:29 +0000http://status.alwaysdata.com/operation/162/detail/Same issue as yesterday (kernel lock), we'll have to upgrade the kernel. In the meantime, the server is being restarted.Tue, 21 Jan 2014 15:39:24 +0000http://status.alwaysdata.com/operation/163/detail/The server will be rebooted in the next few minutes.Fri, 24 Jan 2014 22:59:30 +0000http://status.alwaysdata.com/operation/166/detail/We are investigating.Thu, 30 Jan 2014 23:19:28 +0000http://status.alwaysdata.com/operation/166/detail/A user caused a massive disk usage spike. It's been taken care of.Fri, 31 Jan 2014 00:06:55 +0000http://status.alwaysdata.com/operation/167/detail/The http6 server will be migrated to a new machine today. Each account will be transparently migrated to the new machine, and then a final switch will be made tonight. Total downtime expected: less than 5 minutes. SSH and FTP file accesses may randomly return errors during the migration. It usually doesn't last very long. Don't hesitate to contact us if you have questions.Sun, 09 Feb 2014 10:15:47 +0000http://status.alwaysdata.com/operation/167/detail/The final switch will be made over the next few minutes.Sun, 09 Feb 2014 20:18:12 +0000http://status.alwaysdata.com/operation/167/detail/The services are currently down. Unmounting the /home filesystem is taking an unusual amount of time.Sun, 09 Feb 2014 20:41:33 +0000http://status.alwaysdata.com/operation/167/detail/The migration has completed.Sun, 09 Feb 2014 21:03:24 +0000http://status.alwaysdata.com/operation/168/detail/We're investigating.Mon, 10 Feb 2014 09:13:10 +0000http://status.alwaysdata.com/operation/168/detail/The server is up again, we're stil analyzing what happened.Mon, 10 Feb 2014 09:22:07 +0000http://status.alwaysdata.com/operation/168/detail/Down again.Mon, 10 Feb 2014 09:26:50 +0000http://status.alwaysdata.com/operation/168/detail/A network attack is causing the trouble. We're working on mitigating it and trying to figure out why our firewall didn't work as expected.Mon, 10 Feb 2014 09:39:35 +0000http://status.alwaysdata.com/operation/168/detail/The traffic is still perturbed.Mon, 10 Feb 2014 09:57:36 +0000http://status.alwaysdata.com/operation/168/detail/We're obviously still working on resolving this issue.Mon, 10 Feb 2014 11:08:08 +0000http://status.alwaysdata.com/operation/168/detail/We think we've found the problem.Mon, 10 Feb 2014 11:32:35 +0000http://status.alwaysdata.com/operation/168/detail/We're pretty sure the problem is identified. Another unrelated problem has appeared, we're working on it.Mon, 10 Feb 2014 11:39:45 +0000http://status.alwaysdata.com/operation/168/detail/A quotacheck is running on the /home filesystem, it may take some time.Mon, 10 Feb 2014 11:53:18 +0000http://status.alwaysdata.com/operation/168/detail/The quotacheck is done, but we're still fixing an isuse.Mon, 10 Feb 2014 12:13:48 +0000http://status.alwaysdata.com/operation/168/detail/We're still fighting a mdadm issue. The data is safe.Mon, 10 Feb 2014 12:40:06 +0000http://status.alwaysdata.com/operation/168/detail/We've disabled the RAID for now, the server is up.Mon, 10 Feb 2014 12:58:21 +0000http://status.alwaysdata.com/operation/168/detail/The server is up, everything is working as normal. We'll close the operation for now, but details will be posted later. The RAID is still degraded, but we'll keep it this way until we find a way to fix it properly.Mon, 10 Feb 2014 14:05:34 +0000http://status.alwaysdata.com/operation/168/detail/Here is what happened: yesterday, the http6 server was migrated over a new machine. Everything was working properly until this morning. At 09:45 today, the server started getting random network issues (broken connections, timeouts, weird traffic). Debugging was a bit more difficult than usual since we didn't have a functional SSH access. The only relevant kernel log message was: net_ratelimit: 2127 callbacks suppressed Such messages indicate that the kernel avoided repeating the same messages again and again. We didn't have any other message, though, which is unexpected. Since we couldn't exploit those messages, we tried to discover the cause of the issue by other means. We initially thought it could be a network attack, despite our protections. The network traffic was suspicious. After much investigation, we came to the conclusion it wasn't an attack. We pursued our debugging for a while, to no avail. We decided to downgrade the kernel version from 3.10 to 3.4. The 3.4 kernel was reporting new log messages: IPv4: Neighbour table overflow That helped us quickly notice a typo in a network configuration file: the default gateway was pointing to the server itself, rather than the real gateway (the reality was actually a bit more complex). The neighbour table was filled with non-local IP addresses. Everything worked fine until this morning, when the number of connections increased. We still don't know why the 3.10 kernel didn't report these log messages. We'll have to investigate the RAID issues as well, which started appearing during the kernel downgrade. We'll obviously take lessons from that outage. An automated debugging tool may prove useful in unusually weird situations such as this one.Mon, 10 Feb 2014 18:38:40 +0000http://status.alwaysdata.com/operation/169/detail/Due to a high CPU consumption the server was down. The person responsible for this issue has been suspended.Fri, 14 Feb 2014 17:39:48 +0000http://status.alwaysdata.com/operation/170/detail/We're investigating.Tue, 18 Feb 2014 10:13:59 +0000http://status.alwaysdata.com/operation/171/detail/We're investigating.Sat, 22 Feb 2014 23:22:48 +0000http://status.alwaysdata.com/operation/171/detail/The main IP address for this server was suspended due to abuse. We've redirected the DNS to another IP until the IP is unblocked.Sat, 22 Feb 2014 23:34:58 +0000http://status.alwaysdata.com/operation/172/detail/Due to a memory abuse. We are restarting the server.Mon, 24 Feb 2014 09:59:51 +0000http://status.alwaysdata.com/operation/172/detail/The server is up but experiencing slow downs due to the RAID reconstruction.Mon, 24 Feb 2014 10:21:14 +0000http://status.alwaysdata.com/operation/174/detail/We're investigating.Mon, 10 Mar 2014 11:08:33 +0000http://status.alwaysdata.com/operation/174/detail/It was due to a temporary overload. We will migrate few accounts.Mon, 10 Mar 2014 11:36:59 +0000http://status.alwaysdata.com/operation/175/detail/The NFS daemon is unstable, we'll have to reboot the server. We'll upgrade the kernel as well.Mon, 17 Mar 2014 12:02:52 +0000http://status.alwaysdata.com/operation/175/detail/The server has rebooted successfully.Mon, 17 Mar 2014 12:08:25 +0000http://status.alwaysdata.com/operation/176/detail/We're investigating.Tue, 18 Mar 2014 09:12:24 +0000http://status.alwaysdata.com/operation/176/detail/The database was locked, it's now recovering.Tue, 18 Mar 2014 09:16:58 +0000http://status.alwaysdata.com/operation/176/detail/The server is up again.Tue, 18 Mar 2014 09:20:06 +0000http://status.alwaysdata.com/operation/176/detail/MySQL was hung due to a deadlock, we've had to restart it. We suspect a network issue being the cause, as we've had a similar weird issue with another server at the exact same time.Tue, 18 Mar 2014 09:28:13 +0000http://status.alwaysdata.com/operation/178/detail/We're investigating.Fri, 21 Mar 2014 15:51:07 +0000http://status.alwaysdata.com/operation/178/detail/The server is UP but will experience some slow downs. Fri, 21 Mar 2014 16:06:03 +0000http://status.alwaysdata.com/operation/178/detail/Due to a memory abuse. We will take tough measures against high memory consumers.Fri, 21 Mar 2014 16:13:09 +0000http://status.alwaysdata.com/operation/179/detail/We're working on mitigating the attack.Sat, 22 Mar 2014 22:45:59 +0000http://status.alwaysdata.com/operation/179/detail/The attack has been mitigated.Sat, 22 Mar 2014 23:01:38 +0000http://status.alwaysdata.com/operation/180/detail/We are investigating.Mon, 24 Mar 2014 19:13:43 +0000http://status.alwaysdata.com/operation/180/detail/It was a network issue. The server is reachable.Mon, 24 Mar 2014 19:17:31 +0000http://status.alwaysdata.com/operation/182/detail/IO is saturated, we're investigating.Wed, 26 Mar 2014 16:00:47 +0000http://status.alwaysdata.com/operation/182/detail/The performance is mostly back to normal.Wed, 26 Mar 2014 16:00:48 +0000http://status.alwaysdata.com/operation/183/detail/Due to a crash.Mon, 07 Apr 2014 15:30:01 +0000http://status.alwaysdata.com/operation/183/detail/MySQL is restarting.Mon, 07 Apr 2014 15:32:26 +0000http://status.alwaysdata.com/operation/183/detail/The server is up again.Mon, 07 Apr 2014 15:35:11 +0000http://status.alwaysdata.com/operation/184/detail/Due to a kernel crash. We're restarting the server.Tue, 08 Apr 2014 16:44:47 +0000http://status.alwaysdata.com/operation/184/detail/The server is up again, with an upgraded kernel.Tue, 08 Apr 2014 16:49:56 +0000http://status.alwaysdata.com/operation/185/detail/We're investigating.Tue, 08 Apr 2014 18:55:55 +0000http://status.alwaysdata.com/operation/185/detail/The server has been rebooted and is now reachable. It will experience slow down for a few minutes.Tue, 08 Apr 2014 19:05:09 +0000http://status.alwaysdata.com/operation/185/detail/It was due to memory issue.Tue, 08 Apr 2014 19:39:14 +0000http://status.alwaysdata.com/operation/186/detail/We are investigating.Fri, 11 Apr 2014 18:32:34 +0000http://status.alwaysdata.com/operation/186/detail/Bug kernel, the server has been rebooted.Fri, 11 Apr 2014 18:43:57 +0000http://status.alwaysdata.com/operation/186/detail/filesystem issue, we're investigating.Fri, 11 Apr 2014 18:45:53 +0000http://status.alwaysdata.com/operation/186/detail/we are running a filesystem checksums.Fri, 11 Apr 2014 18:52:09 +0000http://status.alwaysdata.com/operation/186/detail/The filesystem check is still running, it will probably take some time (1 hour or more).Fri, 11 Apr 2014 19:04:16 +0000http://status.alwaysdata.com/operation/186/detail/The filesystem check is now at the final phase (7/7), although it may still take a long time to complete.Fri, 11 Apr 2014 19:28:02 +0000http://status.alwaysdata.com/operation/186/detail/The filesystem check has completed successfully. The filesystem is now mounting, which takes some time as it recalculates the quota.Fri, 11 Apr 2014 19:36:11 +0000http://status.alwaysdata.com/operation/186/detail/The quota is still calculating.Fri, 11 Apr 2014 19:43:43 +0000http://status.alwaysdata.com/operation/186/detail/The mount seems stuck, we're restarting the server.Fri, 11 Apr 2014 19:59:02 +0000http://status.alwaysdata.com/operation/186/detail/We've disabled the quota on the filesystem, it mounted successfully. No file corruption seems to have occurred. The server is now up.Fri, 11 Apr 2014 20:02:29 +0000http://status.alwaysdata.com/operation/186/detail/The server IO is still slow, this is expected.Fri, 11 Apr 2014 20:08:12 +0000http://status.alwaysdata.com/operation/187/detail/The filesystem is in forced read-only due to a corruption. We'll copy the data to a new backup server. New backups on this server are suspended until the transfer is done, which will probably take at least a couple of days.Tue, 15 Apr 2014 16:48:50 +0000http://status.alwaysdata.com/operation/188/detail/mysql1 freezed, we are rebooting the server.Wed, 16 Apr 2014 20:04:18 +0000http://status.alwaysdata.com/operation/188/detail/MySQL is recovering.Wed, 16 Apr 2014 20:07:50 +0000http://status.alwaysdata.com/operation/188/detail/The server is up, mysql is recovering.Wed, 16 Apr 2014 20:08:11 +0000http://status.alwaysdata.com/operation/188/detail/Still recovering.Wed, 16 Apr 2014 20:16:57 +0000http://status.alwaysdata.com/operation/188/detail/MySQL is up again, although it will be slower than usual for a while.Wed, 16 Apr 2014 20:26:12 +0000http://status.alwaysdata.com/operation/188/detail/The server is now mostly back to its normal performance levels.Wed, 16 Apr 2014 21:40:47 +0000http://status.alwaysdata.com/operation/189/detail/We're investigating.Fri, 18 Apr 2014 05:54:29 +0000http://status.alwaysdata.com/operation/189/detail/MySQL crashed, it's restarting.Fri, 18 Apr 2014 05:56:34 +0000http://status.alwaysdata.com/operation/189/detail/MySQL is up.Fri, 18 Apr 2014 06:07:11 +0000http://status.alwaysdata.com/operation/189/detail/A probable filesystem corruption may have caused MySQL to crash. We'll schedule a kernel upgrade and filesystem repair tonight.Fri, 18 Apr 2014 06:08:29 +0000http://status.alwaysdata.com/operation/190/detail/Due to operation #189, mysql1 is slowed than usual. An operation is scheduled tonight to fix the underlying issue.Fri, 18 Apr 2014 08:48:03 +0000http://status.alwaysdata.com/operation/189/detail/Down again, we are rebooting the server.Fri, 18 Apr 2014 09:31:15 +0000http://status.alwaysdata.com/operation/189/detail/We are upgrading the kernel and checking the filesystem. Fri, 18 Apr 2014 09:36:01 +0000http://status.alwaysdata.com/operation/189/detail/MySQL is restarting.Fri, 18 Apr 2014 09:42:31 +0000http://status.alwaysdata.com/operation/189/detail/MySQL is now recovering.Fri, 18 Apr 2014 09:49:43 +0000http://status.alwaysdata.com/operation/189/detail/MySQL is up, but will remain slow for some time.Fri, 18 Apr 2014 09:52:28 +0000http://status.alwaysdata.com/operation/190/detail/We had to do the maintenance operation earlier than anticipated.Fri, 18 Apr 2014 10:22:33 +0000http://status.alwaysdata.com/operation/189/detail/The performance is mostly back to normal.Fri, 18 Apr 2014 10:52:53 +0000http://status.alwaysdata.com/operation/191/detail/Due to a corruption on one database, following yesterday's operations.Sat, 19 Apr 2014 09:19:21 +0000http://status.alwaysdata.com/operation/191/detail/MySQL is being restarted.Sat, 19 Apr 2014 09:24:31 +0000http://status.alwaysdata.com/operation/191/detail/MySQL is up again.Sat, 19 Apr 2014 09:31:06 +0000http://status.alwaysdata.com/operation/192/detail/We're fixing a few corruption issues following the previous issues. MySQL may crash during these operations.Sat, 19 Apr 2014 21:53:27 +0000http://status.alwaysdata.com/operation/192/detail/MySQL crashed, it's restarting.Sat, 19 Apr 2014 21:55:37 +0000http://status.alwaysdata.com/operation/192/detail/MySQL is up again. We have to continue the maintenance, though, so beware of new crashes.Sat, 19 Apr 2014 21:58:24 +0000http://status.alwaysdata.com/operation/192/detail/MySQL crashed again.Sat, 19 Apr 2014 22:05:38 +0000http://status.alwaysdata.com/operation/192/detail/It's up.Sat, 19 Apr 2014 22:06:56 +0000http://status.alwaysdata.com/operation/192/detail/The maintenance is now complete. The database corruptions, which were a result of the filesystem corruptions, should have been fixed.Sat, 19 Apr 2014 22:18:12 +0000http://status.alwaysdata.com/operation/187/detail/All accounts have been migrated to other backup servers. Backups from the last 30 days (that were on backup8) are not readable from ~/admin/backup as usual: if you need them, please open a support ticket.Sun, 20 Apr 2014 10:08:29 +0000http://status.alwaysdata.com/operation/193/detail/http9 is down, we're investigating.Thu, 24 Apr 2014 14:21:05 +0000http://status.alwaysdata.com/operation/193/detail/The server won't reboot, a technician will go and check the machine.Thu, 24 Apr 2014 14:35:37 +0000http://status.alwaysdata.com/operation/193/detail/The server has been hard rebooted manually, it's now up.Thu, 24 Apr 2014 15:04:26 +0000http://status.alwaysdata.com/operation/194/detail/We're investigating.Fri, 02 May 2014 15:06:21 +0000http://status.alwaysdata.com/operation/194/detail/The server is frozen, a technician will go inspect it.Fri, 02 May 2014 15:11:50 +0000http://status.alwaysdata.com/operation/194/detail/The server is up.Fri, 02 May 2014 15:37:28 +0000http://status.alwaysdata.com/operation/195/detail/We're investigating.Wed, 07 May 2014 21:38:58 +0000http://status.alwaysdata.com/operation/195/detail/It looks like a network issue.Wed, 07 May 2014 21:41:08 +0000http://status.alwaysdata.com/operation/195/detail/The whole rack is down. We're waiting for details from our provider.Wed, 07 May 2014 21:42:57 +0000http://status.alwaysdata.com/operation/195/detail/The server has restarted, MySQL is recovering.Wed, 07 May 2014 22:01:58 +0000http://status.alwaysdata.com/operation/196/detail/The kernel will be upgraded and the data partition will be extended.Sun, 11 May 2014 20:34:34 +0000http://status.alwaysdata.com/operation/196/detail/The operation has started.Sun, 11 May 2014 21:30:14 +0000http://status.alwaysdata.com/operation/196/detail/We're investigating a RAID issue.Sun, 11 May 2014 21:54:14 +0000http://status.alwaysdata.com/operation/196/detail/The server is up again, but the data partition extension is cancelled and will be rescheduled.Sun, 11 May 2014 21:55:35 +0000http://status.alwaysdata.com/operation/197/detail/We're investigating.Mon, 12 May 2014 08:25:37 +0000http://status.alwaysdata.com/operation/197/detail/We see an unusual IO activity but have yet to find what/who causes it.Mon, 12 May 2014 09:04:17 +0000http://status.alwaysdata.com/operation/197/detail/We've found the database that caused the high IO activity and suspended it. Everything is now back to normal.Mon, 12 May 2014 10:39:25 +0000http://status.alwaysdata.com/operation/198/detail/We're investigating.Tue, 13 May 2014 15:29:50 +0000http://status.alwaysdata.com/operation/198/detail/The kernel crashed, we're rebooting the server and will install an upgraded kernel.Tue, 13 May 2014 15:32:10 +0000http://status.alwaysdata.com/operation/198/detail/A quotacheck has started, we cannot interrupt it yet.Tue, 13 May 2014 15:37:06 +0000http://status.alwaysdata.com/operation/198/detail/The server is up again. It will be slow for a few minutes.Tue, 13 May 2014 15:47:22 +0000http://status.alwaysdata.com/operation/199/detail/We will have to reboot the server to complete the data partition growth.Mon, 19 May 2014 16:14:21 +0000http://status.alwaysdata.com/operation/199/detail/The operation has started.Mon, 19 May 2014 21:40:58 +0000http://status.alwaysdata.com/operation/199/detail/The server is up again. We'll grow the filesystem (online) in a few minutes.Mon, 19 May 2014 21:44:11 +0000http://status.alwaysdata.com/operation/199/detail/The filesystem has grown successfully. The downtime lasted about 3 minutes for the entire operation.Mon, 19 May 2014 22:07:02 +0000http://status.alwaysdata.com/operation/201/detail/We're investigating.Fri, 30 May 2014 23:29:41 +0000http://status.alwaysdata.com/operation/201/detail/The main IP address for this server has been temporarily blocked by our provider due to abuse. We're updating the DNS to use a new IP address. Clients having a dedicated IP address were not affected.Fri, 30 May 2014 23:42:38 +0000http://status.alwaysdata.com/operation/201/detail/The main IP address has been restored at 5:00.Sat, 31 May 2014 09:44:32 +0000http://status.alwaysdata.com/operation/202/detail/We're investigating.Thu, 05 Jun 2014 12:47:46 +0000http://status.alwaysdata.com/operation/202/detail/There's an issue on this server with our provider. We're still investigating.Thu, 05 Jun 2014 12:53:28 +0000http://status.alwaysdata.com/operation/202/detail/The server is up again (although still slow). For some reason, our provider decided to upgrade the IPMI firmware on this server. We're obviously not happy at all.Thu, 05 Jun 2014 12:58:14 +0000http://status.alwaysdata.com/operation/203/detail/We're investigating.Fri, 06 Jun 2014 13:49:54 +0000http://status.alwaysdata.com/operation/203/detail/The connections can be now established as usual.Fri, 06 Jun 2014 14:02:30 +0000http://status.alwaysdata.com/operation/205/detail/We're investigating.Tue, 10 Jun 2014 12:22:41 +0000http://status.alwaysdata.com/operation/205/detail/The kernel crashed. We have restarted the server, it's up again, although it will remain slower than usual for a while.Tue, 10 Jun 2014 12:36:00 +0000http://status.alwaysdata.com/operation/206/detail/Due to a kernel crash. The filesystem quota is currently being rechecked, which may take some time.Sat, 14 Jun 2014 10:37:18 +0000http://status.alwaysdata.com/operation/206/detail/We've had to cancel the quota recheck. The server is up.Sat, 14 Jun 2014 12:10:56 +0000http://status.alwaysdata.com/operation/207/detail/Due to a kernel crash. We're restarting the server.Sun, 15 Jun 2014 11:13:51 +0000http://status.alwaysdata.com/operation/208/detail/Due to network issues with our provider.Mon, 16 Jun 2014 09:59:40 +0000http://status.alwaysdata.com/operation/208/detail/postgresql1, SSH and FTP are affected for now.Mon, 16 Jun 2014 10:02:59 +0000http://status.alwaysdata.com/operation/208/detail/Actually, postgresql1 and http9.Mon, 16 Jun 2014 10:06:28 +0000http://status.alwaysdata.com/operation/208/detail/It's actually an issue with the power generators.Mon, 16 Jun 2014 10:11:05 +0000http://status.alwaysdata.com/operation/208/detail/http9 is up.Mon, 16 Jun 2014 10:40:34 +0000http://status.alwaysdata.com/operation/208/detail/postgresql1 is up.Mon, 16 Jun 2014 11:17:16 +0000http://status.alwaysdata.com/operation/209/detail/We're investigating.Fri, 20 Jun 2014 13:37:58 +0000http://status.alwaysdata.com/operation/209/detail/A memory spike caused a massive slowdown. It's recovering.Fri, 20 Jun 2014 13:45:57 +0000http://status.alwaysdata.com/operation/210/detail/We're rebooting the server.Sat, 21 Jun 2014 21:49:35 +0000http://status.alwaysdata.com/operation/211/detail/We're restarting MySQL on mysql1 as we suspect a bug.Mon, 23 Jun 2014 11:16:09 +0000http://status.alwaysdata.com/operation/211/detail/Done. The bug is gone.Mon, 23 Jun 2014 11:21:36 +0000http://status.alwaysdata.com/operation/212/detail/We're rebooting mysql.Tue, 24 Jun 2014 06:50:07 +0000http://status.alwaysdata.com/operation/212/detail/mysql is now up.Tue, 24 Jun 2014 06:55:30 +0000http://status.alwaysdata.com/operation/213/detail/We're investigating.Tue, 24 Jun 2014 21:03:23 +0000http://status.alwaysdata.com/operation/213/detail/A switch issue caused the server to be unreachable over the internal network. We've switched to the external network for the time being.Tue, 24 Jun 2014 21:04:57 +0000http://status.alwaysdata.com/operation/213/detail/We've switched back to the internal network.Wed, 25 Jun 2014 14:45:46 +0000http://status.alwaysdata.com/operation/214/detail/The same bug as yesterday made MySQL crash. It's restarting. We've identified the database that causes the bug.Wed, 25 Jun 2014 16:29:47 +0000http://status.alwaysdata.com/operation/214/detail/The database is up. We'll now try to fix the database.Wed, 25 Jun 2014 16:35:05 +0000http://status.alwaysdata.com/operation/214/detail/The database has been fixed.Wed, 25 Jun 2014 16:46:06 +0000http://status.alwaysdata.com/operation/216/detail/Due to high IO usage.Mon, 30 Jun 2014 14:36:39 +0000http://status.alwaysdata.com/operation/220/detail/Internal changes will be made during the day. No downtime expected.Thu, 24 Jul 2014 08:14:00 +0000http://status.alwaysdata.com/operation/221/detail/We're contacting our provider.Thu, 24 Jul 2014 14:32:07 +0000http://status.alwaysdata.com/operation/221/detail/We're still waiting for an update from the technician.Thu, 24 Jul 2014 15:18:19 +0000http://status.alwaysdata.com/operation/221/detail/The server is up.Thu, 24 Jul 2014 15:23:50 +0000http://status.alwaysdata.com/operation/220/detail/We're done for today, we'll continue tomorrow.Thu, 24 Jul 2014 15:32:14 +0000http://status.alwaysdata.com/operation/222/detail/All accounts on backup10 will be migrated to backup8. Older backups will not be accessible over SSH/FTP as usual, please contact us if you need them.Mon, 28 Jul 2014 08:23:02 +0000http://status.alwaysdata.com/operation/223/detail/We're contacting our provider.Tue, 05 Aug 2014 08:03:20 +0000http://status.alwaysdata.com/operation/223/detail/The server is up. The network adapter suddenly hanged, we're investigating.Tue, 05 Aug 2014 08:33:50 +0000http://status.alwaysdata.com/operation/224/detail/We're investigating.Mon, 18 Aug 2014 09:05:12 +0000http://status.alwaysdata.com/operation/224/detail/We've restarted MySQL, everything looks good now.Mon, 18 Aug 2014 09:07:26 +0000http://status.alwaysdata.com/operation/225/detail/We're investigating.Fri, 22 Aug 2014 13:24:52 +0000http://status.alwaysdata.com/operation/225/detail/The kernel crashed, we're restarting the server.Fri, 22 Aug 2014 13:26:32 +0000http://status.alwaysdata.com/operation/225/detail/The server is up, but will remain slow for several minutes.Fri, 22 Aug 2014 13:31:24 +0000http://status.alwaysdata.com/operation/226/detail/Everything should be back to normal in a few minutes.Tue, 26 Aug 2014 10:07:02 +0000http://status.alwaysdata.com/operation/227/detail/Everything should be back to normal in a few minutes.Tue, 26 Aug 2014 10:21:46 +0000http://status.alwaysdata.com/operation/228/detail/The server is down, we are investigating.Tue, 26 Aug 2014 18:56:29 +0000http://status.alwaysdata.com/operation/228/detail/Kernel issue, we are rebooting..Tue, 26 Aug 2014 18:59:34 +0000http://status.alwaysdata.com/operation/228/detail/The server is up, it will experience slow downs for a few minutes..Tue, 26 Aug 2014 19:02:07 +0000http://status.alwaysdata.com/operation/229/detail/The attack has been mitigated.Wed, 03 Sep 2014 00:50:58 +0000http://status.alwaysdata.com/operation/230/detail/postgresql1 is down, we are investigating.Thu, 04 Sep 2014 19:37:24 +0000http://status.alwaysdata.com/operation/230/detail/The server is up, it was due to connection abuse from an account.Thu, 04 Sep 2014 19:46:44 +0000http://status.alwaysdata.com/operation/231/detail/We're investigating.Sun, 07 Sep 2014 17:09:03 +0000http://status.alwaysdata.com/operation/231/detail/The kernel has crashed. The server is now up, the kernel has been upgraded.Sun, 07 Sep 2014 17:14:21 +0000http://status.alwaysdata.com/operation/232/detail/We're investigating.Fri, 12 Sep 2014 15:23:45 +0000http://status.alwaysdata.com/operation/232/detail/It's up again. An IO surge caused the downtime.Fri, 12 Sep 2014 15:39:05 +0000http://status.alwaysdata.com/operation/233/detail/Due to a kernel crash. We're rebooting the server.Mon, 29 Sep 2014 15:33:42 +0000http://status.alwaysdata.com/operation/234/detail/DDoS attack.Wed, 08 Oct 2014 13:17:00 +0000http://status.alwaysdata.com/operation/235/detail/We are investigating.Fri, 10 Oct 2014 09:55:31 +0000http://status.alwaysdata.com/operation/235/detail/The connections can be now established as usual.Fri, 10 Oct 2014 10:12:04 +0000http://status.alwaysdata.com/operation/236/detail/Due to customer abuse. The connections can be now established as usual.Mon, 13 Oct 2014 07:42:05 +0000http://status.alwaysdata.com/operation/238/detail/We'll upgrade the primary router. The traffic will be handled by the secondary traffic during the operation. No downtime expected.Mon, 20 Oct 2014 07:29:02 +0000http://status.alwaysdata.com/operation/238/detail/We'll reschedule the upgrade.Mon, 20 Oct 2014 09:48:25 +0000http://status.alwaysdata.com/operation/239/detail/It's been mitigated.Fri, 24 Oct 2014 11:11:33 +0000http://status.alwaysdata.com/operation/240/detail/We are investigating.Mon, 03 Nov 2014 10:11:07 +0000http://status.alwaysdata.com/operation/240/detail/Due to high memory usage. The server has been rebooted.Mon, 03 Nov 2014 10:25:25 +0000http://status.alwaysdata.com/operation/238/detail/The upgrade is scheduled for this afternoon.Mon, 03 Nov 2014 12:57:05 +0000http://status.alwaysdata.com/operation/238/detail/The upgrade completed successfully.Mon, 03 Nov 2014 15:33:03 +0000http://status.alwaysdata.com/operation/241/detail/Rebooting, due to kernel malfunction.Thu, 06 Nov 2014 07:56:19 +0000http://status.alwaysdata.com/operation/242/detail/We are investigating.Thu, 06 Nov 2014 18:13:35 +0000http://status.alwaysdata.com/operation/242/detail/The server is under DDoS attack.Thu, 06 Nov 2014 18:18:50 +0000http://status.alwaysdata.com/operation/242/detail/The server is now up but may experience slowdowns.Thu, 06 Nov 2014 18:43:49 +0000http://status.alwaysdata.com/operation/243/detail/We will upgrade the DNS management section of our administration panel. Since it requires a database migration, operations on our administration panel that involve modifying the DNS (domains, subdomains, domain records) will be unavailable for about 30 minutes. Our DNS servers are NOT impacted and will continue working as usual during the operation.Mon, 10 Nov 2014 15:38:09 +0000http://status.alwaysdata.com/operation/244/detail/Kernel issue, the server is rebooting.Tue, 11 Nov 2014 09:23:39 +0000http://status.alwaysdata.com/operation/244/detail/The server is up.Tue, 11 Nov 2014 09:27:54 +0000http://status.alwaysdata.com/operation/243/detail/The operation is starting.Wed, 12 Nov 2014 07:30:26 +0000http://status.alwaysdata.com/operation/243/detail/The migration is still running, it will last a bit longer than anticipated.Wed, 12 Nov 2014 08:00:14 +0000http://status.alwaysdata.com/operation/243/detail/The operation has completed successfully.Wed, 12 Nov 2014 08:24:05 +0000http://status.alwaysdata.com/operation/245/detail/We are investigating.Mon, 17 Nov 2014 12:22:35 +0000http://status.alwaysdata.com/operation/245/detail/The server is up again.Mon, 17 Nov 2014 12:22:49 +0000http://status.alwaysdata.com/operation/245/detail/The server overheated, we suspect a datacenter issue. It's back to normal now.Mon, 17 Nov 2014 12:24:39 +0000http://status.alwaysdata.com/operation/246/detail/MySQL has crashed, it's starting again.Tue, 18 Nov 2014 21:58:26 +0000http://status.alwaysdata.com/operation/246/detail/MySQL is up, although requests may be slow for a few more minutes.Tue, 18 Nov 2014 22:28:26 +0000http://status.alwaysdata.com/operation/246/detail/Everything is back to normal.Tue, 18 Nov 2014 22:33:26 +0000http://status.alwaysdata.com/operation/247/detail/We are investigating.Thu, 20 Nov 2014 00:00:48 +0000http://status.alwaysdata.com/operation/247/detail/Kernel issue. It has been upgraded, the server is up.Thu, 20 Nov 2014 00:09:43 +0000http://status.alwaysdata.com/operation/248/detail/Some clients have reported network issues with http7. We're investigating.Mon, 24 Nov 2014 13:37:18 +0000http://status.alwaysdata.com/operation/248/detail/We're contacting our provider.Mon, 24 Nov 2014 13:48:52 +0000http://status.alwaysdata.com/operation/248/detail/The issue is gone, at least for now.Mon, 24 Nov 2014 13:59:22 +0000http://status.alwaysdata.com/operation/249/detail/We are investigating.Mon, 01 Dec 2014 03:36:44 +0000http://status.alwaysdata.com/operation/249/detail/The main IP address for this server was suspended due to abuse. We've redirected the DNS to another IP (176.31.58.27) until the IP is unblocked.Mon, 01 Dec 2014 03:49:47 +0000http://status.alwaysdata.com/operation/249/detail/The old IP (178.32.28.121) is now unblocked.Mon, 01 Dec 2014 09:32:48 +0000http://status.alwaysdata.com/operation/250/detail/The kernel will be upgraded on the HTTP servers.Thu, 18 Dec 2014 23:10:20 +0000http://status.alwaysdata.com/operation/250/detail/All servers have been upgraded.Thu, 18 Dec 2014 23:56:30 +0000http://status.alwaysdata.com/operation/251/detail/MySQL has crashed. It's restarting.Tue, 30 Dec 2014 16:29:52 +0000http://status.alwaysdata.com/operation/251/detail/Still restarting (all .idb files are being read).Tue, 30 Dec 2014 16:43:06 +0000http://status.alwaysdata.com/operation/251/detail/MySQL is up. It will be slow for several minutes.Tue, 30 Dec 2014 16:53:17 +0000http://status.alwaysdata.com/operation/252/detail/We are investigating.Thu, 01 Jan 2015 05:38:47 +0000http://status.alwaysdata.com/operation/252/detail/It was due to a crash kernel. The server has been rebooted.Thu, 01 Jan 2015 06:15:23 +0000http://status.alwaysdata.com/operation/253/detail/The network link is being reset very frequently, which causes downtime. We're investigating with our provider.Sat, 10 Jan 2015 13:08:25 +0000http://status.alwaysdata.com/operation/253/detail/The server is being restarted.Sat, 10 Jan 2015 13:13:58 +0000http://status.alwaysdata.com/operation/253/detail/We'll see if it helps, the NIC hasn't been reset for a few minutes.Sat, 10 Jan 2015 13:22:24 +0000http://status.alwaysdata.com/operation/253/detail/No reset for now. We'll close this incident and reopen it if needed.Sat, 10 Jan 2015 13:39:42 +0000http://status.alwaysdata.com/operation/254/detail/Caused by a heavy IO load.Mon, 12 Jan 2015 16:17:33 +0000http://status.alwaysdata.com/operation/255/detail/MySQL has crashed. It's restarting.Mon, 12 Jan 2015 16:55:40 +0000http://status.alwaysdata.com/operation/255/detail/The server is up.Mon, 12 Jan 2015 17:05:01 +0000http://status.alwaysdata.com/operation/256/detail/We are investigating.Wed, 14 Jan 2015 07:48:54 +0000http://status.alwaysdata.com/operation/256/detail/It's a network issue.Wed, 14 Jan 2015 08:07:33 +0000http://status.alwaysdata.com/operation/256/detail/This is most likely related to incident 253. We've asked our provider to investigate.Wed, 14 Jan 2015 08:15:23 +0000http://status.alwaysdata.com/operation/256/detail/The technician are on their way. They've been asked to resolve the underlying network issue (NIC being reset).Wed, 14 Jan 2015 08:24:24 +0000http://status.alwaysdata.com/operation/256/detail/ETA: at least 30 minutes.Wed, 14 Jan 2015 08:25:54 +0000http://status.alwaysdata.com/operation/256/detail/We're still waiting for an update from our provider.Wed, 14 Jan 2015 08:45:31 +0000http://status.alwaysdata.com/operation/256/detail/Still no news yet.Wed, 14 Jan 2015 09:28:54 +0000http://status.alwaysdata.com/operation/256/detail/All we know is that the technicians are "verifying the network".Wed, 14 Jan 2015 10:17:51 +0000http://status.alwaysdata.com/operation/256/detail/The server is up, although it will be slow for several minutes.Wed, 14 Jan 2015 10:45:29 +0000http://status.alwaysdata.com/operation/257/detail/We're investigating.Sat, 17 Jan 2015 19:45:15 +0000http://status.alwaysdata.com/operation/257/detail/It's a kernel issue, we're restarting the server.Sat, 17 Jan 2015 19:46:50 +0000http://status.alwaysdata.com/operation/257/detail/The server is up.Sat, 17 Jan 2015 19:48:46 +0000http://status.alwaysdata.com/operation/258/detail/The server is slower than usual for now.Mon, 26 Jan 2015 16:38:42 +0000http://status.alwaysdata.com/operation/259/detail/We're upgrading our infrastructure due to CVE-2015-0235. Many services may be available for a couple of minutes.Tue, 27 Jan 2015 22:47:59 +0000http://status.alwaysdata.com/operation/260/detail/We're investigating.Sun, 01 Feb 2015 00:07:55 +0000http://status.alwaysdata.com/operation/260/detail/The primary router seems to be down, the traffic is routed by router2.Sun, 01 Feb 2015 00:17:03 +0000http://status.alwaysdata.com/operation/260/detail/router1 has frozen at the exact same time the monthly RAID check started, for an unknown reason (no traceback or any log whatsoever). We've rebooted it, it's now routing traffic again. The traffic didn't fail over to router2 until 10 minutes after router1 was down, which isn't normal. We're investigating what happened.Sun, 01 Feb 2015 00:41:28 +0000http://status.alwaysdata.com/operation/260/detail/Our logs show that our internal gateway monitoring tool didn't fail-over until 10 minutes after the first downtime was detected, in IPv4. That's obviously not normal. The IPv6 fail-over was done very quickly, though.Sun, 01 Feb 2015 00:49:48 +0000http://status.alwaysdata.com/operation/260/detail/We've found and fixed the bug that caused the slow fail-over in our monitoring tool.Sun, 01 Feb 2015 17:27:15 +0000http://status.alwaysdata.com/operation/261/detail/We are rebooting the server.Mon, 16 Feb 2015 15:38:54 +0000http://status.alwaysdata.com/operation/261/detail/The server is up but will experience slow downs for a few minutes.Mon, 16 Feb 2015 15:49:13 +0000http://status.alwaysdata.com/operation/261/detail/It was due to high memory usage.Mon, 16 Feb 2015 15:52:10 +0000http://status.alwaysdata.com/operation/262/detail/It's restarting.Tue, 17 Feb 2015 14:13:14 +0000http://status.alwaysdata.com/operation/262/detail/MySQL is up. It will be slower than usual for a few minutes.Tue, 17 Feb 2015 14:27:01 +0000http://status.alwaysdata.com/operation/263/detail/We are investigating.Tue, 17 Feb 2015 23:46:13 +0000http://status.alwaysdata.com/operation/263/detail/Kernel issue, the server is up.Tue, 17 Feb 2015 23:55:48 +0000http://status.alwaysdata.com/operation/264/detail/We're investigating.Sat, 21 Feb 2015 15:49:55 +0000http://status.alwaysdata.com/operation/264/detail/It's fixed. Most likely due to a IO spike.Sat, 21 Feb 2015 15:53:46 +0000http://status.alwaysdata.com/operation/265/detail/We're investigating.Sun, 22 Feb 2015 15:34:37 +0000http://status.alwaysdata.com/operation/265/detail/Fixed. We're still investigating.Sun, 22 Feb 2015 15:39:18 +0000http://status.alwaysdata.com/operation/265/detail/router1 froze as we were running commands on it. The automatic failover failed to work properly, we're investigating why.Sun, 22 Feb 2015 15:46:54 +0000http://status.alwaysdata.com/operation/265/detail/We're highly confident that we've found the bug that caused router1 to freeze on both occasions this month. We'll schedule tests to reproduce the bug (with the traffic rerouted to router2) and upgrade the kernel to a newer version with the bug fixed. When the bug is reproduced, we'll also investigate why the failover is malfunctioning, as our laboratory tests obviously didn't reproduce faithfully the situation.Sun, 22 Feb 2015 17:34:01 +0000http://status.alwaysdata.com/operation/266/detail/Piwik is in maintenance mode as it's being upgraded.Thu, 26 Feb 2015 10:17:34 +0000http://status.alwaysdata.com/operation/267/detail/It's restarting.Thu, 26 Feb 2015 12:14:46 +0000http://status.alwaysdata.com/operation/267/detail/It's up.Thu, 26 Feb 2015 12:30:46 +0000http://status.alwaysdata.com/operation/267/detail/It crashed again.Thu, 26 Feb 2015 16:23:39 +0000http://status.alwaysdata.com/operation/267/detail/Up again.Thu, 26 Feb 2015 16:28:00 +0000http://status.alwaysdata.com/operation/267/detail/It crashed once again. We've deleted one database that seems to have caused those crashes.Thu, 26 Feb 2015 16:33:19 +0000http://status.alwaysdata.com/operation/268/detail/Following operations #260 and #265, we'll update our primary router, our failover mechanism and make some tests. No downtime expected.Sun, 01 Mar 2015 11:44:42 +0000http://status.alwaysdata.com/operation/268/detail/The primary router kernel has been upgraded. We're confident that it fixes the crashes, as we couldn't reproduce them (we reproduced them with the old kernel). The failover software has been upgraded as well. Although we were unable to reproduce exactly what happened during #260 and #265 with the old version (which basically froze during a ping), we could reproduce abnormally long delays which are most likely caused by the same bug. It has been fixed in the new version.Sun, 01 Mar 2015 15:27:18 +0000http://status.alwaysdata.com/operation/269/detail/The server is experiencing slow downs due to a DROP SQL command on thousands tables by a customer.Tue, 03 Mar 2015 08:57:11 +0000http://status.alwaysdata.com/operation/269/detail/The server is now responding normally.Tue, 03 Mar 2015 09:14:57 +0000http://status.alwaysdata.com/operation/266/detail/Piwik is up-to-date. Importing the missing data will take at least a few days.Thu, 05 Mar 2015 09:47:36 +0000http://status.alwaysdata.com/operation/270/detail/It's restarted.Fri, 13 Mar 2015 14:41:33 +0000http://status.alwaysdata.com/operation/270/detail/The server is now up.Fri, 13 Mar 2015 14:56:05 +0000http://status.alwaysdata.com/operation/271/detail/It's restarting.Sun, 15 Mar 2015 11:42:22 +0000http://status.alwaysdata.com/operation/271/detail/It's up.Sun, 15 Mar 2015 12:00:35 +0000http://status.alwaysdata.com/operation/272/detail/We are investigating.Tue, 17 Mar 2015 01:51:20 +0000http://status.alwaysdata.com/operation/272/detail/Kernel issue, we are rebooting.Tue, 17 Mar 2015 01:59:01 +0000http://status.alwaysdata.com/operation/272/detail/The server is up.Tue, 17 Mar 2015 02:04:44 +0000http://status.alwaysdata.com/operation/273/detail/It's restarting.Wed, 18 Mar 2015 13:13:08 +0000http://status.alwaysdata.com/operation/273/detail/The server is up.Wed, 18 Mar 2015 13:27:28 +0000http://status.alwaysdata.com/operation/274/detail/It's restarting.Fri, 20 Mar 2015 14:45:20 +0000http://status.alwaysdata.com/operation/275/detail/Due to high IO usage. We're investigating.Sat, 21 Mar 2015 05:58:01 +0000http://status.alwaysdata.com/operation/275/detail/We have found the user who caused that IO usage, the system is now recovering.Sat, 21 Mar 2015 06:27:41 +0000http://status.alwaysdata.com/operation/276/detail/MySQL is restarting.Mon, 23 Mar 2015 14:35:06 +0000http://status.alwaysdata.com/operation/276/detail/The server will be migrated over the next 2 weeks, which should eliminate those recurrent bugs.Mon, 23 Mar 2015 14:39:37 +0000http://status.alwaysdata.com/operation/277/detail/The server hardware will be changed. Databases hosted on mysql1 will be unavailable during the operation.Wed, 25 Mar 2015 09:47:16 +0000http://status.alwaysdata.com/operation/277/detail/The operation is starting.Wed, 25 Mar 2015 23:04:12 +0000http://status.alwaysdata.com/operation/277/detail/The operation has completed successfully. We're still making final checks.Wed, 25 Mar 2015 23:31:12 +0000http://status.alwaysdata.com/operation/278/detail/Due to a high IO load. We're investigating.Mon, 30 Mar 2015 09:47:24 +0000http://status.alwaysdata.com/operation/279/detail/We are investigating.Wed, 01 Apr 2015 17:15:52 +0000http://status.alwaysdata.com/operation/279/detail/The server is rebooted.Wed, 01 Apr 2015 17:19:41 +0000http://status.alwaysdata.com/operation/279/detail/The server is up but will experience slow downs for a few minutes.Wed, 01 Apr 2015 17:25:19 +0000http://status.alwaysdata.com/operation/279/detail/It was due to high memory usage. The responsible account was warned.Wed, 01 Apr 2015 17:42:01 +0000http://status.alwaysdata.com/operation/280/detail/Network issue. A router has rebooted at rbx4.Fri, 17 Apr 2015 12:11:33 +0000http://status.alwaysdata.com/operation/281/detail/One disk has failed, we'll replace it tonight.Mon, 20 Apr 2015 15:16:50 +0000http://status.alwaysdata.com/operation/281/detail/The operation is starting.Mon, 20 Apr 2015 21:49:26 +0000http://status.alwaysdata.com/operation/281/detail/The disk has been replaced. The server will be slow for the next minutes.Mon, 20 Apr 2015 21:57:11 +0000http://status.alwaysdata.com/operation/282/detail/We're seeing a high IO usage.Sun, 26 Apr 2015 21:21:52 +0000http://status.alwaysdata.com/operation/283/detail/Due to a IO spike.Fri, 01 May 2015 22:17:09 +0000http://status.alwaysdata.com/operation/284/detail/We are working on the issue.Tue, 12 May 2015 08:59:08 +0000http://status.alwaysdata.com/operation/284/detail/The issue should be resolved.Tue, 12 May 2015 09:20:52 +0000http://status.alwaysdata.com/operation/284/detail/Here is what happened: we upgraded our internal database schema, as we frequently do, but it triggered an unexpected CASCADE which deleted other tables/views. As a result, the IMAP/SMTP/FTP/WebDAV daemons were unable to perform authentication until we restored a backup. No data has been lost during the operation.Tue, 12 May 2015 09:30:34 +0000http://status.alwaysdata.com/operation/285/detail/We're investigating.Fri, 15 May 2015 16:08:53 +0000http://status.alwaysdata.com/operation/285/detail/The server is up.Fri, 15 May 2015 16:14:33 +0000http://status.alwaysdata.com/operation/286/detail/We are investigating.Tue, 26 May 2015 13:08:06 +0000http://status.alwaysdata.com/operation/286/detail/The server is up but experiences slow downs.Tue, 26 May 2015 13:20:28 +0000http://status.alwaysdata.com/operation/287/detail/We are investigating.Wed, 27 May 2015 20:41:10 +0000http://status.alwaysdata.com/operation/287/detail/The server is up.Wed, 27 May 2015 20:48:56 +0000http://status.alwaysdata.com/operation/288/detail/We are investigating.Thu, 28 May 2015 13:25:31 +0000http://status.alwaysdata.com/operation/288/detail/The server is up, It was due to high memory usage.Thu, 28 May 2015 13:46:01 +0000http://status.alwaysdata.com/operation/288/detail/Down again, we are on it.Thu, 28 May 2015 14:18:08 +0000http://status.alwaysdata.com/operation/289/detail/Due to high IO usage.Sun, 31 May 2015 10:28:18 +0000http://status.alwaysdata.com/operation/290/detail/We're investigating.Sat, 20 Jun 2015 20:50:02 +0000http://status.alwaysdata.com/operation/290/detail/It's most likely a network/hardware issue. We're waiting for an update from our provider.Sat, 20 Jun 2015 20:53:51 +0000http://status.alwaysdata.com/operation/290/detail/The network/hardware issue is confirmed.Sat, 20 Jun 2015 20:54:20 +0000http://status.alwaysdata.com/operation/290/detail/The server is up.Sat, 20 Jun 2015 21:11:07 +0000http://status.alwaysdata.com/operation/290/detail/The server is down again.Sat, 20 Jun 2015 21:23:50 +0000http://status.alwaysdata.com/operation/290/detail/The server is up.Sat, 20 Jun 2015 21:41:09 +0000http://status.alwaysdata.com/operation/291/detail/We are investigating.Fri, 10 Jul 2015 08:42:17 +0000http://status.alwaysdata.com/operation/291/detail/We have rebooted the server. We believe it was a kernel bug but we have to find out the details. The server is up but will remain slow for a moment.Fri, 10 Jul 2015 08:56:25 +0000http://status.alwaysdata.com/operation/291/detail/The server is up and working normally.Fri, 10 Jul 2015 09:08:45 +0000http://status.alwaysdata.com/operation/292/detail/We are investigating.Wed, 15 Jul 2015 07:33:50 +0000http://status.alwaysdata.com/operation/292/detail/The server is rebooting, we are trying to find the origin of the problem.Wed, 15 Jul 2015 07:46:08 +0000http://status.alwaysdata.com/operation/292/detail/The server is now up.Wed, 15 Jul 2015 07:46:33 +0000http://status.alwaysdata.com/operation/293/detail/Many mails are being delayed, we're investigating why.Wed, 15 Jul 2015 09:21:39 +0000http://status.alwaysdata.com/operation/293/detail/The delay was caused by an internal database server overheating (again). We've switched to a secondary server to alleviate the issue.Wed, 15 Jul 2015 09:41:29 +0000http://status.alwaysdata.com/operation/294/detail/We are investigating.Wed, 15 Jul 2015 11:28:15 +0000http://status.alwaysdata.com/operation/294/detail/The server has been restarted and is now up. It will experience slow downs during few minutes.Wed, 15 Jul 2015 11:33:38 +0000http://status.alwaysdata.com/operation/293/detail/Delays are resolved. We believe the cause of this issue is primarily a huge wave of incoming spams (fake invoices) that started this morning.Wed, 15 Jul 2015 11:39:40 +0000http://status.alwaysdata.com/operation/294/detail/It was due to a kernel freeze issue.Wed, 15 Jul 2015 11:42:28 +0000http://status.alwaysdata.com/operation/295/detail/Kernel freeze, the server is rebooting.Thu, 16 Jul 2015 12:39:15 +0000http://status.alwaysdata.com/operation/295/detail/The server is up.Thu, 16 Jul 2015 12:44:54 +0000http://status.alwaysdata.com/operation/296/detail/Kernel issue, the server has been rebooted.Tue, 28 Jul 2015 00:47:28 +0000http://status.alwaysdata.com/operation/297/detail/Some networks (Free, for instance) cannot reach our shared servers, due to an issue with our provider. We're still waiting for more details. Dedicated servers are not affected, as they're hosted in our own infrastructure.Wed, 29 Jul 2015 13:48:57 +0000http://status.alwaysdata.com/operation/297/detail/The issue is still ongoing.Wed, 29 Jul 2015 14:14:48 +0000http://status.alwaysdata.com/operation/297/detail/The issue seems resolved for now, although our provider is still working on it.Wed, 29 Jul 2015 14:38:46 +0000http://status.alwaysdata.com/operation/299/detail/Kernels on shared HTTP servers will be upgraded. Estimated downtime for each server: less than 10 minutes.Fri, 07 Aug 2015 21:22:59 +0000http://status.alwaysdata.com/operation/300/detail/Due to a kernel crash. We're rebooting the server.Thu, 13 Aug 2015 11:16:37 +0000http://status.alwaysdata.com/operation/300/detail/The server is up again.Thu, 13 Aug 2015 11:18:28 +0000http://status.alwaysdata.com/operation/301/detail/We're investigating.Thu, 13 Aug 2015 17:10:59 +0000http://status.alwaysdata.com/operation/301/detail/It's most likely an issue with our provider. We're waiting for more details.Thu, 13 Aug 2015 17:13:59 +0000http://status.alwaysdata.com/operation/301/detail/Our provider has confirmed the issue. We'll update the operation as we have more details.Thu, 13 Aug 2015 17:25:49 +0000http://status.alwaysdata.com/operation/301/detail/Still down, it could be a power issue according to a technician we spoke to.Thu, 13 Aug 2015 17:52:38 +0000http://status.alwaysdata.com/operation/301/detail/The server is up. It was indeed an electrical issue, and the power supply had to be replaced.Thu, 13 Aug 2015 18:45:17 +0000http://status.alwaysdata.com/operation/302/detail/We're investigating.Fri, 14 Aug 2015 13:35:51 +0000http://status.alwaysdata.com/operation/302/detail/There's an issue with our provider again.Fri, 14 Aug 2015 13:37:47 +0000http://status.alwaysdata.com/operation/302/detail/The server is up again. The network interface was down for 9 minutes.Fri, 14 Aug 2015 13:40:18 +0000http://status.alwaysdata.com/operation/303/detail/We're investigating.Sat, 15 Aug 2015 17:54:38 +0000http://status.alwaysdata.com/operation/303/detail/Solved. We had temporarily reached the maximum connections count.Sat, 15 Aug 2015 17:56:00 +0000http://status.alwaysdata.com/operation/304/detail/We're investigating.Thu, 20 Aug 2015 15:49:32 +0000http://status.alwaysdata.com/operation/304/detail/There's a high IO load.Thu, 20 Aug 2015 15:53:25 +0000http://status.alwaysdata.com/operation/304/detail/The server will remain slow for several minutes.Thu, 20 Aug 2015 16:00:03 +0000http://status.alwaysdata.com/operation/305/detail/The kernel will be upgraded on this server.Mon, 24 Aug 2015 21:30:03 +0000http://status.alwaysdata.com/operation/306/detail/Kernel issue, we are rebooting the server.Tue, 25 Aug 2015 07:19:31 +0000http://status.alwaysdata.com/operation/306/detail/The server is UP.Tue, 25 Aug 2015 07:28:41 +0000http://status.alwaysdata.com/operation/307/detail/The server is under a high IO load. We're investigating.Wed, 02 Sep 2015 12:24:31 +0000http://status.alwaysdata.com/operation/308/detail/We're investigating.Fri, 04 Sep 2015 13:45:46 +0000http://status.alwaysdata.com/operation/308/detail/The server is up. An operation manually started on the server did not go as expected.Fri, 04 Sep 2015 13:57:34 +0000http://status.alwaysdata.com/operation/309/detail/The server is under a high IO load. We're investigating.Mon, 07 Sep 2015 09:37:23 +0000http://status.alwaysdata.com/operation/310/detail/We're investigating.Mon, 07 Sep 2015 19:25:56 +0000http://status.alwaysdata.com/operation/311/detail/The network cable will be replaced tonight.Tue, 08 Sep 2015 09:17:39 +0000http://status.alwaysdata.com/operation/311/detail/The operation has started.Tue, 08 Sep 2015 21:34:46 +0000http://status.alwaysdata.com/operation/311/detail/The operation has completed.Tue, 08 Sep 2015 21:39:00 +0000http://status.alwaysdata.com/operation/312/detail/We're making low-level changes on this SSH server in order to eliminate /nfs/httpX/foo symlinks that may appear in some rare conditions (e.g. creating a virtualenv). We've already made those changes on other servers successfully (and transparently), but the specific workload on this SSH server may cause temporary instability issues, such as /home directories not being accessible, or cron tasks not being executed. If you notice issues lasting more than 10 minutes, please contact our support.Thu, 10 Sep 2015 11:07:20 +0000http://status.alwaysdata.com/operation/313/detail/We're investigating.Thu, 10 Sep 2015 15:10:28 +0000http://status.alwaysdata.com/operation/314/detail/We are working on it for a speedy recovery.Fri, 11 Sep 2015 14:08:58 +0000http://status.alwaysdata.com/operation/315/detail/Everything should be back in order in the next few minutes.Mon, 14 Sep 2015 14:24:55 +0000http://status.alwaysdata.com/operation/316/detail/We are investigating.Wed, 16 Sep 2015 00:32:39 +0000http://status.alwaysdata.com/operation/316/detail/The server is rebooting.Wed, 16 Sep 2015 00:43:28 +0000http://status.alwaysdata.com/operation/316/detail/The server is up, there will be slow downs for a few minutes.Wed, 16 Sep 2015 00:50:52 +0000http://status.alwaysdata.com/operation/317/detail/We're investigating.Wed, 16 Sep 2015 10:29:59 +0000http://status.alwaysdata.com/operation/317/detail/The issue is fixed. We're still investigating what happened.Wed, 16 Sep 2015 10:39:43 +0000http://status.alwaysdata.com/operation/317/detail/Post-mortem: we have an internal tool that regularly checks that the permissions of a predefined list of system files and directories are correctly set and if not, changes them. Earlier today, we added a new rule to that tool on one of our internal servers. That rule says that /home should be set to 751. That specific server, which hosts our administration panel, must be able to access all accounts, on both dedicated and shared servers. The directory for that rule should have been set to '/home' in order to change the permissions on that directory only. Instead, we made a mistake and typed '/home/' (with a final slash). Ending the directory with a slash means that we want the tool to apply the permissions recursively, i.e. including all subdirectories. The tool was automatically run at 12:17 and started changing permissions on user home directories at 12:25, in random order. It ran for several minutes until we noticed the error. At 12:37, all user directories had been manually reset to their original permissions. During that 12 minutes window, a little less than half of accounts had wrong permissions. Those permissions prevented user applications from opening files under /home, which resulted in websites showing errors such as "Forbidden". Not all websites were impacted though, mostly those using PHP (which reopens the application files on each new request). Other services (emails, databases) were not affected. We apologize for that downtime impacting an exceptionally large number of our clients.Wed, 16 Sep 2015 11:53:30 +0000http://status.alwaysdata.com/operation/318/detail/We're investigating.Thu, 17 Sep 2015 15:11:13 +0000http://status.alwaysdata.com/operation/319/detail/We're investigating.Thu, 17 Sep 2015 15:40:22 +0000http://status.alwaysdata.com/operation/320/detail/We are investigating.Mon, 28 Sep 2015 12:09:50 +0000http://status.alwaysdata.com/operation/321/detail/We are investigating.Mon, 28 Sep 2015 12:24:04 +0000http://status.alwaysdata.com/operation/321/detail/The server is responding correctly now.Mon, 28 Sep 2015 12:46:10 +0000http://status.alwaysdata.com/operation/322/detail/We're investigating.Tue, 29 Sep 2015 14:34:33 +0000http://status.alwaysdata.com/operation/322/detail/Solved. There was an IO spike.Tue, 29 Sep 2015 14:39:34 +0000http://status.alwaysdata.com/operation/323/detail/We're investigating.Thu, 01 Oct 2015 11:29:59 +0000http://status.alwaysdata.com/operation/323/detail/It looks like there's an issue with our provider. We're waiting for more details.Thu, 01 Oct 2015 11:35:21 +0000http://status.alwaysdata.com/operation/323/detail/It's up. The server has been rebooted.Thu, 01 Oct 2015 11:44:31 +0000http://status.alwaysdata.com/operation/324/detail/A server failure triggered DNS resolution issues, which should not have happened.Fri, 02 Oct 2015 15:31:58 +0000http://status.alwaysdata.com/operation/325/detail/The server is unreachable due to power issues.Fri, 02 Oct 2015 15:42:25 +0000http://status.alwaysdata.com/operation/326/detail/We're investigating.Sat, 03 Oct 2015 15:28:21 +0000http://status.alwaysdata.com/operation/327/detail/We are investigating.Tue, 06 Oct 2015 02:24:19 +0000http://status.alwaysdata.com/operation/327/detail/The server has rebooted and is now up.Tue, 06 Oct 2015 02:47:45 +0000http://status.alwaysdata.com/operation/329/detail/An IO spike is causing a slowdown of the server. We're trying to reduce the load.Wed, 07 Oct 2015 08:52:00 +0000http://status.alwaysdata.com/operation/329/detail/The load is decreasing, but we're investigating on a possible hardware failure.Wed, 07 Oct 2015 11:05:11 +0000http://status.alwaysdata.com/operation/330/detail/We're investigating.Tue, 13 Oct 2015 18:04:23 +0000http://status.alwaysdata.com/operation/330/detail/The server is now up, but will slow down for a few minutes.Tue, 13 Oct 2015 18:20:23 +0000http://status.alwaysdata.com/operation/331/detail/We're trying to mitigate.Wed, 14 Oct 2015 09:58:36 +0000http://status.alwaysdata.com/operation/331/detail/The server is back to a sane load level.Wed, 14 Oct 2015 10:38:31 +0000http://status.alwaysdata.com/operation/332/detail/Load is high again.Wed, 14 Oct 2015 15:36:25 +0000http://status.alwaysdata.com/operation/333/detail/Bug MySQL. We are investigating.Thu, 22 Oct 2015 11:05:32 +0000http://status.alwaysdata.com/operation/333/detail/The MySQL server is rebooting.Thu, 22 Oct 2015 11:06:18 +0000http://status.alwaysdata.com/operation/333/detail/The server is now up.Thu, 22 Oct 2015 11:08:34 +0000http://status.alwaysdata.com/operation/334/detail/Performances are degraded due to a MySQL bug. We'll restart MySQL to restore performances.Wed, 28 Oct 2015 12:03:03 +0000http://status.alwaysdata.com/operation/335/detail/A hard disk failed, which causes degraded performance on this server.Sat, 31 Oct 2015 11:16:18 +0000http://status.alwaysdata.com/operation/336/detail/We're investigating.Sun, 01 Nov 2015 18:47:11 +0000http://status.alwaysdata.com/operation/337/detail/Bug MySQL, the server has been rebooted.Mon, 02 Nov 2015 13:16:22 +0000http://status.alwaysdata.com/operation/338/detail/A hard disk died, it will be replaced tonight.Mon, 02 Nov 2015 13:17:54 +0000http://status.alwaysdata.com/operation/338/detail/The operation has started.Mon, 02 Nov 2015 22:01:28 +0000http://status.alwaysdata.com/operation/338/detail/The disk has been replaced. Performances will be degraded for a few minutes.Mon, 02 Nov 2015 22:06:54 +0000http://status.alwaysdata.com/operation/339/detail/We'll upgrade the kernel and BIOS on several servers tonight. FTP and WebDAV accesses for shared accounts will be unavailable for about 15 minutes.Thu, 12 Nov 2015 12:42:53 +0000http://status.alwaysdata.com/operation/339/detail/The operation is starting.Thu, 12 Nov 2015 22:45:00 +0000http://status.alwaysdata.com/operation/339/detail/The kernels have been upgraded, but not the BIOS. All services are up, we're still considering whether we'll do the BIOS upgrade tonight.Thu, 12 Nov 2015 22:57:07 +0000http://status.alwaysdata.com/operation/339/detail/The BIOS upgrade will be rescheduled.Thu, 12 Nov 2015 23:13:47 +0000http://status.alwaysdata.com/operation/340/detail/We are investigating.Fri, 13 Nov 2015 10:11:18 +0000http://status.alwaysdata.com/operation/341/detail/We're investigating.Mon, 16 Nov 2015 13:31:46 +0000http://status.alwaysdata.com/operation/342/detail/We are investigating.Mon, 23 Nov 2015 08:34:34 +0000http://status.alwaysdata.com/operation/339/detail/The BIOS upgrade is scheduled for tonight.Thu, 26 Nov 2015 16:48:20 +0000http://status.alwaysdata.com/operation/339/detail/The operation is starting.Thu, 26 Nov 2015 22:30:03 +0000http://status.alwaysdata.com/operation/339/detail/The operation has completed successfully.Thu, 26 Nov 2015 22:48:29 +0000http://status.alwaysdata.com/operation/343/detail/We are working on the resolution of this issue.Fri, 27 Nov 2015 13:16:19 +0000http://status.alwaysdata.com/operation/343/detail/The server is up, but may be slower than normal for a few minutes.Fri, 27 Nov 2015 13:17:10 +0000http://status.alwaysdata.com/operation/344/detail/We are investigating.Mon, 30 Nov 2015 14:18:11 +0000http://status.alwaysdata.com/operation/345/detail/We are investigating.Wed, 02 Dec 2015 11:59:29 +0000http://status.alwaysdata.com/operation/345/detail/The server has been rebooted, it will experience slow downs for a few minutes.Wed, 02 Dec 2015 12:14:06 +0000http://status.alwaysdata.com/operation/346/detail/We are investigating.Tue, 08 Dec 2015 20:53:30 +0000http://status.alwaysdata.com/operation/346/detail/Network issue. The server is now reachable.Tue, 08 Dec 2015 21:07:22 +0000http://status.alwaysdata.com/operation/347/detail/We will migrate most of the *.alwaysdata.com websites (including admin.alwaysdata.com) to a new server.Mon, 14 Dec 2015 16:49:54 +0000http://status.alwaysdata.com/operation/347/detail/The operation begins.Tue, 15 Dec 2015 07:01:07 +0000http://status.alwaysdata.com/operation/347/detail/Most of the migration is done, but the administration panel still has issues on some operations.Tue, 15 Dec 2015 07:54:23 +0000http://status.alwaysdata.com/operation/347/detail/All known customer-facing issues have been resolved.Tue, 15 Dec 2015 08:45:37 +0000http://status.alwaysdata.com/operation/347/detail/The latest details have been resolved.Tue, 15 Dec 2015 11:49:44 +0000http://status.alwaysdata.com/operation/348/detail/We are investigating.Wed, 16 Dec 2015 17:36:08 +0000http://status.alwaysdata.com/operation/349/detail/Due to a kernel issue. We're restarting the server.Fri, 18 Dec 2015 19:08:01 +0000http://status.alwaysdata.com/operation/349/detail/The server has restarted, its performance will remain degraded for a few minutes.Fri, 18 Dec 2015 19:13:03 +0000http://status.alwaysdata.com/operation/350/detail/We'll migrate our main internal database to a new server. Several *.alwaysdata.com websites, including the administration panel, may be degraded or unavailable during the operation.Thu, 31 Dec 2015 14:30:54 +0000http://status.alwaysdata.com/operation/350/detail/The operation is starting.Mon, 04 Jan 2016 07:02:39 +0000http://status.alwaysdata.com/operation/350/detail/The operation has completed successfully.Mon, 04 Jan 2016 08:01:23 +0000http://status.alwaysdata.com/operation/351/detail/The server is rebooting.Mon, 04 Jan 2016 13:39:52 +0000http://status.alwaysdata.com/operation/351/detail/The server is up, but will have slow down issues for a few minutes.Mon, 04 Jan 2016 13:56:27 +0000http://status.alwaysdata.com/operation/352/detail/We're investigating.Mon, 04 Jan 2016 16:20:38 +0000http://status.alwaysdata.com/operation/352/detail/This is apparently caused by a network issue our provider is having.Mon, 04 Jan 2016 16:22:51 +0000http://status.alwaysdata.com/operation/352/detail/The network issue seems fixed.Mon, 04 Jan 2016 16:29:51 +0000http://status.alwaysdata.com/operation/352/detail/IPv6 is still down, which may cause timeouts when accessing internal databases. Our provider is trying to fix the issue.Mon, 04 Jan 2016 16:54:42 +0000http://status.alwaysdata.com/operation/352/detail/IPv6 is up.Mon, 04 Jan 2016 17:14:45 +0000http://status.alwaysdata.com/operation/353/detail/IO usage is high.Mon, 18 Jan 2016 14:04:59 +0000http://status.alwaysdata.com/operation/354/detail/IO usage is high.Mon, 18 Jan 2016 15:33:38 +0000http://status.alwaysdata.com/operation/355/detail/The server is experiencing slow down issues. We are investigating.Fri, 22 Jan 2016 10:15:12 +0000http://status.alwaysdata.com/operation/356/detail/These 2 servers have slowdowns, probably caused by some IO. We are investigating.Mon, 25 Jan 2016 16:37:47 +0000http://status.alwaysdata.com/operation/357/detail/We will upgrade the kernel on shared HTTP servers. A downtime of several minutes for each server is expected.Fri, 29 Jan 2016 16:53:16 +0000http://status.alwaysdata.com/operation/357/detail/The operation has started.Fri, 29 Jan 2016 23:01:43 +0000http://status.alwaysdata.com/operation/357/detail/The operation has completed successfully.Sat, 30 Jan 2016 00:00:05 +0000http://status.alwaysdata.com/operation/392/detail/We're investigating.Tue, 09 Feb 2016 16:46:17 +0000http://status.alwaysdata.com/operation/393/detail/Most servers in the roubaix1 datacenter are experiencing a network issue. We're waiting for more details from our provider.Thu, 11 Feb 2016 15:56:32 +0000http://status.alwaysdata.com/operation/393/detail/This seems to be a routing issue, as the servers are accessible from some networks.Thu, 11 Feb 2016 16:03:03 +0000http://status.alwaysdata.com/operation/393/detail/The issue seems resolved.Thu, 11 Feb 2016 16:11:36 +0000http://status.alwaysdata.com/operation/394/detail/We're investigating.Mon, 15 Feb 2016 11:00:17 +0000http://status.alwaysdata.com/operation/395/detail/We're investigating.Fri, 19 Feb 2016 09:14:30 +0000http://status.alwaysdata.com/operation/396/detail/We're investigating.Mon, 22 Feb 2016 15:15:02 +0000http://status.alwaysdata.com/operation/397/detail/We're investigating.Fri, 18 Mar 2016 12:33:32 +0000http://status.alwaysdata.com/operation/398/detail/We're investigating.Sun, 20 Mar 2016 11:19:59 +0000http://status.alwaysdata.com/operation/398/detail/A DDoS attack was responsible for this.Sun, 20 Mar 2016 11:22:37 +0000http://status.alwaysdata.com/operation/399/detail/We're investigating.Thu, 31 Mar 2016 11:30:03 +0000http://status.alwaysdata.com/operation/399/detail/We're still trying to understand why we're having issues.Thu, 31 Mar 2016 12:05:05 +0000http://status.alwaysdata.com/operation/399/detail/We don't have an ETA yet, we're still investigating.Thu, 31 Mar 2016 12:55:26 +0000http://status.alwaysdata.com/operation/399/detail/We seem to have found a temporary workaround.Thu, 31 Mar 2016 13:09:22 +0000http://status.alwaysdata.com/operation/399/detail/We'll continue using the workaround for now and keep investigating for a more permanent fix.Thu, 31 Mar 2016 14:37:21 +0000http://status.alwaysdata.com/operation/399/detail/The workaround is not as effective as yesterday, we're observing a higher system load than usual.Fri, 01 Apr 2016 07:37:28 +0000http://status.alwaysdata.com/operation/399/detail/We're pretty sure the underlying issue is a MySQL bug. Upgrading to a major MySQL version with no prior notification to our clients is not something we want to do. We're instead installing a new MySQL server (running MariaDB 10.1). Clients that want to migrate from mysql1 to this new server should contact us by opening a support ticket from their administration panel.Fri, 01 Apr 2016 08:55:25 +0000http://status.alwaysdata.com/operation/399/detail/The new MySQL server is ready, migrations will start shortly (for clients that opened a ticket).Fri, 01 Apr 2016 11:45:42 +0000http://status.alwaysdata.com/operation/399/detail/We've restarted mysql1 10 minutes ago, it's been running smoothly since then, although we cannot explain why. Fri, 01 Apr 2016 13:02:55 +0000http://status.alwaysdata.com/operation/399/detail/mysql1 is still running fine. Migrations have been suspended for now as some databases refuse to get reimported and we'd like to find a proper solution until we resume the process.Fri, 01 Apr 2016 15:29:17 +0000http://status.alwaysdata.com/operation/400/detail/This server will be migrated to new hardware. Expected downtime: about 5 minutes.Wed, 20 Apr 2016 13:35:20 +0000http://status.alwaysdata.com/operation/401/detail/We are investigating.Wed, 20 Apr 2016 18:56:30 +0000http://status.alwaysdata.com/operation/401/detail/It was a DDoS attack. The attack has been mitigated.Wed, 20 Apr 2016 19:09:51 +0000http://status.alwaysdata.com/operation/400/detail/The operation is starting.Wed, 20 Apr 2016 21:30:15 +0000http://status.alwaysdata.com/operation/400/detail/The migration is done. We're still checking a few more details.Wed, 20 Apr 2016 21:46:54 +0000http://status.alwaysdata.com/operation/402/detail/We are investigating.Wed, 18 May 2016 08:36:13 +0000http://status.alwaysdata.com/operation/402/detail/The server has been rebooted.Wed, 18 May 2016 08:44:50 +0000http://status.alwaysdata.com/operation/403/detail/We're investigating.Wed, 18 May 2016 14:18:08 +0000http://status.alwaysdata.com/operation/404/detail/Performance is degraded due to high IO usage.Mon, 23 May 2016 07:31:24 +0000http://status.alwaysdata.com/operation/405/detail/The maximum connection limit has been reached. We've restarted MySQL.Fri, 12 Aug 2016 11:12:02 +0000http://status.alwaysdata.com/operation/406/detail/A technician is on site.Wed, 24 Aug 2016 22:21:43 +0000http://status.alwaysdata.com/operation/406/detail/The cooling system has been replaced.Wed, 24 Aug 2016 22:29:06 +0000http://status.alwaysdata.com/operation/408/detail/We are investigating.Thu, 06 Oct 2016 05:01:21 +0000http://status.alwaysdata.com/operation/408/detail/Server issue, a technician has been dispatched.Thu, 06 Oct 2016 05:19:25 +0000http://status.alwaysdata.com/operation/408/detail/The server is up. The motherboard and CPU have been replaced.Thu, 06 Oct 2016 06:33:10 +0000http://status.alwaysdata.com/operation/409/detail/Kernels will be upgraded to fix the CVE-2016-5195 vulnerability.Thu, 20 Oct 2016 21:00:16 +0000http://status.alwaysdata.com/operation/409/detail/Kernels have been upgraded.Thu, 20 Oct 2016 21:49:56 +0000http://status.alwaysdata.com/operation/410/detail/IPv6 is mostly down due to a network issue at our provider. Several services are impacted.Thu, 03 Nov 2016 17:55:32 +0000http://status.alwaysdata.com/operation/410/detail/The issue is solved.Thu, 03 Nov 2016 20:46:56 +0000http://status.alwaysdata.com/operation/411/detail/IPv6 is mostly down due to a network issue at our provider. Several services are impacted (including mysql1 on some shared plans).Fri, 04 Nov 2016 10:09:05 +0000http://status.alwaysdata.com/operation/411/detail/The issue seems solved. Impacted users were using deprecated hostnames in their configuration, using the new hostnames would have mitigated the issue for those users (https://blog.alwaysdata.com/2015/03/05/change-of-hostname-for-access-to-our-services/).Fri, 04 Nov 2016 10:26:08 +0000http://status.alwaysdata.com/operation/413/detail/We are investigating.Fri, 02 Dec 2016 18:42:28 +0000http://status.alwaysdata.com/operation/413/detail/The server is up.Fri, 02 Dec 2016 18:48:46 +0000http://status.alwaysdata.com/operation/414/detail/Kernels will be upgraded to fix the CVE-2016-8655 vulnerability.Tue, 06 Dec 2016 22:00:19 +0000http://status.alwaysdata.com/operation/416/detail/We are investigating.Tue, 13 Dec 2016 12:24:11 +0000http://status.alwaysdata.com/operation/417/detail/A faulty hard disk will be replaced.Tue, 13 Dec 2016 14:27:59 +0000http://status.alwaysdata.com/operation/417/detail/The operation is delayed.Tue, 13 Dec 2016 22:30:55 +0000http://status.alwaysdata.com/operation/417/detail/The disk has been changed.Wed, 14 Dec 2016 06:55:47 +0000http://status.alwaysdata.com/operation/418/detail/We are investigating.Wed, 14 Dec 2016 08:04:16 +0000http://status.alwaysdata.com/operation/418/detail/This is an IPv6 routing issue. You can use the IP 87.98.167.229 temporarily to connect to mysql1.roubaix.Wed, 14 Dec 2016 09:51:30 +0000http://status.alwaysdata.com/operation/418/detail/The problem has been solved. Do not forget to change your hostnames again for those who modified it.Wed, 14 Dec 2016 12:15:52 +0000http://status.alwaysdata.com/operation/419/detail/Our provider is having network issues.Thu, 15 Dec 2016 17:07:37 +0000http://status.alwaysdata.com/operation/419/detail/The issue is solved.Thu, 15 Dec 2016 17:10:35 +0000http://status.alwaysdata.com/operation/420/detail/Our provider is having network issues.Fri, 06 Jan 2017 08:32:33 +0000http://status.alwaysdata.com/operation/420/detail/Resolved.Fri, 06 Jan 2017 08:38:55 +0000http://status.alwaysdata.com/operation/421/detail/We are experiencing IPv6 routing issues. We are investigating.Fri, 20 Jan 2017 08:14:16 +0000http://status.alwaysdata.com/operation/421/detail/Only http6.roubaix1 and http7.roubaix1 servers are concerned.Fri, 20 Jan 2017 08:34:44 +0000http://status.alwaysdata.com/operation/421/detail/You can use the hosts mysql1.alwaysdata.com and mysql2.alwaysdata.com temporarily to connect to mysql1.roubaix1 and mysql2.roubaix1.Fri, 20 Jan 2017 08:54:47 +0000http://status.alwaysdata.com/operation/421/detail/The problem has been solved. Do not forget to change your hostnames again for those who modified it.Fri, 20 Jan 2017 09:52:34 +0000http://status.alwaysdata.com/operation/422/detail/We are investigating.Tue, 07 Mar 2017 11:52:20 +0000http://status.alwaysdata.com/operation/423/detail/We are investigating.Fri, 24 Mar 2017 09:54:00 +0000http://status.alwaysdata.com/operation/424/detail/We're investigating.Fri, 31 Mar 2017 13:58:13 +0000http://status.alwaysdata.com/operation/425/detail/We're investigating.Mon, 03 Apr 2017 12:10:06 +0000http://status.alwaysdata.com/operation/425/detail/The server is UP but will experience slow downs for tens of minutes.Mon, 03 Apr 2017 12:20:08 +0000http://status.alwaysdata.com/operation/426/detail/We're investigating.Mon, 10 Apr 2017 13:55:28 +0000http://status.alwaysdata.com/operation/426/detail/The server is UP but will experience slow downs for tens of minutes.Mon, 10 Apr 2017 14:14:54 +0000http://status.alwaysdata.com/operation/427/detail/We'll upgrade our internal database to a newer PostgreSQL version. Our websites (including the administration and API) will be unavailable during that operation.Mon, 05 Jun 2017 10:35:00 +0000http://status.alwaysdata.com/operation/427/detail/The operation has started.Tue, 06 Jun 2017 06:01:53 +0000http://status.alwaysdata.com/operation/427/detail/The operation has completed successfully.Tue, 06 Jun 2017 06:33:10 +0000http://status.alwaysdata.com/operation/428/detail/An on-site technician will check the server.Fri, 16 Jun 2017 12:52:06 +0000http://status.alwaysdata.com/operation/428/detail/It was a network issue. The server is UP.Fri, 16 Jun 2017 13:05:52 +0000http://status.alwaysdata.com/operation/429/detail/We are investigating.Fri, 28 Jul 2017 04:55:49 +0000http://status.alwaysdata.com/operation/429/detail/The server has been rebooted. The concerned account has been alerted.Fri, 28 Jul 2017 06:24:01 +0000http://status.alwaysdata.com/operation/434/detail/The server is under DDoS attack. We are investigating.Thu, 21 Dec 2017 13:30:28 +0000http://status.alwaysdata.com/operation/434/detail/The attack has been mitigated.Thu, 21 Dec 2017 14:09:46 +0000http://status.alwaysdata.com/operation/435/detail/Sites hosted on *.alwaysdata.com (e.g. our administration panel) are currently inaccessible over SSL due to a certificate expiry. We're working on renewing the certificate.Sun, 07 Jan 2018 07:24:37 +0000http://status.alwaysdata.com/operation/435/detail/The certificate has been renewed, all sites are accessible normally.Sun, 07 Jan 2018 07:37:27 +0000http://status.alwaysdata.com/operation/436/detail/Following a significant wave of spam, the proxy server began to fail to communicate with the database containing the declaration of the email addresses. It's a behavior that is not supposed to happen with the architecture set up. We will investigate thoroughly to avoid future recidivism.Tue, 24 Apr 2018 07:59:10 +0000http://status.alwaysdata.com/operation/437/detail/The mysql server crashed. It has been restarted.Wed, 13 Jun 2018 13:15:34 +0000http://status.alwaysdata.com/operation/438/detail/The http server crashed. It has been restarted.Tue, 30 Oct 2018 13:25:41 +0000http://status.alwaysdata.com/operation/439/detail/The http server crashed. It has been restarted and we are investigating.Thu, 22 Nov 2018 11:08:12 +0000http://status.alwaysdata.com/operation/440/detail/To improve server performances we will add RAM.Thu, 22 Nov 2018 13:20:44 +0000http://status.alwaysdata.com/operation/441/detail/The http server crashed. We are investigating.Thu, 04 Apr 2019 08:44:34 +0000http://status.alwaysdata.com/operation/442/detail/The http server crashed. We are investigating.Mon, 08 Apr 2019 12:38:25 +0000http://status.alwaysdata.com/operation/443/detail/The http server crashed. It should quickly be restarted and we are investigating.Tue, 16 Apr 2019 08:16:16 +0000http://status.alwaysdata.com/operation/443/detail/It has been restarted.Tue, 16 Apr 2019 08:17:06 +0000http://status.alwaysdata.com/operation/444/detail/The http server crashed. We are investigating.Wed, 17 Apr 2019 13:03:51 +0000http://status.alwaysdata.com/operation/444/detail/It's fixed.Wed, 17 Apr 2019 13:12:21 +0000http://status.alwaysdata.com/operation/445/detail/The http server crashed. It has been restarted after few minutes.Mon, 27 May 2019 13:37:29 +0000http://status.alwaysdata.com/operation/448/detail/We'll change faulty hardware causing performance degradation.Mon, 12 Aug 2019 09:12:22 +0000http://status.alwaysdata.com/operation/449/detail/The http server crashed. We are investigating.Thu, 22 Aug 2019 08:25:11 +0000http://status.alwaysdata.com/operation/449/detail/It has been restarted.Thu, 22 Aug 2019 08:39:59 +0000http://status.alwaysdata.com/operation/450/detail/Following operation 448, the server will be moved back to its original hardware. Expected downtime: less than 5 minutes.Tue, 03 Sep 2019 14:44:15 +0000http://status.alwaysdata.com/operation/451/detail/The http server crashed. We are investigating.Tue, 17 Sep 2019 09:02:14 +0000http://status.alwaysdata.com/operation/451/detail/It has been restarted.Tue, 17 Sep 2019 09:14:41 +0000http://status.alwaysdata.com/operation/452/detail/We are investigating.Thu, 19 Sep 2019 17:19:43 +0000http://status.alwaysdata.com/operation/452/detail/The server is now responding correctly. . It was due to high memory consumption.Thu, 19 Sep 2019 17:27:35 +0000http://status.alwaysdata.com/operation/453/detail/The http server crashed. We are investigating.Tue, 22 Oct 2019 09:49:06 +0000http://status.alwaysdata.com/operation/453/detail/It has been restarted.Tue, 22 Oct 2019 09:50:31 +0000http://status.alwaysdata.com/operation/454/detail/We are investigating.Wed, 04 Dec 2019 18:19:42 +0000http://status.alwaysdata.com/operation/454/detail/The attack has been mitigated.Wed, 04 Dec 2019 18:23:00 +0000http://status.alwaysdata.com/operation/455/detail/It was an attack that has been mitigated.Fri, 06 Dec 2019 14:12:11 +0000http://status.alwaysdata.com/operation/456/detail/We are investigating.Fri, 13 Dec 2019 11:25:22 +0000http://status.alwaysdata.com/operation/456/detail/Its fixed.Fri, 13 Dec 2019 11:31:56 +0000http://status.alwaysdata.com/operation/457/detail/http8.paris1 is unreachable from several external networks. We're investigating.Sun, 15 Dec 2019 14:58:57 +0000http://status.alwaysdata.com/operation/457/detail/It only affects 185.31.40.18 (IPv6 and dedicated IP addresses are unaffected). It seems to be an issue with a DDoS mitigation from one of ours providers. We're contacting them.Sun, 15 Dec 2019 15:09:24 +0000http://status.alwaysdata.com/operation/457/detail/It should now be fixed. 185.31.40.18 had been too agressively mitigated from a DDoS by our provider.Sun, 15 Dec 2019 15:46:14 +0000http://status.alwaysdata.com/operation/458/detail/Our engineers will add RAM on the server. Expected downtime: less than 5 minutes.Thu, 09 Jan 2020 14:50:32 +0000http://status.alwaysdata.com/operation/459/detail/Our engineers will add RAM on the server. Expected downtime: less than 5 minutes.Fri, 10 Jan 2020 11:58:15 +0000http://status.alwaysdata.com/operation/460/detail/We are investigating.Thu, 23 Jan 2020 10:08:11 +0000http://status.alwaysdata.com/operation/460/detail/SMTP is purging its queue. Expected normal traffic soon.Thu, 23 Jan 2020 10:23:58 +0000http://status.alwaysdata.com/operation/461/detail/As announced in our Migration 2017 plan (https://help.alwaysdata.com/en/advanced/migrations/2017-software-architecture/), the old hostnames formats in the form of [service].alwaysdata.com, are now removed from our platform. imap.alwaysdata.com, pop.alwaysdata.com, ftp.alwaysdata.com and webdav.alwaysdata.com won't be reachable anymore. Those hostnames are deprecated for the last two years. We encourage you to check your configurations match the [service]-[account].alwaysdata.net shape.Wed, 19 Feb 2020 15:41:38 +0000http://status.alwaysdata.com/operation/461/detail/smtp.alwaysdata.com is also removed and won't be reachable anymore.Tue, 03 Mar 2020 09:56:50 +0000http://status.alwaysdata.com/operation/462/detail/This server encounters slow downs. We are investigating to stop the issue.Fri, 03 Apr 2020 15:03:40 +0000http://status.alwaysdata.com/operation/462/detail/We fixed it.Fri, 03 Apr 2020 15:33:33 +0000http://status.alwaysdata.com/operation/463/detail/As of Tuesday, June 2, 2020, it will no longer be possible to log on to our API with your email and password but only by using an access token. - If you connect with your email: you must update your application to connect via your API key (available in the Profile > Authentication view in the administration panel). - If you connect with the API key: you don't have to do anything, a similar access token matching this key will be automatically activated on June 2, 2020.Thu, 23 Apr 2020 12:55:54 +0000http://status.alwaysdata.com/operation/464/detail/Our engineers will add RAM on servers. Expected downtime: less than 5 minutes.Mon, 01 Jun 2020 12:36:46 +0000http://status.alwaysdata.com/operation/463/detail/The update has been postponed to this afternoon.Tue, 02 Jun 2020 10:33:48 +0000http://status.alwaysdata.com/operation/465/detail/Our engineers will add RAM on the server. Expected downtime: less than 5 minutes.Tue, 02 Jun 2020 12:06:10 +0000http://status.alwaysdata.com/operation/467/detail/Languages environment endpoints (accessible on /v1/environment/{LANGUAGE}/ URI) will be merged on a single endpoint : /v1/environment/. Fields name stay unchanged.Wed, 10 Jun 2020 14:23:58 +0000http://status.alwaysdata.com/operation/467/detail/This API update has been rollback and will be postponed.Tue, 16 Jun 2020 09:05:45 +0000http://status.alwaysdata.com/operation/467/detail/API environments endpoints won't changed for now.Thu, 18 Jun 2020 08:36:11 +0000http://status.alwaysdata.com/operation/468/detail/As announced in our Migration 2017 plan (https://help.alwaysdata.com/en/advanced/migrations/2017-software-architecture/), our old shared IP addresses (belonging to OVH) are now removed from our platform. 178.32.28.114, 178.32.28.116, 178.32.28.117, 178.32.28.118, 178.32.28.119, 178.32.28.120, 178.32.28.121, 176.31.58.113, 176.31.58.114 and 178.32.28.91 won't be reachable anymore. You can find the IP to use in the Advanced > Service status menu of your admin interface.Tue, 06 Oct 2020 14:19:11 +0000http://status.alwaysdata.com/operation/470/detail/This server was experiencing issues and we restarted it.Tue, 13 Jul 2021 08:45:04 +0000http://status.alwaysdata.com/operation/471/detail/Our VoIP trunk provider has issues. We remain reachable by ticket or contact forms.Thu, 29 Jul 2021 08:07:34 +0000http://status.alwaysdata.com/operation/472/detail/Let's Encrypt expiration : https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ We are investigating.Thu, 30 Sep 2021 15:06:55 +0000http://status.alwaysdata.com/operation/472/detail/We have just applied a patch.Thu, 30 Sep 2021 15:54:47 +0000http://status.alwaysdata.com/operation/473/detail/Our authoritative DNS servers are under a DDoS attack. We're working on mitigating it.Mon, 04 Oct 2021 20:44:14 +0000http://status.alwaysdata.com/operation/475/detail/The server is under a DDoS attack.Sun, 29 May 2022 18:30:46 +0000http://status.alwaysdata.com/operation/477/detail/More RAM will be added to the hypervisor.Wed, 12 Apr 2023 12:07:40 +0000http://status.alwaysdata.com/operation/480/detail/The server is being attacked by Cloudflare IPs. We've temporarily blocked them.Mon, 15 Jan 2024 16:30:55 +0000http://status.alwaysdata.com/operation/480/detail/Cloudflare IPs have been unblocked.Mon, 15 Jan 2024 17:03:49 +0000http://status.alwaysdata.com/operation/482/detail/Relocation of backup servers in another datacenter.Fri, 21 Jun 2024 13:09:11 +0000http://status.alwaysdata.com/operation/482/detail/We proceed with the relocation of the servers.Tue, 25 Jun 2024 09:51:40 +0000http://status.alwaysdata.com/operation/482/detail/The relocation is complete.Tue, 25 Jun 2024 13:52:08 +0000http://status.alwaysdata.com/operation/483/detail/We are investigating.Mon, 12 Aug 2024 08:15:26 +0000http://status.alwaysdata.com/operation/483/detail/DNS servers are now replying normally, although changes are not applied for now. We're still investigating.Mon, 12 Aug 2024 08:21:40 +0000http://status.alwaysdata.com/operation/483/detail/It's fixed.Mon, 12 Aug 2024 08:58:13 +0000http://status.alwaysdata.com/operation/483/detail/We apologize to our customers whose services or businesses were impacted during this incident. Here is our post-mortem which details what happened. ### Context Every 4 years, we're replacing our software infrastructure, based on [Debian](https://www.debian.org), with the latest available version. That is a long process that typically spans over one year and involves migrating every piece of software we use on their newest version. We've already migrated most of our infrastructure now, and new clients have been running on our new 2024 infrastructure since [early July](https://changelog.alwaysdata.com/417/detail/). Our authoritative DNS servers use [dnsdist](https://dnsdist.org/) (on the front) and [PowerDNS](https://www.powerdns.com/) (on the back). We've started preparing [changes required](https://doc.powerdns.com/authoritative/upgrading.html) to migrate to the latest version of PowerDNS, with one particular [change](https://github.com/PowerDNS/pdns/pull/6838) requiring that we replace how we currently handle SOA records. It's a relatively small change, transparent to our clients, that consists of 2 parts: * modifying how we create SOA records by explicitely setting the serial ourselves, rather than relying on PowerDNS' automatically generated serial * recreating the SOA record for all domains that we handle ### Timeline The first part went into production in early August, with no issue. New domains started using the new SOA format while all previously existing domains still used the old format. After one week, we started the second part, i.e. recreating SOA records for existing domains. The procedure is simple: call the function that creates the SOA record, which we've been using for years. We initially did it on a few test domains only, then on a few dozens domains the next day, and then on a few hundreds the day after. Again, no issue occurred. Today at 9:40 CEST, we made the change for all existing domains (about 40.000). We followed the exact same procedure we used when we deployed on a few hundreds domains. At 9:43 CEST, we received an alert about an internal DNS issue and immediately started investigating it. The overwhelming majority of domains still worked fine, including domains that our procedure had already modified, so we continued our investigation. At 9:48 CEST, as the number of errors and failing domains were increasing, we decided to abort the procedure. Domains that came before `re` (in alphabetical order) had been modified, others were not. It was clear that the procedure had broken something, but we still couldn't understand what exactly. Despite inspecting our database, we couldn't find anything wrong: old SOA records were deleted, new SOA records were present and had a correct format. At 9:55 CEST, we decided to activate our DNS disaster plan. It changes the way PowerDNS operates: rather than querying our database, it uses simple flat zone files that are automatically generated every 30 minutes. It means that DNS changes made during the last 30 minutes are (temporarily) lost. At 9:59 CEST, as PowerDNS had issues starting with the disaster config, we prepared to restore a backup of our DNS records in parallel. At 10:05 CEST, we fixed issues that prevented PowerDNS from starting in disaster mode. We started a few tests before deploying the change. At 10:10 CEST, all production DNS servers were running on the disaster config, everything was working normally again, albeit in degraded mode (DNS changes made since 9:30 CEST were not visible). At 10:56 CEST, the original issue had been found and fixed, all production DNS servers were running the normal config again, and all changes applied during or before the incident were now visible. ### Impact For all domains that come before `re` (in alphabetical order), our DNS servers (`dns1.alwaysdata.com` and `dns2.alwaysdata.com`) may have returned an error (`REFUSED`) between 9:40 CEST and 10:10 CEST. However, due to cache (on our side, thanks to dnsdist, or on recursive DNS servers' side) and other factors (alphabetical order of your domain, TTL expiration, etc.), observable errors were rather random. ### Root cause When a DNS record is added, it's initially in a disabled state. A procedure is then run to determine whether the record must be enabled and if others must be disabled, which happens when a user adds a `A` record that overrides our own, for instance. Finally, the cache on PowerDNS and dnsdist is flushed if necessary. In normal cases, the procedure is almost instantaneous. However, in the very rare cases where we need to change hundreds or thousands of records at once (which is always a manual operation), the procedure is not executed on *each* record but batched and executed at the very end, for performance reasons. This is what happened this morning. As our code iterated on domains, their old SOA record was deleted and a new one was created, in a disabled state. As the procedure that would enable the new record is only run at the end, there was a period of time during which no SOA record was enabled for domains. However, that is not supposed to cause issues as PowerDNS still uses the SOA in its cache as long as it's not flushed, which only happens *after* the new record is enabled. Unfortunately due to [a bug](https://github.com/PowerDNS/pdns/pull/10535) in the PowerDNS version we use, the cache could be flushed when the TTL of the record expired, which is 5 minutes by default on our servers. Although the SOA record is rarely queried directly, PowerDNS refuses to serve a domain if it cannot find a SOA record. Unfortunately, the reason we did not get any issue when we deployed the change on a few hundreds domains is that even a few hundreds was too small to significantly delay the batched procedure, especially as the 5 minutes TTL had to expire before PowerDNS would query the database again. ### Prevention First, that particular PowerDNS bug is fixed in the version we're preparing to upgrade to (ironically). Second, we will test our disaster config more regularly, as not getting errors when starting PowerDNS could have saved us almost 10 minutes. Third, even after a procedure was deemed safe, we should as much as possible deploy by *chunks* rather than all at once (which is what we already do for most of our daily operations).Mon, 12 Aug 2024 15:28:41 +0000