Web Hosting Status

Current System Status

Power outage

15:05 10/02/2016

Current Summary Of Issues (12/02/16 - 08:50)

Linux Shared Hosting - An issue still exists on the following web servers unfortunately; web75, web76, web77 and web78. We are currently looking into the issues with these servers to get them back as soon as we can.

Current Summary Of Issues (11/02/16 - 22:00)

Webmail - The webmail service appears to be back up and running again now, and you should be able to access your mailboxes through this correctly.

VPS & Hybrid Servers - kvmhost248 and kvmhost249 are now fully operational. Please contact support if you have any further problems.

Linux Shared Hosting - An issue still exists on the following web servers unfortunately; web75, web76, web77 and web78. It did appear that this was resolved, but further investigation has shown that there is another issue here. A separate issue has also been identified on web servers 175-183, and is under investigation now.

Premium Hosting - The Premium Hosting service appears to be back up and running again now, and the issue with MySQL database creation has also been addressed and resolved.

UPDATE Thu, 11 Feb 2016 11:20:00 +0000

At present some customers may not be able to log into their control panels due to an overwhelming number of requests, we are working to resolve this. Also on Linux shared hosting we are seeing some MySQL problems with being able to connect to databases, again we are working on this too and should be resolved very shortly.

UPDATE Thu, 11 Feb 2016 12:28:00 +0000

The Premium hosting service is now back online and should be up and running again correctly. Please let us know if you are still experiencing any issues, but this should now be resolved. Webmail is also responding to requests again correctly, though we have had some reports of issues with mailboxes outside webmail which are under investigation now.

UPDATE Thu, 11 Feb 2016 12:40:00 +0000

We have been informed of an issue creating new MySQL databases on Premium Hosting packages here, and are working to investigate and resolve this as soon as possible.

UPDATE Thu, 11 Feb 2016 13:35:00 +0000

Work is continuing with the investigation to get the remaining kvmhost machines up and running, this was initially expected to be resolved within the morning, but unfortunately this work has carried into the afternoon.

UPDATE Thu, 11 Feb 2016 14:10:00 +0000

We are seeing some issues on web servers 175-183 currently, and the admin team are investigating this. We hope to have this resolved shortly.

UPDATE Thu, 11 Feb 2016 15:38:00 +0000

The database creation issue on premium hosting has been resolved now, and MySQL database can be created again correctly.

We of course apologise sincerely for any and all inconvenience caused by the above issues. Further updates will of course be posted here regularly. Below is the historical report and updates for these issues.

- - - - - - - - -

HISTORICAL UPDATES

Wed, 10 Feb 2016 15:05:44 +0000

We suffered a brief interruption in power during works in one of our data centre halls. This has now been restored and all servers are coming back online. We will provide further updates as work progresses every 20 minutes

Wed, 10 Feb 2016 15:50:00 +0000

While a top priority is to keep your websites and services online, our highest priority is the safety of our engineers. When working with equipment such as UPS (massive battery arrays), generators, switching equipment and multiple sources from the mains, we ensure our team are safe. In this case, a voltage sensor was due for replacement, we had switched over to bypass this unit, however for a reason yet unknown a safety warning was triggered which resulted in the shut down of power for about 9 minutes. The full reason is yet unknown, we have our expert data centre team, as well as the equipment providers, working to understand this fully. Right now we can assure you that power is flowing to the data centre as it should be.

Wed, 10 Feb 2016 16:01:00 +0000

Currently our priority is to restore service to infrastructure servers, this is in progress now and will allow us to being work restoring service to the shared platform.

Wed, 10 Feb 2016 16:07:00 +0000

Infrastructure servers are now showing as fully online. Work ongoing on the shared platform and the current ETA for restoration of service is 17:00

Wed, 10 Feb 2016 16:14:00 +0000

Hybrid VPS servers are now being powered on. A tentative ETA for restoration of hybrid servers is 17:30

Wed, 10 Feb 2016 16:38:00 +0000

VPS KVM host servers are now being powered on. Guests are still currently unavailable

Wed, 10 Feb 2016 17:08 +0000

Premium hosting front end webservers are now coming online. Databases are currently still powered down

Wed, 10 Feb 2016 17:37 +0000

Premium hosting database servers are now being powered on.

Wed, 10 Feb 2016 18:09 +0000

Apologies for the delays. Servers are being powered up in stepped process in order to prevent further complications. Once an individual server is powered it needs to be fully checked for unexpected behaviour due to the sudden power loss. Work is ongoing.

Wed, 10 Feb 2016 19:50 +0000

We have identified a failed switch serving a number of KVM hosts. If your VPS server is entirely inaccessible please await a further update.

Wed, 10 Feb 2016 19:57 +0000

A large portion of shared server sites are now online. Some clusters still have inaccessible databases and backend storage. Work is in progress on the remaining clusters.

Wed, 10 Feb 2016 20:16 +0000

We have identified a problem with webmail backend storage. No stored mail has been lost but the webmail service will be unavailable until restoration from backup is complete.

Wed, 10 Feb 2016 20:48 +0000

If your VPS or Hybrid server is entirely unresponsive (does not respond to ping) please await a further update, KVM hosts are still being powered on. If your server is accessible via SSH but the individual sites hosted on it are offline please contact support.

Wed, 10 Feb 2016 21:07 +0000

Mail delivery should now be completely restored. If you have any outstanding issues please contact support. Please note that while mail servers were offline delivery will depend on the retry time of the sender.

Wed, 10 Feb 2016 22:00 +0000

The following KVM hosts are still unavailable due to a failed switch caused by the power outage.

KVMhost107

KVMhost150 to 155

KVMhost159

KVMhost164 to 167

KVMhost196

KVMhost87

KVMhost89

We also have the following KVM servers in alternate cabinets down:

KVMhost102

KVMhost182

KVMhost249

KVMhost4

KVMhost74

KVMhost88

If your VPS is on a host not listed above but is still offline please contact support

Wed, 10 Feb 2016 22:22 +0000

The switch above has been replaced and all hosts apart from KVMhost88, KVMhost249 and KVMhost74 are now responsive. If your individual guest still has problems please contact support. Please note the support team are dealing with a large influx of tickets but we are working through them chronologically.

Wed, 10 Feb 2016 23:05 +0000

Around 60% of shared sites are now online. Work is continuing on backend storage.

Wed, 10 Feb 2016 23:31 +0000

KVMhost69 is still unresponsive. The sysadmin team are currently investigating the cause

Thu, 11 Feb 2016 00:55 +0000

The following kvmhosts still have an issue -

kvmhost153

kvmhost237

kvmhost4

kvmhost69

kvmhost70

kvmhost74

kvmhost248

kvmhost249

Thu, 11 Feb 2016 01:55 +0000

The following webservers still have an issue - web75, web76, web77 and web78. This should hopefully be resolved shortly. We also discovered problems with MySQL on web21, web22, web23 and web24 which has been resolved.

Thu, 11 Feb 2016 03:00 +0000

Webmail - unfortunately we still have problems with the webmail system, it is sadly likely to be unavailable for several hours. Apologies for the inconvenience caused again.

Thu, 11 Feb 2016 04:00 +0000

We are still working our very hardest to resolve the remaining kvmhost and webserver (web75, web76, web77 and web78) issues, many of our staff have been in the office since 8am yesterday and are still here now. We will continue to keep you updated here.

Thu, 11 Feb 2016 04:41 +0000

kmvhost88 which we discovered had some issues is now working.

Thu, 11 Feb 2016 06:05 +0000

We discovered a problem with the webservers web175, web176, web177 and web178. This has now been resolved, sites will be slow to load for a short while due to the sudden influx of traffic but this will calm down soon.

Thu, 11 Feb 2016 06:45 +0000

The problems with kvmhosts kvmhost4, kvmhost69, kvmhost70, kvmhost74, kvmhost153 (this was actually wrongly reported earlier and was fine), and kvmhost237 have now been resolved. There are still issues with kvmhost248 and kvmhost249 unfortunately that will sadly take at least a few hours to resolve.

Planned System Maintenance

There is currently no planned maintenance.