We are very sorry to say that a client's email account on mail1.ecohost.coop has been compromised and used to send a massive volume of spam and this has caused the server to be blacklisted, we have secured the account and are now seeking to have the server de-blacklisted.

We have requested the removal of webarch.email from as many anti-spam blacklists as we can, however it might be up to a week before the server is fully delisted, we are very sorry for any inconvenience caused by this.

The update to webarch.email has been completed and we have also updated the IP address of this status page (see this article for the background) so it soon be available again. We are sorry for any inconvenience caused.

We are rebuilding all our DNS servers, the work on the Sheffield ones, dns0.webarchitects.co.uk and dns1.webarchitects.co.uk has been completed. The European server, dns2.webarch.info has been rebuilt and it is now based in Paris and has a new IP address, 217.70.190.127. Our Icelandic server, dns3.webarch.info, is in the process of being rebuilt and when this is completed it will also have a new IP address, 185.112.146.79.

We are seeing very high loads on host3.webarch.net and we have shutdown Apache and MySQL on the server while looking into this issue. host2.webarch.net is running a filesystem check and should be up soon.

We are having issues connecting to the IPMI on xen2.webarch.net, so it hasn't yet been rebooted, so we are restarting the virtual servers on it and will shut them down again when we are ready to reboot the server, sorry for this delay.

Due to the Meltdown and Spectre exploit critical vulnerabilities we are planning on rebooting all our Sheffield servers during the course of the morning of Wednesday 10th January 2018, we don't anticipate that there will be a great amount of downtime but all services will be affected for a while.

Restoration of all the virtual servers in Iceland is now looking hopeful, the lastest from @1984ehf: "24 hours ago we thought that nearly all VPS data was corrupt beyond repair. We have since found a way to restore disks from the VPSs. This means that we will be able to retrieve data from a lot of the VPS disks and we will help every one of our VPS customers to find their data.".

We don't yet know if the virtual servers in Iceland can be restored, see @1984ehf for the latest news on this. We are starting to restore sites that were hosted on host-b.ecodis.net, from backups, to shared servers in Sheffield.

An update on services in Iceland from @1984efh: "After spending a night with a team of experts in the hardware field, our costly setup was deemed fit for scrap. We have a backup in a distant location and are now working on making it functional from that location.".

We are afraid it is going to be some time before mail.webarch.net is up and running again, we are copying the disk across the network and have copied 41G out of 275G — it is going to take some time to complete.

The file system check on mail1.ecohost.coop is 53% complete and might take another hour or two to complete, then that server will come up again. The server that was running several CentOS servers, cat.webarchitects.co.uk, which had an uptime of 2017 days, had two RAM chips fail after the move, after removing them the server still won't boot, so we moved the disks into another server and used a rescue disk to repair the RAID 0 boot partition on all the disks. The server that the disks from cat are now running in doesn't have enough RAM to run the virtual servers, so we are now copying the partitions over the network to our FreeBDS ZFS file server and when that is completed we will start them up on xen1.webarch.net. This means that it is going to be quite a few hours before these servers are up:

The mail.webarch.net Webarchitects mail server

The Mailman server which hosts email-lists.org and several other domains

We believe all our Sheffield services are up and running again apart from mail.webarch.net, our Mailman server and our MKDoc server, we have had some hardware issues with the physical server these virtual servers run on and are trying to solve the problems.

One account on mail.ecohost.coop was being used to send spam, we have stopped the abuse, but there is a possibility that this will have a negative impact on other users of the server if the server is blacklisted.

We are about to conduct some planned maintainance on mail.webarch.net and webarch.email, this will involve an interuption to your email service, but no email should be lost, we will post up update when the work has been completed.

host1.webarch.net and host3.webarch.net went offline between 1:20am and 1:30am this morning and were not available until 8:30am this morning, we sincererly appologise for any inconvenience and we are investigating the cause of the issue.

We are very sorry to say that we have discovered that our ticketing system was not accepting tickets for 14 days over Christmas and the New Year due to a misconfiguration error. If you tried to contact us during this period you should have recieved a Undelivered Mail Returned to Sender response, please resend your original email as the ticketing system is now up and running again. We appologise for any inconvenience caused by this outage.

The mail1.ecohost.coop server has now been removed from all but one blacklist and for this one "Automatic removal will occur for IPs that are seen to be clean". We have sent an email to users of the server to explain what happened and to appologise for any inconvenience caused.

The mail queue on mail1.ecohost.coop has now cleared but we are sorry to say that the server is now listed on some blacklists due to a compromised account on the server being used to send a large volume of spam — we are now working to have it removed from the blacklists. Sorry for any inconvenience caused by this and please ensure that all the devices you use for email are secure!

We are about to reboot the Webarchitects mailserver mail.webarch.net in order to add additional RAM to it. There will be a short downtime for the MKDoc server as we are removing some RAM from that machine as it is running fewer sites these days.

We believe we have discovered and blocked the abuse of services running on mail1.ecohost.coop and we are now requesting that this server is removed from blacklists. Once again we are sorry for any inconvenience caused by this.

One of our email servers, mail1.ecohost.coop is currently listed as having a 'poor' reputation on the Barracuda Reputation System, this is causing some email delivery issues, we are investigating the cause and when we have tracked it down we will request to be delisted. Sorry for any inconvenience caused by this.

This site and the DNS servers for webarch.info are hosted totally independently from all
other Webarchitects server infrastructure to ensure that any outages elsewhere don't affect this site. The
DNS servers are provided by Gandi and the web hosting by
GitLab.

The code for this site and a script for updating it, can be found at
GitLab. This
site takes a few minutes to be regenerated after an update is committed
to the status.html file,
you can check if there are any outstanding updates by looking at the
jobs page.