ClaraNet customers have been without email access all day, after the business ISP's POP3 system took a lie down this morning.
Engineers are battling to fix the snafu, which struck just after 9 this morning. ClaraNet said it hoped to have a partial fix in place by 6pm this evening.
"It will not however be fully functioning with …

8 Hours to build a POP3 server?

How Long?!

Christ - even if it's not a VM (which takes 20 minutes to restore back exactly as it was before without configuration) you can get an Exchange box up in well under an hour if using a deployment system - which one would hope as an ISP.

Even manually creating a server it shouldn't take more than a few hours....

Working now

At least for me. It was down until around 4:00pm.

I have been with ClaraNet for over 8, yes eight years. This is the first time I have had a problem accessing email. And during that 8+ years the number of times I have been without Internet access must be in single figures.

Expensive they maybe, but they are reliable and when they quote 8Mb bandwidth/download speeds they actually mean it, and it does not depend on the time of day either, in addition there does not appear to be any port blocking or throttling, at least I haven't noticed.

Yes taking several hours to get a pop3 server back up and running does seem a long time. Despite this they are way ahead of other ISP's in so many ways.

It would take more than a few hours of email sever down time for me to class ClaraNet's service as anything other than excellent. I wouldn't give an ISP 8 years of loyalty if they were anything less.

just a thought

Claranet: big

I am a Claranet customer, have been for eleven years, and the only other interruption to service I have suffered on this scale in that time was when BT-operated facility in Ilford caught fire, and broke an awful lot of wholesale ADSL, not just Claranet's.

This is/was not rebuild a POP server. The error message displayed on the webmail login page for a time was: "We are currently experiencing service problems on the Claranet E-mail platform. Claranet Engineers are currently working with our Storage Vendor to rectify the situation.We regret any inconvenience this may causing you."

Claranet, between the business services and the retail services and the services shared with other ISPs they have bought or with which they have partnered, is pretty big. A lot of their stuff used to run on FreeBSD on SPARC hardware; I don't know how much of what still does. It always appeared to me, based on there being multiple public server names that one could use to configure sending or retrieving mail, that there were actual different internal servers providing redundancy and they were all capable of reading to or writing from the same disk files for serverside mail filters, mail filter logs, POP3 mailboxes, and so on -- presumably these files were in some way on a network share, which is not a configuration I'd have chosen, but then what do I know, never having had to spec up a mail handling system even a thousandth the size of this one.

"currently working with our Storage Vendor" I'm going to take a punt: that is to say, the controller card for the RAID went bang, the guy from Sun turned up within x hours as per the Gold/Enterprise/WeLikeJonathansPonytail -level support contract, and yet somehow, despite the customer having the biggest best support contract available, despite knowing what hardware the customer had, despite being told by the customer what component on what system had failed, the guy from Sun did not have the required part on the van.

The interruption to service was a nuisance. As best I can tell no incoming email was lost, and I (and every Claranet customer I know personally) had workable alternative provision for sending and receiving email. The biggest bitch I have right now is that the serverside mail filters aren't available and so 90% of the spam I get sent ought to be deleted before it hits my POP3 mailbox, but currently it's being delivered instead. Anybody want a stack of pointless emails from the likes of sue@casimme.co.uk punting ' ...Business seminar informmation'?

@jake

You mean like making their support staff redundant?

Some people need to learn that resiliency is measured in "number of 9s" for a reason... no-one who's worked in the industry will quote 100% uptime because there's always something that can happen, so lets do the maths:

The poster above says he's been with them for 8 years with no problems, so that's 2922 days

which is 70128 hours

They've now had 12 hours of downtime

12/70128=0.00017112

so 99.983% uptime - that's 3 nines.

Respectable, not exceptional but not too bad.

3 nines is all most people aim for, whilst hoping for more... some people/services aim for 5 nines, which in the same 8 years would equate to 4 minutes of downtime!

To the nines ...

"Some people need to learn that resiliency is measured in "number of 9s" for a reason... no-one who's worked in the industry will quote 100% uptime because there's always something that can happen, so lets do the maths:"

Yeah, yeah, yeah. It's part of what I do for a living, FFS. My own, personal email system has been up and available, non stop, for over a quarter century. Granted, I'm not one of the big-boys ... this is a personal research system, initially built to look into RFC-821 in early '83ish, now used as the family's email, FTP and WWW services. The concepts scale. I have consulted for Fortune 150s, the ones who used my advice should experience similar uptimes.

Note that I said SYSTEM. This isn't a single box located in mommy's basement, and it never was. It started at Stanford, MAE West, and under Bryant Street in Palo Alto. Now I have nodes in three countries. This is a geographically diverse, multihomed, multiple box, multiple OS, multiple MTA system. Redundancy is designed in everywhere I can fit it. Over-kill for a simple family server system? Of course. But as a research platform, it's mostly tax deductible.

Anyone who can ID me from the above, please keep it to yourself ... but feel free to give me a call if you want to get back in touch. I'm in the book, in Sonoma CA.

(I got calls from two old friends last time I posted the above offer ... Thanks, El Reg! :-)

@Jake

@just a thought

claranet's email system

Claranet's mail system is just a *leedle* bit more complex than an exchange server ;) My guess is that it was the storage platform that caused the issue, and rebuilding a filesystem on a storage platform that size might well take several hours.

Weak Communications

What irritates me is the weak explanation of what is going on. They didn't have a PR available to the journalist on the Register to explain ["We couldn't get hold of ClaraNet to find out what went wrong"], their Service Announcement is limited and unclear and does not inspire that they are in control of this problem and earlier they made promises that they did not keep nor did they reply to me from support with anything more than an AutoReply that said nothing - not even a holding standard reply. At one point the web-link to support appeared to fail. These guys are communications clowns.

I appreciate accidents happen but this does not fill me with confidence and a failure to deliver a week of archived material on one email (which I look at weekly on PC and delete on blackberry on a daily basis, relying on 'hard' copies later) will mean I go elsewhere - losing a week of communications is a serious matter if it gets that far. Thanks to Facebook, I am getting a flood of recommendations on alternative suppliers and, I hope, that when I check in in the morning (it is UK time) I will find everything in order or the search starts on Monday.

We need an explanation from the company of who screwed up and why, with assurances that it won't happen again ... no assurances and we and others go.