Posted
by
samzenpus
on Wednesday July 16, 2014 @06:54PM
from the turning-the-lights-back-on dept.

First time accepted submitter Dragoness Eclectic writes Early Tuesday, gamers woke up to find out that they couldn't log in to any Sony Online Entertainment games--no Everquest, no Planetside 2, none of them. Oddly, the forums where company reps might have posted some explanation weren't reachable, either. A bit of journalistic investigation by EQ2Wire came across the explanation: SOE forgot to renew the domain registration on SonyOnline.net, the hidden domain that holds all their nameservers. After 7 weeks of non-payment post-expiration, NetworkSolutions reclaimed the domain, sending all access to Sony's games into an internet black hole. Sony has since paid up. SOE's president, John Smedley, has admitted that the expiration notices were being sent to an "unread email" address.

Hole in someone's head, maybe - after all, a simple spreadsheet to track something this basic or a reminder in a calendar with alerts with someone assigned to keep an eye on things would take care of things like this. They're lucky it wasn't held hostage...

You want to assign someone to keep an eye on things that can be fully automated?

It's a basic accounts payable function, so yes.

Someone has to have authority to maintain and modify the automated payment schedule, otherwise either anything can be added/removed or nothing can be added/removed. Moreso, someone within their accounts payable department should be specifically responsible for all these particular kinds of payments: trademark fees, site ownership fess, official registrations, patent renewal fees, etc. That person should have lost their job today.

Same thing happened to Turbine a couple years back: DDO, LotR, etc all down for exactly the same reason. You wouldn't think this would be that hard to get right, but chances are no one in dev at either company survived from the early days to when the problem happened, so the tribal knowledge was lost.

Wow, that's a non-sequitur (and, BTW, like most things on Slate, the argument fails: the SCOTUS ruling explicitly hinged on the fact that Hobby Lobby was a closely held corporation, and thus no different from a partnership It was not a broad ruling applicable to corporations in general, where the linked argument might have been relevant.)

This sort of lapse has happened in every company I've worked in, big and small, when the person formerly responsible for this kind of thing leaves the company and someone else has to pick up their responsibilities. Sloppy, unorganized? You betcha. Also what I've come to expect.

I've fucked up and forgot when an SSL cert was about to expire. I found out the next morning when their iPhones could no longer access the Exchange server. Shit happens. This time I include SSL, Domain, and Server hardware warranty expiration notices scheduled way in advanced in my calendar as an event.

It wouldn't hurt to have a distribution group for this and then make yourself and others a member of the group, even if it's your boss. Best case scenario, he gets the alert and says, Bob, did you see the alert about... Already took care of it this morning. Good man.

Anyone on Slashdot who gets smugly superior about this and how "stupid companies are" is just being a hypocrite. We have ALL forgotten things in our lives. We've all forgotten an event we were supposed to be at, a bill we were supposed to pay, something we were supposed to bring with us. It happens.

What's more, everyone has been in a situation where something didn't happen because they, and everyone else, assumed someone else was going to deal with it. You don't go and check on everything that ever happens around you or involving you, you mentally categorize things you are and are not responsible for and ignore the latter.

So ya, companies, which are made up of people, can fuck up too. It's amusing, but perfectly normal.

Companies are STUPID because they've gotten hooked on the idea of "employee efficiency" to the point that employee efficiency is being negatively impacted. In the past, when a mistake was made, you could easily nail multiple employees simply because they were supposed to be watching/covering one another. If one (or more) screwed up, it meant the others weren't doing their job so they all got punished. It cost a lot more in payroll, but it made sure the job got done, on time, correctly (as far as procedures were concerned). Nowadays, GM can't even find ANYONE to pin the blame on for the ignition switch recalls.

So yeah, companies can fuck up too. But when you can't even find someone within the company you can point to say "that person is the one who fucked up", what does that say about the company?

Corporations are stupid for simply assuming that people are automatons. You come to work and do it flawlessly, always following the ISO 9001 standard. Yeah. Sure. And monkeys fly out of my butt.

People are people and people are making mistakes. Always. Every single day. Anyone in security learns that VERY quickly. And he also learns quickly that you cannot trust humans to be flawless. Not because people are stupid but because people are NOT automatons and make mistakes. Yes, even (actually, especially) if doing the same job for ages. Show me a person who makes no mistakes and I show you a person who does no work!

Security is FINALLY starting to get wise and build systems that are tolerant of human error. Let's see how long it takes 'til the rest of the system catches on.

Dude/tte, I have five domains. All are paid up through 2020. In 2015 I shall extend them until 2030. It's not hard to do and it's not hard to remember, FFS. For a company that large to neglect is inexcusable. Just buy them for then next 20 years, it doesn't cost all that much.

The whole point of having a corporation (or any other sort of team for that matter) is that you find ways to be less failure-prone than you are as individuals. You have to do this to offset the fact that a failure of the group affects every member - the cost is multiplied.

It's pretty easy to forget when the renewal date is so far off. Plus if they do everything online and via mail it's easy to lose the reminders and email addresses change all the time in corporations, but snail mail often gets to the right department at least. Even if it's on someone's calendar, that person gets laid off or quits or transfers. IT groups especially have high turnover from top to bottom. If someone pays for the upcoming 5 years, that person will almost certainly be gone from that job when

When I fire someone, I redirect their email to their supervisor. It's right there in their employment contracts that their work email address and any correspondence are the property of the company (as if that wasn't obvious, but CYA applies). For things like this we have title addresses like dnsadmin@example.com, noc@example.com etc. which are broadcast to several staff responsible for the management of such affairs.

Also payments such as these are lodged in our recurring expenses ledger and paid by account

Originally I haven't planned on entering this but when I clicked the link I got this.

This bill was introduced on May 12, 2004, in a previous session of Congress, but was not enacted.The text of the bill below is as of Jun 09, 2004 (Reported by House Committee).

To be clear this means president Bush never signed it into law. It also means, that as it isn't a law the other person is right unless you can find one that was.
As for SonyOnline.net the best thing would be a redirect to Piratebay.se

It was done as part of another act in December 2004. If you look at the Trademark Act 1946, it contains the amendments in GP's link.Regardless, it only applies to calculating the damages provided in a trademark violation involving a domain name.

Okay, are you suggesting that accurate reporting to ICANN is covered by the Trademark act? The point I made was the bill he offered and promoted as a refutation was never a law because it died in committee. If you have the other acts name please share

Read it. Its just a booster. It makes intellectual property violations considered 'willful' if involving a fraudulent domain name, and takes on extra penalties if you commit a crime involving a fraudulent domain name.

It doesn't make anything illegal, nor does it give ICANN force of law.

There's no law per se, but there is a recent ICANN requirement called "Whois Accuracy Data Specification". It requires registrars to contact the registrant and click an emailed link as validation that their whois info is correct. The domain can be suspended if the validation isn't done within 15 days.

The intent is good but the implementation is pretty mindboggling. They're expecting every owner of a domain name to check that the email sent to them is not a phishing attempt...how that's supposed to work reliably is anyone's guess.

So, yeah, owners are supposed to verify to the registrars that the info is accurate which you could say is "ICANN's law". But not legally. Here's one of many articles that goes deeper into the issue:

I'm afraid that the current "whois" practices were deliberately set up to allow plausibility deniability, to protect the domain owners from being actually reached by the spammers and numerous sales people or lawyers with cause to contact domain owners. The domain vendors benefit from this: they can follow the letter of the law, but not actually support contacting the domain owners to handle criminal or abuse behavior, and wait for days, weeks, or years while lawyers collect the evidence and chain of repeated contact failures before a court order can be obtained.

In the meantime, they're collecting the registration fees, in bulk, for the relevant domain and all the related domain names. The current system is a critical revenue stream, which the domain and SSL key vendors have no need or desire to encumber by enforcing legitimate contact information.

It looks like that is specifically tied to using false whois info if there is a subsequent copyright or trademark infringement, not if Joe Average decides to put 123 Main St. as his contact address. Seems like the law is a tool that can be used to help prosecution of Lanham violations (there probably aren't many criminals who keep their whois info up to date;)

Not the GP, but yeah, I do - Can you explain what an anti-domainsquatting law that specifically deals with trademarks and identity theft, and absolutely nothing to do with simply giving fake info to a registrar, has to do with your original claim that giving ACCURATE contact info counts as US law?

Now, ICANN can enforce its policies on the registrars themselves, simply by virtue of the fact that a registrar requires ICANN's continued blessing to operate. But the only recourse they have about (non-identity-stealing) fake registration info comes down to taking the domain away from you. For someone like Sony, that might look like an end-of-the-world scenario. For someone who just wants a named place to stick stuff online for my own personal use? Meh, worst case, I've lost $10-$15 and I have to wait three days for a new domain to propagate (and not always even out the money - Much to my surprise, I actually had GoDaddy refund me when I flatly refused to send them a photocopy of my license, three months into a registration).

Just to add my 2c. A while ago I was working on a project which could use data in WHOIS records. Ultimately this failed because the data is very unreliable and mostly unavailable, but I did come accross some laws.

Seems the U.S. is pretty much the only country that has a law on this, and it just says that it is illegal to have inaccurate information in a WHOIS record if and only if you're using that inaccurate information to scam people. So basically you can use inaccurate information all you want but if you

Actually, 10 years is the max registration. And that's exactly what I do. Throwaway domains that I'm experimenting with might only get a year or 2, but once anything becomes important to my business, it gets renewed for 10 years. The same is true for my personal domain. And every couple years I go through and bump it back up to the max. I'd literally have to go 10 years without remembering to renew a domain before one would expire. I can't see why any business would do otherwise.

Network Solutions offers kinda-sorta-100 year registration. Technically it's just a ten year registration that they automatically renew for you every ten years, but it still would've saved Sony a lot of trouble in this case.

20 and 100 Year Domain Registration Service - If the domain name registry of a particular third level domain does not provide for an initial registration term of 20 or 100 years, then Network Solutions will register your domain name on your behalf for the ma

Because something that has to be done every year gets done every year, like taxes.

Something that has to be done every 10+ years is a lot more likely to get lost and forgotten. Sure, you could set a reminder...but where? Staff get replaced, calendars get replaced, software gets replaced, computers get replaced, offices get cleared out, and the people who trained the current employees weren't even around themselves the last time it needed to be done.

How it could happen is pretty simple, someone is working on a new service, they are in a hurry and just buy the domain with a company credit card or a small one time PO or whatever putting their individual work email address as the contact info. They register it for a few years, maybe even the maximum of 10. Maybe they set a reminder for themselves to renew it, maybe they don't bother as they think it unlikely the domain will stay in use that long.

The procrastinator in us invariably assumes there is seemingly an infinite amount of time to take care of this... by the time they send the We really fucking mean it this time! notice, well hell, they've been crying "wolf" so long it doesn't mean anything.

And to be fair, didn't the nerdtastic Mecca website herself forget to renew a certificate recently?

I went to a training session for our new $50k accounting system. They had forgotten to renew their own license for the training classroom. Took an extra hour to get their tech in there to get it fixed. Yup, should have got up and went home at that point.

sigh

We bought it cause it was industry specific (well focused at least) and by a small company that only did this for 20 years. Next year they are bought by a national company and instead of being 1 of 200 customers now we were 1 of 20000 on a minor product.

This is why you don't directly use employee email addresses for certain business activities. These activities get their own emails which forward to whoever the responsible person or persons are. Ex. domain_registration@sony.com. Note "forward to", these would not be standalone email addresses that someone has to log in to.

So, forward domain_registration@sony.com to former_employee@sony.com. Let us know how that works out for you.

That's why I wrote person or persons. Plus when someone is told they are now responsible for or involved in domain registration they go update the recipient list for the email address. There is no need to update some outsider's records. There is no need to get into the former employee's email. It really is an improvement over using employee emails directly.

Whoosh. That only make it easier. It doesn't fix the process, which still requires tracking and making changes to make it effective.

Actually you might want to re-think who is having the woosh moment.:-) I never said it fixed the problem. I offered a practice that is an improvement, i.e. forwarding and multiple recipients, that reduces the opportunity for unread emails.

It still requires tracking and making changes. It's easier to change the local email system than a registrar's database, but in either case, updates must be made to be effective. With 10 year registrations available, there's no guarantee that former_group_members@example.com is much better than former_employee@example.com, especially in fast moving industries. If company X acquires company Y, dns@y.com is apt to be forgotten, too.

You're suggesting a tactical solution to a process issue. Better to have the

You're suggesting a tactical solution to a process issue. Better to have the responsible group track and update necessary renewals on a regular basis, instead of depending on notifications from external parties being received.

I only hold a couple of dozen domains, but this is exactly what I do. I get notifications from the registrar directly to a specific e-mail address I've set up for that purpose, but I also automatically generate an email to my personal account on the first of each month reminding m

With 10 year registrations available, there's no guarantee that former_group_members@example.com is much better than former_employee@example.com, especially in fast moving industries.

Stop thinking in terms of employees, that's the point of this exercise, the email addresses on the distribution list can include functional roles. company_web_site_manager@sony.com, senior_web_admins@sony.com, etc. Basically the slots in the corporate org chart come with an email address based on the function so you don't necessarily have to know who the person in that role is nowadays.

You're suggesting a tactical solution to a process issue. Better to have the responsible group track and update necessary renewals on a regular basis, instead of depending on notifications from external parties being received.

So your calendar server has a list of people rather than your email server, that's not much of a difference.

So, forward domain_registration@sony.com to former_employee@sony.com. Let us know how that works out for you.

It works a lot better because if domain registration emails are being sent directly to former_employee@sony.com, then only he knows that domain registrations are being sent to him. There is no record at Sony saying that he was the one getting those emails.

If you instead have it sent to domain_registration@sony.com with a forwarder, when former_employee is fired, the sysadmin can look at the enti

So, your plan is that former_sysadmin@sony.com makes the change. OK, but how is keeping track of what emails need redirection when an employee arbitrarily changes more reliable than keeping track of when registrations expire, which is known well in advance? Is it somehow easier to remember to grep email accounts for "dns@example.com" than to query a database for "domain_expiration<90days?"

See the difference? One places responsibility for a mission critical function on an external party, and the other d

I long for the good ole days when they actually send out paper invoices in envelopes!;^)

And from the archives:

"In December 1999, Microsoft forgot to renew the domain name Passport.com,and so rendered its Hotmail service partially crippled. A Linuxprogrammer, Michael Chaney, paid the $35 fee and promptly handed overownership to Microsoft."

I long for the good ole days when they actually send out paper invoices in envelopes!;^)

You actually still look at your paper mail? I tend to assume it's all just spam. Then again, I tend to assume that of my email, too.
What was the last year we had a communications system that had more signal than noise? It seems to have been a while.

If the address was unread now, it must have been monitored originally.

Not necessarily - I have a domain. It has a "real" administrative contact email (a throwaway GMail account). I haven't checked it since I had to confirm it as valid (the registration just autorenews - Pssst, SCEA, you live off subscription models, ever thought of using the same damned idea to keep your domains/certs/etc active?).

Administrative contacts for a domain amount to nothing more than a pre-confirmed spam address. Why the

come on guys.. There's lots of reasons to hate on SOE. Hell, I haven't bought an SOE product in 10yrs because of the Foglok fiasco... I was actually banned from their forums for a few months back in the day for suggesting they didnt exist, only later find out I was right. The title of the freek'n thread to announce the disappointment was "CharlieMopps was right, not a troll, there are no frogloks!!!" (paraphrased, the threads been deleted for some time now) If you don't know what thats about you've no reason to hate on SOE. Ok ok, I'm just tryning to point out I have no love for them...

Anyways... Managing a domain is a pain in the ass. I've worked in a few places with large website, I'm sure a few of you have. Maintaining that domain registration is deceptively difficult. Think about it as if you were the one in charge of it.

You tell your staff "Register out domain!"They go off and come back "well, it appears we can register it for anywhere from 1yr to 5yrs, which you would like?"You say "5yrs of course!"They tell you "how would you like it billed? We can pay it one time now... or put it on the company credit card?"You say "The company card of course! It will renew!"***5yrs later your site goes down***How could this happen?!?! An in-depth review shows that the entire team you assigned to take care of that task has either moved on or transfered elsewhere in the company. Doh! Even worse, credit cards only last for 5yrs before they are canceled and reissued, you were doomed from the start. All the phone numbers you gave them were moved, the people gone, and those that answered barely knew what a domain was in the first place. You're biggest fault was apparently setting the renewal so far out. If you'd set it for 1yr at least you could have a repeating process for people to get use to as newhires rolled in and out.

But wait! There's a "contracts" department that should have cought this!Well "contracts" kind of sorts things in order of importance by cost and that domain registration cost what? $20? So that out it between free Twinkie Friday and the new coffee pot... not really on their radar.

As many times as I've seen this happens it still baffles me to this day why there isn't a service that went something like "$10k per year and you'll never have to worry about any of your domains... ever... pay us, we take care of it"

anyways, whatever... point is, it's not as simple as it appears on the surface.

IT department.List of all domains.Expiry date of those domains, culled from WHOIS.

How hard is it? Ten minute job. And you KNOW what domains you have to use - you've been including them in game titles, software on the systems you put out, and keeping those domains running somewhere.

This is NOT a huge task. Even for a multi-million dollar company with 10,000 domains. Hell, it's barely an IT task... more an office admin kind of thing (did they have to "renew" their subscription to the newspapers and tech j

I haven't bought an SOE product in 10yrs because of the Foglok fiasco... I was actually banned from their forums for a few months back in the day for suggesting they didnt exist, only later find out I was right.

What do you mean froglok's don't exist, I've seen them. There was a HUGE underground fortress/city FULL of them down south near the Ogre/Troll town in EQOA. High level Froglok paladins and such.

They were NPC's in the original EQ release too. Ykesha made them playable.

Another big company that I worked for, curiously also starting with S, had exactly the same problem. With an internal server, so nobody buy the people working there noticed it. Why? I have no idea.

But once you spend a few years working, you notice that the term "professional" only means "doing it for money". Not "doing it professionally". So please, don't think corporations are in any way more efficient than you are, or that they would or could do a better job. And how should they? It's just people doing fo

I have been doing web work for a decade, and I can tell you this happens all the time. In fact, older employees in marketing have told me horror stories about 800 numbers and mailing addresses that were never set up, misprinted, or never updated.

I always tell clients that they should set up emails that describe the job/function, like marketing@example.com and webmaster@example.com, and make sure that those emails go to a distribution list that goes to at least two people.

You wouldn't believe how often critical accounts and webforms are only accessible with the email addresses of Sally the Secretary or William the Webmaster. When they leave, no one knows there is a problem, until it is a big problem.

Exactly. Unless you know someone or have some inside connections, it is virtually impossible to contact someone, who actually knows something, using publicly available information. And I'm sure that NetworkSolutions really doesn't want to spend time calling everyone who lets their registration lapse.

The real problem is that Sony couldn't be arsed to register the domain names using a working e-mail address that actually goes to the person at Sony who is responsible for such a thing.

]The real problem is that Sony couldn't be arsed to register the domain names using a working e-mail address that actually goes to the person at Sony who is responsible for such a thing.

Not quite, it should be a special purpose email like domain_registration@sony.com rather than an employee email. However the special purpose email should forward to those responsible, involved or overseeing the particular thing. The special purpose email should not be something that someone is supposed to log in to.

How on earth do you figure this is a "blow" in the console war? Are you suggest that Microsoft was somehow behind this? Or is everything that gets reported on and is related to Playstation\Xbox now some sort of insidious plot to discredit one or the other console?

In reality it sounds like pure incompetence at Sony (and the same in the story you link about Microsoft) and I think when many people are affected by this sort of thing it's fair enough that it's covered on tech sites. It doesn't have to be part o

Didn't affect PSN/SEN on the PS3/PS4/PSP/Vita, it doesn't use SOE logins. In fact, as far as I know, only one sony console game used an SOE based login, and that was Everquest Online Adventures, which is no longer running.

Somewhere in Sony, some IT guy KNOWS what would happen if that domain went offline. He knows that it shouldn't be allowed to happen. The beancounters know that it costs to much for the consequences, so they'd have to authorise it whatever.

Thus, someone, somewhere KNOWS this critical business element is a possible point of failure but NOBODY bothers to create that documented procedure.

It's not like the process is opaque, or that nobody could have predicted it...

The thing you aren't taking into account is things like reorgs and reductions in force. You have the process and procedure - and a distribution list is set up for domain-renewal@mycompany.com which has as members the manager Jane and the senior sysadmins John, Jill and Juan. Jane gets promoted and the group gets put under a different manager - Scott from Business Systems. Scott says "why am I getting all this junk from this address -- take me off the list" - he's the new manager and they follow instruc

I'm usually not a grammar-, spelling- or other language Nazi. But where the fuck does that "of" come from? That's really a mistake I've never seen from anyone but a native speaker, nobody who learned English as a second language would ever think that this could for some odd reason function.

I don't know about your country, in mine MS would probably get into hot water with that procedure. Suing for what the other party already willingly offers is not something judges enjoy presiding over. I would see something like this in the pre-ordeal:

Judge: MS, what's your claim?MS: He registered our domain.Judge: To keep it?MS: No, he pointed it at our servers.Judge: So... it worked like you wanted it to?MS: Yes, but he still had the domain.Judge: Did he plan to keep it?MS: No, he wanted to hand it over t