Search

Subscribe

Disgruntled Employee Holds San Francisco Computer Network Hostage

Childs created a password that granted him exclusive access to the system, authorities said. He initially gave pass codes to police, but they didn't work. When pressed, Childs refused to divulge the real code even when threatened with arrest, they said.

He was taken into custody Sunday. City officials said late Monday that they had made some headway into cracking his pass codes and regaining access to the system.

Childs has worked for the city for about five years. One official with knowledge of the case said he had been disciplined on the job in recent months for poor performance and that his supervisors had tried to fire him.

"They weren't able to do it - this was kind of his insurance policy," said the official, speaking on condition of anonymity because the attempted firing was a personnel matter.

Authorities say Childs began tampering with the computer system June 20. The damage is still being assessed, but authorities say undoing his denial of access to other system administrators could cost millions of dollars.

EDITED TO ADD (8/10): According to another article, "officials say the network so far has been humming along just fine without admin access by the city." So it's not a complete shutdown as much as an admin lock out.

EDITED TO ADD (8/13): This is getting weirder. Terry Childs gave the right passwords, but only to the mayor personally.

I suspect Anonymous is referring to the fact that IEEE 1394 requires DMA access, so a configured remote device can use that to directly access the memory on the machine, no matter what password protection may be in place.

"Prosecutors say Childs ... tampered with the city's new FiberWAN (Wide Area Network), where records such as officials' e-mails, city payroll files, confidential law enforcement documents and jail inmates' bookings are stored."

That doesn't sound like it's just one database file that he's shut everyone out of, it seems that he's simply revoked administrative rights to anyone else within the network. According to another article, "officials say the network so far has been humming along just fine without admin access by the city." So it's not a complete shutdown as much as an admin lock out.

In far too many places I've been, one or two people can exercise far too much control over systems. These folks may not intend anything bad to happen, but:
a) work being what it is, if one nice admin can do everything you need without all that red tape and paperwork - you're sure going to ask.
b) these same places are too lax on policy & procedure, so nothing gets documented. When the old admin leaves [ promoted | hit by bus | vacation| forgets his own password ] all the knowledge goes away, too.

Bryan Feir understood what I was saying. I had not meant to post anonymously.

I was trying to say that physical access to the machines (not the disgruntled worker) should grant them the access they need with the exception of a few rare cases. In my next post I was trying to show that even questioning the worker may not be enough to acquire a password. He may very well have chosen not to type a password he could remember.

As for hosing him - it won't get a password he does not remember back and there are civil rights in this nation that might cost them bad press/lawsuits. If they can get away with it I understand their data is worth a lot. They should try to be utilitarian. There might be really cheap non-controversial methods of retrieving access once more.

Physical access to the data is almost never enough. It's better than nothing, but if you actually want the enterprise to run effectively you need the data and the applications and the communications, and all the twiddly little configuration hacks that people have accumulated over the years.

DMA has been shown consistently that it is more than effective at bypassing local authentication screens. Because you have access to the first 4 GB of RAM you may write "success" bytes over auth-completed-yet bits or nops over the checks altogether.

"City officials said late Monday that they had made some headway into cracking his pass codes and regaining access to the system." sounds like they are running brute force scripts to recover the password.

DMA can grant access to the systems needed to restore the network to what it once was. Abandon questioning Childs? No, don't do that, but if you want your network back up and running don't launch a brute force tool if you have physical access to the machines.

All this talk about "physical access to the systems makes this a moot point", and no one examining what has to be done when resetting these systems physically:

You actually have to reset them, toasting all active connections, toasting all data in RAM, causing network devices to re-route (or just plain drop) traffic.

How much is that downtime going to cost, and who is going to pay for it? Is there a plan in place for how to do this without causing huge problems? Can they verify that when things go down, they are going to come back (or did he put a poison pill in place)?

They are doing EXACTLY what they should be doing, trying to get back the password and building contingency plans for if they can't get it back. If you work in an environment where you CANNOT reboot a machine without possible disastrous consequences, you would know this.

The really scary part is that he gave them a password that didn't work. I wonder if he can't remember the password...

I think paul, at 1:27 PM, explained what was puzzling me about this story. It seemed unlikely that the data itself had been encrypted, so how could this be a big deal? But if the City's been locked out from all the screens, preset queries, and other special-sauce apps they've spent millions developing over the course of years, the data is probably nearly useless to them.

What a jackass. I hope his favorite color is orange, because that's what he'll be wearing for a while.

So, does anyone know what type of systems he's locked, and how come someone with system root-level privileges can't just run some utility to override/overwrite the problem authentication?

Perhaps you asked it more clearly than I have been. Why can't someone with root access just run a utility to override the problem with the authentication? Even if the boxes are off site, they are somewhere and a police mandate will get them access to the box.

I don't want to beat a horse to death here, but DMA will give you root access. You can go through a card bus or something else if it has no firewire. There is no reboot needed. Fire up a shell and use DMA to search for the shell and set its UID to root.

@Ross: probably what the guy did was run the script/utility to handle root/admin password changes, then not tell anybody what password he picked. Nobody can run an override because nobody's *got* root access anymore but him, they don't know the password. DMA... won't help much. Technically it'll let you get in, but the process will be a long and intricate one.

@Ross: probably what the guy did was run the script/utility to handle root/admin password changes, then not tell anybody what password he picked. Nobody can run an override because nobody's *got* root access anymore but him, they don't know the password. DMA... won't help much. Technically it'll let you get in, but the process will be a long and intricate one.

If as reported it looks as though everything is still running, then presumably no critical data is gone. So he locked out other folk from administrative access - this should be no more than an inconvenience. Given physical access to the equipment in question, is there any vendor that does NOT offer instructions on how to recover and reconfigure in the event of a lost password?

The real question:
Why has no other tech working for the city already corrected the problem?

Sounds as though the city has only one guy of any ability, and for some reason they REALLY pissed off their only capable guy. The story that lead up to this should be (more) interesting.

From the article it would apear that whatever he has done has not been destructive, as the systems are all still running...

So the question is what are the "systems", the article implies it's the network not servers (could just be usual journo translation prob).

Also the SF authorities appear to be "talking it up" in that they are talking about a "third party" with the ability to gain access to any and all systems (aparently without any evidence of such)...

One thing of note is they mention that he was monitoring what other sysadmins where saying...

What is the betting that the other sysadmins have been very very lax and have emailed passwords and or access certificates to each other, which he then used.

I'm thinking that he may not have been a senior sysadmin but just a network bod who obtained access over and above his job requirments from his monitoring activities. And possibly the SF authorities are trying to hide their real problem (lax behaviour at senior sysadmin levels).

Also why have the SF authorities not mentioned what the issue was with the guy that caused him to be nearly fired and put under observation?

Perhaps he was just rocking the boat about lax security and got told to can it by the seniors. And (daftly) just decided to prove the point and for some reason it has gone pear shaped.

Perhaps one of the seniors who became aware of his activities was gungho and said "no problem I can fix it". And in the process did something that stoped the passwords (or whatever) from working...

I'm just making guesses here based on way to little information that just smells fishy or as Bruce puts it "hinky".

Therefor it will be interesting to see what comes out in the court case. As I have the gut feeling that there is a significant amount that is not being told by the SF authorities...

Then again we may never find out it might just plea bargin out to a minor offence or less to cover the blushes of "those that be".

At least one article, which I can't dig up at the moment, mentioned that the city was working with Cisco to recover. That tends to confirm "network equipment" (although C. does also sell network storage gear these days).

So, password recovery time on a bunch of routers, maybe? Generally possible -- it's a rare piece of networking equipment that DOESN'T have a recover-the-admin-password function -- but often disruptive; it tends to involve reboots, and in some cases loss of configuration information as well.

People still sometimes try to claim that allowing someone with physical access to a box the ability to reset the admin password is a big security hole, ooooo scary. Incidents like this help show why this is silly (and why "Availability" is one of the cardinal attributes of security).

Now, once you look at it like that, each single item is simple to fix in the situation described. Particularly since everything seems to be WORKING right now.

People log into
Workstations that are wired to
Switches or hubs that direct their traffic to
Servers that run
Applications or databases including the
Directory Service that they are using.

Imaging the workstations would be a costly project (and fairly time consuming).

Resetting the switches would be EASY because you'd do all the HARD work AHEAD of time and just dump the settings to the boxes. Someone has to physically be there, but that would be a weekend project. Maybe two weekends.

Active Directory / eDirectory? More difficult. But that also requires fairly high level knowledge to lock out ALL the options. And I'm not talking about just removing everyone in the admin group except yourself.

>A lot of people seem to equate the appearance of "functioning normally" with it being secure, stable data.

" the network, which contains ... law enforcement, and inmate booking files' apps and data, according to a published report."

Actually there's a good question in there about potential problems with criminal trials for people who were arrested and booked in these few days. Since the data system in question is technically out of the control of the authorities and there has certainly been speculation about the possibility of an accomplice, a clever lawyer could argue the break in chain of custody and potential for tampering with evidence by persons unknown renders any information processed during the time of this lock out at least suspect, at most null and void.
Worth considering.

@ sooth sayer and Rich B
Exactly. Why is this terrorist treated any differently than those who are held in Gitmo? Is he not terroristic enough? Why don't they waterboard all the prisoners in the USA? Are they not terrorists? Hell, I'm more afraid of convicted rapists and murderers than random muslims and arabs, but what do I know. I might be a terrorist myself..

I would think the concern about data integrity is legit, but on the other hand, if he just wanted to tamper with the data, why lock everyone out first? It is way to high-profile. It would be a lot smarter to mess with the data silently, instead of attracting a lot of attention.

I think that fact that he locked everyone out doesn't change whether or not he has the ability to mess with the data; if it is a concern now, it should have been a concern in the past.

As many have said, IEEE 1394 requires DMA access, so a configured remote device can use that to directly access the memory on the machine, no matter what password protection may be in place. I am not surprised at the initial action given the increasingly lousy treatment of IT personnel by managers who are technologically ignorant. I am just surprised that this guy was so stupid as to get caught, and then not give the correct pw. I am also surprised this doesn't happen more often. The overwhelming majority of times I have seen this kind of thing in private sector, it was never reported or prosecuted because publicizing it would cost more in lost revenue than just silently correcting the issue and improving internal checks/security rights. It's just lucky for all of us that most IT experts are morally responsible and basically use their power ethically.

While reading the article and the comments, I started to think about how to prevent this situation from happening(as best a possible) on my systems. I work for an un-named organization and one of our strategies is to separate server, network and backup(including offsite) sysadmin functions and people such that recovery is not too difficult or time-consuming. And of course this takes more people to implement but does help mitigate damage from a single rogue sysadmin.

More media exposure is more payback and dollar contracts awareded to "fix" the problem.
Typical dirty laundry political world we live in.
I really doubt that the network is hard to crack with physical access, by contacting designers of system.

It would be interesting if the guy is sophisticated enough that a certain wrong password, enables some bad stuff silently. Now if this guy built the system, and people can not hack it or deal with the problems, he *might* have some "insurance" although there are MUCH better ways to get recognition and job security.

Any basic details on the system would be interesting, haven't looked hard for it yet. Yawn, liberal hysteria.
Millions of dollars damage, what a LAME racket, costs little to fix, but claimed lots of deals. Typical, who really are the crooks? Must make a larger can of worms to hide their scamming of the system...

#1 preventive strategy, RESPECT your IT people and pay them well. #2, have another good IT guy review others work for contigency handling and data handling. #3, Have access to those who are the best. Perhaps #3 really is equivalent to #1.

IIRC, last time the "FireWire DMA Hack" went around, it was mentioned that although the spec requires that an adaptor be capable of DMA, it does not forbid that capability being disabled by the host. Apple reportedly does this, so they would only be vulnerable between the time the device "wakes up" on a power-on, and the BIOS or OS turns off the ability. Presumably other professional organizations take similar precautions.

I find it interesting how many people are skeptical of the information being released by the city government. How can a society ever achieve a reasonable level of security if there exists an ever widening level of distrust between the people responsible for security and their government?

It's also interesting how many people on the blogs (more so on other blogs than here) seem to support the admin and assume the problem is with his management.

In the early days of the Web, whenever I came across a site with security issues, I would email the admin. I was always thanked. Today, I just ignore such problems because I am afraid I might be accused of hacking and have no faith in the judicial system when it come to computer crime. Even worse, I generally don't report suspicious behavior unless I know the individual to whom I report the activity personally, and I know they won't over react.

When all the facts are known about the San Francisco case, I suspect the root cause will be a breakdown of trust somewhere in the organization.

First of all I really have no ideal what the guy allegedly done. The news accounts are pretty vague on the issue. Did he tamper with routers, multiple email servers, file servers, or some authentication server? I don’t know.

But, one of the strikes me is that some posters have committed on the question of: Why doesn’t Terry Childs turn the passwords over to the police?

If he was guilty why would he? That would be admitting guilt. It would best to assert that he was locked out as well. That he gave the police what he thought were valid passwords. But he was unaware that he was locked out by the intruder as well. The only way the state could get the passwords out of that guy is to guarantee that it cannot be used as evidence against him in a court of law.

Now, if this Terry Childs has a competent criminal mind, he has taken into account the sever logs. He most likely knows what the backup schedules are. I can probably guess that the log files have all been deleted or altered in someway.

Just finding the tangible evidence of his involvement could take years. Terry Childs might walk because California cannot provide him with a speedy trial as required by law.

But even if the state does have the tangible evidence against Terry Childs the state by law must disclose it to the defense. So why should he fold before he knows what the state’s hand is?

"In Richmond Newspapers v Virginia (1980), the Court found the First Amendment gave the press a right of access to courtrooms, absent a compelling government need (such as to protect a minor's privacy or national security). In a concurring opinion, Justice Stevens stressed the significance of the holding: "This is a watershed case. Until today the Court has accorded virtually absolute protection to the dissemination of information or ideas, but never before has it squarely held that the acquisition of newsworthy matter is entitled to any constitutional protection whatsoever.""

Now the entire system goes from being legal to technical. It's easy to see whole states shutting down because all the Google systems are down or somebody or some geek decides we don't need judges or trials. We just need software and search tools to guide us into a blissful future where everything is easy and work is a memory. Paper is a luxury and water is only a dollar a gallon and we all drive electric cars and watch Al Gore on TV run by solar power cells. The Constitution is so 1776!

I don't feel the rubber-hose idea should be entirely off limits. It seems to me that there is more than one category of physical punishment and/or torture. There are at least these three:

(1) Torturing someone who has not yet been put on trial, to make him confess or rat out his mates.

(2) Physical punishment, for instance whipping, handed down as a sentence by a judge after the criminal has been properly tried and convicted.

(3) Torturing a criminal after he is convicted, not as punishment per se, but to make him give back some person or thing he's stolen and hidden.

In my view, only category (1) is clearly wrong (and an exclusionary rule, though probably necessary, is NOT sufficient to protect against it). I can see making exceptions in "ticking bomb" situations, but only if the person giving the order is open to prosecution and lawsuit if he's wrong.

(2) has mostly been eliminated in the US, but I would like to see it brought back, especially for violent crimes.

And as for (3), which is this case, I believe it is clearly already both legal and right. Compelling a convicted criminal to give back what he's taken cannot qualify as "cruel and unusual punishment" no matter how much force is required, because making the victim whole trumps any right of the criminal.

I really wonder if they will be able to bring the guy to justice - if he is half as "badass" as the article paints him, all the incriminating material (incl. logs and backups) was gone the moment those guys entered "a password that didn't work"

No odder than San Francisco not having a formal information security policy until a couple of months ago.

As I understand it, Childs was the only CCIE employed by the city, so he was readily able to set up ninja-level security that only he understood.

My prediction (and hope) is that San Francisco has just now learned a few expensive lessons and will build a proper information security policy including goodies like change control and key/password escrow.

Giving the passwords to the Mayor sounds like just what a person who felt he was being wronged would do when approached by someone they feel is trustworthy.

As for the hope that "San Francisco has learned a few expensive lessons and will build a proper information security policy" - why? Policies don't provide security. Emphasis on compliance and policy development in the last few years has just about killed information security as a profession. It's a field full of products that don't work and are overpriced and best. Managers judge skill by counting the number if initials attached to a name and true security is all but forgotten because it's a hard job that few can do and even fewer managers can understand. Compliance is easy, security requires planning and understanding requirements ;which is why compliance dominates the profession.

the solution for preventive protection is easy and right there on the market though: Oracle's Database Vault. eliminates the superusers power. all superaccess must be granted by a second semi-superuser and vice versa...

Mr. Schneier is quoted in several articles on this matter saying that the admin in question is obligated to give those passwords up.

I question the reasoning here.

By way of analogy, if you administered a government network which contained taxpayers personal information, and were told by your boss to do something that would irrevocably compromise that information, are you really obligated to follow those instructions?

Would it not be professionally negligent to do so?

If you worked in HHS, consider your boss asking you to take the medical records of everyone and mail them, in cleartext, on CD to each doctor in the state - to share the information. Presumably, as an administrator, you would have to refuse this request.

If a tram mechanic for the state was told by his boss to disconnect some of the safety systems on the trams, presumably the mechanic could refuse, if they felt the action would be negligent.

So surely, if this admin felt that sharing the passwords with the large group of people would be negligent in their duty as network administrator - a duty to protect and safeguard the potentially confidential information within that network - they would be ethically justified in not sharing those passwords.

And, in fact, could perhaps later by prosecuted for negligence, if they shared the passwords as directed.

I'm not saying that this is what occured in this case.

I'm just saying that it is not as simple as Mr. Schneier has stated when he said that:

"The passwords are owned by the city, so as an employee he's obligated to give them up to his boss," Schneier added."

"In Richmond Newspapers v Virginia (1980), the Court found the First Amendment gave the press a right of access to courtrooms, absent a compelling government need (such as to protect a minor's privacy or national security). In a concurring opinion, Justice Stevens stressed the significance of the holding: "This is a watershed case. Until today the Court has accorded virtually absolute protection to the dissemination of information or ideas, but never before has it squarely held that the acquisition of newsworthy matter is entitled to any constitutional protection whatsoever.""

I have to agree with Justice Stevens, State of California -v- Terry Childs will be a "watershed case" for the entire information technology service industry. My fear is the state of California has blurred the lines between Hollywood/fiction and the Law.

That is unless someone with half a mind stops this crazy train! Where is James West and Artemus Gordon when you REALLY need them!