Posted
by
samzenpuson Monday November 21, 2011 @11:12AM
from the abc-easy-as-123 dept.

Trailrunner7 writes "In an e-mail interview with Threatpost, a hacker who compromised software used to manage water infrastructure for South Houston, Texas, said the district had HMI (human machine interface) software used to manage water and sewage infrastructure accessible to the Internet and used a password that was just three characters long. The hacker, using the handle 'pr0f' took credit for a remote compromise of supervisory control and data acquisition (SCADA) systems. Communicating from an e-mail address tied to a Romanian domain, the hacker told Threatpost that he discovered the vulnerable system using a scanner that looks for the online fingerprints of SCADA systems. 'This was barely a hack. A child who knows how the HMI that comes with Simatic works could have accomplished this,' he wrote in an e-mail."

Except Randall Munroe underestimated how good that is. If there are 6000 "common words", then a four word password is out of 6000 * 5999 * 5998 * 5997 = 1.3 * 10^15 combinations. That's more than 50 bits of entropy (2^50 = 1.1 * 10^15), his time to guess should be multiplied by 2^6, or 35,000 years by his 1000 guesses a second (and no login will allow that many, multipy by a thousan more for 35 million years!)

Except that there aren't going to be 6000 "common words" as the base. You're going to see the same inanity as current passwords, you're going to see dictionary file attacks using an actual kiddy dictionary with 1000 words or less. This will break through most passwords. You're going to see users allowed to create their own password, which means "jebusisgodone" and "onelittlefishyswim" followed "jebusisgodtwo" and "twolittlefishyswim" and so on and so forth.

"Bitwise", it sounds secure, until you realize it can trivially be attacked on the token level rather than the bit level.

If you let users pick all four words in the password, yes, but that's not the suggestion. The actual idea is that the password is a 44-bit random number created by a cryptographically secure RNG. You can hex-encode this in 11 hex digits (difficult to remember), or use 4 base 2^11 digits. Since we don't have 2^11 easy to remember/type symbols, we use one word per symbol, defined in advance. The end user does not pick the alphabet/word-set used for this.

I hate passphrases because I once used one for PGP and then forgot which verbs were in the past tense and which weren't. I guess I could have hacked my passphrase easily, but I didn't really need it, so I gave up.

This is the second one I've heard of in a week, there was one a few miles from here last week (it was covered in slashdot). SCADA operators better start smartening up or there'll be some serious trouble.

Someone should at least get a reprimand out of this. Three letters? WTF?

Richard Feynman had a story about how his hobby was safe cracking. He cracked a cabinet that had a combination lock on it and then told the people who mattered the security hole. Did they upgrade the security on the cabinet? No, they banned him from the room. Problem solved.

The weak point is always going to be the human being. Pile on as much security as you want and people are going to find ways to disable it and make themselves vulnerable. Thousands of jobs in the tech support industry depend on it.

That is annoying, forcing me to change my password at the end of the month from H@cker1 to H@cker2 to H@cker3, and H@cker4 before I can go back to the password I like, but they IT work preventers at my work are really good, so when I am working on the road for 2 weeks, they make sure I can't change my login password without being on the intra-net, and once I am 2 days passed the expire date, the prevent me from launching VPN, joining web meetings... So then I have to use gmail to email a co-worker my passwords so he can change them for me on connected laptop first. Lots of fun.

I think the goal is that even if it is compromised but they haven't realized it yet at least it will only be at most X days before it is changed again. Changing the password frequently removes some of the risk/incentive to hack it in that you as the hacker would have to know what you want and it would have to already exist. It also makes the hacking problem harder since you have to be able to cycle through the key combinations in X days not in an infinite number of days, so after the password reset you don't know if something you already tried is now the password or not, you pretty much have to start all over again. Once you've cracked the password you can't just camp out and see every engineering drawing that the company is working on forever. I realize in most cases this doesn't matter the hacker will be more than happy to grab the easy stuff first and see what is useful or use the password to delete/otherwise disrupt things. But an enemy camping on a whole collection of passwords and bringing whole systems down.

At a larger scale: say your China and you are hacking power plant passwords to be able to shut them off (not blow them up). If the passwords are cycled frequently you likely will always have some passwords you've cracked and some you haven't, but the chances that you'll get a sufficient subset of the passwords cracked so you could completely bring the power grid down in a geographical area is remote.

Yeah, thar's yer problem. Just because these things are second nature to us, doesn't mean that non-experts are any good at making these decisions.

I'd like to see the investigation focus on who approved putting a SCADA system directly on the Internet, why, and then see structural changes to ensure that that sort of person can't make those sorts of decisions anymore.

Yeah, all SCADA systems should use ssh-quality authentication, but in the meantime we have millions of units deployed that need to be secured.

Hey, maybe I should market the pfSense firewalls I sell as SCADA secure access controllers...:P

It's an improvement. Slight, but an improvement none the less. I personally prefer the SEL 3620 and similar devices for this type of task (ethernet gateway for up to 16 serial links).

The biggest problem with most systems is that they need device-level passwords vs user-level passwords (and the default management passwords are almost always in place). If you at least set up a VPN with certificate-based authentication you have created a significant barrier to the network, without complicating response time

Would you believe that the SCADA system that runs the water utility I work for has such a button on the screen? You would? Cool. I have this used bridge I'd like to sell you in New York City.

You criticize people for not having an effective password and then promptly say something so ignorant that any plumber, civil engineer, or a fifth grader could tell you you're wrong. Ignorance happens. If you need proof, look in a mirror.

Now as for the password, yes, it is foolish. There is more truth in the character

I would have assumed that a major water utility would have employed some IT specialists to run their IT systems (that is, "some of us"). I sincerely doubt that the SCADA system was installed and administered by a sewage treatment chemist.

There really is no excuse for it; I very much hope the idiot in charge of their IT security is fired.

The Plant manager needs to be fired on the spot. there is ZERO need to have a full connection from a SCADA system to any internet accessable networks.

An airgap for data is standard operating proceedure for these things. Hell even crap SCADA software like "wonderware" supports a unidirectional ethernet cable and UDB broadcasting of the data stream so that you can airgap it from the administrative computers doing data collection.

Note: if you don't know what a "unidirectional ethernet cable" is, think standard Cat 5 with the TX wires clipped off on one end http://www.stearns.org/doc/one-way-ethernet-cable.html [stearns.org] and YES they do work PC to PC with the right settings or by using a switch where you can force a port on without negotiation.
No hacker on this planet can crack a system that is at the other end of this type of cable, unless he has physical access.

sadly, what is common is for so-called "isolated SCADA network" to be hooked to a card in a PC that also is also on LAN, and then the guy install remote access software so he doesn't have to come into work if there is a problem at 3am....... or just cracking the PC into a router is all it takes to p0wn the works

Understanding what the term means is completely different from understanding how it is accomplished.

I've been building and maintaining networks for over a decade and have never even considered a uni-directional connection before I read this today. Of course, the systems I'm familiar with are specifically for internet access, so bi-directional communication and firewalls had become my norm.

That's funny, because it's the second idea that came to my mind, and I don't work in networking specifically. The first idea was a diode, but that seemed like a lot of work for something done simply by not crimping one wire.

I'm kind of glad, really. It would be very interesting to be in a meeting with someone as seasoned as yourself offering thousands of $currency's worth of new kit as the best solution, and for a "Lumpy" like me to say "Why not cut the Tx pair?".

I was referring to the person who had something constructive and informative to say.

Simply cutting the TX pair won't do the trick, there are many more configurations necessary for the network to accept this type of connection. Negotiation is a process where two end points determine the capabilities of the other end and "negotiate" a connection. Without bi-directional communication, you must configure the transmitting end with static values, then inform the receiving end what those values are. Simply cutt

Yeah, but this could have just as easily been one of those *Monster* unidirectional cables. You know, the ones with the arrows printed on them telling you which way the bit flow. These only have one arrow printed.

Oh really, when he says that virtually all of the SCADA software is designed to handle blind UDP broadcast over that kind of cable, your counter is "TCP wouldn't work and the software would have to be specifically written to handle it"? Astonishing.

I don't see how TCP could possibly work over a unidirectional ethernet cable. Only UDP. And even then only if the higher level network code was designed to handle generic broadcast to an IP address without anything initiating the connection or any kind of handshaking, etc. My point being that virtually no software would work with such a cable unless it was specifically designed to handle that scenario.

syslog, in continuous use since the 80s. The advantage of being old is everything old is new again. I'm sure someone will reinvent syslog and sell it for millions to SCADA operators.

And yes, having done this, you do have to hard code the ARP table entries in the sender on the local lan, hence the appeal of putting a router in front of the doctored up cable such that it's the only device than need be configured with the MAC address of the syslog sink machine.

You are assuming that the SCADA system has the hardware capability to swap TX and RX. It would be extremely easy for the hardware designers to leave that feature out by using an ethernet controller with it disabled via hardware config pins or completely absent. Plus I reckon that most of the systems already deployed use humble 10Mbps ports designed before autoswitching was even a standard.

I'm in this line of work.. The password was not the problem. Even the hacker is thinking like 'corporate IT' would think in terms of security. The plant floor is different.

Here's the rule: A computer that controls industrial machinery should not be connected to the Internet. The only part of an industrial process that can even possibly be connected to the Internet is historical data and alarming.

HMI software is typically a set of screens representing the automation parts of a plant process. This means that in order to start/stop a motor or energize a valve, the screen is required. It is insecure to put a password on that screen. Yes.. insecure. The priorities at a plant are different. It is always the most secure to allow control of the plant to the people at the plant. There are physical E-stop buttons on control panels in case of emergency, but the E-stop is not the end all to prevent industrial disasters. For example, if a person has his hand caught in a valve, hitting the E-stop may cause the valve to move. Another example would be an exothermic process where explosive gases could accumulate in the wrong parts of the process, hitting the E-stop may not get rid of the gas. The operator at the plant is in charge of the process - it is critical that he or she always have control over the system.

Therefore, don't connect your plant floor to the Internet.. unless you want China to be able to control it. If white-collar executive-type people want to see pretty screens, give them historical data.

what's wrong with using physical keys for this kind of situation -- proves you're there at the console, proves you've been entrusted with access (or took a wrench to somebody who had been entrusted with access).

Physical keys are used for the lockout/tagout procedure during maintenance cycles. But, there is usually no reason for physical keys at the operator terminal. Usually, you have to check in at a guard shack before you enter the plant. Then, often, you have a key card that swipes you into the area you are authorized to be. After that, further security starts to just get in the way of plant operations. Plants can typically trust the people that have physical access to the area.

Its just engineering malpractice, pure and simple. No different than trying to claim we don't need those OSHA required safety guards because no one would ever do something stupid or malicious in the plant.

The other way to hook up to the internet, as described to me by a guy who works at a "real" chemical plant where dangerous stuff is done, is you use two separate systems both of which would have to be hacked to cause damage, plus non-SCADA automatic control.

In this scenario, where they blew the water pump up by power cycling it, there are two series control relays supplying power to the VFD and if EITHER scada system decides there is a problem with the plant or the other SCADA, that scada cuts input power to the VFD until its convinced its OK. Most VFDs like a 0-10 volt DC input to control their output, and its not all that difficult to hard wire a physical time delayed relay that says you need to output more than a volt for more than a minute to close the relay contacts connecting the VFD to the SCADA and start the pump, so the SCADA literally cannot physically turn the pump on and off more often than once per minute. You can also drive the time delayed relay off the other SCADA system, so one system decides to turn on the pump, while the other decides how fast to run the pump, and either can shut down the pump if they feel the need. Most VFDs can be configured to not allow operation outside certain limits, like drawing more than X amps where X is larger than normal but less than theoretical VFD limit, and not to turn on if a thermocouple says its too hot or a pressure gauge somewhere has an open loop signal. Similar design such that NPSH and output pressure have to be within certain limits or again, the time delayed relays open circuit the AC input to the VFDs and/or the control input to the VFDs. Finally its no heroic effort to wire up two safety bypass relays in series so that if you have control of both SCADA systems, and both independent scada systems agree, you can bypass the safety relays (and the enabling of this bypass also turns off a green light inside the safety directors office, resulting in management involvement, formal written reports and investigation, etc)

This is cheaper to install and operate than you think, because both suppliers know darn well they can be replaced individually with no real impact to plant operations, unlike the traditional "one ring to bind them all" scada design where the consultants and suppliers know they've got you over a barrel and can charge what they want.

And a guy I know at another plant described "adversarial SCADA" to me where two separate systems from two separate mfgrs and two separate consultants, one run by an "operator" and reporting up the operations management chain all the way to the board, and another run by "safety" and reporting up the safety management chain all the way up to the board.

The operations guy and his SCADA system do whatever they want whenever they want, but if the safety guy and his SCADA detect an overspeed or an overtemp or underpressure then safety guy and his scada cuts power to the operations guy and his scada. Also operations guy can "get even" with safety guy because he has relays installed that can simulate sensor failure, and the safety guy has to respond within X minutes following whatever procedures, and the operations guy is presumably intelligent enough to only perform those tests when operationally convenient.

Also although technically either the safety guy OR the operations guy can punch the "give up" buttons, because the safety guy does not answer to the bean counters, that means the dump tank and suppression buttons are for all intents and purposes exclusively operated by the safety guy... The operations guys have training issues in not bothering to even know how to operate the fire suppression valves, for example. Which is bad, because the centers are geographically separate, so if a tornado wiped out the safety center, or even just a failure or a hack event took it out, the ops guys might literally not know how to put out a fire at the plant, even though they are technically capable.

This is a fail when weird plant conditions require jury rigging and close coordination, and also a financial failure because the independent supplier of the operations scada knows the plant shuts down if they try to change out, so he's free to charge as much as he pleases.

Hack our safety scada yesterday? who cares, ops will safe the plant. Hack our ops today? who cares, safety will safe the plant. Hack both separate systems with separate designs and separate manufactures tomorrow at the same time? who cares, that has to be an inside job...

Here's the rule: A computer that controls industrial machinery should not be connected to the Internet. The only part of an industrial process that can even possibly be connected to the Internet is historical data and alarming.

Therefore, don't connect your plant floor to the Internet.. unless you want China to be able to control it. If white-collar executive-type people want to see pretty screens, give them historical data.

Tell that to Iran. I heard their nuke program was set back a few years, even though t

Some government sites have these onerous password requirements e.g no fewer than 15 characters, no consecutive characters even if they are a different case, at least one numeric and at least one punctuation. It's not surprising that coming up with something you can remember that fulfills these requirements is a bitch. Oh, and you have to change it periodically. IMHO, this naturally leads to writing the damn thing down somewhere.

A password difficult enough not to get cracked is a password difficult enough that it can't be remembered. Smart card and relatively simple password is probably better but that costs money (readers and cards but also lost time because "I left my card at home") where as password complexity requirements are just a simple software configuration away.

Network admin for another city govt in Texas here... albeit a very much smaller city.

1) first of all, it's absolutely nuts to place your water purification SCADA (or even your wastewater plant's SCADA) onto any network segment that's accessible from the public Internet, and we in the IT department know that all too well, however we're not "in charge" of the SCADA systems and have essentially zero authority to do anything about it. Part of the problem here is that the folks who *are* in charge of these systems are thoroughly aware that we in IT know how to better secure their systems, but do not want us involved in any way because our security will "make things too hard for them to do their jobs".

2) The folks who run the SCADA systems on a daily basis know only two things about systems security: 1) diddly and 2) squat. They are water process and industrial chemistry people, not computer people, and it shows big time.

3) The vendors who supply and support the SCADA systems feverishly demand that the SCADA systems be easily accessible over the Internet for their convenience for remote support, and frankly do not give a rat's ass about the customers' security... their response is that security is not their problem it's ours.

So, it's no wonder these systems are getting hacked and it's going to get worse as time progresses.

3) The vendors who supply and support the SCADA systems feverishly demand that the SCADA systems be easily accessible over the Internet for their convenience for remote support, and frankly do not give a rat's ass about the customers' security... their response is that security is not their problem it's ours.

Can't allow VPNs instead of wide open access? Even the place I'm at now, has exclusively VPN access for "outside engineering suppliers"

Historically, back when dial up support was the way to go, I worked at a place where IBM had remote access to "our" multiple mainframes only when a orange cable was draped across the desk of our security officer (this is before orange meant fiber, it was just orange "silver satin" 4 conductor modular phone wire).... Being a "mahogany row" level management position, this cab

As usual, blame the owners and operators of the target, not the hacker. Because if I don't lock my front door, it's totally OK for you to come in and run up my utility bill and eat out of my fridge, help yourself to my stereo and tv while you're at it... and if I have a spare key under my hood that you find on my car, by all means, how could anyone be held accountable if they take it for a joy ride and/or steal it?

If I see you bent over, I'm guna kick you up the ass.
That's just how this world works. It's my fault, but your at some fault for bending over so easily. Good and Bad in black and white forms doesn't exist, in reality there a number of parties at blame for this.

Yeah, if your car gets stolen and you say to your insurance company "Yeah I keep the keys in the ignition and never lock it, don't you fucking assholes blame the victim here, he had no right to steal my property!" you're not getting your claim approved. Just like if, say, a prison's doors aren't designed to be kick-proof, and a prisoner kicks one down and escapes, that's his fault, not the prisons! Don't blame the victim!

The most telling thing, for me, was this section of the linked article:

“DHS and the FBI are gathering facts surrounding the report of a water pump failure in Springfield, Illinois,” according to a statement released by DHS spokesman Peter Boogaard. “At this time there is no credible corroborated data that indicates a risk to critical infrastructure entities or a threat to public safety.”

So...in the instance of a single shoe bomber, stopped by his own stupidity and the efforts of other airline passengers, TSA (a section of DHS) responds by calling it a systemic risk to air travel, and we must all take off our shoes. In the instance of a plot to use liquid explosives, which probably wouldn't have worked and was stopped in the planning stages, TSA responds by calling it a systemic risk and we must all limit ourselves to 3oz bottles of liquids that fit in a quart size bag. In the instance of a single underwear bomber, stopped by his own stupidity, TSA responds by calling it a systemic risk to air travel, and we must all be subject to X-ray/millimeter wave scanners and/or the big Grope.

In the instance of SCADA hacking, which could conceivably harm our infrastructure on a significant and systemic level from afar, with little/no risk of the perpetrators being caught, DHS responds by saying, "No big deal."

I don't know what attacks the other poster was referring to, but the UK courts established that the liquid bomb plot would probably have worked, based on evidence from explosives experts, including testimony and video of experimental explosions.

Here's a pretty convincing video. [bbc.co.uk] Note the bomb is liquid held in one 500ml soft drinks container, which could be carried through most airport security checks. To make the point that 100ml was just as unsafe, the liquid for this particular bomb was actually combine

OK. I was unaware of that. From what I remembered, the whole binary liquid bomb had been debunked as 'difficult to mix ahead of time because it's unstable; impossible to mix on the plane because of the long times involved and the specific conditions needed to avoid premature detonation (which would not be explosive enough), and very hard to detonate without attracting a whole lot of attention.'

Apparently that notion was wrong.

Nonetheless, I stand by the rest of my statement. DHS/TSA responds to awkward a

To elaborate, TSA is afraid that people could bring separated liquid components of a binary explosive, mix them together in flight, and end up with an explosive. Mixing TATP on an airplane would be very challenging, and likely to arouse suspicion before it could actually explode:

Also remember that nitroglycerin would be detectable quite easily with regular explosive detectors (highly oxidized nitrogen compound), so just because of this incident where would be still no reason to continue confiscating water bottles.

Things that need external service technicians often have very simple passwords. For example I work in health care and I know of at least two major companies who's components have the same login for every site for administrator access. You probably as a customer could insist on changing it but the vast majority of sites don't. So need to give someone some radiation? You know the password. That said it isn't going to affect a whole community but the 30-100 patients that get treated before the problem is detected? Very doable. Similarly wifi routers from ISPs almost always have a default password most people I know change the WPA key but don't touch the admin account password. So anyone allowed into the network (or who can plug a network cable into the back of the box for a couple minutes) can take it over pretty easy. Not a real big deal I realize because if they change the password to login (since they don't know yours presumably that is what they would do to get internet access) you'd realize it isn't working and work to set it back. But if you are running a wired network primarily but it is a wifi device could be an issue.

I've seen "wtr" used, my cousin had part time job at village water works. he didn't get to choose or change the passwords.....however, on the bright side that was only for monitoring, the pumps actually were so very old they had the old "knife throw swiches", the wood panel with those was roped off lest someone touch the open metal and get zapped

For usablility reasons. Remember Buisness Requirements are more important than technical ones. This way the security guard for the building could sit at his desk, with solitare on screen and IE open with the web client in the other.
Clearly they got caught knapping!

I'm no security expert, but humor me and point out the flaws in my logic below.

Disabling access after X tries might be enough where the token to uniquely identify access is relatively well-defined, like say your ATM card, and disabling access for that user doesn't de-facto terminate the system (i.e. other ATM users can still use the machine with their credentials after it eats your card).
For admin-access to such systems over the internet it's dangerous to disable the admin account after X tries, because

Bad people are the problem. Start hunting down and exterminating bad people. If the prize for hacking into a water plant is 15 minutes of fame followed by an early grave we'll see the population of scumhackers nosedive.

Wow, you must be horribly short-sighted and ignorant. First of all, there is no universal definition of "bad" when talking about people. Different cultures, religions and even different kinds of families can have very different views on what it means to be "bad", and as such what would you do if someone just went to your mother's house, shot her, and said that she was deemed "bad" and that's why she was put down. And then the person who shot her was free to go. Would you then feel it was such a great idea?