Krebs on Security

In-depth security news and investigation

‘Shellshock’ Bug Spells Trouble for Web Security

As if consumers weren’t already suffering from breach fatigue: Experts warn that attackers are exploiting a critical, newly-disclosed security vulnerability present in countless networks and Web sites that rely on Unix and Linux operating systems. Experts say the flaw, dubbed “Shellshock,” is so intertwined with the modern Internet that it could prove challenging to fix, and in the short run is likely to put millions of networks and countless consumer records at risk of compromise.

The bug is being compared to the recent Heartbleed vulnerability because of its ubiquity and sheer potential for causing havoc on Internet-connected systems — particularly Web sites. Worse yet, experts say the official patch for the security hole is incomplete and could still let attackers seize control over vulnerable systems.

The problem resides with a weakness in the GNU Bourne Again Shell (Bash), the text-based, command-line utility on multiple Linux and Unix operating systems. Researchers discovered that if Bash is set up to be the default command line utility on these systems, it opens those systems up to specially crafted remote attacks via a range of network tools that rely on it to execute scripts, from telnet and secure shell (SSH) sessions to Web requests.

According to several security firms, attackers are already probing systems for the weakness, and that at least two computer worms are actively exploiting the flaw to install malware. Jaime Blasco, labs director at AlienVault, has been running a honeypot on the vulnerability since yesterday to emulate a vulnerable system.

“With the honeypot, we found several machines trying to exploit the Bash vulnerability,” Blasco said. “The majority of them are only probing to check if systems are vulnerable. On the other hand, we found two worms that are actively exploiting the vulnerability and installing a piece of malware on the system. This malware turns the systems into bots that connect to a C&C server where the attackers can send commands, and we have seen the main purpose of the bots is to perform distributed denial of service attacks.”

The vulnerability does not impact Microsoft Windows users, but there are patches available for Linux and Unix systems. In addition, Mac users are likely vulnerable, although there is no official patch for this flaw from Apple yet. I’ll update this post if we see any patches from Apple.

US-CERT has a list of operating systems that are vulnerable. Red Hat and several other Linux distributions have released fixes for the bug, but according to US-CERT the patch has an issue that prevents it from fully addressing the problem.

The Shellshock bug is being compared to Heartbleed because it affects so many systems; determining which are vulnerable and developing and deploying fixes to them is likely to take time. However, unlike Heartbleed, which only allows attackers to read sensitive information from vulnerable Web servers, Shellshock potentially lets attackers take control over exposed systems.

“This is going to be one that’s with us for a long time, because it’s going to be in a lot of embedded systems that won’t get updated for a long time,” said Nicholas Weaver, a researcher at the International Computer Science Institute (ICSI) and at the University of California, Berkeley. “The target computer has to be accessible, but there are a lot of ways that this turns accessibility into full local code execution. For example, one could easily write a scanner that would basically scan every Web site on the planet for vulnerable (Web) pages.”

This entry was posted on Thursday, September 25th, 2014 at 6:56 pm and is filed under Other.
You can follow any comments to this entry through the RSS 2.0 feed.
Both comments and pings are currently closed.

“Am I from Mars? • September 26, 2014 4:55 PM
@JaneBob:
You will just have to wait for Apple to push out a fix. If you are not running a public web server from your computer with a (IMHO) bad implementation of its web services, you don’t have much of a vulnerability.

— Expounding on the obvious follows —
#1, for a web service to be vulnerable, the service must accept input that is not cleaned up.
#2, the input must be placed into an environmental variable before a Bash script is executed.
#3, the Bash script has to be executed with root privileges, or with privileges elevated enough to do some harm.

So there are at least three things that must be skipped in the implementation, setup, and deployment of the web service. I stand by the statement that it’s a problem with incompetent developers and administrators. Really, it’s like Bobby Tables. Stop leaving the door wide open, there’s real jerks out there.

The other part is a dhcp client that doesn’t check its input, and uses a Bash script for some of its settings.

This “bug” was implemented as a feature, and it has been in the manual pages for as long as Bash has had this feature. Just now someone has noticed it, and of course others have capitalized on it. I learned the importance of checking input before the web became popular. And now people are noticing that maybe its a good idea now??

No insults please, I’m a layperson not a red white or black hat of any sort.

So am I correct in my understanding that this vulnerability primarily effects servers but can also possibly effect routers & wireless printers. So although I have a Mac the likelyhood of my being effected by this is low.

I’d state it a little bit differently. This vulnerability _affects_ a great many devices, from web servers to connected home devices (smart TVs, DVRs, media players, printers, toys, toasters, you name it). However the risk (as in, the damage that can be done) is greatest for Unix-based web servers and for those that run Linux on their laptops. In the home, I’d be most concerned about the router or wireless access point; any network-attached storage (i.e. hard drives that connect straight to the network instead of to a PC); any audio or video monitoring devices (nanny cams, baby monitors, and such); and Linux – based laptops and PCs (far less common). I wrote a lay interpretation of the bug at http://dnlongen.blogspot.com/2014/09/a-shell-of-bash-shellshock-in-lay-terms.html

* afaict, iOS doesn’t come with bash, and is thus fine
* iOS jailbreaks apparently include a vulnerable version of bash, and thus jailbroken devices are probably somewhat at risk (but unless you make bash the default shell, you shouldn’t be terribly at risk)

Lisa, it depends on which Mac OS you’re running. The issue (re-read the article carefully) is whether or not BASh is your default command line interface.

Open Terminal and type the command given in the article.

Also, see here: https://support.apple.com/kb/ta27005
If you’re using an OS that is Panther (10.3) or later (Tiger for sure), then your default terminal shell is the vulnerable one. I haven’t looked at Mavericks to make sure about it, but for anything after Panther (Tiger, Leopard, Snow, Lion, Mountain, Mav’s), it’s probably still BASh.

Many people repeat the mantra that you gave, “So although [sic] I have a Mac the likelyhood [sic] of my being effected [sic] by this is low.” In other words, the “Mac’s are safer” lie that get’s people doing things on websites to give up their info, which has nothing to do with your platform. Mac OS X is a Unix-type (Linux-type) system. The likelihood of your system being affected is BECAUSE it’s a Mac.

The problem isn’t just that your Mac might or might not be vulnerable; the problem is that your cable modem, router, the stoplights in your town, hospital equipment, security system, or uncountable numbers of other things might be, many not even remotely capable of being quickly or cheaply fixed if the companies that made them even still exist.

The bug has been there 22 years, and bash has been used hundreds of millions of devices. It’s been a very popular choice for exactly the kind of CGI scripts that form the worst case scenario for this exploit.

There’s some confusion in these messages about what “vulnerable” means in this case. Let’s separate the issues. There is the *bug* in bash; and there is the *vulnerability* to attack.

The simple test that’s made the rounds is for the *bug*. Pretty much every version of bash up through those patched specifically to fix it have this bug. The test only shows you that the bug is present. *In and of itself*, the bug as tested is harmless: It allows someone who *already has the ability to execute bash commands* (like the test code) to … execute a bash command.

The *vulnerability* occurs when some *external interface to the system* allows an attacker to get some strings into the stream of data that bash will execute. This is very similar to SQL injection: It’s a way to take what was intended to be pure data into something that gets executed.

It turns out that there are plenty of ways to do that, which is what makes gives bug such a potential to cause mayhem. But *not all systems that have the bug also have a vulnerability*. Systems that don’t allow incoming connections, for example, are inherently immune. Typical “client” systems – like Mac or Linux laptops – don’t run a web or DHCP server (both already demonstrated to open up the vulnerability) and so are *probably* safe, even though they have the bug.

Of course, at this point the best one can say is “probably”. Even client systems allow *some* incoming connections – e.g., for various messaging systems. I just checked my Mac laptop, and it has 17 ports available for incoming connection. Some are obvious – AFP (file sharing), ssh (remote login); ipp (printer sharing)I could even turn some of these off if I wanted. Most I’m not sure what most of the rest are. Could AFP or one of the others have the vulnerability? I don’t know; presumably there are people checking this, but I haven’t yet found a comment either way.
— Jerry

That’s true, but it misses the point. There’s absolutely nothing you can do on your own machine to protect a DHCP server on some *other* machine – or to protect *you* from damage as a result of listening to that DHCP server. (Actually, it’s not obvious how a bad DHCP server could cause anything but network failures to its clients, but one needs to be careful about this – there may be subtle attacks.)

What it comes down to is that most *clients* are probably safe from direct attack, but many *servers* are not – and since clients rely on servers, they are definitely subject to *in*direct attack. (The indirect attacks would be just as significant against Windows boxes, which don’t generally run the affected code, so cannot be *directly* attacked.)

Putting it another way: Most *clients* probably don’t need, and would not be helped by, patching; many, perhaps most, *servers* *must* be patched. Until the servers you rely on are patched, you as the user of a client system is in danger and there’s not much you can do to protect yourself.

Jerry, I think you miss the point, If you go with your Mac to the starbucks down the street and if shop has an unpatched infected router than dhcp response from that router will spawn a child bash process on your Mac and that bash can be exploited by malware installed on starbucks router.

Or I could pull into a starbucks or whatever parking lot with a laptop acting as a wifi router and offer a connection and start performing man in the middle attacks based on the client vulnerability to my rogue dhcp server. This has always been possible but the bash bug leverages more than just trafffic sniffing allowing the rogue server to install malware.

That article does point to some other vulnerable applications, including VMWare Fusion. More are likely to be found. However, most of these are vulnerable to *local* attacks – i.e., you have to already be logged in the Mac. (Presumably they let a non-privileged user run privileged code; otherwise, there’s nothing to attack!)

Brief summary in English: The Shellshock bug can be used to attack a system only when the system performs particular operations that make it possible for an outsider to “reach” the bug. The most common operations are part of “being a server” – e.g., running a Web server. These are things laptops don’t typically do.

On Linux systems, the implementation of DHCP – an internal part of the system used to assign network addresses – proved to execute those particular operations. However, it appears that OSX uses a different implementation, and is safe.

If you want to be extra safe, go into System Preferences, go to the Sharing pane, and turn everything off. None of these services are currently *known* to be vulnerable, but if you don’t actually need them, it’s good policy to have them turned off anyway.

You can go a step further by going in the Security & Preferences pane, choosing the Firewall tab, turning on the firewall, and under Options selecting “Block all incoming connections”. (This actually allows “essential apps and services” to connect.)

Caution: Either of these steps, and especially the second, *may* cause some software to break. But you can always put stuff back as it was once Apple issues a patch.
— Jerry

I think it will affect Windows somewhere down the line. I will say thanks to Troy Hunt who said the following:
“There are non-Microsoft components sitting in front of their Microsoft application stack, components that the traffic needs to pass through before it hits the web servers. These are also components that may have elevated privileges behind the firewall – what’s the impact if Shellshock is exploited on those? It could be significant and that’s the point I’m making here; Shellshock has the potential to impact assets beyond just at-risk Bash implementations when it exists in a broader ecosystem of other machines.” at http://www.troyhunt.com/2014/09/everything-you-need-to-know-about.html . I think it will be one of those things that you will see linger for a while, and you may not see the impact of right away.

When I ran the test on a recent build of kali linux (from a few weeks back) it was vulnerable, running an “apt-get update && apt-get dist-upgrade,” caused the script to generate just:

a “this is a test,” message

with no vulnerable message.

I think the key thing to determine first is weather you have internet facing linux/unix boxes and patch these first. It appears updating fixes the issue.

I also believe that those running linux maybe lulled into a false sense of security because it is in my opinion more secure then windows and mac, with that said users of linux may not update there systems as often as they should.

I don’t disagree with what you say Chris, but the average Windows home user also does not run updates that often (a few, never) either. The typical (and uninformed) Windows user treats the computer like any other home appliance. Turn it on, do e-mail, facebook, play games, etc. They are more apt to change the oil in their car every couple of years than worry about updates! Also, keep in mind that most posters on these types of forums are not those typical home users, but more technically astute.

Thank you Bob.
Personally I like to think of myself as somewhere in between. What I lack in technical knowledge I try to make up for through research & questions. I appreciate when those of you with that knowledge translate your thoughts for me. My questions may sound stupid to some of you but they’re asked with a sincere desire to learn & understand. Snarky responses to those who ask questions serve little purpose other than to make the snarker feel superior & prevent the questioner from asking for help in the future. The goal of everyone with knowledge should be to share it with & enlighten those without not belittle them.

There is a difference between having a bug and being vulnerable. I have a small web server which has the bug, AND I run Apache AND I use some simple BASH CGI scripts.

So far I have not seen a cogent explanation of how just anyone on the net can inject BASH commands into just any BASH CGI script. Or can just anyone force Apache to execute just any old BASH CGI script they want?

Please someone explain the exact mechanism by which this bug is turned into an exploit so that I can tell if my buggy server is actually vulnerable.

Well with my limited understanding, and no programming knowledge what so ever,

when environmental variables are set, if a script calls for bash, an attacker can use these to bypass restrictions and execute commands in another environment, when he would otherwise not be able to do so.

When a web server invokes a CGI script, it passes the values of various parameters in the environment. For example, SCRIPT_NAME is the path to the script that will be invoked, and QUERY_STRING contains the stuff that was in the URL after the “?” – the parameters to the query.

The bug relies on the fact that (a) functions are stored in the environment, as strings that start with “()”; (b) if there’s junk on the line after the function definition, when a new shell starts up and reads its environment, *it will execute that junk”. Look at the test:

env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”

env is a program that takes a bunch of definitions of environment variables of the form variable_name=string; everything after the last assignment is passed to a shell. So this puts into the environment a variable named x that will be interpreted as function – the leading “()” – with a body – the stuff inside “{}” – that does nothing (the “:” is a no-op command; don’t ask), followed by a “;” to terminate the definition, and then the “echo” command. After putting this into the environment, env will run the bash shell directly, with the single command “this is a test”. If not for the bug, all you would see is “this is a test”. But because of the bug, while bash is starting up, it will define the bogus x function and then execute the “echo vulnerable” command – and you’ll “vulnerable”.

Word has it that now that people have started to look at bash, they’re rapidly finding all kinds of other issues. Bash is one of those programs that works when it gets what it’s expecting to get – but bash programmers have long known that if you hand it illegal stuff, it may not respond well. This is the first broad security problem that’s been reported as a result of a bash bug, but it may well not be the last.

“Please someone explain the exact mechanism by which this bug is turned into an exploit so that I can tell if my buggy server is actually vulnerable.”

Specific to Perl and CGI only, here’s the deal. The request information (HTTP headers, URI, etc) is essentially put into environment variables and passed into the CGI script. If the CGI invokes a bash shell (e.g. backticks, open, system, exec) then the malicious code being passed in from a HTTP header will execute. For example, the following Perl code can be exploited in most cases, even though it doesn’t touch any of the environment variables.

“…if Bash is set up to be the default command line utility on these systems, it opens those systems up…” I manage various network devices, the majority of which run some version of Linux or Unix in the background. Looking at the quote above, it seems to me that if the device has a bash shell running in it somewhere, but it isn’t the default on the CLI, then it’s OK? Does that sound right?

No, that is not correct. Programs can call the buggy version of bash directly, no need to rely on the default login shell. Whether they do or not is the question, and good luck getting any answers from vendors on that one. I expect that if bash is installed, then they’ll release a patch for it and move on without telling you one way or the other.

Would a “fix” be to make the default user shell “csh” or “tcsh?” – one can easily invoke bash from the command-line. Or does one have to change the “default system shell” – or even uninstall bash (somehow)?

Without knowing exactly how the affected code is written, it’s impossible to say. However, keep in mind that the whole reason to use a shell from one of these programs is to execute a script. Shells have different ways to accomplish the same thing (though with significant overlap), and there’s no way to know if a script that works with one shell will work with another.

There’s no “system default shell” as such. Each user has a default shell. Some low-level processes probably use /bin/sh as the shell to invoke. On Linux systems, /bin/sh is often just a link to /bin/bash. On Mac OS, the two are distinct, but it isn’t clear exactly what /bin/sh is – it’s very close in size to /bin/bash (1228304 vs 1228240 bytes on my MacBook) which suggests they are pretty much the same, but they aren’t *identical*.

It’s probably not worth the effort to go this route. Better to get a patched bash.

Until the code is completely audited, the real fix is more or less removing “bash” from systems (including suffering from every bash-ist application breaking when you replace /bin/sh and /bin/bash with some other shell).

This is more or less the same problem that we ran into with OpenSSL after the discovery of Heartbleed. The presence of a single bug is a canary hinting that there may be more bugs.

Brian: Several years ago you recommended the following: A “live CD” is a free and relatively painless way to temporarily boot a Windows PC into a Linux environment for financial transactions on a general use computer. Does ‘shellshock’ put this recommendation at risk? Are the same steps to be used to look for the presence of a vulnerable ‘BASH’ in the live CD environment? Keep up the great work. You are the best white hat that I am aware of.

The Live CD version won’t be running any Web services, so they can’t be compromised by this bug. Unfortunately, it *will* almost certainly be running the DHCP client, which is itself vulnerable. (It would be possible to create a version of the CD with a hard-coded IP address and a no DHCP client, which would be safer for a number of reasons – but much less convenient to use, perhaps even unusable on some networks.)

Right now, it would take a blended attack to make this an issue: Someone would have to attack through DHCP (which, with a properly configured network, should only be possible locally, not over the Internet); *and* leverage the resulting access to attack Web connections to financial institutions. It can certainly be done, but it will likely take some time before anyone produces such an attack: There’s so much more low-hanging fruit in people who aren’t running off a LiveCD. So this wouldn’t keep me up at night. You’re certainly *much* safer doing things that way than using mixed-purpose PC directly.

On the other hand, once the smoke has cleared and new, patched versions of the necessary software are available, it would certainly be prudent to go make a new CD….
— Jerry

The thing to keep in mind about “Live CDs” is that they’re “dead” as soon as you burn them.

If you burned one before Heartbleed, then every time you boot your live cd, you’re vulnerable to Heartbleed.

This means that you have to plan to burn new Live CDs regularly (and somehow ensure that your burning process isn’t compromised).

There isn’t really a “solution” to this problem. (More recently Brian reviewed a device which has some ability to update itself, but if you’re unlucky, it’s possible that it might be exploited in the interim — races suck.)

The lesson I’d like to underscore here is that if you choose to go the Live CD route, then you really do need to plan to replace your Live CDs regularly.

(One thing that might help would be if Live CDs used BusyBox instead of Bash, BusyBox is used by many embedded distributions of Linux, and I’d like to claim that it probably had more eyes than Bash did — until today, at which point Bash probably grew many more shallow eyes.)

Unfortunately, this only works if none of your applications / scripts call Bash directly.

The product I worked on last week is full of scripts each of whose first line is:

#!/usr/bin/env bash

Which basically says “dear OS, please ask ‘env’ for help finding ‘bash’, and then ‘env’ will run ‘bash’ and tell it to run the script”.

The reason that this line is present instead of “#!/bin/sh” is that the scripts we wrote have “bashisms” including “$BASH_SOURCE”.

If you want to take this approach, you should instead replace /bin/sh and /bin/bash with /bin/dash, and then watch as each application breaks and try to fix the application to not be bashist. Unfortunately, in our case, I couldn’t easily find a non bashist (or better a posix sh — http://pubs.opengroup.org/onlinepubs/009604599/utilities/xcu_chap02.html ) way to do what we were trying to do, and thus instead of removing our bashism, I declared our script bashist.

Patches were made available almost immediately.
What is so hard about patching?

Servers are the main concern.
Servers use static IPs, not DHCP. If afraid to patch, just don’t run any services that use bash. Is that really hard?

Desktops are a little harder. They use dhcp, cups, but they don’t usually run corporate-critical applications – so just patch them.

Problem solved for 99%.

For the other 1% running servers with bash interfaces that are corporate critical – fix it quick. Perhaps adding a patched, reverse-proxy that looks for the attempts is all it required?

Most routers don’t run bash – they run a stripped down borne shell or dash, so many people can just relax over that. dd-wrt uses busybox, not bash.

Mac users – you pre-paid $500-$1000 for the support. Walk into an apple store and demand that support. Apple has a habit of delaying patches for known issue by multiple years. Might be better just to change shells – zsh users love it, though I’ve never tried zsh.

For certain values of “almost immediately” that statement was true. We didn’t have to wait until some future Tuesday. 😉

Anyone with big issues (lots-o-systems) has a method to automatically patch select (or all) systems. Heck, we only have 30 and have been doing “devops” management for years. Even writing a trivial sh+ssh script to perform the patching shouldn’t take more than 20 min for any competent Linux/UNIX Admin (5 min is more likely). 30 seconds if you are already using a devops tool like ansible.

Not to be rude, but if writing a script for this is difficult, then that admin IS incompetent and should improve those skills. This facility is built-in, not extra cost involved.

Ran patches 3 times this week. 15 min each run, no reboot needed. Generally, we only patch during weekend maintenance periods, but this was a special situation – Wed, Thu, Sat. It was a slight inconvenience only.

As Jeff said the patches are incomplete, expect more to come imo. There is probably too many variables to account for all of them.

To be 100% safe people would have to read every script on their pc and rewrite them if nescessary lol, not realistic at all.

For home Desktop’s using cups and DHCP> to be safer set a static ip address and totally disable dhclient. If using an HP printer, install hplip from hp’s website and totally disable cups. with your printer ethernet into the router and its wirless off.

As for our routers, well thats another story. The source code of the firmware for mine supposedly doesn’t seem to have dash in it, and it doesn’t seem to have a telnet on, but who knows…. More and more of the routers the ISP’s give their customers have backdoors on them and prevent you from blocking them. Heck, the firmware on mine tries to stop me from turning off UPNP lol.

I am using OS X Yosemite 10.10 and the bug still available. Apple shoule fix this too because there are many developers who allready install the new OS X to test application. I am more curious if there are signs that this bug has been allready used by bad guys.

@Alexandru Dan Balan: No, you are *not* using “OS X Yosemite 10.10”. That hasn’t been released yet. You’re using a beta, which Apple consistently warns is for testing, *not* production use. If you have any significant value on a system running a beta, any loses are *your* fault, not Apple’s.

Apple has already announced that when OS X Yosemite 10.10 actually ships (which appears to be imminent), it will contain a patch for this issue. But they aren’t issuing a patch for the beta as with proper use of the beta, there’s no problem.

If you’re really concerned, instructions for building you own bash have made the rounds. You could also take the patch for Mavericks and pull out the pieces by hand – messy but it would probably work.

Although there’s likely been much scrambling to patch or throw up some sort of defense against this bug in the past couple of weeks, here’s a link to very interesting document by one security wonk on his investigation of its breadth and penetration: