Share this story

Remember when we said that a new patch had fixed the problems with the last patch to fix the rated-highly-dangerous “Shellshock” bug in the GNU Bourne Again Shell (bash)? You know, that bug that could allow an attacker to remotely execute code on a Linux or Unix system running some configurations of Apache, or perhaps the Git software version control system, DHCP network configuration or any number of other pieces of software that use bash to interact with the underlying operating system? Well, the new patch may not be a complete fix—and there may be vulnerabilities all the way down in the bash code.

Further Reading

Here's how the Shellshock vulnerability works, in a nutshell: an attacker sends a request to a Web server (or Git, a DHCP client, or anything else affected) that uses bash internally to interact with the operating system. This request includes data stored in an environmental variable. Environmental variables are like a clipboard for operating systems, storing information used to help it and software running on it know where to look for certain files or what configuration to start with. But in this case, the data is malformed so as to trick bash into treating it as a command, and that command is executed as part of what would normally be a benign set of script. This ability to trick bash is the shellshock bug. As a result, the attacker can run programs with the same level of access as the part of the system launching a bash shell. And in the case of a web server, that's practically the same level of access as an administrator, giving the attacker a way to gain full control of the targeted system.

David A. Wheeler, a computer scientist who is an acknowledged expert in developing secure open-source code, posted a message to the Open Source Software Security (oss-sec) list this evening urging more changes to the bash code. And other developers have found that the current patch still has vulnerabilities similar to the original one, where an attacker could store malicious data in a variable named the same thing as frequently run commands.

Norihiro Tanaka, a Japanese open-source developer, noted the problem in an e-mail to the bug-bash list today. By using an environmental variable called cat—the same name as a Unix utility that can concatenate files—he was able to bypass the fixes in the latest bash patch and pass through executable commands.

Wheeler noted this vulnerability as well, in an email to both oss-sec and the bug bash list:

I appreciate the effort made in patch bash43-026, but this patch doesn't even BEGIN to solve the underlying shellshock problem. This patch just continues the "whack-a-mole" job of fixing parsing errors that began with the first patch. Bash's parser is certain have many many many other vulnerabilities; it was never designed to be security-relevant…John Haxby recently posted that "A friend of mine said this could be a vulnerability gift that keeps on giving.” Bash will be a continuous rich source of system vulnerabilities until it STOPS automatically parsing normal environment variables; all other shells just pass them through! I've turned off several websites I control because I have *no* confidence that the current official bash patches actually stop anyone, and I am deliberately *not* buying products online today for the same reason. I suspect others have done the same. I think it's important that bash change its semantics so that it "obviously has absolutely no problems of this kind".

In other words, “Shellshock” may be partially patched, but it’s still highly dangerous on systems that might use bash to pass information to the operating system or to launch other software. And it may take a significant change to fix the code.

Wheeler made two specific recommendations for fixes to bash that will essentially break backward compatibility with older bash versions. The first is an idea suggested by German computer security expert Florian Weimer to add a prefix to bash functions, preventing them from being named the same thing as system variables (one of the mechanisms used to exploit the current bug). The second recommendation, put forward originally by NetBSD developer Dr. Christos Zoulas, is to allow bash to only import environmental variables into bash when explicitly requested—eliminating the risk of any other bugs in bash’s command parsing from accidentally bringing in malicious code or irrelevant data lying around in random defined environmental variables. Both of these changes, Wheeler said, are already being made by developers—function importing has been disabled by default in FreeBSD’s bash implementation in the latest fix.

These fixes technically break backward compatibility for bash, but Wheeler and others believe that they won’t affect the vast majority of bash-dependent systems. However, they will require developers who’ve used bash—and there are many, many software systems that rely on bash in some way—to thoroughly test and possibly modify their own code.

All of this means that the Shellshock bug will likely require many older services to be simply shut off until patches are fully implemented and tested. And there are many devices in the field—embedded systems that run versions of the Linux operating system that don’t use a streamlined utility system such as BusyBox or some other shell—that will require attention as well.

Promoted Comments

One thing that I would really like to know is a list of services I should shut down (like apache2) until this bash madness has been completely fixed.

Is there such a list of possibly affected services ?

Thanks,a Debian user

If you are using at least Debian squeeze (6.0), you're actually probably in better shape than anyone using Red Hat. Debian links /bin/sh to /bin/dash, so unless a script or executable explicitly calls bash, you're safe.

If you use "ForceCommand" directives in ssh, you can't rely on these if you let your users run bash. (You are effectively letting them get un-restricted shell access instead). But I think you could mitigate this by forcing them to run /bin/sh instead (or let them run mksh it's significantly more user friendly). And if you don't use "ForceCommand" it doesn't matter.

dhclient runs shells scripts in /etc/dchp/dhclient-enter-hooks.d, /etc/dchp/dhclient-enter-hooks.d but unless something insane was done, it will execute them with /bin/sh unless the script explicitly calls bash.

telnet, rsh, also are good things to disable, but you shouldn't have been using them in the first place.

You should be careful about the login shells for users. If you have some user that really should be a system user with /bin/no-login, or /bin/False as their shell, but some reason needs an actually posix shell as their login shell, set it to /bin/sh, make user it's not /bin/bash

Share this story

Sean Gallagher
Sean was previously Ars Technica's IT and National Security Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland. Emailsean.gallagher@arstechnica.com//Twitter@thepacketrat

147 Reader Comments

Here's how the Shellshock vulnerability works, in a nutshell: an attacker sends a request to a Web server (or Git, a DHCP client, or anything else affected) that uses bash internally to interact with the operating system. This request includes data stored in an environmental variable.

This is the problem. Remote requests should not have direct access to the local shell, period, just as remote requests should not have direct access to the database or else SQL injection happens. If bash is to be used inside the application in a way that handles user input, it should be given no permission to access the rest of the system so that it isn't a general purpose shell. If a local shell is invoked, user-specified parameters passed to it should be sanitized.

This is what permissions are for. It's a feature. Use them correctly. But I would think these would have to be very old servers (or code.) Who haven't been rewritten or updated in.. a decade, more?Isn't this reasonably remote nowadays? This almost seems more like FUD. I wear a pretty heavily lined hat too.

And in the case of a web server, that's practically the same level of access as an administrator, giving the attacker a way to gain full control of the targeted system.

What? Are you running your webserver on minix?

Typically, the default config in any *nix I've worked with.. the web server runs under its own account. On more security-conscious ones, it runs inside a chroot jail!

That's not to say there aren't major issues.. once an attacker has access to run arbitrary code as a user, that attacker can use local privilege escalation vulnerabilities to get higher level access... but no, the web server should never be 'basically an administrator'.

Where I work we ran into similar shell type security vulnerabilities. (No, not in the parser, but the script) One of the problems I see with scripting in general isn't that it can't be secure, but most scripts start off as nothing but some glue between different systems. Most shell stuff was never written with security in mind.

In my case we were passing properly escaped arguments to a shell script. Somewhere in the guts the escaping got lost. When confronted with the results the owners of the script were "we never intended this to be used how you were using it." Except we weren't the only folks using it like that.

So the code/glue works when it's tested the first time and eventually someone pushes it to production. Then more things start to depend on it... then you can't change the semantics of it...

Here's how the Shellshock vulnerability works, in a nutshell: an attacker sends a request to a Web server (or Git, a DHCP client, or anything else affected) that uses bash internally to interact with the operating system. This request includes data stored in an environmental variable.

This is the problem. Remote requests should not have direct access to the local shell, period, just as remote requests should not have direct access to the database or else SQL injection happens. If bash is to be used inside the application in a way that handles user input, it should be given no permission to access the rest of the system so that it isn't a general purpose shell. If a local shell is invoked, user-specified parameters passed to it should be sanitized.

A DHCP client receiving a response from a malicious DHCP server is not exactly a remote attack.

This almost seems more like FUD. I wear a pretty heavily lined hat too.

I wouldn't call it FUD.. I'd call it a legitimate issue of unknown scope. You might think "I don't use cgi anymore, so noone can remotely execute bash without credentials on MY box".. but the number of services on your typical *nix that use the shell to run scripts based on network connections is probably a lot higher than you think. Less so on servers, hopefully.. but (cr/h)ackers are very good at finding those rare exceptions to the rule.

A DHCP client receiving a response from a malicious DHCP server is not exactly a remote attack.

It is remote because it's from another machine and unless you trust every coffee shop wireless network, it's in your interest for your computer to protect itself against attacks originating from DHCP servers.

I see it often, and people tell me there's some other security mechanism which makes that safe, but I've never heard it explained clearly and succinctly in a way that made it stick to my poor tired old brain.

A DHCP client receiving a response from a malicious DHCP server is not exactly a remote attack.

It is remote because it's from another machine and unless you trust every coffee shop wireless network, it's in your interest for your computer to protect itself against attacks originating from DHCP servers.

It may not even stop there. You're thinking DHCP servers only listen for UDP broadcast packets which don't traverse routers. But remember that a DHCP server might also receive, parse, and respond to TCP packets routed to them from DHCP relay agents. So such a DHCP server may be vulnerable to attack over the internet.

I'd only just reenabled CGI (for mailman and awstats; nothing else in our cgi/bin except symlinks to those applications) literally minutes before reading this, but have now turned it back off again. At least they can keep processing in the background; it's just the web interface I lose.

What about SSH connections? I presume one would have to authenticate before a bash process were started? (Ubuntu 14.4; ssh seems to be using pam for authentication; pretty default setup.)

I agree with bash functions only working in envars with a specific prefix or suffix or both (FUNC) as it easily allows clearing the vars and they could be disabled from import into subshells by default.

I don't agree with disabling envar import en masse. It breaks backwards compatibly and forwards compatibility badly. (Things may start depending on a variable being present in the future, but if an app that was not updated sits in the middle, the var would be stripped).

Finally, I ask people to just not update bash for a week and wait until someone is sure the issue is finally, actually fixed for good. Exceptions if you use mod_cgi on your internets facing webserver as that's the most vulnerable. For other users, just wait for testing to be complete… please…

I can't agree more that it is a real threat. I am just surprised those old boys are still around. Heck RedHat 6 had apache running at very low privileged levels. As for CGI, I can believe theirs still some crud here and there, piled, on and on and an su gets stuck in ther. It just seems like massive explorations are the rage of the day, and this should be so rare.

Social engineering is the problem. Stop blaming the software (quite so hard.)

You're really missing the point.

1. Anything exploiting this can run arbitrary code as the same server the webserver runs as. In most cases this means that it has access to, at least read otherwise non-public files, like config files and such with plaintext passwords in them (say creds to a SQL database).

2. #1 opens the door to local priv escalation, which are much more common these days than remote access vulnerabilities.

This is about a software bug, not about social engineering, and it's not just about CGIs or even webservers in general. CGIs are just the most direct route to exploitation, easiest to demonstrate.

I'd only just reenabled CGI (for mailman and awstats; nothing else in our cgi/bin except symlinks to those applications) literally minutes before reading this, but have now turned it back off again. At least they can keep processing in the background; it's just the web interface I lose.

What about SSH connections? I presume one would have to authenticate before a bash process were started? (Ubuntu 14.4; ssh seems to be using pam for authentication; pretty default setup.)

Correct, you must authenticate first with SSH. However, SSH is vulnerable if you use ForceCommand in sshd_config to restrict the commands a user can use (such as a git account) because it allows users to bypass ForceCommand and run arbitrary commands.

(If you don't use ForceCommand, then the user can already run arbitrary commands when logged in via SSH, so nothing changes)

I...really have to be honest; Heartbleed I was pretty understanding of, that occasionally serious issues get through in any system. I was a bit surprised to hear of more vulnerabilities so soon afterward, but now hearing that the Bash vulnerabilities go even further - AND it's not some extreme edge-case security situation - makes me genuinely question that fundamental knowledge geeks have had for years that Linux is more secure than anything else.

I'm not about to claim Microsoft or Apple writes things more securely, but I don't think it can really be considered an advantage of Linux by any large margin anymore.

Correct, you must authenticate first with SSH. However, SSH is vulnerable if you use ForceCommand in sshd_config to restrict the commands a user can use (such as a git account) because it allows users to bypass ForceCommand and run arbitrary commands.

(If you don't use ForceCommand, then the user can already run arbitrary commands when logged in via SSH, so nothing changes)

So is SSH only vulnerable through something like Git IF they can actually achieve SSH authentication with the server? Meaning there is less chance for exploits if all users are trusted?

I...really have to be honest; Heartbleed I was pretty understanding of, that occasionally serious issues get through in any system. I was a bit surprised to hear of more vulnerabilities so soon afterward, but now hearing that the Bash vulnerabilities go even further - AND it's not some extreme edge-case security situation - makes me genuinely question that fundamental knowledge geeks have had for years that Linux is more secure than anything else.

I'm not about to claim Microsoft or Apple writes things more securely, but I don't think it can really be considered an advantage of Linux by any large margin anymore.

Anyone who genuinely believed that Linux had a huge security advantage hasn't been right since Windows 9x and classic Mac OS ceased being developed. They also can't call themselves knowledgable if they never once paid attention to security advisories or followed the kernel mailing list and seeing (shockingly!) that fixing bugs is still kind of a thing in a large software project. And if they never did that, it's almost a certainty they never bothered to configure their system to take advantage of the security features Linux does provide.

Here's how the Shellshock vulnerability works, in a nutshell: an attacker sends a request to a Web server (or Git, a DHCP client, or anything else affected) that uses bash internally to interact with the operating system. This request includes data stored in an environmental variable.

This is the problem. Remote requests should not have direct access to the local shell, period, just as remote requests should not have direct access to the database or else SQL injection happens.

What's so special about a shell? It's just a programming language interpreter. Whichever language you write the server application in, the program will have direct access to at least that language's runtime. Keeping the remote users from executing arbitrary code is always a must.

dash is a POSIX-compliant implementation of /bin/sh that aims to be as small as possible. dash is a direct descendant of the NetBSD version of ash (the Almquist SHell), ported to Linux in early 1997. It was renamed to dash in 2002.

However, who really runs any mod_cgi stuff nowadays and calls bash directly?

Amidst all the hysteria ("your Android phone could be hacked! Your smart lightbulb could be vulnerable! Up to 500 million systems run Bash and every one of them could be spying on you and watching you take a poop!") it's really difficult for an ordinary person to understand the facts.

This is what I seem to ascertain, please correct me if I'm wrong:

In order for a Shellshock attack to take place, an attacker has to connect to a network-facing service running on the target computer. Normally this would be a web server with PHP or something similar; or a CGI script. Other types of services may also work with more effort, but for simplicity we'll assume it's a web server with PHP. The attacker than has to append shell script commands to a web page request that involves PHP - not every PHP script will work, but some will if they call functions that use a shell somewhere. The attacker also must hope that the PHP or CGI script does not sanitise its inputs before processing them - very poor programming practice for a web-facing service, but sadly fairly common. The administrator must not have changed the default non-interactive shell away from its default (Bash).

If all these conditions are correct, the attacker's shell commands will run, typically as the ultra-limited user account of the web server, and sometimes within a chroot jail or a VM that does not run anything else. An additional unrelated attack is needed to get into a less limited user account or to Root in order to do anything useful.

EDIT: It's been pointed out to me on another site that the attacker does not need to get into another user account in order to steal your database username and password, if these are read from a PHP script in the site hierarchy.

So... if I understand things correctly, Bash itself is only part of the problem. An important part to fix, but still only part of the problem. Web apps should always sanitise any data that comes from users. You might have a fully patched Bash, but until your inputted data is sanitised you could be vulnerable to a buffer overflow or an unrelated bug in something else.

This should bring to light another problem: Why is the default non-interactive shell something as big and complicated as Bash? I understand the interactive shell being Bash - convenience for admins and desktop users - but why isn't the default system shell something light and simple and locked-down to decrease attack surface? Or are these web apps explicitly calling Bash rather than just "whatever the default shell is"?

No question about it. Even if there's a legit reason for a remote client to be running shell commands, limit the ability to execute code of any kind from an environment variable. in fact, for remote access or web server access,I'd disable eval, exec and any other commands that allow de-referencing and execution of code.

The more I read about this, the more I think that this is not necessarily a bash issue, but a webserver/cgi/whatever issue. Simply put, the webserver or application should not be allowed to pass commands or data directly to shell. The webserver should be sanitizing it's inputs before it allows data or commands to be sent elsewhere.

Here's how the Shellshock vulnerability works, in a nutshell: an attacker sends a request to a Web server (or Git, a DHCP client, or anything else affected) that uses bash internally to interact with the operating system. This request includes data stored in an environmental variable.

This is the problem. Remote requests should not have direct access to the local shell, period, just as remote requests should not have direct access to the database or else SQL injection happens.

What's so special about a shell? It's just a programming language interpreter. Whichever language you write the server application in, the program will have direct access to at least that language's runtime. Keeping the remote users from executing arbitrary code is always a must.

I don't think it's being used as a shell, at least not the way a user would. I think it's probably being used as an API to get stuff from deeper in the system, probably because it's stable and easy to do. As for why? Probably your options to do stuff like that are BASH or whatever your builds API's to do this (Redhat would be different to Ubantu etc) and there's no option to get it straight from the kernel, oh and some systems might not have anything but BASH plus a minimal layer to support whatever program you want to run (And Apache server doesn't need an X window system)

Amidst all the hysteria ("your Android phone could be hacked! Your smart lightbulb could be vulnerable! Up to 500 million systems run Bash and every one of them could be spying on you and watching you take a poop!") it's really difficult for an ordinary person to understand the facts.

This is what I seem to ascertain, please correct me if I'm wrong:

In order for a Shellshock attack to take place, an attacker has to connect to a network-facing service running on the target computer. Normally this would be a web server with PHP or something similar; or a CGI script. Other types of services may also work with more effort, but for simplicity we'll assume it's a web server with PHP. The attacker than has to append shell script commands to a web page request that involves PHP - not every PHP script will work, but some will if they call functions that use a shell somewhere. The attacker also must hope that the PHP or CGI script does not sanitise its inputs before processing them - very poor programming practice for a web-facing service, but sadly fairly common. The administrator must not have changed the default non-interactive shell away from its default (Bash).

If all these conditions are correct, the attacker's shell commands will run, typically as the ultra-limited user account of the web server, and sometimes within a chroot jail or a VM that does not run anything else. An additional unrelated attack is needed to get into a less limited user account or to Root in order to do anything useful.

So... if I understand things correctly, Bash itself is only part of the problem. An important part to fix, but still only part of the problem. Web apps should always sanitise any data that comes from users. You might have a fully patched Bash, but until your inputted data is sanitised you could be vulnerable to a buffer overflow or an unrelated bug in something else.

This should bring to light another problem: Why is the default non-interactive shell something as big and complicated as Bash? I understand the interactive shell being Bash - convenience for admins and desktop users - but why isn't the default system shell something light and simple and locked-down to decrease attack surface? Or are these web apps explicitly calling Bash rather than just "whatever the default shell is"?

Not quite, given that DHCP and other services used, from what I get certain values that are supposed to come in with the message are passed straight on to BASH. Based on your example a username and password are sent to the server for validation, the PHP script passes them off to BASH for some reason but they have special meaning to BASH and cause it to do something different then what the PHP script excepts

I'm kind of curious if similar exploits could work against other shells or scripting environments. Shells aren't generally seen as attack vectors and I haven't really heard much about security in regards to them.

Do the other popular shells on the *nix platforms suffer from the same issues?

What about Powershell or CMD? Even though applications don't depend so heavily on the command line interfaces on Windows platforms, I have seen it from time to time.

Norihiro Tanaka, a Japanese open-source developer, noted the problem in an e-mail to the bug-bash list today. By using an environmental variable called cat—the same name as a Unix utility that can concatenate files—he was able to bypass the fixes in the latest bash patch and pass through executable commands.

Here's the code in question (note: I changed "echo rm -rf /;" to "echo This is an env variable, not cat.;"):

You cannot reproduce this with the Red Hat patch to bash (which you mentioned in the previous "Shellshock" article). I've tried. The patch prevents anyone except bash itself from using envrionment variables to store function definitions, which is the plumbing it uses for exporting functions to subshells (see: Shellshock: How does it actually work?). If you try, the result is treated simply as a string and never treated as executable code. If you add 'echo $cat' to test.sh, you can see there is indeed a variable named cat containing the string '() { echo This is an env variable, not cat.;}', but it does not overide the cat program in PATH and you cannot execute it. As far as I can tell, this article is still discussing the patch to CVE-2014-6271. Unless there's some other patch for CVE-2014-7169 that isn't the Red Hat patch.

Quote:

The first is an idea suggested by German computer security expert Florian Weimer to add a prefix to bash functions, preventing them from being named the same thing as system variables (one of the mechanisms used to exploit the current bug).

This is precisely what the Red Hat patch does. Although, to clarify, bash now adds a prefix and a suffix to variables it uses for exporting function definitions to subshells, not to the names of functions being exported (overriding system commands with shell functions is perfectly legitimate).

Quote:

Well, the new patch may not be a complete fix—and there may be vulnerabilities all the way down in the bash code.

I do not understand what this even means? The original bug was "all the way down in the bash code".