I write about disruptive ideas in science, technology and culture that change the future, the new discoveries in science and technology that change how we interact with ourselves, society and our planet.
My technology industry experience at IBM, Microsoft and Amazon includes work in account management, software testing management, localization editing and online content production. I have also worked as a consultant in the software and gaming technology space.
My work on technology, science and culture has appeared in publications such as Wired, Ars Technica and Massive Online Gamer, among others. For story ideas, requests to review new technologies or simply to get in touch, please feel free to email me at mpvenables@gmail.com. Follow me on Twitter, Facebook or Google +.

[Updated] How They Popped The Penguin: One Bash Tactic And What It Means For Linux Data Security

Screen capture of output from the listener (attacker side) of the Linux bash tactic. Image courtesy David Schwartzberg

Update, Oct. 11.

I’ve included text below that is taken directly from the post from Mark Kikta’s blog on the Viopoint company website. The post explains Kikta’s recent presentation at GrrCon, BSides Las Vegas, BSides Chicago, and BSides Detroit on the subject of maintaining persistence in Linux and using Bash to test for simple vulnerabilities in Linux. It also addresses some questions that have been raised by the Linux community in the comments of my post.

“Popping the Penguin” is my introductory talk on the principles of maintaining persistence in Linux. It began with a simple question:

How much of an attack can I demonstrate with Bash; a shell that has been around since 1989 and is found on virtually every Linux server?

I use demonstrations such as those in my talk to illustrate how remote access trojans or remote access tools behave on the network. The use of simple Bash commands lowers the perceived risk because no malware is required or installed on the servers. It does not get more fundamental than Bash commands. If we can achieve a reverse connection in a demonstration, using Bash, what can real-world attackers do with real-world weaponized malware?

I have heard some common questions, when giving the Popping the Penguin talk.

“Is this new?”

No. As Michael wrote, it is “new take on an old tactic”. What is new is my talk, discussing fundamental concepts and how they still relate to us today. I have given this presentation in various forms at GrrCon, BSides Las Vegas, BSides Chicago, and BSides Detroit. For additional resources, you might be interested in Reverse Shell with Bash, Linux shells using built-in tools, and IO redirection. I list these websites in the references slide of my presentation.

“Why did you not write a script using a language like C, Python, Perl, etc?”

First, the simplest path remains the most useful in a demonstration, allowing people familiar with Linux to quickly identify all components of these techniques. Second, with some clients, installing potentially malicious software or downloading scripts is a concern. As a former Linux admin, if someone came and asked to run scripts as root, I would be less than accommodating. Keeping it as simple as possible makes the demonstration more meaningful.

“Why base64 instead of actual encryption?”

Again, the simplest path remains the most useful when demonstrating the principles behind particular technique for a few reasons. First, base64 encoding can be used quickly, with minimal configuration on the host side. Second, it’s easier for the untrained observer to understand encoding than it is for them to understand cryptography or cryptanalysis. Third, base64 encoding doesn’t require symmetric or asymmetric key generation, distributio,n and protection. Thereby again making this technique easily accessible and reproducible for anyone with access to a Linux operating system. When performing these demonstrations, accessibility of the information to our audience is key.

“What about firewalls? What about intrusion detection systems (IDS)?”

When we are testing the ability of an attacker to maintain access during an exercise, it is assumed that all prior controls have failed. This matters because controls will fail. Our goal is not to see how well we can break into clients’ systems, but rather to see how well clients can detect, prevent, and correct each phase of an attack. This particular demonstration, in Bash, assumes the initial breach and escalation of privileges has already occurred.

“There are many ways to block this type of method of persistence.”

There are definitely many ways to block this. In my talk, I suggest people go ahead and try these Bash commands and see if a connection can be created and persistence can be achieved. If persistence can be achieved, we look into host-based IDS, network IDS, firewalls, and other technical controls. In a perfect world, none of this would need to be tested. But, as any admin can tell you, controls fail and processes breaks down.

“You used a Minecraft server, seriously?”

Yep! I have played with these attacks in Bash a number of Unix-based and Linux-based operating systems, in a number environments. Thanks to the help of David Schwartzberg, I even did a demo from the Minecraft server that sits in his very impressive lab, behind layers of controls.

In conclusion, find simple ways to test for simple vulnerabilities. Remote access tools are still actively used to create botnets, maintain access, and enable security breaches. Using Bash to demonstrate these types of techniques on Linux is one way to test. You can even use PowerShell to do similar persistence and IDS evasion tests. (Check out Matt Johnson and Ben0xa’s DerbyCon videos for the Windows equivalent, and more.) I appreciate the interest generated by this article and hope we can find other fundamental methods to use as stepping stones to evaluate the security posture of systems you and I maintain.

I recently learned of the discovery of Mark Kikta’s new take on an old tactic of creating a remote connection and a reverse shell that establishes persistence on a Linux server. This tactic is just one of the stages of a security breach, those being: reconnaissance, exploitation, elevation of privileges and the final one — persistence. This particular technique is for demonstrating persistence, and assumes that the other steps have already been completed.

His company, Viopoint, performs vulnerability assessment and network penetration testing of high-level enterprise networks with sensitive data. Kikta is a security consultant with the company (and former Linux engineer) who is a frequent speaker at information security and hacker conferences such as SecTorCa and various BSides events, Grr Con and others.

Kikta gave me a detailed rundown of how this Linux tactic works. He is what is known as a White Hat hacker, aka an “ethical hacker,” one of the good guys whose job it is to think of what are known as “penetration attacks” on company servers. His penetration testing evaluates the weakness of company data servers that are protecting valuable information technology assets such as company databases, personnel files, proprietary technology schematics and other crucial data related to the company’s intellectual property.

I asked Kikta to provide a general description of his demo of his attack with Bash, a shell that has been around since 1989 and is found on virtually every Linux server. Attacking a computer system is a fairly straightforward process. First, the attacker discovers as much information as possible about a potential target. This could include socially engineering the information out of people to using Nmap to identify open ports. Typically, during a malicious hack at this point, no laws have been broken; this changes during the next steps. The attacker, once the information has been gathered, proceeds to utilize this information to exploit vulnerabilities in the infrastructure, which are almost always there. Once the attacker is in the system, he or she wants to make sure that they do not lose access to that system before continuing on to attack the network from within. This stage in the attack is usually referred to as establishing persistence, or simply persistence. Knowing that this is a key step for an attacker to “own” a network, many security officers and engineers spend a lot of time attempting to prevent this from occurring. It was this concept that got me thinking about what nobody would expect.

Kikta described to me just how his Linux tactic was executed. He considered many different ideas before settling on the scripting language called “bash”. Bash stands for “Bourne Again Shell” and is the primary system administration language on most Linux systems, present on nearly all *NIX systems by default. Since it is so commonplace in the Linux world, Kikta figured common system administration programs and commands, which on their own are innocent, could be combined in such a way as to work against the system. It ended up working even better than he expected in terms of detectability, even evading detection on enterprise grade firewalls. This method of maintaining access is not something that needs to be uploaded, configured, or snuck past firewalls. It is simply a combination of routine administrative tools.

According to Kikta, the reason this tactic evades security so well is because it has no “signature” to alert an intrusion prevention system, and since the compromised host is creating an outgoing connection to your host, it looks like a simple outgoing TCP/IP connection. There is only one way to stop this tactic from working in your environment, and that is to prevent your host from talking to the internet. Above, at the top of the post is a screenshot taken that views the output from the listener (attacker side) of the “Bourne Again Shell” tactic. Kikta adds that one should note the ability of the attacker to send commands and receive the output on the other side. This particular connection was established from a web server in Chicago, through a layer 7 firewall/IPS, to a listener at Kikta’s lab in Michigan. Acknowledgement goes out to Kikta’s friend, David Schwartzberg , who sacrificed the integrity of his Minecraft server for the sake of the research experiment.

Kikta explained to me that the whole concept behind this is that it utilizes flow-based programming versus object oriented programming. The way it works is fairly simple. The attacker starts the listener on the box they want the attacked host to connect back to. The compromised box will then attempt to open a TCP/IP tunnel back to the listener using a function of bash, /dev/tcp, which is a pseudo device file that establishes a connection on a specific port and IP. During the initialization of the listening command on the attacker’s machine, a file called a First in First Out pipe is created. This allows the attacker to send commands into the data flow, sending it across the network to the remote host. The cool thing about this is that it is protocol agnostic, since all the processing is handled within the processes on the hosts.

The attack was tested, without the base64 encoding, against an enterprise layer 7 firewall with a Snort based IPS. Kikta and his team were expecting it to be blocked as an attempt at a nonstandard telnet connection. As it turns out, the firewall was unable to block the traffic for the following reasons:

1. The connection was outgoing (the compromised machine being the one to initiate the connection, utilizing an unrestricted, high level port) 2. It is protocol agnostic, i.e., there was no protocol traffic going across the connection such as ssh, telnet, etc. 3. Were an IPS to be set to look for strings such as ‘ls’, ‘echo’, ‘passwd’, etc. it would be constantly alerting the moment an admin attempted to Google for something relating to this.

The remediation step for this that the team has worked out is to use stateful filtering. Assuming that your host should not be talking to the network (using something like RHN Satellite for updates and package management), block any attempt at sending a SYN packet outbound.

Post Your Comment

Post Your Reply

Forbes writers have the ability to call out member comments they find particularly interesting. Called-out comments are highlighted across the Forbes network. You'll be notified if your comment is called out.

Comments

This is stupid–no, I’m sorry–this is beyond stupid. You are saying using Bash you can open up a network connection and send commands? Or is it the super secure base64 encoding used that “bypasses IDS rules” that is exciting? And this is supposed to be called a “bash attack”?

Any decent programming language has network functionality, and the ability to Base64/rot13/whatever your network traffic. I wouldn’t call it a “Python attack” or “C attack” if I managed to get a shell on a server and write a program to execute more commands. That is what computers do–they run commands, sometimes *gasp* even on networks!

How is this interesting? How is this news? Who thought this was worth writing an article about?

While it’s nice that you’re looking to bring attention to Linux Security, I’d like to call to light several issues with this article. Most notably its technical accuracy.

“There is only one way to stop this attack from working in you environment, and that is to prevent your host from talking to the internet.”

In this sentence, the reader is lead to believe you need to isolate your web server (for instance) from the internet entirely. This is not true. A responsible system administrator is likely filtering traffic outbound down to RELATED and ESTABLISHED connections, or to known devices such as a database server. (emphasis added as those are among the many connection types that a TCP/IP socket can use.)

Additionally, iptables (the built-in software firewall in Linux), has the ability to filter based off of UID and GID (User and Group IDs, respectively). A very brief google search for “iptables uid gid module” returns many results on how this is done. Secondly, the entire article assumes that the attacker has access. The initial attack to gain access to the server is likely to set off a properly-configured IDS like a Christmas tree.

“Were an IPS to be set to look for strings such as ‘ls’, ‘echo’, ‘passwd’, etc. it would be constantly alerting the moment an admin attempted to google for something relating to this.”

While this is technically true, a competent administrator is going to put their public-facing servers in a DMZ, and have an IDS/IPS monitor that leg of the network. So searches for ‘ls’, ‘cd’, and so forth *SHOULD* cause the IDS/IPS to alert in a properly configured network.

Please do some research into how a network should be properly designed before making broad statements like this in your articles.

Nothing about this is news. Nothing about this is even grounded in reality. What tiny tiny minuscule bits of truth it contains are nothing more than basic primitives of computer networking. The author forgets to mention entirely that this isn’t an exploit at all, by any means, but rather just a ‘stealthy’ (heh) means of phoning out from an already compromised machine using known tools to do exactly what they are meant to do.

As the author of this article states, “data security is something that most people don’t want to think about.” I wouldn’t consider myself to be in this group of people that he’s referring to. I take security very seriously, and was kind of amused by this article. I’m going to try to outline every brazen point that Mr. Venables makes in this article, and give a counterpoint based on my experience in the industry.

The first thing that I’d like to point out is that one of the major topics in this article is “establishing persistence”. The technique outlined in this article describes something called a Bash connectback shell. The term “connectback” is used, because the target machine makes a connection back to the attacking machine to provide a Bash shell. This is generally a pretty decent way to get around firewalls if they are incorrectly configured, but any competent sysadmin or netadmin knows the implications of allowing any outbound connection to be made from a server. The reason I dispute the fact that this method provides any means of “persistence”, is that for this attack to be successful, you need to have a way to run the commands (in this article) on the target machine in the first place. This method only allows you to gain access to a shell on a machine that you can already remotely run shell commands on (what’s the point?).

I was also very confused by the statement, “It ended up working even better than he expected in terms of detectability, even evading detection on enterprise grade firewalls.” Especially after seeing that this technique was tested from a Minecraft game server. I would not expect to see any kind of hefty firewalls used on a server of this type, and as no specific details were disclosed about the firewall, I have reason to doubt that any “enterprise grade firewall” was tested at all. I did see that Snort was “tested” later in the article, but because Snort is a rule based Intrusion Detection/Prevention System, the actual configuration of the service is vitally important for it to be effective.

In the attack code itself, I can see that this connectback method relies on the existence of the /dev/tcp pseudo-device directory. Many newer Linux systems have specifically disabled this pseudo-device in the kernel. Furthermore, some other *NIX systems don’t even support it. Due to this fact, this “new ‘bash attack’” would be rendered useless on many different Linux distributions and *NIX systems.

Now I’d like to comment on the statement, “the firewall was unable to block the traffic for the following reasons…” As for #1 in the list, “The connection was outgoing”, any good stateful (or even stateless) firewall ruleset should be able to effectively filter both incoming and outgoing connections. Just because the connection was made outbound, does not mean for even a second that it _COULD NOT_ be blocked. If the firewall felt like it, it could have started dropping the packets right then and there. #2, “It is protocol agnostic”, yeah, it’s raw freaking text… I believe there are already pre-existing Snort rulesets that can look for connectback shells (based on signatures commonly seen when running shell commands) and start setting off red flags. And then #3, “Were an IPS to be set to look for strings ….” This is a pretty ridiculous assumption to make. Any layer 7 protocol analyzer/firewall should be able to determine what protocol the traffic it is seeing is, and whether or not the traffic should be allowed. If it starts seeing raw ASCII commands and responses that look like command output, with absolutely no HTTP (or other protocol) headers, something must clearly be wrong.

Then there’s the “bash attack”…. I would have to agree with Tyler Nighswander on this. In all the years I’ve worked in security and system administration, I’ve never heard the term “Bash attack”. Tyler does a pretty good job at “bash attacking” this point, so I’ll leave it at that.

Finally, I’d like to point out that this kind of attack is like 20 years old and hardly newsworthy.

I welcome the obvious investment and zeal of the Linux community of developers and users in addressing their concerns regarding the methodology of Kikta’s bash tactic. I think that substantive deliberation is a good thing, so I wanted to address the comments that have been posted so far. After further consultation with Viopoint, I’ve clarified the language in my description of Kikta’s tactic in my post, and have removed any misleading or imprecise descriptors that lent the impression that Kikta’s tactic was a newly-invented phenomenon.

I’ve also invited comments from Viopoint to address any ongoing questions that may arise within the comments of this post for the sake of continuing the discussion.

This tactic isn’t new? It’s new to me and I have a PhD in CS and 20 years in security. This is bleeding edge stuff. Looking forward to more articles of unconventional vectors for attack in the future. Great work!

Mr. Venables, great work! I’d like to say I have a PhD. in CS and found this article extremely informing. This is next level stuff! Keep up the great work and I hope to read more articles from you soon.

Really? Next level stuff? Other ‘plebs’ have ‘little to no security knowledge?’ REALLY? What kind of shitty CS program gave you a PhD without understanding simple scripting and TCP clients/servers? Please go back to whoever issued you your degree and demand an immediate refund. Either that, or stick to what a PhD is good for: marketing via buzzword bingo with your Cyber Big Data Loss Prevention Appliances and UltraLaserNextGenSuperDooper Firewalls and Advanced Mega Heuristic AntiVirus – because those are the things that keep us cyber safe. /facepalm

You want to go plebs? I’ve forgotten more about security than you noobs have learned. I’ve been smashing stacks since you were cuddling a Cabbage Patch. You could stand to learn a thing or two from Mr. Venables.

Wow, Forbes – I sure hope you put more critical thought into your financial articles. Has it ever occured to you, that if I bribed/extorted/whatever someone with check writing authority in your company, that I could get them to send me a big fat check? Better get on that.

Some great clarifications and comments above. Let’s add in some context.

“Popping the Penguin: An Introduction to the Principles of Linux Persistence”, that is the title of Mark Kikta’s SecTor talk. The Introduction part really is key.

Mark walks the audience thru example attacks, using Bash to illustrate the principles involved in persistence. At one point, as illustrated in this article, he sets up a Bash connectback shell. Mark then explains how the reverse shell can be caught by IDS such as Snort and IDS installed on enterprise firewalls. At another point, he shows the basic bypass of using encoding, like Base64. Mark’s goal for the talk is to outline the principles by providing example commands and tactics used within Bash.

Everything old is new again. For those new to information security, introductory talks can be eye opening. For those of us who have been in the field a while, let’s keep the discussion going while reaching out to the broader and younger community.

I’ve updated the post with the complete text from Mark Kikta’s blog on the Viopoint company website that explains his recent presentation at GrrCon, BSides Las Vegas, BSides Chicago, and BSides Detroit on the subject of maintaining persistence in Linux and using Bash to test for simple vulnerabilities in Linux, as well as addressing some common questions that have been raised in the Linux community in the comments of my post.

“Kikta reminds me that Linux is no more or less secure than any operating system.”

Which casts doubt upon his competence/knowledge.

30 years ago, when I was working in secure systems, there was something usually referred to as the Orange Book, which gave criteria for categorising operating systems into six *distinct* security levels (in ascending order) C1, C2, B1, B2, B3, A1.

Clearly there were difference between these categories in terms of security.

The article is misleading in seeming to suggest that this is an attack which allows an attacker to gain access to a target server. The attacker needs to have privileged access to the target server already to start the connectback shell on the target — basically, you’d need to have shell access in the first place — and a smart attacker would probably choose to drop something a little more persistent and stealthy than an extremely suspicious-looking shell session that is piping stuff from an arbitrary port right into bash. This is going to stick out like a sore thumb to anyone who happens to check a list of running processes, will go away the moment something happens to end the shell session it’s running in (such as a reboot or routine culling of long-running jobs), and will definitely be detected very quickly by any halfway competently-configured intrusion detection system.

From the article, “This tactic is just one of the stages of a security breach, those being: reconnaissance, exploitation, elevation of privileges and the final one — persistence.”

Opening the Bash session comes during the exploitation stage. For Linux servers on the Internet hosting web sites, typically, this is achieved by taking advantage of vulnerabilities in the web pages. There are other ways, of course, but the bottom line is that the exploitation comes before the techniques Mark Kikta covers in his talk.

“This particular technique is for demonstrating persistence, and assumes that the other steps have already been completed.”