I was recently in a client engagement where we had to rebuild / redeploy some ESXi 4.x servers as ESXi 5.1. This was a simple task, and quickly done (thanks VMware!), but before we were finished I realized that we had missed a critical part - the remote managent port on the servers. These were iLO ports in this case, as the servers are HP's, but they could just as easily have been DRAC / iDRAC (Dell), IMM or AMM (IBM) or BMC (Cisco, anything with a Tyan motherboard or lots of other vendors). These "remote management ports are in fact all embedded systems - Linux servers on a card, booting from flash and usually running a web application. This means that once you update them (via a flash process) they are "frozen in time" as far as Linux versions and patches go. In this case, these iLO cards hadn't been touched in 3 years.

So from a security point of view, all the OS version upgrades and security patches from the last 3 years had NOT been applied to these embedded systems. What can you do with access to these remote managment cards? Well, anything from take over the console keyboard and screen, mount a CD or DVD over the netowrk, to powering the server off or reconfigure or delete RAID arrays. The goal of these cards is to enable you to do almost anything you can do from the console, but from a remote location. Couple this with difficulty in patching them (or just forgetting to patch them), and youv'e got a serious exposure.

How can you mitigate a situation like this? The obvious answer is to patch as updates come out. For many server vendors however, this means booting the server from a CD or DVD. This is often a tough sell to management, as it's not only an outage for a production server, but if the firmware update fails or causes some new problem, that could cause another (unplanned) outage later, or in the best case a planned outage to back out the update. Plus you need to convince them every time the topic comes up that you need remote management at all, which eventually starts to sound like too much work. But *not* updating critical server components is a ticking time-bomb.

And all this assumes that your server vendor actually fixes known security issues in their OS or management interfaces. Since these interfaces are normally web-based, not only does this mean OS patches, but it's VERY common to see XSS, CSRF, authentication bypass and even command injection vulnerabilities built-in to the web interface - so you get your web vulnerabilities for free before you even deploy your own web application! And server vendors aren't always keen to hear about or fix security issues in these interfaces - from their point of view they might be asked to fix an interface issue in a product they stopped selling years ago, so to the sales folks it might seem like lost money to fix these things.

What many folks do is put the remote management cards in their dedicated management VLAN, which has ACLs and other protections on it. This certainly isolates these cards from attackers and targetted malware, but if that VLAN is ever breached, these cards become the low-hanging fruit for the attack, which can then be used as a pivot to attack more hardened interfaces such as the Hypervisor admin consoles or SAN management interfaces that are also commonly in these Management VLANS. I'd suggest both patching your server hardware and segmenting these interface cards off, possibly on a dedicated VLAN just for them.

What other devices in your datacenter should be considered embedded systems, with the same risks?

The obvious ones in my mind are KVM (Keyboard / Video / Mouse) switches, which now often have IP interfaces for access, and in some cases operate over IP with dongles attached to the servers - in this case both the KVM unit and the individual dongles are all embedded systems.

On the network side - routers, switches and fiber channel switches all have these same risks. These devices and risks are generally more well-understood though, and in most security conscious environments are patched annually or (hopefully) more frequently. But in security assessments, it's not uncommon at all for me to find a core routing switch that the entire organization depends on running 10 year old code (just to pick an example from last month).

Many of the more advanced SCSI RAID, SCSI HBAs or Fiber Channel HBAs (Host Bus Adapters) are now web-enabled, with their own IP addresses and management interfaces (no risk there!). Folks tend to see these web interfaces as great features, and not make that next cognitive leap to see how they can easily be turned into silent killers.

Oh, one more thing - please change the passwords on all of these! All the patching in the world won't help you if you're attacker can google for the administrative credentials. I can't tell you how many SANs, Bladecenters or FC Switches I've seen with the default administrative credentials still in play. If your admin password is still "password", it's time to change it!

The list of embedded devices in your datacenter goes on - door locks, lights and HVAC controls of course, but I'm sure there are other embedded systems that could be turned to evil in our server racks. Take a walk down the cold aisle (or better yet, down the hot aisle) in your server room and take a look - please, post anything interesting that you might find in our comment form!

Thanks to Gebhard for pointing out the article by Heise about a new spider focusing on finding web application vulnerabilities [1]. "Punkspider" runs essentially a vulnerabiliy scan on random web sites. The results are then searchable. I am not sure about the quality about the results (it doesn't find anything for isc.sans.edu ... ) but you may want to check your own site. There is also a simple, non documented at this point, json API:

http://punkspider.hyperiongray.com/service/search/domain/

Which accepts the following GET parameters:

searchkey: url|title
searchvalue: the url or title you would like to search for
pages: 0
pagesize: how many results (10 by default)
pagenumber: which page (1 by default)

The Heise article below has more details. Evidentially it is possible to block the spider via robots.txt but I haven't seen the user agent documented. (need to check my logs). Of course, you could block it in robots.txt, or return overly large, or wrong results based on the user agent. Maybe some fake vulnerabilities to see who is exploiting them later.

Just got another interesting phishing e-mail. This time around it is security company Trustwave that is being phished. I am not a customer, so I am not sure how well these e-mails reflect the real thing, but they confused me for a while. The give away that this is a fake is the from e-mail address as well as the link leading to a different site then advertised.

Click on the image for a full size example.

[Update:] An analysis of this phish by Trustwave's own Spiderlabs can be found here: http://blog.spiderlabs.com/2013/02/more-on-the-trustkeeper-phish.html

Usually, we find that e-mail used to trick users to malicious or spam sites is either not customized at all, or manually tailored for a particular recipient. A couple years ago at our RSA panel with Alan Paller and Ed Skoudis, I eluded to "mass customized" malware. Malware that automatically harvests social networking accounts or other open source information to find out how to best target you. For example, the malware may see that you "Like" Star Trek on Facebook and then will send you a link to a new movie trailer.

For a while now, I am seeing simple e-mails that appear to be doing something like that. The emails follow the same pattern. The "Real Name" displayed is the name of a person I know. The from e-mail address however has no relation to the person, and is usually some kind of free email 'yahoo'/'gmail' style address. The body of the e-mail itself is just a one liner with a link.

I did suspect Facebook as the source of the information. For most of the "senders" I had gotten these e-mails from in the past, there are other ways then Facebook that link me to them. But wasn't sure about it until now, when I received the e-mail below.

"Orlando Fermi" is the name for the Facebook page of my cat. I don't think there is anything else that links me to this particular name. The URL no longer works (for me at least... getting a 404 right now). But I would suggest you don't try it out.

In the past these e-mail led to various exploit kits, and on occassion spam. But it may also happen that an exploit kit will redirect you to spam if it doesn't have the right exploit for you. My cat's Facebook profile is public (sort of on purpose) and well, Mr. Fermi is a pretty mean cat so I wouldn't click on any link he sends me anyway which is one reason I didn't fall for this one.

And BTW: If you do happen to visit RSA: Ed, Alan and myself will have this years panel on Wednesday 9:20am in room 134. Lots of interesting stuff this year about targeted attacks, DDoS attacks and things like mobile malware stealing two factor auth tokens. (as usual, check the program guide for changes in room/time).

Last weekend I attended a presentation by Liam Randall (@hectaman) on the Bro networking language. It helped break though many of the assumptions I had about it and encouraged me to take a second look at using it the lab. His talk is available on you-tube and slides are available here: http://www.appliednsm.com/shmoocon-2013-bro-slides-and-video/

"I have snort, why do I need another IDS?"

That pretty much summed up my thoughts about BRO. Liam described most people's NSM stack as: tcpdump for capture, wireshark for analysis, argus for flow data, snort for alert data, and python to script interactions. When he siad that BRO could replace each of these tools I was a mix of incredulous and intrigued. The key point that helped me understand was the explanation that Bro is a domain-specific language for networking applications and that Bro-IDS (http://bro-ids.org/) is an application written with Bro.

So, what else does it do?

Basically Bro generates Events from traffic, and these Events drive Actions or generate Structure Output. If you've ever had a need to script something quickly to process the output of tcpdump, you'll see the appeal of Bro that dumps traffic out in an orderly fashion that's very UNIX command-line friendly.

Using something like Liam's fire-scripts (https://github.com/LiamRandall/bro-scripts/tree/master/fire-scripts) you can explore how protocols are being implemented on your network. While wireshark does an outstanding job of coloring protocols and identifying flows, Bro scripts do a better job of identifying the order of events and counts of events in a session (helpful for looking bots that are pretending to be Internet Explorer or SSL/TLS shenanigans.)

Bro can be scripted to extract every executable that flies by on the wire. While this can be done with a few key-clicks in wireshark or batched by using tcpflow, Bro allows you to make it part of the analysis process which you can then kick off other static analysis or additional alerts.

How do I get started?

What are you using Bro for?

While Googling around to verify the links for this entry, I see a lot of interesting SSL/TLS projects and APT1-related modules and scripts. For those of you who are using Bro in your processes, leave a comment below.