Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

An anonymous reader writes "It is no secret that SSH binaries can be backdoored. It is nonetheless interesting to see analysis of real cases where a trojanized version of the daemon are found in the wild. In this case, the binary not only lets the attacker log onto the server if he has a hardcoded password, the attacker is also granted access if he/she has the right SSH key. The backdoor also logs all username and passwords to exfiltrate them to a server hosted in Iceland."

Unless you're auditing said sources, you're no better off than installing binaries.

You know that Gentoo checks the SHA256 SHA512 and WHIRLPOOL digests of the downloaded sources before compiling right? You also know that the digests stored in Gentoo's repository are signed using a PGP key of the package maintainer, right?

So can you please explain me again why I need to audit such sources? Sure sources can be compromised at upstream's servers and Gentoo maintainers can also make mistakes but this is not what TFA is about.

Everyone replying here seems to be obsessed in comparing binary vs source and totally miss the point. The point is thatGentoo users do not have the habit of installing binary packages at all, for that reason this trojanized sshd BINARY does not concern us.Wooooooosh!

The trojaned ssh isn't the one installed from the repo, it's installed later by the bad guy, so it doesn't matter how you installed . Again, the trojaned ssh isn't the one you installed. The ONLY difference between source vs. binary packages in this case is that people who installed binaries could be alerted that the hash of the existing file doesn't match the correct binary. So binary installs are SAFER as far as this trojan.

How does the bad guy get the trojan on your system, if not from the repo, yo

How does the bad guy get the trojan on your system, if not from the repo, you ask? He gets access when someone else who is infected logs into your machine - your sysadmin, your hosting company, a vendor, etc.

Dude, this does not make any sense at all!If person X logs into my machine (which is not infected) and that X is infected with the trojanized daemon. Then the attacker will not be able to find out the X's account password in my machine because the connection was made to my uninfected daemon and not to X's infected daemon. Get it?

I was asked to clean some exploited servers in the wild that had compromised SSH binaries in them. This was about 10 years or so ago, so this is really a not current news story.

Thankfully most script kiddies are exactly that, and their script left the source on the machine for me to review. Why? I guess to compile on the machine to make sure it used the local libraries. They generally didn't have passwords hard coded though, they used SSH keys. The passwords they stole were usually stored locally in a file, and sent to somewhere in eastern Europe or Asia. I have no idea if that was by the script kiddie design or the person who rolled it up. It's a pretty slick trick though. You get the script kiddies to gather intel from compromised machines, and feed it back to you. Now you have access to every machine the script kiddies compromised.

The funny part about most of the cleanups was, the version of SSH they put on was newer than the remotely exploitable version they got in with.

Apparently it's quite common for attackers to actually patch the security holes they used to get in, once they've installed their backdoors. Don't want someone else getting in the same way, after all...

I've seen that too. Mostly with fairly open hosting machines. Like, letting anyone sign up for free or low cost hosting. I saw a lot of them that were built for one platform (usually Redhat), where the libraries didn't quite line up with the running system.

At one place, I was collecting them. The script kiddie would put their SSH server on a high port. It worked great, except they couldn't get root with it, and they couldn't get to the daemon through the firewalls. It was rather entertaining.

Yeah, I hate most Linux exploit articles. They always seem to be about what happens after the machine was rooted. Binaries changed out, configurations changed, permissions set, yadi yadi yada... They never seem to be about how they got rooted in the first place.

That's why ssh is trojaned - it's how they got in. Once they get into one box some other way, the trojan gets them into every box a user connects to via ssh. So it spreads like a virus. At some point, an admin with access to a lot of machines, like a hosting company admin, uses ssh to rsync / scp to their main machine. Then the bad guys get access to every machine hosted there.

Yeah, I hate most Linux exploit articles. They always seem to be about what happens after the machine was rooted. Binaries changed out, configurations changed, permissions set, yadi yadi yada... They never seem to be about how they got rooted in the first place.

Unlike the MS world where typically some software maker (and often MS itself) has screwed up, Linux tends to be rooted because some administrator or user got careless. That is not very interesting and the details do not matter.

You can never clean up a system. MD5s help, but you know what one of the first things I'd do when rooting a system is? After making sure my rootkit didn't show up in directory listings, I'd patch md5, shasum, perl, and ruby to return the exact MD5 I wanted for every file I defined a magic string for.

You gonna catch me on some systems? Sure. You gonna catch me on an extremely common distro like Debian without installing out-of-tree software? Probably not.

Rule #1 of investigating a compromised system is you don't use the tools on the compromised system.

Security is a process. Running a scanner on a compromised system has no guarantee of finding out that the system is compromised. But 98% of the time it will work just fine - these systems are usually compromised by script kiddie tools that don't spend a great deal of effort to find the rootkit scanners.

I suppose it would be better if somebody came up with a well-developed package for cross-scanning systems o

I suppose it would be better if somebody came up with a well-developed package for cross-scanning systems over shared storage (does it already exist?) but that's also only going to reduce your 98% to 98.5%. I suppose all you need to really do is to validate the integrity of the host system's scanner (or rpm or dpkg, etc.) but then again the remote system access method could also be compromised (today it's TLA stuff to compromise e.g. nfsd such that it will only return the wrong (but valid) md5sum and sha1sum and [randomhashsum] for only the scanners you might check for) but it's not likely.

Taking down every system on a frequent schedule for an offline scan would be a good idea but then again the BIOS could be compromised and you start to run into the collision of business needs and absolute security.

I use NFS root internet servers connected to a intranet NFS server. On the last I use rsync to do backup and detection of any files change. The only permanent memory on the internet servers machines are the BIOS, ok, but I have yet to see an exploit that hack the BIOS in a way to compromise a TFTP + NFS boot without any detectable trace from the NFS server point of view. If your concern really go into this level of paranoia, then you can simply hack the write protection signal of your BIOS memory chip (see

To clarify - you should be continually monitoring all your machines for unusual traffic. By unusual, i mean for example any traffic from a port that shouldn't be open. This is a good example of why it is safer to be fucking paranoid with your firewall rules on your edge, and do egress filtering as well as ingress - and be very specific with regards to what you open up and the source/destination as well as what ports you allow. If some daemon is running on a port that it shouldn't be running on, or connec

HTTP/HTTPS sessions will have a limited lifetime (the exception being some HORRIBLE streaming technologies you shouldn't be using anyways) and SSH should only have incoming connections from KNOWN sources with very few exceptions. If you can't block a "hacker" (read: script kiddy) using GoToMyPC, you need to find another profession.

The nfs could be hacked to give the original binaries to your outside server.

Absolutely impossible because the hacked file will have his checksum changed from the NFS server point of view. As long as the NFS server is not compromised, this is very safe. I personally use NFS root internet servers since more than 12 years now and I can say that's a very efficient way to detect and recover from external attacks. In addition, it very reliable, comfortable to manage, and easy to backup.

To archive a permanent modification of one of the internet server box, a file must be modified on the NFS root mounted filesystem. This modification is very easy to detect from the NFS server point of view. I personally use a simple script that parse the rsync log while it backup the internet servers

I have experimented a very successful way to solve the problem: Internet connected servers using NFS root from an intranet NFS server. All the machines are using a standard distribution. The intranet NFS server handle the backup of all the internet servers using rsync. It's really easy and efficient as it's just moving data internally of the RAID array. It's easy to detect unexpected files change, to compare internet machines binaries with the intranet machine binaries, and to

I will ALWAYS find your rootkit. This is because you're trapped in a VM, and I can always checksum the files from another uncompromised system (LiveCD / USB).

This is, of course, assuming that you yourself are not running on another compromised virtual machine.

(There was one hack I was involved in where an investigator tried to get clever and started calculating MD5 checksums with a universal Turing machine operated using pencil and paper. Fortunately, I'd already trojaned base logic itself and managed to subvert alphanumeric characters to return the 'correct' values. Hacking the logical representations of arabic numerals? Now that's pretty advanced stuff. But then, there's always the worry that my own consciousness is running on something other than what I think is my own brain...)

I myself have set up modified versions of the sshd dameon/ssh client on various rooted machines in the past (mine was simply allowing root login without logging anything with a hardcoded password + writing logins/hosts/passwords, after encryption, deep inside a fake shared library on the system - this took less than an hour to implement in the openssh source).

In all cases, despite the extreme simplicity of this mechanism, it was very, very easy to then gain access to a lot of other boxes on the network, and eventually to all the network itself. Just wait for the clueless sysadmin to log in (dumping his pass), then wait for him to use ssh to log in to another box in the network (dumping his pass as well). Then install the modified version of the daemon/client on to this new box. Rince, repeat.

It is in a way terrifying to see how most sysadmins are clueless. So, IMHO, a few measures that should be compulsory on any network:

- Do not allow write access to any essential binaries (like sshd, ls, and so on). If you have to, make sure you have a stealthy daemon checking the hashes of all critical binaries at regular intervals to make sure they have not been tampered with.- Never, ever, log in to a server in your network from another server in the same network. Use port redirection, then log in from your initial box.- Always use a dedicated server for system logs (syslog handles that very easily - sending logs over the network). That way, the logs of the initial system compromise will not be deletable. Do not ever log in remotely to this dedicated machine.- After the initial system install, make a dump of the syscalls table of your kernel. Check it regularly to make sure it has not been tampered with (kernelspace rootkits usually hijack this table).-...there are also other ways to implement an effective kernelspace rootkit, like VFS hijacking, so make sure you disable modules loading in the kernel before compiling (and check the hash of the kernel image itself regularly as well).

Those are the main sane measures that come to mind (this is all based on real life experience intruding into large companies/laboratories/university systems, without ever having been detected). Posting anonymously for obvious reasons.

- Do not allow write access to any essential binaries (like sshd, ls, and so on). If you have to, make sure you have a stealthy daemon checking the hashes of all critical binaries at regular intervals to make sure they have not been tampered with.

I'm sure there are plenty of other such systems, but Tripwire ( http://www.tripwire.org/ [tripwire.org] ) is one of the more popular tools to keep a check on your system and warn you immediately if it detects tampering attempts.

- After the initial system install, make a dump of the syscalls table of your kernel. Check it regularly to make sure it has not been tampered with (kernelspace rootkits usually hijack this table).

Personally I think a password is a very good idea if combined with a key. If a laptop gets stolen the thief can log in directly with the stolen key saved on the laptop. If they have to work out a password as well they only get what is on the laptop.

Security is usually one of three things: Something you know (password), something you have (key, token) or something you are (biometry). Alone, none of them is really a big security obstacle, at the very least you should combine two of them. If you're paranoid, require all three.

Using two of the same class is patently useless, as practice has shown. If one password can be sniffed, so can two and more. And people tend to store their keys and dongles and other work related gadgets in similar places (if you get their keys, you usually also get their ID dongles), so if one is gone, usually they are all gone. And I guess I needn't explain why fingerprinting and retina scans aren't really a sensible combo either (aside from the cost).

And preferably request them through two separate and independent channels. It's amazing how rarely this rather essential minimum of security is enforced.

To a non-sysadmin what they see during the ssh login process is a request for a password even when it is really the key phrase for the key. They'll take that "don't use a password" advice above as logging in with keys that have no passphrase.What I wrote above should be obvious but I seem to have attracted five people that have decided I'm a pedant so want to get pedantic about me using "password" to make the my post easier to read.

There is no way to enforce this, except to scan people's home directories for unsecured keys. It's also awkward if they are using "ssh-agent" or other passprase unlocking tools on a shared host, such as a server where I have administrative access or when someone has set up software to use passphrase-free keys as part of a regularly scheduled task, such as a backup system for databases. I've used these approaches to borrow other user's SSH keys when needed, not always with their advance knowledge. (I always

As others have suggested, I will confirm. Do not use username/password as the authentication method for your ssh server. If you want 2-factor encrypt the private keys or go hardcore with PAM and have them enter one-time codes from a pad or separate comm channel (sms, etc) after the key is verified.

Just put a password/passphrase on the key as designed and it all works as intended for general use.Single factor authentication with a stealable key can be even worse than single factor authentication with a password in some cases.

No - the above poster wrote "don't use passwords". Taken at face value that sounds like recommending not to encrypt your private key with a password - because that would mean "using a password" - thus bad advice.

The parent is talking about logging in with password authentication, rather than public key authentication.

No - that is called reading between the lines, adding something that is not there and making an assumption that somebody that doesn't know better will not

No - the above poster wrote "don't use passwords". Taken at face value that sounds like recommending not to encrypt your private key with a password - because that would mean "using a password" - thus bad advice.

No, it doesn't. Everyone with half a brain regarding SSH should know what 'not using passwords' with SSH means - and it doesn't mean *not* using a pass 'phrase' for your SSH key. He never said that. I'm afraid your splitting hairs here to have something to say.

No - that is called reading between the lines, adding something that is not there and making an assumption that somebody that doesn't know better will not make - thus it doesn't change that the post above is bad advice.

No. You have totally misunderstood the post, and please don't tie yourself in knots like that. It was very clear what he wrote and you were the one who added in 'I think a password is a very good idea if combined with a key'. It's also not called a 'p

Please read his second post. He's describing single factor authentication using a key alone instead of single factor authentication using a password alone. That's why I called it bad advice and gave an example to demonstrate why the first can be just as bad an idea as the second.

The private keys can be considered part of ssh, so at face value it does come out as "don't use passwords for your private keys" - thus very bad advice. I don't know why you are trying to complicate this.

This makes no sense. Once somebody has root and can write to those binaries you are already fucked and they can change access to anything they like, and then write over it to their hearts content.Putting them on read only media sounds like a cool idea but is difficult to implement for anything other than the short term and can be circumvented if somebody roots your box anyway. There's the script kiddie toy of changing file attributes (and thus announcing

Now that's something more sensible than read-only media that's only going to have a short lifetime before you have to change things on it. Of course locking root out from changing the mount point is going to be a challenge in both cases.

So? You have root, write a script to change those flags, have it run on startup in single user mode, and reboot in the middle of the night to single user mode so nobody notices. Of course your script could then change the box back to the right runlevel when it's finished and let you back in.Once somebody has root (or full physical access and a screwdriver) they own the box, and about all you can do is make sure they can't easily get into other ones.

Except you can't. The securelevel [wikipedia.org] feature in FreeBSD (maybe others) disables the ability to write to files with the immutable bit set. The only way to write to these files is to reboot the box into single user mode, before the kernel securelevel is raised. Single user mode = no network running.

Setting securelevel is a one way thing, once it has been raised, you can not lower the securelevel.

If they are not distro(i.e. ubuntu, redhat,centos, etc.) servers, or openssh servers, this is a non-story. I can create a random domain, compile a version of openssh to sends passwords to me and host it on my server as an official ssh binary.

If I install stock wordpress with access to the file, would that then count a compromised?

I never use passwords with ssh. If ssh asks me for a password, I know something is wrong. I made this policy decision because of the endless attempts to log into my machine from all over the world. It was either something like a yubikey, or only using public key authentication. I went for the public key authentication.

Denyhosts can be set to block all, not just ssh, if wanted. I decided not to because some might have ISP NAT and that would block many people from my webserver... Maybe unlikely though, and leaving them an option to find an exploit in apache or wordpress might not be that good idea...

Sorry it took a while to get back to you.

My point is I would rather have all traffic from an offending IP address be DROPped at the kernel by an iptables rule, than have the machine continue to receive packets only to have connections rejected by TCP Wrappers and the hosts.deny file.

Of course with TCP Wrappers you do have more fine-grained control. You can, as in your example, decide to block one service and not another. In my own use-case there is no scenario where I would want to deny SSH access t

Yup. There are actually several reasonably priced hosting services in Iceland. I was shopping for international homes for some of my data, until I thought about it and none of my data is worth spending money to hide.:) Iceland was pretty high on the list though.

I don't think so. I suspect an extraterrestrial ship parked in a geothermal vent somewhere. They're out of sight, they get free energy, and they're close enough to a server to hack into it with their advanced computers. They've given up on those abductions and anal probes, now they're just hacking into the network to discover whatever might be on our minds.

Or, maybe it's occurred to them that we don't keep our brains next to our rectum?

You'd like to think so, but when things like this [debian.org] happen, you can see that having thousands of unqualified people looking at code and sometimes modifying it is not really any benefit to security at all.

The bad guys only had to compromise one machine, then the trojan spreads. Say for example my co-worker Jeff has him home machine infected. He uses ssh to connect from home to his office. The bad guys now haveaccess to infect his office machine. Jeff is a sysadmin at the office, so from his office desktop he logs into various servers. That spreads the infection to the servers. I then use scp (ssh file copy) to pull some files on to a server from my work desktop. Now my desktop is infected. Later, I s