Posted
by
Soulskill
on Sunday April 12, 2009 @05:39PM
from the password-123456-letmein dept.

Peter N. M. Hansteen writes "In real life, zombies feed off both weak minds and the weak passwords they choose. When the distributed brute-force attempts stopped abruptly after a couple of months of futile pounding on ssh servers, most of us thought they had seen sense and given up. Now, it seems that they have not; they are back. 'This can only mean that there were enough successful attempts at guessing people's weak passwords in the last round that our unknown perpetrators found it worthwhile to start another round. For all I know they may have been at it all along, probing other parts of the Internet ...' The article has some analysis and links to fresh log data."

"IMHO if the passwords are strong enough there is nothing to worry about"

The problem is that most users are not capable of choosing a strong password. Even when you try to enforce policies about minimum password strength, users will manage to choose weak passwords; aside from the world's most common password (password1), there are plenty of people who use their own username as a password -- and requiring non-alphanumeric symbols won't stop them: jane123 will just becomes j@ne123. Minimum password leng

I disagree. Should a new bug arise in openssh, I sure feel a lot better knowing that while I do enforce key-only authentication, I also restrict access to specific IP addresses. It's pretty hard to crack a service that you can't reach on the network due to packet filtering.

I don't allow password-based logins either (SSH keys only), allow SSH only from specific IP addresses, and I use fail2ban [fail2ban.org] across all services that involve any kind of authentication (mail, ftp, http auth, etc). I've got it set to "two strikes and you're out"; every day I still get hundreds (some days thousands) of IPs banned in the logs. It's pretty sad.

From a quick look at fail2ban it looks like one of it's features is that the blocking only lasts until the next log rotation. Considering that attacks are temporary and automatic firewall rule generation could become a script kiddies playpen this is actually a good thing. If they work out you have this and spoof a few adresses as a denial of service attack the system will recover over time without needing someone to go through all the firewall rules.I'm still a bit nervous about allowing malicious third p

From a quick look at fail2ban it looks like one of it's features is that the blocking only lasts until the next log rotation.

It's configurable, you can select any period of time for the ban to remain in effect.

I'm still a bit nervous about allowing malicious third parties to effectively write firewall rules for me.

That I completely understand. It's not without its potential hazards, but I think the benefits outweigh them.

some of us don't really know where the next legitimate connection is going to come from

I've been thinking about something like a variant on port knocking, wherein a machine would be make several connections attempts to a non-existent service port from source ports whose numbers add up to some magic number. Filtering would then be disabled for the life of that connection. Maybe someone's already done it

It's a good idea in theory, but the botnets are smarter than that these days. My server networks get portscanned multiple times a day, and it's inevitably followed up with login attempts (even with non-standard ports) on any hosts that aren't taking aggressive defense measures.

It's still worth doing, it cuts the rate of attacks by at least one order of magnitude (if not two) and your log files will be a lot cleaner. Yes, the worms are getting more sophisticated, but they also have to leave additional footprints in the IDS logs (because of having to port scan).

(We run public key only authentication, with SSH on a nonstandard port. The former is a bit of a PITA, but the latter is a simple change.)

I only allow public key connections, and am only listening on port 2022 (I have no issues telling the world that).My auth.log is completely empty of password attempts. Am I missing something simple-stupid or is the bot net only going after port 22?

I would suspect it's going after port 22 only. If your smart enough to move the port from 22, your probably smart enough to use key pairs and then what is the point of trying to brute force you?
Focusing on default 22 is a good strategy because you'll find those who have completely defaulted settings, weak passwords, etc.

Something interesting about your usage of tcp/2022. I did the same thing recently on an occasion when I could not use tcp/22, since it seemed an obvious choice. Probable most automated attacks only concentrate on tcp/22 (the obvious target and if you can move it, you probably know enough to secure it properly), but I've been wondering if someone might start considering scanning on 2022 or any other "obvious" choice.

Right now it seems that nobody is scanning for them, but is anyone else setting ssh servers o

Likewise - I use fail2ban with iptables to drop any packets from someone who fails auth about 5 times in a few minutes. I've toyed with the idea of adding them to a global blacklist for all servers in all locations, but in reality this solution works just fine.

Likewise - I use fail2ban with iptables to drop any packets from someone who fails auth about 5 times in a few minutes. I've toyed with the idea of adding them to a global blacklist for all servers in all locations, but in reality this solution works just fine.

If you RTFA, they tell you that these attacks are coming from different machines, presumably so they don't trip such things as fail2ban et al.

Looking at the logs he supplied, this is a _very_ slow attack, the attempts are many seconds, or even minutes apart. You would have to have a very guessable username / password combination for it to work.

I would comment though that I'm not seeing anything like this attack in my logs. I personally use IPTables rules (using hashlimits) to limit 1 connection / IP per m

Nice in the short term but giving people an easy way to add rules to your firewall may create hassles later once miscreants know that is what you are doing. Some people have scripts that implement temporary blocking so it doesn't hurt much on the day that some script kiddie decides to have fun with them by forging attacks from different addresses.

passwords are still the most portable lightweigth authentication method out there, that's why. I had a system that got caught by this when a user with shell access set a weakish password. the user was sandboxed with only a limited whitelist of applications they could execute, so no harm done, but i did log all of the bot's attempts and the IRC channel it connected to along with passwords to other bots. it was very interesting to take the attack a part, at it's core it runs a simple dictionary attack and onc

I disagree. I both cases, password auth or key auth, barring any security problem with the protocol, the weakest link is the user. A passphraseless ssh key is tempting to the user for many reasons and unless you audit the passphrases selected by your users, key auth is no more secure than password auth. In fact password auth may actually be more secure if you enforce complexity on the system(s). You have no control over the passphrase on a u

Passwords are very dependent on the user. They are generally selected by the user and if any of your users recycled a username password pair from http://www.google.com/search?ie=UTF8&q=myspace1.txt.bz2 even though they are eight characters with a letters and numbers and mixed case they are essentially known.

Passwords depend on users not recycling username password pairs that have have been phished.

Don't go spouting nonsense like that! Storing SSH keys is only as secure as the physical security of the client. If you're travelling (esp international) and need remote access to servers, passwords are the only way to go.

Or they should be. Unfortunately, there are a lot of "how-to"s on the web that use a null key passphrase to make scripting easier. That openssh allows this is a problem, but lots of people have written scripts that depend on it.

But that doesn't have to matter, since anyone with any sense is storing their.ssh directory in an encrypted partition anyway. Along with their browser history, local copies of their email, and so forth. Why password-protect just one type of sensitive data, when you could protect everything at once?

What is the advantage of having an extra VPN in terms of SSH usage?(I know there are other advantages, but you seem to imply that's just to allow SSH access from everywhere but a list of known addresses)

The odds of them getting into a system like this must be quite low, but I guess they're after the low-hanging fruit. Running your services on a high port rather than the default reduces this, as does disabling password login and using 2-factor authentication. Quite easy to do, and very, very secure.

We did remapped SSH to a higher unused port and then took it a step further blocking access to that port on the hardware firewall from every IP address except for the office. If I need to connect to the server, I first have to connect to an OpenBSD box in the office.

We have 3 - 4 PCI scans a day (seems like every payment gateway we support for our clients scans the server daily. None of them even see SSH.

I had to re-install one of those low hanging fruit which was taken over via a dictionary attack about four years ago.For some reason the guy that was looking after what should have been a simple mail server and incredibly basic web server decided to change things so that everyone with a mailbox had shell access. Then he put a compiler on there. Then, sick of having to "su" all the time he changed the permissions of everything in/etc (system configuration files) and who knows where else so that any user o

I know the way to do this to allow only those specific users that need ssh access (one or two in the above case - and never root on an external facing box), use keys instead of passwords, and only accept connections from the IP addresses that legitimate users will be on. None of those things are difficult. The above example shows what can happen with a far too relaxed approach.

Use SSH keys in addition to passwords. Disable ssh root logins. Use the AllowUsers command in sshd_config to restrict what accounts can log in with ssh. Edit/etc/hosts.deny and add IP ranges [iana.org] for where you are unlikely to login from. Use iptables rules to block people [itwire.com] who are hammering your ssh server from the same address. Use tools like Fail2ban [fail2ban.org] and DenyHosts [sourceforge.net] to block other abusers and share abuser information with other victims.

While there might be good reasons to allow root access with a restricted key, that's hardly wise and there usually there is no need to do so.

As for the pointlessness of keys AND password, I think you're rather uncreative. There are a few uses of that scheme, the first one to come to my mind is a random password tightly controlled by the IT staff and periodically changed (and the user can do nothing about that) plus a key, under user control. The password allows enforcement of the security rules and insures

Until your account is compromised and the attacker can now ssh into your other accounts without having to enter a password. I'd rather have to enter the password when the key is used. Better safe than sorry.

it's compromised.And having to type a password too often is not added security, because it only adds minimal, easily circumvented security in the rare case where you're already fucked, but it annoys the hell out of users *all* the time, causing them to have unsafe practices.

No its not. Disabling root logins means the attacker has to brute force the username as well as the password, thus increasing the chances of the attack failing by a power of two. In actuality it is even more effective, as the attacker will usually try "root" and move on.

Same rationale exists on Windows. If you don't rename the "Administrator" account on setup, (or create a new admin account with a different name and delete the old one if you have a pre-installed box) you are asking for a world of hurt.

Look up defense-in-depth and the reasons it is used. Encrypted keys may be the strongest link in the chain, but if they are the only defense you have then you are SOL the instant a vulnerability is discovered.

Why would you sit back after implementing private key access without any further consideration to the other techniques for making ssh even safer? For home servers this is probably sufficient, but if it is your job to secure systems then this would be negligence - possibly criminally so, depending on th

mostly good advice. you might consider using ssh keys instead of passwords, depending on your environment. the only thing i'd outright disagree with is pre-denying IP ranges based on a guess of where you're likely to log in from. i've had to leave the country on business unexpectedly on very short notice; it'd suck to have been locked out when i landed.

If you don't have a password as well then if your account is compromised the attacker will be able to access your accounts on other servers without entering a password. I know generating a ssh key without a password is convenient but it also creates risk.

By the way, you can look at the
traffic statistics from DenyHosts [denyhosts.net], you can clearly see that ssh password-guessing traffic increased about 10 fold on Apr 6. (And since I configured DenyHosts to email me every time it blocks an IP, I've been very aware of this attack)

I've always wondered why someone doesn't do something with the DenyHosts IP list. It should be impossible to forge IP's for ssh, due to handshaking and key exchange. So doesn't DenyHosts have a pretty good map of somebody's botnet? Is any

Same here with denyhosts and emails. I noticed immediately that three different systems were being slowly plucked away with ssh attempts. mainly to 'admin', which I suppose is typical of home routers.

you bring up a very good point about the denyhosts data being a good map of the botnet. I think it would be up to the private sector to initiate something before law enforcement would bother though. It seems most of the blackhat stuff is reported/stopped by a few whitehats instead of by government entities.

Use a script like denyhosts, and I'm sure there are a ton of others out there that are just as good if not better. Unless your password is weak enough to be guessed in five attempts and the attacker isn't already in the denyhosts list, you shouldn't have to worry about too much. And, most importantly, just peruse your auth logs every now and then, it's not really that big of a chore.

For those already familiar with Peter Hansteen's website [home.nuug.no], I'll offer a Thumbs Up recommendation for his Book of PF [amazon.com].

There's already been several stories on Slashdot either submitted by or about him, and I don't recall any mention of his book. I'd say his efforts if not his humility deserve some kind of reward, and the reduced sale price of $19.77 is a bargain.

I've used a script to block servers that failed a certain number of attempts along with AllowUsers. That worked well for a couple of years, but was annoying in that you could see the attempts being made and knowing that if you made a config error you could be vulnerable. It seems to me that even after I got several hundred systems in my block list it wasn't making a difference since the pool of zombies was so large.

Now I just use key only access and AuthUsers and feel a lot more secure. I'm thinking I may add a white list of IP addresses as well. That would really lock things down pretty well.

This has been going on for years. Really. I've been seeing this crap in my logs since we started running an Internet-facing SSH host nearly ten years ago. It's always the same password based login attempts with the same dictionary/script used in the attacks. This is probably just some training exercise for Chinese hackers at some state-run school to see who can break into the running-dog Yankee Imperialist's computers the fastest.

The botnet owners have large lists of likely usernames/passwords and IP addresses to try. In the past they just assigned a bot to try out a complete list of usernames/passwords to try for a range of IP addresses. Now, they just give each bot a range of usernames/passwords for a complete list of IP addresses. It doesn't affect the processing time required, but makes it harder to ban hosts.

Just checked my auth logs and I'm seeing hundreds of various IPs, some of which are connecting up to 20 times. Definitely a new twist. I'll have to do some poking around to see what kind of machines are doing the probing. ( Is it compromised windows boxes?)

Be proactive on port 22 as well. At the advice of another comment I saw on/. a year or so ago I'm running a honeypot, with three static ports (one of them 22) and 4 roving ports. Establishing a TCP connection to any of them causes your IP to be instantly added to an iptables blacklist. It's worked pretty well; I've got about 2-3 unique addresses trying per day, and about 294 have been blocked since mid-December 2008. It takes care of both port scanners and bots deliberately connecting in order to try and g

Is there any difference with respect to a PKCS#11 token?I've been thinking of using one of these tokens as a "road warrior" SSH key, but then realised that since they need drivers to be useable, that wouldn't be practical to use on machines not owned by me.

Also, why not S/KEY [freebsd.org] instead of one of those yubikeys (or at least the random password)?

While it's true there are a variety of techniques that can increase security, I've found simply moving to a new high-numbered port eliminates all random login attempts. Yes, security through obscurity is all it's cracked up to be, but for now, I've eliminated the problem with a pretty quick fix.

There sure are a lot of people who didn't bother to read the article.The point of these attacks are that it's a coordinated botnet attack. Meaning if you block any single IP, or even a large subnet, you've cost the attacker nothing. Fail2ban, denyhosts, all of these won't even slow these attacks down.

Which means this won't work, since what I see in the logs (at least in the last go-round with this stuff) is one attempt per remote IP, yet a coordinated distributed dictionary attack. What we need is a ruleset that properly spots this attack pattern, then locks down port 22 entirely except for defined, fixed-IP administrative exceptions, just if there is any pattern of failed attempts over a longer period.

In a slightly-different recent case, I saw someone get ahold of an ftp login. The uses of that login were one-off per remote IP. The IPs were in Ukraine, Russia, China, Texas... with hours between successive attacks. From what was attempted on the logins, it was obviously a generic attack, not well-directed at the particular server. Quite likely an infected client system divulged the login to the botnet that infected it, and the botnet then focused the distributed attack. As with ftp, so with ssh. They don't want you to even notice it in the logs. Each IP will be used for one transaction attempt with you, spaced hours apart.

We need tools to spot these slow-moving distributed attacks. The single bad login attempt needs to be correlated with others from completely different IPs, over a span of days, not just minutes.

I get about 5 of these a day, on a relatively small site. I wrote a small shell script out of sheer boredom that parses hosts.deny and gives me country and hostname info. Here's the output from the past week or so. It seems to confirm that most of these are from public isps overseas.

I was having similar brute force attacks.I've made some alterations to protect my server from brute force SSH attempts.

1) Moved SSH to another random port2) Bound the SSHD to an IP address that is not used for Web/Mail/FTP, etc.. So the IP should generally see less traffic3) Disable Password Authentication, Users who are given SSH access must use a password protected key file4) Disabled Root SSH Login5) Setup the system that 3 failed logins add the entire IP Subnet(X.X.X.0-X.X.X.255) for 15 minutes, 5 failed attempts 1 week, anything else is a never ending ban. (iptables and hosts.deny, just in case)

Obviously ssh should be off by default. Many people use a different port for ssh, as long as you understand that that is security trough obscurity, that is fine. It is not a real option if your system is a multi user system. Imagine if each webserver on the internet ran on a random port. Not very nice.

So then you must ask yourself if everybody in the whole world is allowed to run ssh to you or just a few.You can already use your hosts.allow to block many and allow other IP adresses.Then you can ask yourself

Some years ago, I worked for a partner of a then-enormous national ISP that shall remain unnamed, but its initials are AOL. At one point, I had access to a large list of usernames and passwords of their users. Out of curiosity, I performed a statistical analysis of the data and discovered, to my not very large surprise, that about 20% -- I forget the exact figure -- had passwords that were either "password", "secret", or their username. In other words, if you know one of their usernames, you have a roughly

Um. You realize, of course, that remote desktop is a lot less secure than ssh, right?

It doesn't matter if people are trying to pick the lock on your door. What matters is whether they can pick the lock. Use RSA-based authentication, and no amount of brute force is going to improve the odds of their breaking in to the point where it's worth bothering.

Remote desktop, on the other hand, is completely brute-forceable. If you're not seeing brute force attacks, it's because nobody's bothering, not because

I had something like this (but far less automated) going back in my one-man-IT department days. I also had good responses from admins who generally felt bad and wanted things fixed up on their end ASAP. My employer told me it was a waste of my time and to stop because I wasn't "the Internet police." He looked stupid after I was gone and my incompetent replacement managed to get them hacked. He actually allowed someone to set a password to "password" because they complained that my old password requireme