Posted
by
kdawson
on Saturday March 06, 2010 @08:51PM
from the some-definitions-of-managed dept.

An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"

This is really cool until you find yourself trying to log in from the same access point where somebody with a virus was attached earlier in the day. Better to just use crypto (key-based authentication only) and rate-limiting.

Not only use something like DenyHost or Fail2Ban, but firewall off all IP classes from Asia, South America and any other region that you know you'll never SSH from. I can easily get the/8s and such that are in use by each region (just look up Apnic, RIPE, etc...).

I find fail2ban to be much more effective since I can use it for more than just SSH (on my system: ftp, imap, pop3, ssh, smtp). Some of the newer botnets will attempt to crack the password using another service and then try the resulting password on ssh so it's important to have more complete coverage.

You can also use the hashlimit module for iptables. I find it works pretty well for allowing people in who need to access and block things like the ssh worm that was causing ssh to run out of resources.

You might check out the end of this presentation that I made 4 years ago.

I've been using hashlimit successfully for years in a web hosting environment.

I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!

I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator.

That's probably the most relevant thing I've seen. I'm amazed at how many places don't want or have people with a clue working on their equipment. TFA said they have *NO* sys/net admin, and the guy who attends to those task is a "n00b".

"I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!"

Quite to the point. There's a rising trend on questions like "I'm having problems about X but I prefer being cheap on coping with X, what should I do?". And then there's the rising trend on answering something else than "if you depend on X you'll need to invest on X".

I am using fail2ban, but I do not think it's a particularly great tool when you see how many different IP addresses the attacks come from, so you're somewhere stuck in the middle trying to optimise how to make fail2ban more effective, while not being a problem for your own users:

- you configure blocking from 1 login attempt and a long block time cuts down on most of the outside attacks, but if some legitimate user mistypes his password, he'll be locked out for that same long block time...- you configure sev

No but they can steal if from your users' computers. It's one thing to have staff use keys when they are on your secure network but having users who are out on the web using keys when you can't control the security on their machines is only as secure as your dumbest client. Of course if they can hack and steal keys, they can probably steal passwords as well. For remote access one of the securest ways is using a security token. Every time user logs in they have to enter a different number.

"Because stealing RSA/DSA SSH keys is so much more common than dictionary/brute attacks..."

You seem to forget that if you are seeing those attack levels is because of user computers already rooted, so anything on the client's side it's already at the attacker disposal. What do you prefer? Seeing thousands of failed login attempts in your logs or not seeing anything because the attack succeded by taking control of the ssh keys on the client's machine?

The best authentication relies on two factors, what you have and what you know.

A rooted client gives the attacker access to at least one part of that.

A separate physical RSA token is probably the only thing to prevent that, since they work without being attached to the authenticating system and even full scale keylogging monitored by a squad of hackers in rotating shifts will not really suffice.

bfd (and apf if you like it) are probably the best software solutions your gonna find, but if you're facing 1 million+ auth failures, I would seriously consider a hardware firewall and VPN of some sort.

Let me see if I have this right. If iptables can keep count of incoming connections within the kernel and drop incoming connections as they happen without any intervention from a userspace program, that's _not_ automatic.

But running a gigantic shell script to scrape text messages out of your system logs and then call randomly chosen commands which may or may not have any effect on the observed problem some time after it occurs, then that's "doing that stuff automatically"?

I use a variation on tarpitting that has worked very well for me.It cut the attempts down from 60,000/day to 20 or 30 per day.

I just add a small delay between the initial connection attemptand when I send the username/password prompt. The delay (inseconds) is the number of attempts in the last 30 minutes,squared. This makes all but the most determined attackergive up and go away very quickly.

I have been using this with both FTP and SSH for the lastyear, and it works great.

Yes, I think that this is your best solution. Instead of using iptables directly, you should check your distribution software relative option.Generally is is easier (but anyway, it will ultimately result in the same iptables command)

That, and create only a single user who can log in, that takes you to the real log in prompt. That way an attacker has to guess the one user+password, and have a legitimate userid+password to gain access.

It's not foolproof, but it foils the vast majority of script attackers.

if you can manage the set of users of your server, you can use OpenVPN and then SSH. OpenVPN has a "feature" that if each packet of the VPN is not digitally signed by a previously arranged (and distribuyed) key, then the packet is "ignored". After the VPN session is established/authenticated, your users can log in using ssh. There are even some virtual appliances and special distros (untangle.com) that have a "openvpn appliance" built in for this purpose. The how-to for openvpn is also easy to follow.

Here's my excuse to ride my usual hobbyhorse about passwords being obsolete. SSH supports certificate-based authentication, which is not only more secure, it's less of a hassle for the user. Don't know if it would be practical for your application (you might tell us more about that) but it's worth a look.

I agree with both parent posts. PKI Certs are certainly the way to go, but it's really hard to do this right.

This is a case where some consulting to (a) set up the PKI stuff; (b) train our (unfortunately anonymous) questioner on how to disseminate the certs; and (c) apply the appropriate tarpit/other firewall settings, would probably be money-well-spent.

I agree that PKI would be more secure, but it is a LOT more hassle for most users. The simple fact is, a tarpit is *extremely* effective. Even a relatively weak password is going to be nearly impossible to break if the attackers are limited to only a few tries per hour or day or whatever.

I agree completely. We use fwknop as a part of a required two-factor authorization scheme for a US Government customer. It is a minor inconvenience to use it to open the ssh port before connecting, but it virtually eliminates attack attempts. How can you hack a closed port?

Welcome to the internet -- this happens to every site with a public IP.

First off, do not change your SSH port. It won't do a whole lot for you, and it will be more hassle than it works.

There are a whole lot of programs available to deal with SSH brute forcing. sshguard is one of them, and it's not too bad (you can apt-get install it). It's a bit of a hack -- it just watches your logs and takes appropriate action -- but it does work.

The other thing you can do immediately is simply turn off password authentication in ssh. Anyone getting in will need a key. This is what sourceforge and github both do. This isn't always practical for every site, but it will damn sure keep your passwords from being brute-forced.

Use public key authentication. It's super easy and much more secure, unless your random passwords happen to be over 300 characters long (ssh-keygen makes 2048-bit keys by default; assume 6 bits per character in a random password if each character is in the base64 set). Also, sshguard is your friend. After a small number of invalid logins - 4 by default - that IP gets firewalled for a short period of time.

Fail2ban [fail2ban.org] or denyhosts [sourceforge.net] activly target ssh. fail2ban includes rules for other services, but denyhosts includes a mechanism for sharing lists of your denied hosts w/ other admins, as well as using their ban lists to protect your ssh logins.

If your not willing to invest money into intrusion detection, vulnerability scanning, or moving to a more responsive provider, accept that your server is going to constantly be scanned by the droves of script kiddies out there throwing everything but the kitchen sink at any server that responds to a ping and keep monitoring those logs, which hopefully are stored on a separate server in case you are ever actually compromised. In the mean time, try installing Fail2Ban on your server. It will block an IP addre

add the ip address and/or hostname of all the hosts you use to access your servers into/etc/hosts.allow. If denyhosts picks up 3 failed logins from a single ip address, that address is added to/etc/hosts.deny, if this happens to be your machine (and you're having a bad day entering your password), you could get locked out of your system.

If you really have secure passwords, the random guessers aren't going to get them. Log it and move on.

I get thousands of Chinese hackers attempting to break into the battery monitor in my tool shed, big deal. I don't know why my battery voltage and solar current is so important to them, but they can knock themselves out all day.

If the logs bother you then install fail2ban and configure it to lock people out after a few bad guesses. (Then be ready to unlock yourself from an alternate IP when you inevitably lock yourself out.)

I'm surprised I had to scroll down half the page to find this. Seriously, what is everyone so worried about? Just set a secure password and don't even worry about it. Assuming you have a secure password (8 characters, upper and lower case letters and numbers), even a million attempts per second will take over 3 years on average [google.com] (and ssh won't let you try that often anyway). Bump that up to 12 characters and you're looking at millions of years [google.com].

I use one or more of these on my public facing servers.1. Move the default ssh port to a higher order port (5000+)2. Use Denyhosts http://denyhosts.sourceforge.net/ [sourceforge.net] to block repeated attempts3. use key exchange instead of username/password4. use network based IPS.Just moving the ssh port reduced the ssh brute force attack for me. Either stop being a noob or hire a sys admin.

They do not do portscans, at least not of random ports. They do not attempt to find ssh elsewhere.

I swear, I keep having this idiotic discussion. Have it about spam too. 'But why would you block email servers that don't appear to understand the SMTP protocol? Spammers can just write better servers.'

I used to feel the same way. But really, moving the port lets you focus on the real threats.

Look at it this way: if you are being targeted with a sophisticated hack, any dangerous attempts to log in to port 22 will get lost in the massive amount of bot traffic. But if your SSH server is on port 12344, then the reports in your logs will be more meaningful. You'll know that someone was conducting real reconnaissance and you can start worrying. There's no reason that you can't change the port as well as r

You're missing the point. The goal of moving it is to stop the botnets. Now if you see failed attempts to login via SSH on port 12345 you know that it's a much more sophisticated directed attack on your host. Moving the port is more about filtering out the noise.

Changing the port might be troublesome if you're trying to SSH from public locations like airports and such. They usually leave port 22, 443 and 8080 open for their own services and block all others, so you will have to either set up VPN to get into your servers or wait until you're elsewhere. Additionally, an evenly-slightly determined hacker can still find where your sshd process is listening on by running a basic nmap scan.

Key exchange and disabling root login is probably the best way to go. Fail2ban wor

We started using BlockHosts to feed iptables rules, and our failure logs went from 30-50k per day to 100. Basically, with more than 'x' failed logins within 'y' time frame, the source IP is blocked for 'z' time period. Since it uses iptables, you could block it from just the ssh port, or the entire system (we do the latter).All three variables are configurable, and we also have whitelisted a few select standby IPs for contingency use. (As another poster said, you **will** lock yourself out eventually.)

I have a similar situation and cannot limit to very specific IP ranges. I have done the following with good success. I pulled some examples from my configuration that can be tweaked for yours if you like.

Although the solutions like fail2ban and denyhosts are undeniably useful, you will cut down well over 90% of the drive-by ssh attempts by simply changing the port.

None of the scripts, that I'm aware of, actually portscan your machine looking for the SSH login headers, they all hit port 22 and try to log in. It takes too long to scan a machine so if nothing is listening on port 22, they will move on.

You will still get the odd attempt from someone who has portscanned you - these will be relatively rare. In th

I actually go one step further: after IPv6-enabling my site, I only allow v6 ssh inbound. Since Teredo makes it possible to get v6 from nearly anywhere, it doesn't cause any inconvenience, and ssh attacks have basically vanished. 'course, it won't last forever, but it works great right now.

RSA keys are notoriously time intensive to brute-force. As far as I know, no one is crazy enough to automate an attack. If you disable password authentication altogether, you eliminate your risk of an automated dictionary attack. It might confuse the hell out of some users, which is a con.

fail2ban can sometimes be ok, but be aware that it creates a denial-of-service vulnerability that is exploitable by attackers who can can spoof source addresses.

Yeah my server went off line a week ago and while debugging the problem I noticed the kernel complaining at startup that a file included from my pf configuration was corrupt. Its a script I wrote which updates that file, not fail2ban. The file was my blocked hosts table.

I hate to sound unsympathetic here, but the Internet is a hostile place. If you have port 22 open to the world, you should expect to get pummelled with password cracking attempts more or less constantly.

Either learn to live with it, or at least take some steps to slow them down. I find that throttling back the number of connections any one IP address can make in a short time pretty much slows it down to a reasonable level. Alternatively, you can also put some "trap" logins on your system. Usernames lik

If you are randomly generating your passwords and they are of a decent length then you don't really need to do anything. If your passwords contain lower-case letters only (not recommended), but are eight characters long then your million authentication attempts would represent only a 0.0005% chance of success. If you passwords contain numbers and upper-case characters too, then the likelihood is 1000 times less.

I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla

Right off the bat, you're far too focused on the security of least concern, ssh. Unless you set an idiotic password, it's not going to be guessed. It sounds like you have that covered.

But.. the part you've missed is your FAR more vulnerable applications. You say your ISP is "actively managing" your security, but how certain are you they're doing a decent job of it? Do they have experts

Ok let's assume a few things:1) your passwords are 8 chars long and use upper and lowercase alpha and numbers only2) they are essentially random passwords that are NOT in the dictionary.

This would mean that there are 2^62 possible passwords in your password space. That's 4,611,686,018,427,387,904 possible passwords. The chances of guessing one password in that space within a million random tries is 1,000,000/4,611,686,018,427,387,904. Tha

That should keep the CPU usage down, and your system secure from that type of attack. You can’t do much more than that.

You could run a script to find the owner of all those IPs... maybe do a bit of statistics to see what you can find out... But in the end it will be someone that you can not sue or contact in any way anyway.

I can't believe the person who set the server up left this port open to the world. If you want the server secure then you will close all the ports that are not actually needed - in your case this likely means you keep 80 and 443 open.

For remote management make get a few IPs for the desktops that are meant to connect to it and make sure the ISP opens SSH port only to the IPs.

You can tunnel X over SSH and do file management, whatever you need so it should be enough.

Don't bother changing it to some random port, security through obscurity is total bullshit in this age of port scanners.

Then the answer to the question is the link to someone who can provide that.

I doubt the responses would have been quite so hostile if the topic had been car repair - poorly maintained vehicles are a menace on the roads!

The answer does not necessarily have to be "here is what you do" - it can be "find someone who knows what they are doing". Trolling him is not likely to create any positive effect, except among the already insular server admin clique. Save the "oh my god, so this noob totally connected a 54-5

If he asked "How can I stop my house being burgled?" and you said, "Without a full alarm, CCTV and patrol system, don't even bother. Leave it to the professionals."

Some others might see some practical value in suggesting, "Maybe lock your door at night."

My opinion in this situation is first to take what advice you can and do something yourself as soon as possible (since any extra security is better than nothing). Next, as you suggested and when he has the resources to employ someone to do it properly, seek professional help. Simply doing nothing until he can do it properly sounds a bit dangerous.

It's not a sign of being "psychologically mature" to hold the notion that human life is a fungible commodity.

A poor person in the slums of Bangalore is economically worth more dead (for his organs) than alive. You state that society places an economic value on everything, including people. Yet for some reason, society has outlawed murder, even if the result is a net monetary profit for society.

Your complete failure to comprehend that a human life is worth more than its economic value to society is why I cal

Depends how realistic we are being with our range of possible scenarios.

Worst not-ridiculously-unlikely case: ill configured service, service minus important patches, or poorly chosen authentication/privilege setup, or something similar, allows an automated hack in and therefore the server becomes yet another bot in the net or possibly even a C&C box for the botnet.

A zombied server is usually more trouble for the rest of the 'net then a zombied home machine because they generally have access to bet

Yeah this is just normal for me. I have a script which hooks into syslog and blocks offending hosts in pf but frankly I don't think it is worth the risk of doing that. Make sure sshd has a secure configuration and don't worry about the brute forcing.

1 million per year over 50 sites == 20,000 per year per site, or 54 per day per site. Change the passwords every few months (and use different ones for each site).

Further - 1,200 different remote hosts means what, 17 attempts per remote host per site per year. Probably randomly p0wned PCs that hit addresses at random, make a few attempts using default or ocmmon passwords, then move on.

But a million cars haven't driven past. Only ~1200 --- in the past year. This is no big deal. Hell, I start thinking something is wrong when I DON'T see many failed SSH attempts in the daily log reports.