Sometimes you may log into a machine, and find that the SSH key has changed. This can happen in a man-in-the-middle attack, but more likely the host has been rebuilt and new keys generated. (This is a good reason to save and restore the SSH keys during a rebuild.)

So how do you check the fingerprints?

You can do it over the network by pulling down the public keys from the remote server:

(Replace remote_host with the appropriate name or IP address.) Then you generate fingerprints from this file:

ssh-keygen -l -f /tmp/ssh_host_rsa_dsa_key.pub

However, if you are investigating why your keys changed mysteriously, you don’t want to do this over the network. Log into the actual server through other means (such as a management console or a KVM console) and generate fingerprints directly:

Then compare this with what you got over the network. If they match, you should be alright; if not, there are some serious problems.

If the fingerprints don’t match, you should do a network scan for ARP requests and see who is answering the ARP requests for the relevant IP address. Scan for ARP requests as you do a ping. If there are two hosts, then they will “fight” each other for an ARP entry, and you should see both responding.

Once you know the Ethernet address of the mysterious host, then chase it down by tracking where the ARP traffic is going by looking at router (or switch) interfaces.

Recently, I found myself trying to understand what was going on in the shadow file. For one thing, the “password” field in Ubuntu Lucid Lynx is much longer than it is in Red Hat Enterprise Linux 5. Then there were the exclamation points in front of some password entries.

Originally, there was no /etc/shadow file – the passwords and salts were contained with other user information in /etc/passwd. The problem was that /etc/passwd was readable by anyone, and that attacks became more common and more powerful. It was necessary to move the password into a separate file that would only be readable by root.

Here is an example of a shadow file line:
ddouthitt:$6$QR3drPrQ$JLolPKyiVuXvea1F2IpbPx9F9PEV0s/IGcNCpm6ZrBA6AFDvwHPQG7EsTQHuUqxfCvlsuKRb.O7w5RLPyj8nS/:15119:0:99999:7:::

Each field is separated by a colon (:). There are a number of fields:

User name. This must match up with a user name in /etc/passwd; it is the key.

Password. This field is a hash of the password (not the actual password itself) combined with an id (specifying which form of encryption was used) and with a salt.

Last changed date. This is the day that the password was last set (in days since January 1, 1970).

Minimum days. This is the minimum number of days that a password must be active before a password can be changed again. A user’s password cannot be changed by them until this number of days has elapsed.

Maximum days. This is the maximum number of days for a password to be valid. Once the maximum number of days has elapsed, the user is forced to change their password.

Warn days. Before a password expires, warn the user this many days ahead of time.

Inactive days. Once a password has expired, it will be disabled after this many days.

Expire date. On this date (in days since January 1, 1970), the account is disabled and the login can no longer be used.

Rather than trying to compute the date from the number of days since 1970, use the chage command:

$ chage -l ddouthitt
Last password change : May 25, 2011
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7

This also explains what each number means, and incorporates any defaults.

The password is more complicated. The actual password is not stored anywhere; the stored value in the /etc/shadow file is a hash generated from the password.

Originally, the password hash was just the hash itself; later, a two-character salt was added at the beginning.

The password hash today comes in three parts, separated by dollar signs ($):

Id. This identifies the encryption hash method used. A value of 1 denotes MD5; 2 or 2a is Blowfish; 3 is NT Hash; 5 is SHA-256; and 6 is SHA-512.

Salt. This is used by the encryption algorithms, and could be up to 16 characters.

Lastly, the password entry could be missing entirely (which means no password) or it could have an invalid entry (locked account).

Traditionally, locking was done with the asterisk (*) by convention or by the word LOCKED. However, this has been extended – by convention – to a variety of new suggestions. System logins that never had a password use a double exclamation mark (!!) and accounts that have been locked have the valid password entry invalidated by a exclamation mark prefix. Using the prefix in this way allows the password to be reinstated as it was.

FreeBSD uses a slightly different setup, with a file called /etc/master.passwd, an extended /etc/passwd file with password aging included, and support for an extended DES format (which starts with an underline). Extended DES is more secure than the original DES, but is no longer used in favor of the new modular format described above. FreeBSD standardizes on MD5 – like Red Hat, and unlike Ubuntu.

The finger daemon has been maligned for many years, and not without cause. The original daemons had bugs that caused them to dump core and became widely known as one of the ways the original Internet worm (the Morris Worm) spread across the Internet.

The only problem the finger daemon has today is that it provides user data to anyone on the network. This “information leak” will never be resolved as it is why finger exists in the first place.

However, if you put aside the fact that showing user information may be bad, finger can also be quite useful. It is also possible to run it without any network at all, using local data only. If there is no network access to finger (that is, no finger daemon is running), then the risk from finger is minimal.

There remains the ability to run a finger daemon on a network – cfingerd was written in order to provide secure finger services, including minimizing information leaks and giving people the ability to turn off finger access to particular users, or to create fake users. Those who wish to run a finger daemon would do well to choose cfingerd; it is enormously configurable.

Some systems include finger by default and some do not. Certainly, none come with a finger daemon active or present.

I will probably try cfingerd just to see how it works – but I wouldn’t recommend it on most production systems.

What is your take? I’d be interested in hearing from people who are actually running finger daemons and finding out why (and how).

Logging every shell command that a user makes turns out to be more difficult that initially imagined. The shell’s history function was designed to aid the user in using previous commands. We all know the use case: you just typed in a long name, and mistyped one character. The history allows you to fix the one character without typing all of the rest.

However, for auditing purposes, shell history makes life difficult: it was not designed to be secured against the user.

For bash, things are particularly difficult as its goal is to make life easier for the user – in whatever way possible – so it has all the “bells and whistles.” All of these multiple features must be accounted for and changes to the history file prevented.

Korn shell is simpler, and makes it easier to secure the shell history.

To lock down the history in these shells, there are a number of steps to take.

First, lock the shell history file itself. Change its attributes to append only with chattr +a .sh_history or chattr +a .bash_history – this makes it impossible to delete or change the data in the file. Not even the user can alter the attributes – only root can.

Secondly, insure that the history variables are appropriately set and cannot be changed, these include most importantly HISTFILE HISTCOMMAND HISTIGNORE. To do this, use the shell’s typeset command with the -r option: this makes the specified variables read-only. For good measure, make all history environment variables read-only. For example:

The HISTTIMEFORMAT is a bash extension that will provide timestamps in the history file.

For bash, change some of the standard options for history:

shopt -s cmdhist
shopt -s histappend

Setting cmdhist will put multiple line commands into a single history line, and setting histappend will make sure that the history file is added to, not overwritten as is usually done.

Also for bash, set the PROMPT_COMMAND:

PROMPT_COMMAND="history -a"
typeset -r PROMPT_COMMAND

This is because bash actually writes the history in memory; the history file is only updated at the end of the shell session. This command will append the last command to the history file on disk.

Lastly, create a SIGDEBUG trap to send commands to syslog. VMware’s ESXi already does something like this with its version of the ash shell. In short, create a function that will log the current command (pulled from the history file) and send it to syslog with the logger command. This will work both in bash and in Korn Shell.

Now all of these steps will take you a long ways towards recording everything your users do – but both bash and ksh have new features to make this all so much more simpler. GNU Bash introduced logging to syslog in version 4.1 – all that is required to activate it is a shell that was compiled with this feature enabled.

Korn Shell has had auditing since the introduction of ksh93. Similar to bash 4.1, user auditing is a compile-time feature. To see if your version of ksh93 has auditing installed, do one or the other of the following commands:

echo ${.sh.version}
echo $KSH_VERSION

In Ubuntu Maverick Meerkat, I get this output from ksh93:

# echo ${.sh.version}
Version JM 93t+ 2009-05-01

If auditing was enabled, the feature string (JM) would also have the letter A (auditing enabled) and possibly the letter L (per-user auditing enabled). Both IBM DeveloperWorks and Musings of an OS Plumber have fantastic articles on Korn Shell auditing.

It is also unlikely that bash includes auditing; the version on Maverick Meerkat is 4.1.5(1)-release.

For those who still use C shell (and tcsh in particular) there is a variant of tcsh called “tcsh-bofh” which supports logging to syslog. Unfortunately, tcsh-bofh hasn’t been maintained in a long time and the FreeBSD port of tcsh-bofh was removed from the FreeBSD ports tree back in January 2010.

It is also possible to get at this information without using the shell directly. Two commands can be used to get the same details: lastcomm (from the acct package, found in the Ubuntu Main repository) and auditctl (from the auditd package, found in the Ubuntu Universe repository). Linux Journal had a good article on Linux process accounting way back in 2002. There is also the rootsh and snoopylogger packages, but neither of these are in the Ubuntu repositories. rootsh is like a enforced version of typescript, and snoopylogger is a system library that you add to user environments. (Many of these tips come from a question asked on serverfault.com.)

If you see mysterious IPs on the network – and they don’t seem to be actually be doing anything like SMTP or DNS or SSH, there’s an angle you may not have considered.

Do a Google search for the address and the word “default” and see if anything is using the IP address for a default setting. Many routers or other devices come with a default setting and you may have one or more of these on your network which has not yet been configured.

In my case, the culprit was the IP 192.168.0.120 – which turned out to be a Dell iDRAC which was not yet configured.

There have been numerous problems that have slowed down SSH connections over the years. All are resolvable today – and could be fixed in modern operating systems, but typically aren’t. Most or all of these resolutions are related to reverse IP lookups: that is, given an IP address find out what DNS name is associated with it. These fixes turn off reverse IP lookups for the SSH client and SSH server.

The most often suggested fix is to add the following to your server configuration (sshd_config):

UseDNS no

This prevents the SSH server from making a reverse DNS lookup – that is, looking up the IP address and finding what DNS name is associated with that IP. Since the reverse lookup often does not respond, then there is a time-out involved and this is what generates the normal delay seen in SSH connections.

However, there are other things as well. In some distributions (such as Debian and Ubuntu) the system is set up to use Avahi. When SSH attempts to connect to a host, it uses Avahi as part of the authentication process. When the system uses mDNS (part of Avahi) to resolve an IP address, this resolves in a large delay until the system comes back.

Fixing this is simple as well: add this line to the SSH client:

GSSAPIAuthentication no

This authentication method triggers Avahi; by disabling GSSAPIAuthentication this skips Avahi altogether. It is also possible to rewrite the /etc/nsswitch.conf file to do the same thing. Changing the nsswitch.conf file affects the entire system, so be sure this is really what you want.

On my Ubuntu 10.04 system, the original line in nsswitch.conf looks like this:

hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4

To stop using mDNS entirely, change the /etc/nsswitch.conf file hosts line to this:

hosts: files dns

Reverse IP lookups are a common source of delays for all manner of services (including PPP, Sendmail, and many others!).

It almost goes without saying that servers should never run telnet or rlogin but rather should have SSH instead. Today’s UNIX and Linux server operating systems, whether commercial or open source, come with SSH (usually OpenSSH) installed.

The question is: is password authentication more secure than public key authentication? This question has been asked before; consider these two questions from serverfault.com, one from November 2010 and one from June 2010. The instinctive response from many will be “Of course not!” – but the question remains. What are the actual downfalls to password authentication – or are there any?

There are multiple problems that can exist with public key authentication that password adherents will point out.

Firstly, there is the possibility that a system can be compromised and the keys taken. This risk is not so high as it might sound; if you are running Linux or UNIX on the system and have it properly secured, the risk of system compromise is lower than you might think. It is also possible (and recommended) to encrypt the keys with a passphrase.

Secondly, those who recommend passwords will note that passwords exist only in your mind (as long as you don’t write it down). However, is an easily memorable password truly secure?

There are numerous advantages to using a key, including many not directly related to security: you can have different keys for different systems, and keys can be restricted in various ways by the server administrator. It is also possible to revoke access to a user without affecting any other user – no more having to tell everyone the new password after a password change.

Keys are also not susceptible to brute force attacks. A hacker can attempt to break into an SSH server by brute force by trying a variety of passwords and usernames; this is not possible with keys. This sort of attack can come from anywhere, which makes it easier for hackers to do and more likely to occur. If there are a lot of users using SSH, then the likelihood of someone using a weak password is much higher – making the risk of server compromise that much higher as well.

Passwords can also be stolen remotely. If a server is compromised, the next time you log in you may be connecting to a hacked SSH server which copies all of your passwords. If you used public key authentication, then your private key would never be seen (or compromised) by the remote server.

So how do you properly secure your private key? Here are a number of things you can do:

Password protect your private key.

Put the key on a removable device and remove after using.

Put the key on an encrypted volume.

Use ssh-agent to store the key instead of using it over and over.

If you have already created a key without a password, you can add a password by using ssh-keygen:

$ ssh-keygen -p -f id_mykey
Key has comment 'id_mykey'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.
$

There are a number of good sources of information on SSH – and OpenSSH in particular. Try some of these:

Mick Bauer’s articles in Linux Journal, “The 101 Uses of OpenSSH”, Part 1 and Part 2

The nicest thing about having Lastpass store accounts is that you don’t have to try and remember all the places you used a particular password. Using the Lastpass audit feature, it will scan your list of stored passwords and determine which web sites have duplicate passwords, and you can change them all – one by one, of course.

One other nice thing about Lastpass is their Android application – it provides all of the nice capabilities of their browser extensions but on the small Android platform and for all applications as well.

With Lastpass you can have a different random password for all of your web service accounts – and don’t even have to remember them. Just remember your Lastpass account password and you are good to go.

Without Lastpass… I dread the thought of having to remember all the web services, much less try to change all the passwords to go with them.

Side note to web service designers: Could you make it just a little easier to change passwords? Please? Several services had the change password selection hidden quite well – strangely, msn.com was one of these! Yahoo.com wasn’t any better – and Yahoo has the most exasperating password setup I’ve ever seen – requesting your password almost every two minutes. There simply must be a better way.

Recently, DNSSEC has been rolling out into major DNS servers, including those that service the .org zone and now the root zone. This sounds at first glance like a good thing: all responses from DNS servers are validated, and it becomes impossible for man-in-the-middle attacks to take place.

However, there are commercial uses for “man-in-the-middle” operations; OpenDNS is one that comes immediately to mind. Indeed, OpenDNS is opposed to DNSSEC and has implemented DNSCurve instead.

The main problem (for this discussion) is that DNSSEC completely removes the possibility of a man-in-the-middle – that is, it is impossible for a DNS server like OpenDNS to return a different IP address than the actual DNS address of a machine.

The OpenDNS article also suggests that Akimai and the NTP Pool Project will both be affected by this as well. In these cases, the problem is that when a name is presented to the DNS server, it chooses a particular IP address based on parameters of its choosing – so a one-to-one mapping of DNS name to IP address is irrelevant and impossible.

This also suggests that DNS round robin for clusters would be impossible to implement with DNSSEC active as well.

Mei Douthitt

Mei is an experienced UNIX and Linux system administrator, a former Linux distribution maintainer, and author of two books ("Advanced Topics in System Administration" and "GNU Screen: A Comprehensive Manual").