Recent releases of AIX installation media
(for 7.1 and 6.1) now contain the OpenSSH base installation filesets. This is
very handy; we no longer need to download or locate the software from other
sources.

One thing to consider is what this means
for future AIX migrations.

If you are migrating a system (that already
has a version of SSH installed) to AIX 7.1 then you may notice that the first
time you attempt to connect to the server (after the 7.1 migration) the
following ssh message appears:

root@nim1
: / # ssh aixlpar1

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@WARNING: REMOTE HOST IDENTIFICATION HAS
CHANGED!@

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT
IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone
could be eavesdropping on you right now (man-in-the-middle attack)!

It
is also possible that a host key has just been changed.

The
fingerprint for the RSA key sent by the remote host is

59:68:05:71:60:b5:d1:96:87:df:f6:9c:ca:9a:14:3e.

Please
contact your system administrator.

Add
correct host key in /.ssh/known_hosts to get rid of this message.

Offending
RSA key in /.ssh/known_hosts:17

RSA
host key for aixlpar1 has changed and you have requested strict checking.

Host key
verification failed.

In the
output above I’m attempting to SSH from another system to the newly migrated
AIX 7.1 LPAR. This is essentially informing us that the SSH host keys on the
AIX 7.1 server don’t match the host key stored in the local systems
/.ssh/known_hosts file. Something has changed.

Now of
course I could simply accept this change and update my known_hosts files, like
so:

root@nim1
: / # ssh-keygen -R aixlpar1

/.ssh/known_hosts
updated.

Original
contents retained as /.ssh/known_hosts.old

With
known_hosts updated, I’m able to SSH to the AIX 7.1 system successfully.

cgibson@nim1
: /home/cgibson $ ssh aixlpar1 date

Mon
Aug 20 19:44:20 EET 2012

But that’s
just for my SSH known_hosts file only. What about all the users that connect to
this system via SSH/SFTP/SCP? Do I really expect all of them to update their
known_hosts file with the new host key information?

This could
create problems for automated tasks, like file transfers. If these transfers
stop working then their could be “hell to pay”. So the question I’m often asked
is what can I do to prevent this from happening in the first place? Luckily
there is a way.

In this
example, we are using nimadm to migrate from AIX 5.3 to 7.1. The
AIX 7.1 lpp_source resource was created using the AIX 7.1 installation media
DVDs. All filesets were copied from the DVDs, verbatim, to the new 7.1
lpp_source resource on the NIM master.

First we
verify that the openssh* filesets are in fact in the AIX 7.1 lpp_source on the
NIM master.

root@nim1
: / # nim -o showres lpp_sourceaix710101 | grep -i ssh

openssh.base.client5.4.0.6100IN usr,root

openssh.base.client5.8.0.6101IN usr,root

openssh.base.server5.4.0.6100IN usr,root

openssh.base.server5.8.0.6101IN usr,root

openssh.man.en_US5.4.0.6100IN usr

openssh.man.en_US5.8.0.6101IN usr

openssh.msg.EN_US5.8.0.6101IN usr

openssh.msg.en_US5.4.0.6100IN usr

openssh.msg.en_US5.8.0.6101IN usr

On the NIM
client (running AIX 5.3), we verify there is an older version of SSH already
installed. The migration will remove these filesets (and the associated
/etc/ssh_host_* files). The newer version of SSH will be installed and new
ssh_host_key* files will be generated (hence the problem with the remote SSH
clients known_hosts files no longer holding the correct host keys).

Rather than
update these filesets manually after the migration, you can include this step
as a post migration task with nimadm.

An
alternative way to work around this problem (after the fact) would be to
restore the original ssh_host_key* files from a backup.For example, I copied the original
ssh_host_key* files to my home directory before starting the AIX migration.

aixlpar1
: / # cd /etc

aixlpar1
: /etc # cp -pr ssh /home/cgibson/ssh_orig/

In the
output below, I discover that my ssh_host_key* files have all been recreated
during the migration.

aixlpar1
: /etc/ssh # ls -ltr

total
352

-rw-r--r--1 rootsystem1288 May 01
2007ssh_config

-rw-r--r--1 rootsystem1155 May 04
2007sshd_banner

-rw-r--r--1 rootsystem2867 Oct 29
2008sshd_config

-rw-r-----1 rootsystem7 Aug 20 21:00
sshd.pid

-rw-r--r--1 rootsystem2341 Aug 20 21:19
ssh_prng_cmds

-rw-------1 rootsystem132839 Aug 20 21:19
moduli

-rw-r-----1 rootsystem382 Aug 20 21:45
ssh_host_rsa_key.pub

-rw-------1 rootsystem1679 Aug 20 21:45
ssh_host_rsa_key

-rw-r-----1 rootsystem630 Aug 20 21:45
ssh_host_key.pub

-rw-------1 rootsystem965 Aug 20 21:45
ssh_host_key

-rw-r-----1 rootsystem590 Aug 20 21:45
ssh_host_dsa_key.pub

-rw-------1 rootsystem668 Aug 20 21:45
ssh_host_dsa_key

I copy the
original files back to the /etc/ssh directory. The sshd subsystem is also
restarted to pick up the updated ssh_host* files.

aixlpar1
: /etc/ssh # cp -p /home/cgibson/ssh_orig/ssh_host_* .

aixlpar1
: /etc/ssh # ls -ltr

total
352

-rw-r--r--1 rootsystem210 Feb 03
2006ssh_host_rsa_key.pub

-rw-------1 rootsystem887 Feb 03 2006ssh_host_rsa_key

-rw-r--r--1 rootsystem319 Feb 03
2006ssh_host_key.pub

-rw-------1 rootsystem515 Feb 03
2006ssh_host_key

-rw-r--r--1 rootsystem590 Feb 03
2006ssh_host_dsa_key.pub

-rw-------1 rootsystem668 Feb 03
2006ssh_host_dsa_key

-rw-r--r--1 rootsystem1288 May 01
2007ssh_config

-rw-r--r--1 rootsystem1155 May 04
2007sshd_banner

-rw-r--r--1 rootsystem2867 Oct 29 2008sshd_config

-rw-r-----1 rootsystem7 Aug 20 21:00
sshd.pid

-rw-r--r--1 rootsystem2341 Aug 20 21:19
ssh_prng_cmds

-rw-------1 rootsystem132839 Aug 20 21:19
moduli

aixlpar1
: /etc/ssh # stopsrc -s sshd

0513-044
The sshd Subsystem was requested to stop.

aixlpar1
: /etc/ssh # startsrc -s sshd

0513-059
The sshd Subsystem has been started. Subsystem PID is 3997822.

Starting with AIX 7.1, CSM is no
longer supported or available. It has been replaced by Distributed Systems
Managment (DSM).Section 5.2 of the IBM AIX 7.1 Differences Guide Redbook provides
details of the new DSM capabilities.

Fortunately DSM still provides access
to the dsh command.I’ve written about how I’ve used this utility
in the past. The new dsh command (and other tools) are
provided in the new DSM filesets named dsm.core
and dsm.dsh.

These filesets are NOT installed by default. You must
manually install them. They can be found on your AIX 7.1 media.

If dsh is something you use, then I recommend you read the section on
DSM in the Redbook. Also take a look at section 5.2.7 Using DSM and NIM,
in which it describes how you can integrate DSM and NIM and completely automate
the installation of AIX:

“The
AIX Network Installation Manager (NIM) has been enhanced to work with the Distributed
System Management (DSM) commands. This integration enables the automatic
installation of new AIX systems that are either currently powered on or off.”

Although I’ve written about the dsh command before, there’s one usage
I’ve not covered.And that is using dsh to manage users across a group of
LPARs. In particular, changing a user’s password.

Before I go any further, I should
state that for the following to work you must first configure ssh keys on your
NIM master (or central mgmt AIX system) so that you can communicate with all of
your AIX systems via SSH, as root,without being prompted for a password. Read my article on dsh to find out how to do this if
necessary.

In the following example, I use dsh from my NIM master. It is my
central point of control for my AIX environment.

My ssh keys for root on my NIM master
have been generated and distributed to all of my LPARs.

root@nim# ssh-keygen -d

Generating public/private dsa key pair.

Enter file in which to save the key (/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /.ssh/id_dsa.

Your public key has been saved in /.ssh/id_dsa.pub.

The key fingerprint is:

ed:18:e9:00:37:13:7c:7c:74:6a:a9:e0:ad:c0:09:a9
root@nim

The key's randomart image is:

+--[ DSA 1024]----+

|... .. .|

|...o .+|

|o .
=. .+|

| . o = = =|

|E+ o
S .|

|.
+ +|

|.
o .|

||

||

+-----------------+

root@nim# ls -ltra

total 40

-rw-------1 rootsystem214 17 Sep 2010authorized_keys

drwxr-xr-x7 rootsystem4096 16 Nov 11:43 ..

-rw-r--r--1 rootsystem3615 16 Nov 12:04 known_hosts

-rw-r--r--1 rootsystem601 16 Nov 12:06
id_dsa.pub

-rw-------1 rootsystem672 16 Nov 12:06 id_dsa

drwx------2 rootsystem256 16 Nov 12:06 .

On my AIX LPARs, the authorized_keys file has been updated with
the public ssh key from my NIM master:

On the NIM master, the root user was
configured for the DSH environment. The following entry was placed in roots .profile:

root@nim# cat /.profile

ENV=$HOME/.kshrc

The following entry was placed in
roots .kshrc file:

root@nim# cat /.kshrc

export
DSH_NODE_RSH=/usr/bin/ssh

export
DSH_NODE_LIST=/usr/local/etc/nodes

A /usr/local/etc/nodes
file was created on the NIM master. This file contains a list of each of the
nodes that dsh can communicate with
from NIM:

root@nim# cat /usr/local/etc/nodes

aixlpar1

aixlpar2

aixlpar3

aixlpar4

aixlpar5

aixlpar6

aixlpar7

aixlpar8

aixlpar9

aixlpar10

aixlpar11

The first time that the dsh command is run against a new host,
the following message will be displayed. dsh
uses the FQDN, and the FQDN needs to be added to the known_hosts file for ssh. Therefore you must make an ssh connection first with FQDN to the
host:

root@nim# dsh uptime

aixlpar1.cg.com.au
: Host key verification failed.

dsh:2617-009 aixlpar1.cg.com.au remote shell had exit code 255

It is necessary to ssh directly to each node using its
FQDN. This step is only required once for each node. For example:

root@nim# ssh aixlpar1.cg.com.au

The authenticity of host 'aixlpar1.cg.com.au (172.1.6.17)' can't be established.

I set the users password to abc123,
using the chpasswd utility. I also
remove the ADMCHG flag so that the
user is not prompted to change their password on their first logon attempt.

root@nim# dsh 'echo cg:abc123 | chpasswd -c'

I confirm that I can logon with the
new user with the specified password, on one of the AIX LPARs.

root@nim# ssh cg@aixlpar1

cg@aixlpar1’s password:

Last login: Thu Mar1 20:05:01 CST 2012 on /dev/pts/1 from aix71

$
id

uid=204(cg)
gid=1(staff)

Another nice feature of dsh is the dshbak utility. This utility presents formatted output from the dsh command. For example:

root@nim 520 [/.ssh]# dsh errpt | dshbak

HOST:
aixlpar1.cg.com.au

------------------------------------

IDENTIFIER TIMESTAMPT C RESOURCE_NAMEDESCRIPTION

AA8AB2411116110811 T O OPERATOROPERATOR NOTIFICATION

A6DF45AA1104135011 I O RMCdaemonThe
daemon is started.

2BFA76F61104134111 T S SYSPROCSYSTEM SHUTDOWN BY USER

9DBCFDEE1104134111 T O errdemonERROR LOGGING TURNED ON

HOST:
aixlpar2.cg.com.au

-------------------------------

IDENTIFIER TIMESTAMPT C RESOURCE_NAMEDESCRIPTION

DE9A52D11111012611 I S rmt10AAA1

4865FA9B1111012211 P H rmt10TAPE
OPERATION ERROR

DE9A52D11110233511 I S rmt0AAA1

4865FA9B1110225511 P H rmt0TAPE
OPERATION ERROR

DE9A52D11109180311 I S rmt0AAA1

4865FA9B1109180011 P H rmt0TAPE
OPERATION ERROR

DE9A52D11108180411 I S rmt2AAA1

4865FA9B1108180211 P H rmt2TAPE
OPERATION ERROR

DE9A52D11108165711 I S rmt6AAA1

4865FA9B1108165111 P H rmt6TAPE
OPERATION ERROR

A22058611102085311 P S SYSPROCExcessive interrupt disablement time

F7FA22C91031134111 I O SYSJ2UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

DE9A52D11030163411 I S rmt0AAA1

4865FA9B1030163411 P H rmt0TAPE
OPERATION ERROR

....etc....

WARNING: Please be VERY CAREFUL when using the dsh command. Issuing
the wrong command can cause damage to all your AIX LPARS!

The dsm.dsh package contains the following utilities:

# lslpp -f
dsm.dsh | grep /usr/bin

/usr/bin/dcp
-> /opt/ibm/sysmgt/dsm/bin/dcp

/usr/bin/dsh
-> /opt/ibm/sysmgt/dsm/bin/dsh

/usr/bin/dping
-> /opt/ibm/sysmgt/dsm/bin/dping

/usr/bin/dshbak
-> /opt/ibm/sysmgt/dsm/bin/dshbak

If you are a
fan of the dping command, you are
going to be disappointed. Although the command is currently included in the dsm.dsh fileset, it probably won’t be
for much longer.

The command
works, “sort of”:

root@nim#
dping aixlpar1

aixlpar1: ping (alive)

But if you
run ‘dping –a’:

root@nim# dping
-a

dping: 2651-095 CSM license has
expired or has not been accepted. Run csmconfig -L if you have installed a new
release.

According to
the developers, dping is no longer
supported and will eventually be removed from the DSM package. The response
from the developers was as follows:

"The
reason "dping -a" is failing with the license check is because the
command is calling “/usr/bin/runact-api –c IBM.DmsCtrl::::isLicenseValid"
and the license is not set. So the command fails. Since CSM is not
supported anymore and went end of life. "

“...
please consider the dping command as being "deprecated" code pending
removal from the dsm.dsh package.”

When I asked why the command was listed in the AIX 7.1 online
documentation if it was no longer available, I was informed: “We are
in the process of working with component owner regarding the DOCs and updating
them.”. At this
stage I’ve not been able to find an alternative command (in AIX). If I find
one, I’ll update this post.

If you are
planning on migrating to AIX 7.1 please be aware that CSM is no longer
supported or available with AIX 7.1. CSM is now ‘end of life’.