While attending the IBM Power Systems
Symposium this week, I learnt that starting with AIX 7.1 (and AIX 6.1 TL6) JFS2
logging is disabled during a mksysb restore. You may be familiar with disabling
JFS2 logs, if not, take a look at this IBM technical note:

I’ve been unable to find any official
documentation from IBM that mentions this new enhancement to the mksysb restore
process. However, when I checked my own AIX 7.1 system I found the following
statement/code in the /usr/lpp/bosinst/bi_main script:

A customer was attempting to move an LPAR from one POWER7 750
to another POWER7 750 using Live Partition Mobility (LPM). But he was receiving
the following error message when performing an LPM validation.

This was an
easy one to fix. The reserve policy on the hdisk on the VIOS (yes, it was a
single VIO server not dual VIOS, don’t ask why!) was set to PR_exclusive instead of no_reserve.

You are receiving that mail because we have been either in touch in regards
of LPA2RRD tool about year ago or you are on LPAR2RRD mailing list.

I’ve decided to make professional paid support of LPAR2RRD.
Here are my reasons which led me to that thought:
- I have not touched the source code for nearly 1.5 year (looks like the
last version is quite stable :) ). No time at all and lack of motivation...
- Only what I trying to keep is support of the tool, but my response times
are not what I am proud of
- the tool itself would need big time investment to implement new
functionalities, rewrite the web face (would need to hire a web designer
for that) etc.

Only the solution what I see how to keep continuity and do not let project
slowly died is paid support.
Basically I feel that as the last chance for any further development or
support from my side.
I simply cannot manage to do everything I actually do around and still work
on LPAR2RRD.
If I would get some money from the project then I will be able to
prioritize my other activities in favour of LPAR2RRD development and
support.

From that reason I’ve created new LPAR2RRD home here:http://www.lpar2rrd.com
I am about to found a company which would be responsible for support. Based
on your feedback I will decide go or no-go.
I could even imagine working on LPAR2RRD full time in case of enough
support subscribers.

I do not intend to force you ordering support. If you are happy with the
actual version, you have no issues etc ... that this makes me also happy.
Make people happy was the main reason why I spent my free time by
developing of the tool since 2006. I do not regret that at all! It was fun.

OK, so I had
a NIM master with two network interfaces, en0 and en1. The en0 interface was connected
to the 192.168.10.0 network and the en1 interface was connected to the 10.1.1.0 network. When the
master was initially configured, we chose en0 as the primary install interface.
The master could now install NIM clients on the 192.168.10.0 network, without
any additional NIM network configuration.

There was
a single NIM network definition for the 192.168.10.0 network (net_192_168_10) in the NIM database.

root@nim1 /
# lsnim -c networks

net_192_168_10 networksent

Sure
enough, the time came when we needed to install NIM clients on the 10.1.1.0
network. Rather than adding NIM routes, we chose to configure the NIM master
with an additional install interface on the 10.1.1.0 network. Fortunately, the
NIM master was already “directly” connected to the 10.1.1.0 network on its en1
interface. So all we had to do was update the NIM configuration.

First we
added a new entry in the /etc/hosts file for the NIM master’s 10.1.1.0 network
address. The hostname for this interface was nim1i.

root@nim1 /
# grep nim1 /etc/hosts

192.168.10.10nim1

10.1.1.10 nim1i

root@nim1 / #
host nim1i

nim1i is 10.1.1.10

Next we
defined a new network install interface for the NIM master via the smit
fastpath, ‘smit nim_mac_if’.

We entered
the hostname of the NIM master on the 10.1.1.0 network i.e. nim1i.

Define a Network Install Interface

Type or
select a value for the entry field.

Press Enter
AFTER making all desired changes.

[Entry Fields]

* Host Name
of Network Install Interface[nim1i]

We named
the new NIM network, net_10_1_1 and entered the appropriate subnet mask for
this network. We did not enter a default gateway for the client or the NIM
master (it is unnecessary, as the master is “directly” connected to this
network).

Define a Network Install Interface

Type or
select values in entry fields.

Press Enter
AFTER making all desired changes.

[Entry Fields]

* Machine
Namemaster

Network Install Interface

*Cable TypeN/A+

Network Speed Setting[]+

Network Duplex Setting[]+

*NIM Network[net_10_1_1]

*Network Typeent

*Ethernet TypeStandard+

*Subnetmask[255.255.255.0]

Default Gateway Used by Machine[]

Default Gateway Used by Master[]

*Host Namenim1i

Network Adapter Hardware Address[0]

Network Adapter Logical Device Name[]

There were
now two network definitions in our NIM database. The Nstate attribute was set to ‘ready
for use’, so our definition was OK to use.

root@nim1 /
# lsnim -c networks

net_192_!68_10networksent

net_10_1_1networksent

root@nim1 /
# lsnim -l net_10_1_1

net_10_1_1:

class= networks

type= ent

comments= Generated during definition of nim1i

Nstate= ready for use

prev_state = information is missing from
this object's definition

net_addr= 10.1.1.0

snm= 255.255.255.0

The NIM
master has two network interfaces defined.

root@nim1 /
# lsnim -l master | grep if

if_defined= chrp.64.ent

if1= net_192_168_10 nim1 0

if2= net_10_1_1
nim1i 0

Now we
could install new NIM clients using this network. We verified this by adding a
new NIM client to the NIM database.

# smit
nim_mkmac

Define a Machine

Type or
select a value for the entry field.

Press Enter
AFTER making all desired changes.

[Entry Fields]

* Host Name
of Machine[lpar22i]

(Primary Network Install Interface)

After
pressing enter on the previous smit panel, we were immediately presented with a
panel to modify the attributes of the NIM client and (more importantly) the
correct network and hostname were automatically selected as part of the clients
definition.

Define a Machine

Type or
select values in entry fields.

Press Enter
AFTER making all desired changes.

[Entry Fields]

* NIM
Machine Name[lpar22]

* Machine
Type[standalone]+

* Hardware
Platform Type[chrp]+

Kernel to use for Network Boot[64]+

Communication Protocol used by client[nimsh]+

Primary Network Install Interface

*Cable TypeN/A+

Network Speed Setting[]+

Network Duplex Setting[]+

*NIM Networknet_10_1_1

*Host Namelpar22i

Network Adapter Hardware Address[0]

Network Adapter Logical Device Name[]

IPL ROM Emulation Device[]+/

CPU Id[]

Machine Group[]+

Managing System Information

WPAR Options

Managing System[]

-OR-

LPAR Options

Identity[]

Management Source[]+

Comments[]

If we had
configured NIM incorrectly i.e. it was not aware of the 10.1.1.0 network, then
we would have been prompted for network information before we could configure
the NIM client (as shown below).

I don’t know if there are any plans to fix this but I wanted to
share this information in case anyone else encounters this issue. iSCSI is not
all that popular on AIX systems (perhaps this is a good thing?), so it may not
impact enough people to warrant a quick fix.

Need to update the running attributes of a virtual SCSI adapter on
an AIX system? Try these steps.

I want to change value for the vscsi_err_recov attribute
to fast_fail.
I can’t do this if the adapter is busy i.e. has open/available paths to a disk device.

#
lsattr -El vscsi2

vscsi_err_recov
delayed_fail
N/A
True

vscsi_path_to
0 Virtual
SCSI Path Timeout True

#
chdev -l vscsi2 -a vscsi_err_recov=fast_fail

Method
error (/usr/lib/methods/chggen):

0514-029 Cannot perform the requested function because a

child device of the specified device is not in a correct state.

I have two paths to my disk devices. This will allow me to take
one path offline while I change the attributes of the adapter.

#
lspath | grep vscsi2

Enabled
hdisk4 vscsi2

Enabled
hdisk5 vscsi2

#
rmpath -l hdisk4 -p vscsi2

path
Defined

#
rmpath -l hdisk5 -p vscsi2

path
Defined

#
lspath | grep vscsi2

Defined
hdisk4 vscsi2

Defined
hdisk5 vscsi2

With the path offline, I can now change the attributes of the
device and then bring the path online again.

#
chdev -l vscsi2 -a vscsi_err_recov=fast_fail

vscsi2
changed

#
lsattr -El vscsi2 -a vscsi_err_recov

vscsi_err_recov
fast_fail N/A True

#
cfgmgr<
Bring the path online.

#
lspath | grep vscsi2

Enabled
hdisk4 vscsi2

Enabled
hdisk5 vscsi2

Now I can do the same for the other VSCSI adapter.

#
lsattr -El vscsi3 -a vscsi_err_recov

vscsi_err_recov
delayed_fail N/A True

#
chdev -l vscsi3 -a vscsi_err_recov=fast_fail

Method
error (/usr/lib/methods/chggen):

0514-029 Cannot perform the requested function because a

child device of the specified device is not in a correct state.

#
lspath | grep vscsi3

Enabled
hdisk4 vscsi3

Enabled
hdisk5 vscsi3

#
rmpath -l hdisk4 -p vscsi3

path
Defined

#
rmpath -l hdisk5 -p vscsi3

path
Defined

#
lspath | grep vscsi3

Defined
hdisk4 vscsi3

Defined
hdisk5 vscsi3

#
chdev -l vscsi3 -a vscsi_err_recov=fast_fail

vscsi3
changed

#
lsattr -El vscsi3 -a vscsi_err_recov

vscsi_err_recov
fast_fail N/A True

#
cfgmgr

# lspath
| grep vscsi3

Enabled
hdisk4 vscsi3

Enabled
hdisk5 vscsi3

All paths are available again and the adapters have been updated.
All of this was done without taking an outage.

#
lspath

Enabled
hdisk4 vscsi2

Enabled
hdisk4 vscsi3

Enabled
hdisk5 vscsi2

Enabled
hdisk5 vscsi3

#
lsattr -El vscsi2 -a vscsi_err_recov

vscsi_err_recov
fast_fail N/A True

# lsattr
-El vscsi3 -a vscsi_err_recov

vscsi_err_recov
fast_fail N/A True

If I’d chosen not to take the paths offline, one at a time, I
would have needed to change the adapters attributes by specifying the –P flag
with the chdev command. Then I would need to reboot the system.

-P

Changes
the device's characteristics permanently in the Customized Devices object class
without actually changing the device. This is useful for devices that cannot
be made unavailable and cannot be changed while in the available state. The
change is made to the database, and the changes are applied to the device when
the system is rebooted. This flag cannot be used with the -T flag. Not all
devices support the -P flag.

Before
the change, I was unable to compile anything. Note the highlighted text below.

#
/usr/vac/bin/xlc cgc.c

The
license for the Evaluation version of IBM XL C/C++ for AIX V11.1 compiler
product has expired. Please send an email to compiler@ca.ibm.com
for information on purchasing the product. The evaluation license can be
extended to 74 days in total by either a) setting an environment variable XLC_EXTEND_EVAL=yes; or, b) specifying a compiler command line option
-qxflag=extend_eval. The extended evaluation license will expire on Sat
Jun 18 10:26:33 2011. Use of the Program continues to be subject to the terms
and conditions of the International License Agreement for Evaluation of
Programs, including the accompanying License Information document (the
"Agreement"). A copy of the Agreement can be found in the
"LicAgree.pdf" and "LicInfo.pdf" files residing in the root
directory of the installation media. If you do not agree to the terms and
conditions of the Agreement, you may not use or access the Program.

With
the environment variable in place, I was able to compile again.

#
/usr/vac/bin/xlc cgc.c

#
./a.out

Hello
World!

Of
course I could also export the environment variable as required instead e.g.

#
export XLC_EXTEND_EVAL=yes

#
/usr/vac/bin/xlc cgc.c

#
./a.out

Hello
World!

If
you do place this environment variable in /etc/profile, you may need to restart
any processes on the system that need to call the compiler.

Someone installs SAP onto an AIX system and decides to use TCP
port 3901 as an SAP service port. This is the same port used by nimsh. In some
rare cases, nimsh may not be active on the LPAR, which makes it easy for the
SAP installation to hijack port 3901. If nimsh is active, the person installing
SAP may consciously stop nimsh and use port 3901 for SAP anyway. Hopefully that
doesn’t happen. Hopefully, they will talk to the AIX administrator and discuss
the best way forward. Hopefully...

In either case, if the port is taken by SAP, nimsh will no longer
work. If you love using NIM as much as I do, this is a real problem! We could
revert back to using rsh but no-one will do this anymore because of concerns
around security. And rightfully so!

The ports used by nimsh (3901 and 3902) are registered to Internet
Assigned Number Authority (IANA). These port numbers appear in the
/etc/services file.

nimsh3901/tcp# NIM Service Handler

nimsh3901/udp# NIM Service Handler

nimaux3902/tcp# NIMsh Auxiliary Port

nimaux3902/udp# NIMsh Auxiliary Port

Considering these port numbers are registered with IANA, we can usually
persuade our SAP colleagues to change their SAP installation to use a different
port number. However, depending on the skills/experience of the SAP resource,
one of two things usually happens 1) They take an outage, re-install SAP and
choose a different port number or 2) The more experienced/confident SAP basis
resource will take an outage and modify the instance to use a different port:
without reinstalling SAP.

Perhaps SAP need to include a warning in their install notes,
advising customers not to use port 3901 on AIX systems (i.e. best practice)?

Now, if you must change nimsh to use a different port number, it
is possible. But not recommended.

To do this, you must change the /etc/services file on the NIM master
and the NIM client to reflect the same port numbers for nimsh. This will work
until the NIM master or the NIM client have their services file overwritten by
way of install or fileset updates. After which, the default values for nimsh
will be reinstated.

You would also need to change the services file on all of your NIM clients. Every time you
performed a NIM fileset update, you would need to remember to change the /etc/services
file again. This is painful and bound to catch someone out eventually!

In the following example I’ll demonstrate how to change the port
number used by nimsh.

We start with a typical nimsh configuration using port 3901. On
the NIM client, nimsh is listening on port 3901.

We can confirm that we have connected to the NIM client on port
39011 by looking at the output from lsof
and netstat. There is a TCP session
established between the master and the client on port 39011.

A colleague was attempting to recreate a WPAR. The WPAR had previously
been removed using the WPAR Manager Web GUI. This worked, or so he tells me!

Anyway, when he tried to create the WPAR again from the GUI, using
the same WPAR name (wpar20), it
failed. The error claimed that /wpars/wpar20
already existed. But it didn’t!

Now, before I go any further, looking back at the issue now, it is
pretty obvious what the problem might have been. However, the solution was not
evident at first. This proves one thing for certain: some problems cannot be
resolved without TWO cups of coffee in the morning.

Of course, I resorted to the command line to see what was REALLY
happening.

The mkwpar command was
indeed complaining that /wpars/wpar20 already existed.

#
mkwpar -n wpar20

mkwpar:
0960-288 The /wpars/wpar20 file system already exists.

mkwpar:
0960-417 Specify -p to create this workload partition using the existing file
system data.

I could not find any directory or file system with that name,
anywhere under /wpars or elsewhere.

#
ls -l /wpars/wpar20

/wpars/wpar20
not found

#
ls -l /wpars

total
0

#

#
df –g | grep –i wpar

#

#
lsvg | lsvg –il | grep –i wpar

#

I ran the mkwpar with truss to try to find out where it was
finding the reference to /wpars/wpar20.
At this point you have probably already determined where the problem might be!

The truss output
pointed me in the direction of /etc/filesystems.....of
course! Another “facepalm“ moment!

#
truss mkwpar -n wpar20 > /tmp/wpar20 2>&1

#
vi /tmp/wpar220

statx("/wpars/wpar20",
0x30008AB8, 128, 010) = -1

kopen("/etc/filesystems", O_RDONLY|O_LARGEFILE) =
4

Whatever my learned colleague had done when he “removed” the WPAR
with the GUI, it left entries for the WPARs file systems in /etc/filesystems.

I was working at a client site today, on a NIM master that I
configured a month or so ago. I was there to install the TSM backup client
software on about 30 or so LPARs. Of course I was going to use NIM to
accomplish this task.

The software install via NIM worked for the majority of the LPARs
but I noticed a few of them were failing. This was very odd, as the last time
I’d use the same NIM method to install software, everything was fine.

I suspected that perhaps something had changed on the client
LPARs...maybe with their /etc/niminfo
file for instance. So I performed the following steps to reconfigure the /etc/niminfo file
and configure the nimsh subsystem on
the client LPAR.

lpar1#
mv /etc/niminfo /etc/niminfo.old

lpar1#
niminit -a master=nim1 -a name=`hostname`

lpar1#
stopsrc -s nimsh

lpar1#
smit nim_config_services

Configure Client Communication Services

Type
or select values in entry fields.

Press
Enter AFTER making all desired changes.

[Entry Fields]

* Communication Protocol used by client[nimsh]+

NIM Service Handler Options

*Enable Cryptographic Authentication[disable]+

for client communication?

Install Secure Socket Layer Software (SSLv3)?[no]+

Absolute path location for INSTALLP package[/dev/cd0]/

-OR-

lpp_source which contains INSTALLP
package[]+

Alternate Port Range for Secondary
Connections

(reserved values will be used if left
blank)

Secondary Port Number[]#

Port Increment Range[]+#

The last step failed with the following error message:

0042-358
niminit: The connect attribute may only be assigneda service value of "shell" or
"nimsh".

I checked the NIM client and confirmed it was configured for nimsh
and it was fine. However, I did notice something odd when I ran the following
command:

lpar1#
egrep 'nimsh|nimaux' /etc/services

lpar1#

The entries for nimsh
were missing from the /etc/services
file!

Somebody had decided that these entries were not required and had
simply removed them! Gee, thanks so much for that!

After adding the following entries back into the services file,
everything started working again!

nimsh3901/tcp# NIM Service Handler

nimsh3901/udp# NIM Service Handler

nimaux3902/tcp# NIMsh Auxiliary Port

nimaux3902/udp# NIMsh Auxiliary Port

I’ve also encountered this error when there is another
process (other than nimsh) using
port 3901 or 3902.

Another error message you might confront, if those
entries are either missing or commented out, is on the NIM master:

nimmast#
nim -o showlog -a log_type=lppchk lpar1

0042-001
nim: processing error encountered on "master":

0042-006 m_showlog: (From_Master) connect
Error 0

poll:
setup failure

I thought I’d also mention another error message that
can potentially drive you insane (especially if you haven’t had your morning
coffee!). The error doesn’t relate to nimsh
at all but I thought I’d describe it anyway. The message appears when running
the nim –o showlog command against a
client LPAR.

nimmast# nim -o showloglpar1

0042-001
nim: processing error encountered on "master":

0042-006 m_showlog: (From_Master) connect
Error 0

0042-008 nimsh: Request denied – wronghostname

I’ve modified the output a little to make it easier to
identify the problem. Can you see it? I thought so! Upon investigation you may
find that the IP address for the NIM master is resolving to a different
hostname on the client. For example:

On the NIM master:

nimmast# host nimmast

nimmast
is 172.29.150.177

nimmast# host 172.29.150.177

nimmast is 172.29.150.177

nimmast# grep 177 /etc/hosts

172.29.150.177nimmast

On the NIM client:

lpar1# host nimmast

nimmast
is 172.29.150.177

lpar1# host 172.29.150.177

wronghostname is 172.29.150.177

lpar1# grep 172.29.150.177 /etc/hosts

172.29.150.177wronghostname

172.29.150.177nimmast

In this example, someone placed two host entries in /etc/hosts with the same IP address. The client was resolving the IP address
to an incorrect hostname. This resulted in our nim –o showlog command failing.

If restoring a workload
partition, target disks should be in available state.

So I tried the command again,
this time with the –O flag as suggested in the error message. It failed again
stating that it could not remove cgvg from hdisk5. This was also good. At this
point, an experienced AIX admin would look for the existence of a volume group
on hdisk5. However, a junior AIX admin might not! :)

Global# mkwpar -O -D
rootvg=yes devname=hdisk5 -n rootvgwpar1

mkwpar: 0960-620
Failed to remove cgvg on disk hdisk5.

Global# lsvg -l cgvg

cgvg:

LV
NAME
TYPE LPs
PPs PVs LV STATE
MOUNT POINT

cglv
jfs2 1
1 1 open/syncd /cg

loglv00
jfs2log 1
1 1 open/syncd N/A

But what if the file systems
were already unmounted prior to running mkwpar
with the –O flag. What would happen?

So, to test my theory, I
unmounted my file system so that the logical volumes in cgvg were all now
closed.

This could be a trap for “first time” users of RootVG WPARs! So
look out! :)

Apparently this is working as designed, as the –O flag is really
only meant to be called by WPAR tools such as the WPAR Manager.

The man page for mkwpar
states:

-O This flag is used to force the overwrite of an existing volume group
on the given set of devices specified with the -D rootvg=yes flag directive. If not
specified, the overwrite value defaults to FALSE. This flag should only be specified
once, for its setting will be applied to all devices specified with the -D
rootvg=yes flag directive.

I was
collecting a list of WWPNs for a bunch of new AIX LPARs I was installing from
scratch. There were two ways I could find the WWPN for a virtual Fibre Channel
adapter on a new LPAR i.e. one that did not yet have an operating system
installed.

I started by
checking the LPAR properties from the HMC (as shown below).

To speed
things up I moved to the HMC command line tool, lssyscfg, to display the
WWPNs (as shown below).

I gave these
WWPNS to my SAN administrator so that he could manually “zone in” the LPARs on
the SAN switches and allocate storage to each. He then, half-jokingly said,
“Gee, it would be nice if you could insert the colons into the WWPNS for me! J”. Of course, this got me thinking and
after a few minutes of playing with sed, I came up with a way to do this
quickly.

# cat lpar1_wwpns.txt | sed
's/../&:/g;s/:$//'

c0:50:76:03:a2:92:00:7c

c0:50:76:03:a2:92:00:7e

c0:50:76:03:a2:92:00:78

c0:50:76:03:a2:92:00:7a

Now my list
of WWPNs was ready to be cut’n’paste by my SAN admin. He was happy.

This is something
that I experience on all new Power/AIX systems that I install:

System migrated to AIX
5.3 (or later) might experience double boot

When booting AIX Version 5.3 (or
later) on a system that has previously been running an earlier release of AIX, you
may notice that the system automatically reboots and restarts the boot process.
This is how the firmware processes changed information in the boot image. This
reboot also occurs if the process is reversed. A system previously running AIX
5.3 (or later) that is booting a release of AIX prior to 5.3 goes through the
same process. This
″double boot″ occurs only once; if the stored value does not change,
then the second boot does not occur. If you install AIX 5.3 (or later) and
continue to use only that version, this double boot occurs once, and it occurs
only if your system was running a pre-AIX 5.3 release before you boot AIX 5.3
(or later). Systems
that are preinstalled with AIX 5.3 (or later) and use only that version do not
experience the ″double boot.″

I received
an email from one of my customers recently that simply said:

“mate
…on lpar30, for some reason the IBM.DRM and others are failing to start .. um
.. any chance you could have a quick look at that ?”

So I asked, “OK, so this use to work right?”. To which I received a
relatively confused reply, ”.....yep it
did....actually no....it’s never worked....or has it???...I’m not sure...”.

Based on my experience, the most common issue that prevents DLPAR
operations from working are network problems. Before diving into the deep
end and trying to debug RSCT, it’s always best to start with the basics. For example, can you ping the HMC from the
LPAR? Can you ping the LPAR from the HMC? If either of these tests fails, check
the network configuration on both components before doing anything else.

– Check the LPAR
communicationsbox in HMC configuration screen for LAN adapter
that is used for HMC-to-LPAR communication.

– By the way, unlike POWER4 systems, LPARs on POWER5 and POWER6 systems
do not depend on host name resolution for DLPAR operations.

·Check routing
on the LPAR and the HMC.

–
Use ping and the HMC’s Test
Network Connectivity task to verify the LPAR and the HMC can communicate
with each other.

If you check the network and you are happy that the LPAR and the HMC
can communicate, then perhaps you need to re-initialise the RMC subsystems on
the AIX LPAR. Run the following commands:

# /usr/sbin/rsct/bin/rmcctrl –z

# /usr/sbin/rsct/bin/rmcctrl –A

# /usr/sbin/rsct/bin/rmcctrl –p

Wait up to 5
minutes before trying DLPAR again. If DLPAR still doesn’t work i.e. the HMC is
still reporting no values for DCaps, and
the IBM.DRM subsystem still won’t
start, try using the recfgct command.

Only run
the rmcctrl and recfgct commands if you believe something has become corrupt in the
RMC configuration of the LPAR. The fastest way to fix a broken
configuration or to clear out the RMC ACL files after cloning (via alt_disk
migration) is to use the recfgct
command.

These daemons should work “out of the
box” and are not typically the cause of DLPAR issues. However, you can try
stopping and starting the daemons when troubleshooting DLPAR issues.

The rmcctrl -z command just
stops the daemons. The rmcctrl -A command ensures that the subsystem group
(rsct) and the subsystem (ctrmc) objects are added to the SRC, and
an appropriate entry added to the end of /etc/inittab and it
starts the daemons.

The
rmcctrl –p command enables the
daemons for remote client connections i.e. from the HMC to the LPAR and vice
versa.

If you are familiar with the System
Resource Controller (SRC) you might be tempted to use stopsrc and startsrc
commands to stop and start these daemons.

Do not do it; use the rmcctrl commands
instead.

If /var is 100% full, use chfs
to expand it. If there is no more space available, examine subdirectories
and remove unnecessary files (for example, trace.*, core, and so forth). If /var is full, RMC subsystems may fail
to function correctly.

The polling interval for the RMC
daemons on the LPAR to check with the HMC daemons is 5-7 minutes; so you need
to wait long enough for the daemons to start up and synchronize.

The Resource Monitoring and Control
(RMC) daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and
are controlled by the System Resource Controller (SRC). These daemons run in
all LPARs and communicate with equivalent RMC daemons running on the HMC. The
daemons start automatically when the operating system starts and synchronize
with the HMC RMC daemons.

The
daemons in the LPARs and the daemons on the HMC must be able to communicate
over the network for DLPAR operations to succeed. This is
not the network connection between the managed system (FSP) and the HMC; it is
the network connection between the operating system (AIX) in each LPAR and the
HMC.

Note:
Apart from rebooting, there is no way to stop and start the RMC daemons on the
HMC.

The
following links also contain some (out dated) information relating to DLPAR
verification and troubleshooting. Even though it is quite old some of it is
still relevant today and is good a place to start.

The previous
link (above) provides some information relating to the values for DCaps and
what they mean (also out dated):

0 - DR CPU capable(can move CPUs)

1 - DR MEM capable(can move memory)

2 - DR I/O capable(can move I/O resources)

3 - DR PCI Bridge(can move PCI bridges)

4 - DR Entitlement(POWER 5 can change shared entitlement)

5 - Multiple DR CPU (AIX 5.3 can move 2+ CPUs at
once)

0x3f = max, and 0xf is common for AIX 5.2

If you are
interested in how HMC and LPAR authentication works with DLPAR, then read on.
Otherwise, happy DLPARing!

HMC
and LPAR authentication (RSCT authentication)

The diagram
below outlines how the HMC and an LPAR authenticate with each other in order
for DLPAR operations to work. RSCT authentication is used to ensure the HMC is
communicating with the correct LPAR.

Authentication
is the process of ensuring that another party is who it claims to be.
Authorization is the process by which a cluster software component grants or
denies resources based on certain criteria. The RSCT component that implements
authorization is RMC. It uses access control list (ACL) files to control user
access to resources.

The RMC
component subsystem uses cluster security services to map the operating system
user identifiers, specified in the ACL file, to network security identifiers to
determine if the user has the correct permissions. This is performed by the
identity mapping service, which uses information stored in the identity mapping
files ctsec_map.global and ctsec_map.local.

The
RSCT authorization process in detail:

1.On the HMC:DMSRM pushes
down
the secret key and HMC IP address to NVRAM when it detects a new CEC; this
process is repeated every five minutes. Each time an HMC is rebooted orDMSRM is
restarted, a new key is used.

2.On the AIX LPAR:CSMAgentRM, through RTAS (Run-time Abstraction Services),
reads the key and HMC IP address out from NVRAM. It will then authenticate the
HMC. This process is repeated every five minutes on a LPAR to detect a new HMCs
and if the key has changed. An HMC with a new key is treated as a new HMC and
will go though the authentication and authorization processes again.

3.On the AIX LPAR:After
authenticating the HMC,CSMAgentRM will contact the DMSRM on the HMC to
create aManagedNode
resource
in order to identify itself as a LPAR of this HMC.CSMAgentRM then
creates a compatibleManagementServer resource on AIX. This can be displayed
on AIX with the lssrsrc command.
e.g.

root@aix6 / # lsrsrc
"IBM.ManagementServer"

Resource Persistent
Attributes for IBM.ManagementServer

resource 1:

Name= "192.168.1.244"

Hostname= "192.168.1.244"

ManagerType= "HMC"

LocalHostname= "10.153.3.133"

ClusterTM= "9078-160"

ClusterSNum= ""

ActivePeerDomain = ""

NodeNameList= {"aix6"}

4.On the AIX LPAR: After the creation of theManagedNode and ManagementServer resources on
the HMC and AIX respectively,CSMAgentRMgrants HMC
permission to access necessary resource classes on the LPAR. After granting the
HMC permission,CSMAgentRM will change itsManagedNode, on the
HMC, Status to1. (It
should be noted that without proper permission on AIX, the HMC would be able to
establish a session with the LPAR but will not be able to query for OS
information, DLPAR capabilities, or execute DLPAR commands afterwards.)

5.On the HMC: After theManagedNode Status is changed
to1, LparCmdRM establishes
a session with the LPAR, queries for
operating system information and DLPAR capabilities, notifies CIMOM about
the DLPAR capabilities of the LPAR, and then waits for the DLPAR commands from
users.

I’ve
written about multibos before, here and here. But recently I
started experimenting with multibos mksysb migration. A customer asked me how
this worked and apart from a high-level view I wasn’t able to provide any real
world experience, so I thought I’d give it a try. What follows is just a ‘brain
dump’ from my quick test.

First of all
this isn’t really a migration. It just simply populates a second instance of
AIX with a higher-version. It doesn’t really migrate (or merge) your existing
configuration into the second instance. So I’m not sure how useful this feature
really is right now.

Starting with
5.3 TL9 you can add a 6.1 TL2 (or above) instance. This is done with the new –M
flag. You must be running with the 64bit kernel.

This isn’t really a migration because it populates the second instance using a
mksysb based on the new release.

In 6.1 TL2 a new flag (-M) was added to the mksysb command which allows you to
create a mksysb for use with multibos. It creates a backup of BOS (/, /usr,
/var, /opt).
bos.alt_disk_install.boot_images must be installed.

It is not advised to run in this environment for an extended period of time.
There could be problems if tfactor or maps are used. Be aware that 6.1 specific
attributes may not be reflected in the standby instance.

So in my
lab environment I have two AIX LPARs. One is running AIX 6.1 and the other
running AIX 7.1.

First I
take a mksysb (with the –M flag) of the AIX 7.1 system to a file. This file
will be called by multibos to populate the second instance.

aix7[/] > mksysb -Mie /data/aix7-mksysb

Creating information file
(/image.data) for rootvg.

Creating list of files to back up.

Backing up 71643 files.....

71643 of 71643 files (100%)

0512-038 mksysb: Backup Completed
Successfully.

aix7[/] > ls -ltr /data

total 4276112

drwxr-xr-x2 rootsystem256 Feb 21 20:59
lost+found

-rw-r--r--1 rootsystem2189363200 Feb 21 21:06
aix7-mksysb

I copied this
file over to my AIX 6.1 system. This was the system that was to be ‘migrated’.
The next step was to perform a preview of the multibos operation.

Upon
checking my bootlist output, I
noticed (as expected) that the list now contained two extra entries for bos_hd5. These were the boot logical
volume entries for the second instance. If I was to boot from this LV I’d be
booting into AIX 7.1. Cool.

root@aix6 /# bootlist -m normal -o

hdisk0 blv=bos_hd5

hdisk0 blv=bos_hd5

hdisk0 blv=hd5

hdisk0 blv=hd5

So at this
point, I’d created a second instance of AIX running 7.1. My current version of
(running) AIX was AIX 6.1. All I had to do now was reboot the LPAR and let it
restart as an AIX 7.1 system.

root@aix6 /# oslevel -s

6100-01-05-0920

root@aix6 / # shutdown –Fr

The LPAR
rebooted successfully and I found I was now running AIX 7.1, just as I’d hoped.

aix6[/] > oslevel -s

7100-00-01-1037

If I wanted
to go back to AIX 6.1, I would change my bootlist setting again and restart the
LPAR.

Now that
I’ve actually tried this method of migration, I’m not sure I’d actually use it
in its current form.

Although
the migration keeps my hostname and IP address, the file systems are not shared
between instances. Most of the target systems configuration is not retained.
For example, any user accounts I create on my AIX 6.1 system would also need to
be created on the existing AIX.7.1 system which I used to create the AIX 7.1
mksysb image. It reminds me a little of a preservation install.

Recently I
had the pleasure of configuring a couple of POWER7 720s for a customer. Each
720 was to host roughly 12 LPARs each. There were two VIO servers and a NIM
server per 720.

Everything
went along nicely and according to plan. In a few days we had both systems
built. My steps for building the systems were almost identical to those
described by Rob McNelly in a recent post.

All four
VIO servers were running the latest VIOS code i.e. 2.2.0.10-FP-24 SP-01. All the client LPARs were running AIX 6.1 TL6 SP3. Each VIOS was
configured with two FC paths to the SAN and the SAN storage device was an IBM DS5020.

Native AIX
MPIO was in use on the VIOS and the AIX LPARs. I did not deploy SDDPCM on the
VIOS as this is currently unsupported with the DS5020.

Once the
LPAR builds were complete we performed a number of “integration tests”. These typically
involve disconnecting network and SAN cables from each VIOS and observing how
the VIOS and LPARs respond to and recover from these types of conditions.

One of the
integration tests required that BOTH fibre cables be disconnected from the
first VIOS and confirm that the client LPARs were not impacted i.e. that all
I/O travelled via the second VIOS.

During the test we notice the following:

I/O on
the client LPAR would hang for approximately 5 minutes. Our test was simple
enough, create a file in a file system. Eventually the I/O would continue
after the 5 minute delay.

The VIOS
became sluggish and took some time to respond. The lspath command would take a very long time to return (or in some cases it would never
return). During one test, the VIOS actually hung and had to be restarted
(however we were not able to reproduce this issue again).

When we
reconnected both the cables, the paths would recover on the first VIOS
after another 5 minutes. On the client however, it took roughly 20 minutes
for the paths to recover. And yes, the hcheck_interval
was set to 60 on all disks.

What's even more puzzling was that if we simply rebooted the first VIOS,
everything worked as expected i.e. the client LPARs were not impacted, I/O continued
as normal and when the first VIOS was back up, the paths on the client LPARs
recovered quickly.

After
doing some research, we discovered the following post on the IBM developerWorks AIX forum:

This post
highlighted a number of things we needed to check and also confirmed several
decisions we’d made during the design process, such as SDDPCM not being
supported with DS5020 storage and VIOS (this was good as some people were
starting to believe we should have installed SDDPCM to resolve this problem.
I’d only be happy to do this if it was a supported combination and it’s not).

Finally we
found the following IBM tech note that related directly to our issue.

"For active/passive storage device, such as DS3K, DS4K, or DS5K if
complete access is lost to the storage device, then it may take greater than 5 minutes to fail I/O. This feature is for
Active/Passive storage devices, which are running with the AIX Default A/P PCM. This includes DS3K, DS4K, and DS5K family of
devices.”

The new feature was
described as follows.

“Added feature which health checks
controllers, when an enabled path becomes unavailable, due to transport problems.
By default this feature is DISABLED.
To enable this feature set the following ODM attributes for the active/passive storage
device. Enabling this feature, results
in faster I/O failure times.

cntl_delay_time:
Is the amount of time in seconds the storage device's controller(s) will be
health checked after a transport failure. At the end of this period, if
no paths are detected as good, then all pending and subsequent I/O to the
device will be failed, until the device health checker detects a failed path
has returned.

cntl_hcheck_int:
The first controller health check will only be issued after a storage fabric transport failure had been
detected. cntl_hcheck_int is the amount of time in seconds, which the next
controller health check command will be issued. This value must be less than
the cntl_delay_time(unless set to
"0",
disabled).

If you wish to allow the storage
device 30 seconds to come back on the fabric (after leaving the fabric), then
you can set cntl_delay_time=30 and
cntl_hcheck_int=2.

The device, /dev/hdisk#, must not be in use, when setting the ODM values (or
the chdev "-P" option must be used, which requires a
reboot).

CAUTION: There are cases where the storage
device may reboot both of the controllers and become inaccessible for
a period of time. If the controller health check sequence is enabled, then this
may result in an I/O failure. It is recommended to to make sure you have
an mirrored
volume
to failover to, if you are running with controller health check enabled (especially
with under 60 second cntl_delay_time). “

And as I
suspected the issue was related to the type of storage we were using. It
appears the I/O delay was attributed to the following attributes on the DS5020
hdisks on the VIOS:

After
making the changes to the hdisks on all VIOS, I performed the same test i.e.
disconnected BOTH fibre cables from the first VIOS and continued to write a
file to a file system on the client LPARs. By modifying these values on all the
DS5020 disks, on all the VIO servers, the I/O delay was reduced to seconds
rather than five minutes!

The following
attributes were used for the hdisks and adapters in the final configuration.

-j identifies the volume group that will be
used on the NIM master to create file systems.

-Y Agrees to required software license
agreements for software to be installed.

-N specifies the name of the new AIX 7.1 NIM
mksysb resource to be created after migration.

Before running this
command I made sure that I had installed the bos.alt_disk_install.rte fileset into my AIX
7.1 SPOT. If this fileset was not installed I would have received the following
message immediately after issuing the nimadm
command:

0042-001 nim: processing error encountered on
"master":

/usr/bin/lslpp: Fileset bos.alt_disk_install.rte not installed.

0505-204 nimadm:
SPOT spotaix7tl0sp2 does not have bos.alt_disk_install.rte installed.

0505-205 nimadm:
The level of bos.alt_disk_install.rte installed in SPOT

spotaix7tl0sp2 (0.0.0.0) does not match the
NIM master's level (7.1.0.2).

Cleaning up alt_disk_migration on the NIM
master.

I also had an AIX 7.1
lpp_source and SPOT ready and waiting on my NIM master.

root@nim1 / # lsnim -t lpp_source

aix61tl6sp3resourceslpp_source

opensshresourceslpp_source

aix7tl0sp2resourceslpp_source

root@nim1 / # lsnim -t spot

vio1-spotresourcesspot

spotaix61tl6sp3resourcesspot

spotaix7tl0sp2resourcesspot

OK, back to my nimadm operation. Upon initiating the
migration I saw the following output on my screen (click here
for a copy of the complete log file):

I didn’t encounter any
issues with my nimadm operation but
if I had I could have reviewed the nimadm
log file located in /var/adm/ras:

root@nim1 /var/adm/ras/alt_mig # ls -ltr

total 856

-rw-r--r--1 rootsystem435009 Jan 06 11:00 cg-aix61_alt_mig.log

If you ever encounter
problems with nimadm, I highly
recommend that you re-run the operation with the –V and –D flags. This will
enable verbose output, as well as debugging information which can assist you
greatly in determining the root cause of your issue.

I was able to use the
new, migrated, AIX 7.1 mksysb image to install a new LPAR.

By the way, if you are
interested in migrating from AIX 4.3 to AIX 7.1, then you’ll need to consider
the mksysb migration method. I’ve not
done this myself (i.e. 4.3 to 7.1 migration) but the feedback I received from
the AIX development team indicates that it does work as expected.

“... you had expressed some interest
in mksysb migration. Since we were not aware of anyone having attempted
it with 7.1. I went ahead and created a 4.3.3.75 system and created a
mksysb and then used that mksysb in conjunction with nim and successfully
installed another system using the 4.3 mksysb and 7.1 nim resources to perform
a mksysb migration to 7.1. I just followed the steps as noted in the
Installation guide for mksysb migration. “

Refer to the following
document for further information on mksysb migration with NIM.

I received an email this week from a
colleague that worked with me on the NIM
Redbook back in 2006. He was experiencing an issue with DSM and NIM. He was
attempting to use the dgetmacs
command to obtain the MAC address of the network adapters on an LPAR. The
command was failing to return the right information.

I experienced this very issue during
the writing of the AIX 7.1 Differences
Guide Redbook. And given that I was in Austin, sitting in the same building
as the AIX development team, I was able to speak with the developers directly
about the issue. At that time they provided me with the following workaround.

First they asked me to check the size
of the /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat message
catalog file.

# ls –l /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat

-rw-r--r--1 binbin3905 Aug 08 09:54

They were surprised to find that the
file appeared to be “too small”. They promptly sent me the catalog file from
one of their development AIX 7.1 systems.I replaced the file as follows:

# cd/usr/lib/nls/msg/en_US/

# ls -ltr IBMhsc*

-rw-r--r--1 binbin3905 Aug 08 09:54
IBMhsc.netboot.cat

# cp -p IBMhsc.netboot.cat
IBMhsc.netboot.cat.old

# cp /tmp/lpar1/IBMhsc.netboot.cat.new IBMhsc.netboot.cat

# ls -ltr IBMhsc*

-rw-r--r--1 binbin3905 Aug 08 09:54
IBMhsc.netboot.cat.old

-rw-r--r--1 binbin26374 Dec 23 11:24 IBMhsc.netboot.cat

This fixed the problem for me during
the residency.

So I asked my friend to do the same
(after I sent him the message catalog file). He ran the dgetmacs command again and this time it returned the MAC address
for all the network adapters in his LPAR. Success!

A colleague of mine contacted me during the week to ask how one could
determine if a POWER7 system was capable of Active Memory Expansion. I thought
I'd share my response with everyone who follows my blog.

You can check if the system is capable of providing Active Memory
Expansion (AME), from the HMC command line using the lssyscfg command, as shown here:hscroot@hmc1:~>
lssyscfg -r sys -m Server-8233-E8B-SN1000 -F active_mem_expansion_capable1

Alternatively you can view the capabilities of
the managed system from the HMC GUI, under System properties/Capabilities, as
shown in the following image.

Once you’ve concluded that your POWER7 system is
AME capable, the next step is to check if AME is enabled or disabled for
an LPAR. This is easily done from the AIX command line with the lparstat and/or amepat commands, as shown in the following example:;Running
lparstat and amepat on a LPAR with AME
disabled.

Note
that I deliberately ran these commands as a non-root user to highlight the fact
that you don’t need root access to ascertain whether or not AME is active on an
LPAR.

By the
way, did you know that AME is now supported in SAP production environments running
on AIX and POWER7? Both SAP application servers, as well as database servers
running DB2 LUW are supported. Unfortunately there’s no support for LPARs with
Oracle RDBMS at this stage. Thanks to George Manousos from IBM Australia for
providing me with this update.

As AME is not
recommended with large page support, AME
will disable AIX

64KB pages by default.

AME monitoring
capability in CCMS

Monitoring
capabilities are available with saposcol v12.46 for more details see SAP note
710975.

It looks as
though SAP have been quick to support AME on AIX/POWER7. This is a great
benefit to SAP customers running with PowerVM on the IBM POWER platform. The
following screenshot shows the SAP CCMS with AME statistics being reported.

The SAP AIX porting team continues
in further improving the integration of PowerVM into SAP system monitoring. While
in the past already most processor and AIX virtualization metrics could be
monitored via CCMS, now POWER7 Active Memory Expansion (AME) has been
included. Customers will find a new memory section in the respective
CCMS-panel as depicted in this screen shot.

For those
who are not familiar with what AME actually is and how it can benefit you, I
suggest you take a look at the IBM AME Wiki site first: