Someone installs SAP onto an AIX system and decides to use TCP
port 3901 as an SAP service port. This is the same port used by nimsh. In some
rare cases, nimsh may not be active on the LPAR, which makes it easy for the
SAP installation to hijack port 3901. If nimsh is active, the person installing
SAP may consciously stop nimsh and use port 3901 for SAP anyway. Hopefully that
doesn’t happen. Hopefully, they will talk to the AIX administrator and discuss
the best way forward. Hopefully...

In either case, if the port is taken by SAP, nimsh will no longer
work. If you love using NIM as much as I do, this is a real problem! We could
revert back to using rsh but no-one will do this anymore because of concerns
around security. And rightfully so!

The ports used by nimsh (3901 and 3902) are registered to Internet
Assigned Number Authority (IANA). These port numbers appear in the
/etc/services file.

nimsh3901/tcp# NIM Service Handler

nimsh3901/udp# NIM Service Handler

nimaux3902/tcp# NIMsh Auxiliary Port

nimaux3902/udp# NIMsh Auxiliary Port

Considering these port numbers are registered with IANA, we can usually
persuade our SAP colleagues to change their SAP installation to use a different
port number. However, depending on the skills/experience of the SAP resource,
one of two things usually happens 1) They take an outage, re-install SAP and
choose a different port number or 2) The more experienced/confident SAP basis
resource will take an outage and modify the instance to use a different port:
without reinstalling SAP.

Perhaps SAP need to include a warning in their install notes,
advising customers not to use port 3901 on AIX systems (i.e. best practice)?

Now, if you must change nimsh to use a different port number, it
is possible. But not recommended.

To do this, you must change the /etc/services file on the NIM master
and the NIM client to reflect the same port numbers for nimsh. This will work
until the NIM master or the NIM client have their services file overwritten by
way of install or fileset updates. After which, the default values for nimsh
will be reinstated.

You would also need to change the services file on all of your NIM clients. Every time you
performed a NIM fileset update, you would need to remember to change the /etc/services
file again. This is painful and bound to catch someone out eventually!

In the following example I’ll demonstrate how to change the port
number used by nimsh.

We start with a typical nimsh configuration using port 3901. On
the NIM client, nimsh is listening on port 3901.

We can confirm that we have connected to the NIM client on port
39011 by looking at the output from lsof
and netstat. There is a TCP session
established between the master and the client on port 39011.

I was performing a volume group re-org i.e. changing the
INTER-POLICY of a logical volume from minimum to maximum.

# lslv
fixeslv| grep INTER

INTER-POLICY:minimumRELOCATABLE:yes

# chlv
-e x fixeslv

# lslv
fixeslv| grep INTER

INTER-POLICY:maximumRELOCATABLE:yes

I attempted to run the reorgvg
command. I was greeted by the following error message!

# reorgvg
tempvg fixeslv

0516-966 reorgvg:
Unable to create internal map.

I ran the command again, this time with truss. I found that the /usr/sbin/allocp
command was being called and was failing. I determined this must be because of
a lack of space at the logical volume layer.

#
/usr/sbin/allocp -?

/usr/sbin/allocp:
Not a recognized flag: ?

0516-422
allocp: [-i LVid] [-t Type] [-c Copies] [-s Size]

[-k] [-u UpperBound>] [-e
InterPolicy] [-a InterPolicy

The truss output showed:

statx("/usr/sbin/allocp", 0x2FF21ED8, 76, 0)= 0

statx("/usr/sbin/allocp", 0x20009E70, 176, 020) = 0

kioctl(2, 22528, 0x00000000, 0x00000000)Err#25
ENOTTY

kfork()=
3735812

_sigaction(20,
0x00000000, 0x2FF21F20)= 0

_sigaction(20,
0x2FF21F20, 0x2FF21F30)= 0

kwaitpid(0x2FF21F90,
-1, 6, 0x00000000, 0x00000000) = 3735812

And yes, my volume group was indeed out of free PPs!

# lsvg
tempvg

VOLUME
GROUP:
tempvg
VG IDENTIFIER: 00f6027300004c0000000130773bdb73

VG
STATE:
active
PP SIZE: 512 megabyte(s)

VG
PERMISSION: read/write
TOTAL PPs: 99 (50688 megabytes)

MAX
LVs:
256
FREE
PPs: 0 (0 megabytes)

LVs:
1
USED PPs: 99 (50688 megabytes)

OPEN
LVs:
1
QUORUM: 2 (Enabled)

TOTAL
PVs: 2
VG DESCRIPTORS: 3

STALE
PVs:
0
STALE PPs: 0

ACTIVE
PVs:
1
AUTO
ON: yes

MAX
PPs per VG: 32768
MAX PVs: 1024

LTG
size (Dynamic): 256
kilobyte(s) AUTO
SYNC: no

HOT
SPARE:
no
BB POLICY: relocatable

PV
RESTRICTION: none

cgaix7[/opt]
>

Silly me, it clearly states in the reorgvg man page that there must be at least one free PP in the
volume group for the command to run successfully.

2At least one free physical partition
(PP) must exist on the specified volume group for the reorgvg command to run
successfully. For mirrored logical volumes, one free PP per physical
volume (PV) is required in order for the reorgvg command to maintain logical
volume strictness during execution; otherwise the reorgvg command still runs,
but moves both copies of a logical partition to the same disk during its
execution.

So I shrunk the file system in question (there was a large amount
of allocated but unused file system space, so it was safe to shrink it).

I can see when my reorgvg
failed (rc=1) and when it succeded (rc=0). This is also a good way of
determining when a reorgvg command
is issued and when it finished. Of course, an easier way would be to start the reorgvg command with the time command. It will produce a nice
little summary of the time taken.

# time
reorgvg tempvg fixeslv

0516-962
reorgvg: Logical volume fixeslv migrated.

real3m12.94s

user0m1.52s

sys0m4.60s

But if I forgot to use the time
command, I can look at the lvmcfg alog file for an answer. In the following
example, the reorgvg.sh process is
started at 23:49. The entry in the log file begins with an uppercase S. The entry that starts with an
uppercase E indicates the end of the
reorgvg.sh process.It is the information in the third field that
tells me how long the process ran for in seconds:milliseconds.

Before
the change, I was unable to compile anything. Note the highlighted text below.

#
/usr/vac/bin/xlc cgc.c

The
license for the Evaluation version of IBM XL C/C++ for AIX V11.1 compiler
product has expired. Please send an email to compiler@ca.ibm.com
for information on purchasing the product. The evaluation license can be
extended to 74 days in total by either a) setting an environment variable XLC_EXTEND_EVAL=yes; or, b) specifying a compiler command line option
-qxflag=extend_eval. The extended evaluation license will expire on Sat
Jun 18 10:26:33 2011. Use of the Program continues to be subject to the terms
and conditions of the International License Agreement for Evaluation of
Programs, including the accompanying License Information document (the
"Agreement"). A copy of the Agreement can be found in the
"LicAgree.pdf" and "LicInfo.pdf" files residing in the root
directory of the installation media. If you do not agree to the terms and
conditions of the Agreement, you may not use or access the Program.

With
the environment variable in place, I was able to compile again.

#
/usr/vac/bin/xlc cgc.c

#
./a.out

Hello
World!

Of
course I could also export the environment variable as required instead e.g.

#
export XLC_EXTEND_EVAL=yes

#
/usr/vac/bin/xlc cgc.c

#
./a.out

Hello
World!

If
you do place this environment variable in /etc/profile, you may need to restart
any processes on the system that need to call the compiler.