AIX performance tool enhancements - Filtered output and XML reports

High End
POWER7 systems, like the POWER7 795, support configurations of up to 256
physical processors. Most of us can only dream of working on a system of this
size. For those of us that are fortunate enough to play with these big boxes,
you may find collecting and reviewing CPU related performance data a challenging
task.

With 256
physical POWER7 processors, where each processor has 4 threads enabled (SMT 4),
there will be up to 1024 logical CPUs active on the system. As a consequence, a
lot of performance monitoring data will be generated, which presents challenges
to the OS to filter and effectively present relevant performance data to the
administrator.

Many AIX
performance tools provide per-CPU statistics for the administrator to analyse
application and system performance. With the introduction of 1024 logical CPUs
on a system, these tools are impacted. All standard AIX performance tools have
been modified in AIX 7.1 (and AIX 6.1 TL6) to support AIX partitions running as
many as 1024 logical CPUs on a POWER7 system.

The
following tools have been modified to support the increase in logical CPUs and
now provide options for filtering and sorting data: mpstat, sar and topas.

Both the mpstat and sar commands have been updated to allow for the sorting and
filtering of output. The –O flag provides
this new feature. The following options can be supplied to the –O flag (from the man page):

-OOptions

Specifies the command option.

-O options=value...

Following are the supported
options:

*sortcolumn = Name of the
metrics in the mpstat command output

*sortorder = [asc|desc]

*topcount = Number of CPUs
to be displayed in the mpstat command sorted output

For example,
to see the sorted mpstat output for
the cs column you would enter the
following command.

# mpstat -d
-O sortcolumn=cs 1 3

Another
example, to see the list of
the top 10 CPUs, you would enter the following command.

# mpstat -a
-O sortcolumn=min,sortorder=desc,topcount=10 1 3

Some
examples of this new option with the sar
command are shown below.

-OOptions

Allows users to specify the command
option.

-O options=value...

Following are the supported
options:

*sortcolumn = Name of the
metrics in the sar command output

*sortorder = [asc|desc]

*topcount = Number of CPUs
to be displayed in the sar command sorted output

To display the sorted sar output for the column cswch/s
with the -w flag, you would enter the following command:

# sar -w -P
ALL -O sortcolumn=cswch/s 1 3

To list the top ten CPUs, sorted on
the scall/s column, you would enter the following command:

# sar -c -O sortcolumn=scall/s,sortorder=desc,topcount=10 -P ALL 1 3

Some
performance tools have also been enhanced to provide the capability to generate
XML reports. The following tools now have this feature: sar, mpstat, vmstat, iostat and lparstat.
Specifying the –X flag with these
commands will produce XML output. You can specify an output file with the –o flag.
If you don’t specify an output file, a default file is generated with the
following naming convention, command_DDMMYYHHMM.xml.

Here are
some examples of using the –X flag
with the updated commands:

# lparstat
-X

# sar -X 1
5

# mpstat -X
1 5

# vmstat -X
1 3

# iostat -X
1 3

*** stack
smashing detected ***: program terminated

IOT/Abort
trap(coredump)

#

# ls -lr
*.xml

-rw-r--r--1 rootsystem50027 Nov 25 22:00 vmstat.xml

-rw-r--r--1 rootsystem50852 Nov 25 21:59
sar_2511102159.xml

-rw-r--r--1 rootsystem44570 Nov 25 21:59
mpstat_2511102159.xml

-rw-r--r--1 rootsystem18926 Nov 25 21:59
lparstat_2511102159.xml

In my tests
I noticed a couple of things. First the vmstat
output is named vmstat.xml, this
doesn’t match the expected naming convention. This could be a bug? Also, the iostat command dumped core. This is
meant to be supported so this was totally unexpected. And yes I was running the
latest service pack for AIX 7.1 (7100-00-01-1037). This could also be a bug. Anyway,
I digress!