IBM released the AIX 6
Open Beta in July 2007. This is the first time IBM has allowed customers to
download a beta release of their AIX operating system. IBM state they have
taken this approach to give customers a chance to gain experience with the new
OS and to help IBM make improvements, prior to the official release of AIX 6.1.

Several new enhancements
have been introduced in this beta release. They include Workload Partitions, AIX
Security Expert, Role Based Access Control (RBAC), Dynamic Tracing, IBM Systems Director Console for AIX and the Name Resolver
Caching Daemon. The area of greatest interest to me was Workload Partitions
(WPARs). They are similar in concept to Solaris Zones, in that they partition a
single instance of the Operating System into multiple virtual instances of the
OS. WPARs allow for the usage of separate virtual partitions with only one AIX
image. This provides you with the flexibly of having separate environments
without the burden of having to manage separate images of AIX. This could be
useful at sites with large numbers of AIX OS instances (such as ours, which has
over 150 real instances of AIX to manage).

Since downloading and installing
the AIX 6 beta, Iíve focused my attention on testing WPARs. In this paper, I've
captured as much information as I could from my testing (so far), in order to
provide an overview of WPARs, describing the differences between stand-alone
AIX and WPARs and the various types of WPARs i.e. system and application. Also covering the basics, such as creating a WPAR,
starting/stopping a WPAR, configuring networking, making software/applications
available to a WPAR and managing file systems within a WPAR. I'll also
touch on the 'Live Application Mobility' feature which allows you to relocate
live WPARs from one physical server to another physical server.

AIX 6 Open Beta.

AIX 6 is binary
compatible with previous releases of AIX including AIX 5.3, 5.2 and 5.1. AIX 6
includes new capabilities for virtualization, security, continuous availability
and manageability. The open beta can run on any IBM System p or pSeries system
that is based on POWER4, PPC970, POWER5 or POWER6 processors. You can download
the AIX 6 Open Beta from the following site:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/ibm/aix6beta/

The key features included
in the beta release are:

- Workload Partitions introduce
a new, software-based, virtualization of AIX. This complements the existing IBM
System p LPARs by reducing the number of operating system images that have to
be managed when consolidating workloads. WPARs allow the system administrator
to consolidate multiple instances of AIX within a single running instance of
AIX 6 on an LPAR. The LPAR can consist of real or virtual I/O adapters and
dedicated or shared processors.

- Role Based Access
Control (RBAC) enables administrators to grant authorization for management of
specific AIX resources to users other than root by associating those resources
with a role that is then associated with a particular system user. RBAC can
also be used to associate specific management privileges with programs, which
can reduce the need to run those programs under the root user or via setuid. Another security enhancement is the 'AIX Security Expert'.
This was introduced with AIX 5.3 TL 5. It provides the capability to manage
over 300 system security settings from a single interface (smit). In AIX 6,
this tool has been enhanced with an option to store security templates directly
in a LDAP which may simplify implementation of consistent security across many
AIX systems.

- AIX 6 provides a new
dynamic tracing capability that can simplify debugging complex system or
application code. This dynamic tracing facility, known as Ďprobevueí will allow
a developer or system administrator to dynamically insert trace breakpoints in
existing code without having to recompile the code. I have no idea why IBM
didn't take up the offer to incorporate Dtrace into AIX (http://blogs.sun.com/bmc/entry/dtrace_on_aix), but probevue hasn't made
it to the beta release as yet. Hopefully it will appear in a future release of
the beta.

- Name Resolver caching
daemon. The network resolver caching daemon caches requests to resolve a
hostname, service or netgroup to improve the efficiency of subsequent requests
for the same information. Doesnít seem like a big deal but I guess itís new so
itís worth mentioning?

Installing the AIX 6 Open Beta.

For the purposes of my
testing, I installed the beta onto a Power4 p650 LPAR. The LPAR had the
following hardware configuration:

System Model: IBM,7038-6M2

Processor Type: PowerPC_POWER4

Number Of Processors: 1

Processor Clock Speed: 1452 MHz

CPU Type: 64-bit

Kernel Type: 64-bit

LPAR Info: 1 uataix6

Memory Size: 4096 MB

Platform Firmware level: 3K041029

# lsdev -Cs scsi

cd0††† Available
1Z-08-00-1,0† SCSI DVD-RAM Drive

hdisk0 Available 1Z-08-00-8,0† 16 Bit LVD SCSI Disk Drive

hdisk1 Available 1Z-08-00-9,0† 16 Bit LVD SCSI Disk Drive

You have the option of
either downloading the beta as a single DVD image or as 3 CD images. I
initially tried installing the beta via CD but had issues with the 3rd install
CD. After downloading the DVD image and burning it to a DVD (with Nero), I was
able to perform a successful installation. Installing the beta is not very
different from a typical AIX installation via media. The beta image is
basically a mksysb which you install. The 'AIX
6 Open Beta Getting started' guide provides step by step information on
installing the beta. It is important to note that (at this stage) NIM
installations are not supported. Also migration installations are also not
supported and TCB is not enabled in order to support the WPAR functionality.

If you need more
assistance or information on installing the beta, please review the information
at the following sites:

You will find that the
User Forum may already have answers to some of the common issues with the beta
release.

After the beta has
finished installing, I verified the level of beta code with the oslevel
command:

# oslevel -r

6000-01 BETA

It is worth noting that
new fixes and functionality for the beta will be released as completely new ISO
images which will need to be downloaded and installed. The initial open beta
images are referred to as "Beta Driver 6000-01". The next image would
be 6000-02, and so on. Individual fixes for problems are not being provided.
New release notes, outlining new fixes and functionality, will be included with
the new images when they become available.

Perform the following, to
avoid any issues using the beta image (these issues are also listed in the
release notes):

# chmod 1777 /tmp

# rm /etc/resolv.conf †

# rmfs /maria

During the first boot of
the beta image, you may see the system 'hang' at LED 581 for a while. Fixing
some of the issues above will prevent this from occurring again.

Now that the beta is
installed, I can set about testing the WPAR technology.

Workload Partitions.

A WPAR is a software
created, virtualized OS environment within a single AIX 6 image. Each workload
partition is a secure and isolated environment for the application it hosts.
The application in a WPAR thinks it is being executed in its own dedicated AIX
instance. The AIX operating system instance that hosts the workload partition
is known as the Global environment. Creating WPARs within an LPAR does not
restrict the use of the hosting AIX instance. It is possible to log into the
global environment and perform normal AIX functions or run applications. It is
no different to a stand-alone instance of AIX. This global environment can be
hosted within a dedicated LPAR or a micropartition, using real or virtual I/O
adapters. All WPAR administration must be performed from the global
environment. WPARs cannot be created within other WPARs. The global environment
owns all physical resources of the LPAR i.e. network adapters, disks adapters,
disks, processors, memory. It allocates CPU and memory resources to the WPARs
and provides access to the network and storage devices. It is possible from the
global environment to see (and control) the processes executing within the
WPARs, and to see the file systems used by the WPARs. Most performance
monitoring and tuning activities are performed from the global environment.

To most applications the
WPAR appears as a normal AIX environment. In general, applications can run
without modification in a WPAR. Applications within a WPAR have private
execution environments, are isolated from other processes outside the WPAR, have
their own signals and file systems, have dedicated network addresses and they have
inter-process communication which is restricted to processes executing in the
same WPAR.

There are two types of
workload partitions that can reside in a global environment i.e. System and
Applications WPARs. A System WPAR looks and feels like a complete instance of
AIX. An Application WPAR is a light weight environment suitable for execution
of one or more processes.

A system WPAR is similar
to a normal AIX environment. Each System WPAR has dedicated writable file
systems, although by default it shares the global environment /usr and /opt file
systems in read only mode. When a system WPAR is started, an init process is created for this WPAR,
which in turns spawns other processes and daemons. For example, a system WPAR
contains an inetd daemon to allow
complete networking capability, making it possible to remotely log into a
system WPAR. It also runs a cron daemon, so that execution of processes can be
scheduled.

In an application WPAR,
an application or group of applications can be started with one command. This
command is passed as an argument to the wparexec
command (which creates the application WPAR). As soon as the command exits, the
WPAR is terminated. An application WPAR shares the file system of the global
environment. It does not have itís own file systems nor does it have itís own
network environment. An application partition can run daemons. But application
partitions will not run any of the system service daemons, such as inetd, srcmstr, etc. It is not possible
to remotely log into or remotely execute a command on an application WPAR.

The following figure
depicts the global environment with several WPARs within a single AIX instance.

What follows is a list of
some of the potential advantages I can see when using WPARs:

- Consolidation
(Virutalisation). You can consolidate workloads into one LPAR and instance of
AIX. Using WLM you could host different workloads within a single system. This
means less LPARs (hardware) and instances of AIX to manage. For example, you
could consolidate your development and test LPARs into one physical (or virtual
I/O) LPAR.

- Workload Isolation.
Using WPARs and WLM you can consolidate workloads on a single system without
any impact to the other workloads running on that system.

- Security. Each WPAR can
have itís own security configuration based on itís specific needs. One WPAR
cannot interfere with or access another WPAR within the same system. Also,
using the new RBAC features of AIX 6 it is possible to provide access to
certain secure functions for each WPAR e.g. allowing users of a WPAR the
ability to reboot the WPAR or to gain root access to all or some privileged
commands (without compromising the security of the other WPARs running in the
LPAR).

- One installation of an
application can support multiple workloads (WPARs). Reduce number of software
installations.

- Quickly create a WPAR
for testing or development purposes. No need to acquire new hardware or to
install another copy of AIX (or an application). Does not impact workload in
other WPARs. WPARs can be built in a few minutes.

- Separation of admin
roles, based on WPARs. Using, root (privileged) commands can be given to users
without impacting the global environment or other WPARs.

- Live Application
Mobility (LAM). Can move live WPARs (applicatons/services) from one physical
server to another. Good for large outages like microcode upgrades or AIX
maintenance. But is this practical in itís current
incarnation?

So far so good. WPARs are
certainly a useful and flexible way to deploy and manage AIX systems. There a
few things to consider however:

- Actions in the global
environment can impact a WPAR i.e. unmounting a WPAR file system.

Note: You can use the uname ĖW command to determine if you are
in the global environment (returns 0) or within a WPAR (returns non-zero).

- WPAR mobility requires the
purchase of additional software (the WPAR Manager). It also relies on an NFS
server (which would need to be highly available). It may require the purchase
of additional server hardware (utility server) for the WPAR Manager (and may be
more than one server for it to be highly available Ė possibly clustered with
HACMP, at additional cost?). More on WPAR mobility later.

- Software is updated in
the global environment only. The good news is the once the software is updated
in the global environment, all the WPARs are also updated. The bad news is that
all the WPARs are also updated! For example, if you apply AIX fixes to the
global environment, the WPARs will also receive the fixes (via the shared /usr and
/opt file systems). This means that all the WPARs applications must be able to
support the same level of AIX. This may, or may not, be an issue depending on
your application requirements. If you are applying fixes for DB2, again all the
WPARS will receive the fixes. Again this may be an issue for you and the
ramifications should be considered first. However, if you wish to install a
different version of a software product, it may be possible to install the software
to a different location than the default. This will allow you to have multiple
versions of the software in the global environment, allowing each WPAR to run a
different version of the application if desired. This makes testing new
versions of software easy as you can continue to use the old version in the other
WPARs while you test it one WPAR only.

- NFS servers are not
supported within WPARs. Also, itís not advisable to export WPAR file systems
from the global environment because as soon as you stop the WPAR the file
system will be unmounted and no longer accessible.

- To create a System
WPAR, the following disk space will be required (either in rootvg or another
volume group).

- 512MB in rootvg
(default location) per WPAR.

- Non-shared /usr + /opt:
2.5 to 3G per WPAR.

It also is advisable to
create a new file system called /wpars
(the default root directory for WPAR mount points) so that it is separate from
the / (root) file system.

You will see the name corrals
sometimesmentioned
when working with WPARs. Makes me wonder if WPARs were originally going to be
known as something else?

Creating a System WPAR.

Creating a WPAR is fairly
easy. In the following example, I create a WPAR (using the mkwpar command) called wpar1, give it an IP address of 10.1.1.81 and
specify en1 as the real network
interface on which an IP alias will be created for this new WPAR. The creation
process takes a few minutes.

In the following examples
I demonstrate how I made several software applications available to the WPARs.
Software is installed via the Global environment. Since creating my first WPAR
Iíve created another WPAR named wpar2. Both wpar1 and wpar2 will be used in the
following examples. I installed the following software: OpenSSH, Sudo, Apache
and DB2 V8.

Installing OpenSSH and Sudo.

OpenSSH was installed
into the global environment via installp and Sudo was an RPM install. After
performing a standard installation of both utilities, I ran the syncwpar command to synchronise the
software in the global environment with the WPARs.

To start the workload partition, execute the following as
root: startwpar [-v] 'wpar1'

I reported this problem
on the AIX 6 Beta User Forum. The AIX dev team promptly replied stating this
was a known issue (602196). Until a fix is available, you can work around it by
exporting INUBOSTYPE=1 before running mkwpar. I also discovered that if you run
syncwpar, after starting the WPAR, that the fileset installs correctly and the
issue is resolved.

From within in the WPAR I
could now see the OpenSSH and Sudo packages.

I now have two WPARs,
both with different IP addresses, running distinct instances of httpd. This was fairly impressive
considering the short amount of time it had taken to setup and configure the
environment.

As a final example, I
installed DB2 and created a DB2 instance on wpar1. I followed the installation
documentation for DB2 V8 and installed it in the Global environment first. A
typical installation installs all of the necessary utilities for configuring
DB2 8 in a WPAR. Documentation for installing DB2 V8 on AIX can be found here:

http://publib.boulder.ibm.com/infocenter/db2luw/v8

After you install DB2 V8,
you can make it available to other WPARs with the syncwpar command.

I found that I needed to create
a nodelock file within the WPAR and copy the contents of the nodelock file from
the global environment. Without it, db2 would not start, claiming it did not
have a valid product license.

WPAR resource control provides
methods for controlling CPU and memory resource allocation to WPARs. This can
help to minimise the impact each WPAR can have on each other, in terms of
resource utilisation and performance. WPAR resource control is based on WLM. However
the inner workings are hidden from the system administrator and they do not need
to have in-depth knowledge of WLM in order to use WPAR resource control. For
people who found WLM confusing and hard to manage, WPAR resource control makes
it very easy.

WPAR resource control can
be used to specify the amount of CPU and memory resources allocated to a WPAR, the
number of processes and threads that are allowed in a WPAR and the amount of
virtual memory that a single process within a WPAR can consume.

A working example:

There are two ways to
control the usage of the CPU: share based and percentage based approach. It is
recommended to start with the share based approach to control the resources. In
almost all cases, it should satisfy your requirements. The share based approach
means that you are allocating the available resource to the workload partitions
based on their relative importance. WPAR resource control settings can be adjusted
dynamically.

In the following example
Iím using CPU shares:

80% for wpar1 (20/25)

20% for wpar2 (5/25)

# lswpar -R

=================================================================

wpar1 - Active

=================================================================

Active:††††††††††††††††††††††††††† yes

RSet:

CPU Shares:††††††††††††††††††††††† 20

CPU Limits:††††††††††††††††††††††† 0%-100%,100%

Memory Shares:†††††††††††††††††††† unlimited

Memory Limits:†††††††††††††††††††† 0%-100%,100%

Per-Process Virtual Memory Limit:† unlimited

Total Processes:†††††††††††††††††† unlimited

Total Threads:†††††††††††††††††††† unlimited

=================================================================

wpar2 - Active

=================================================================

Active:††††††††††††††††††††††††††† yes

RSet:

CPU Shares:††††††††††††††††††††††† 5

CPU Limits:††††††††††††††††††††††† 0%-100%,100%

Memory Shares:†††††††††††††††††††† unlimited

Memory Limits:†††††††††††††††††††† 0%-100%,100%

Per-Process Virtual Memory Limit:† unlimited

Total Processes:†††††††††††††††††† unlimited

Total Threads:†††††††††††††††††††† unlimited

I started some CPU
intensive jobs on both WPARs and monitored it from the global environment with topas:

Some AIX tools have been
changed to cater for WPARs. In particular, commands relating to system status
and performance have been modified. e.g.†
ps and topas are two examples already shown in previous examples, both of
which now take the @ symbol as a new
flag for displaying WPAR information.

Also, the topas ĖC (Cross Partition monitor) command
can monitor WPARs e.g. on my p650 I could see both my WPARs.

Along with the lswpar
command, you can also use lssrc to see if your WPARs are active.

# lssrc -a | grep cor

†cor_wpar1†††††††††††††††††††††††† 655592†††††† active

†cor_wpar2†††††††††††††††††††††††† 462974†††††† active

The wlmstat command also takes the @
symbol as flag for displaying WPAR information in relation to resource usage
and WLM.

# wlmstat -@ 1

† CLASS††† CPU†††
MEM†† DKIO

† wpar1†† 0.15††
0.03†† 0.00

† wpar2†† 0.15††
0.01†† 0.00

† TOTAL†† 0.30††
0.04†† 0.00

† CLASS††† CPU†††
MEM†† DKIO

† wpar1†† 0.46††
0.03†† 0.00

† wpar2†† 0.20††
0.01†† 0.00

† TOTAL†† 0.66††
0.04†† 0.00

† CLASS††† CPU†††
MEM†† DKIO

† wpar1†† 0.41††
0.04†† 0.00

† wpar2†† 0.21††
0.01†† 0.00

† TOTAL†† 0.62††
0.05†† 0.00

Special Files and Directories
related to WPARs (corrals?).

Iíve listed the files
here purely to make you aware that they exist. The IBM Redbook on WPARs (see
the references section) contains detailed information about each of these files
if you need more information.

/etc/rc.wpars

/etc/wpars/

/var/adm/wpars/

/wpar

/etc/wpars/devexports

WPAR Administration via pConsole.

IBM have introduced a new
Web interface to SMIT and several other management and administration tasks.
This new interface is known as pConsole and it can be used to manage and
administer WPARs. The release notes for the beta contain extra information on
pConsole. See the document 'Open_Beta_Driver_Release_Notes_6.0.0.0-11'.

Unfortunately, Live
Application Mobility (also known as WPAR mobility) is not available for testing
under the AIX 6 Open Beta program, so I can only offer some descriptive information
on what it is supposed to do and offer up my opinion on itís
feasibility.

Live Application Mobility
refers to the ability †to
relocate a WPAR from one physical system to another. It is also referred to as
WPAR mobility or WPAR relocation. An additional Licensed Program Product,
called Workload Partitions Manager for AIX, is required to enable this feature.
Thatís right, itís not part of the base AIX 6 operating system. This is
disappointing as it is probably the most anticipated feature of AIX since the
ability to shrink file systems! Why we should have to pay for this has got me
beat! There are 3 components required: The WPAR Manager server (a Java
application that runs on the AIX system), the WPAR management agent (that runs
in the global environment of the target WPAR and a web browser (with a network
connection to WPAR Manager).

The WPAR Manager uses a
checkpoint and restart feature to move WPARs. The checkpoint saves the current
status of the application and then restarts it on a new system or
operating-system instance at the previously saved state. For this to work
successfully, the systems must be at the same
AIX level and hardware/processor type.
The WPAR checkpoint feature lets you capture a snapshot of an application
as it runs. This snapshot is saved in a file that you can use to restore the
state of the application and then resume itís execution.

The premise here is to
allow for planed migrations of workloads from one system to another without
interrupting the application. This could be used for example to perform a
planned firmware installation on a server. Applications do not need to be aware
that a WPAR is being or has been relocated. But IBM urges you to perform proper
planning and testing before attempting a relocation of a production system!

The NFS dilemma!

The relocation of a WPAR
consists of moving its executable code from one LPAR to another one, while
keeping the application data on the same storage devices. It is therefore
mandatory that these storage devices are accessible from both the source and
target LPARs hosting the WPAR. In the initial version of AIX 6, this dual
access to the storage area is provided via NFS. All files that needs to be
accessed by the application must be hosted on NFS. All other files, including
the AIX operating systems files can be stored in file systems local to the
hosting global environment.

The following figure
depicts WPAR mobility and the necessary infrastructure and components to enable
itís functionality.

The following table
highlights the need for NFS within a System WPAR to support WPAR mobility. Both
System and Application WPARs require the application specific files be hosted
on NFS.

The† WPAR Manager requires the following
pre-requisite software for it to function: IBM AIX Version 6.1, Java version 5
and DB2 version 9.

Some considerations with
respect to WPAR mobility:

I was very excited when I
first heard about moving live WPARs between physical systems. The advantages
and flexibility it could provide were amazing, on paper. Like I said, I canít
test any of this yet but I must say I †have a few concerns with the
implementation and design.

The first concern is that
it relies on NFS. NFS is slow and at times unreliable. Most organisations are very
unlikely to host business critical applications via NFS and itís these critical
systems that would benefit the most from WPAR mobility! Performance is a big
factor. Iíd never recommend NFS as an ideal candidate for blindingly fast I/O
performance, so I canít see myself putting a large database on NFS hosted file
systems.

The second concern is the
time it takes to Ďfreezeí a WPAR before migrating itís workload to another
system. According to the WPAR documentation a snapshot of the WPAR is taken by
interrupting the processes within it. The process context is saved to a state
file. The checkpoint state file is written to shared
storage via NFS. To restart the WPAR on the other system, first the state file
is loaded (via NFS) and then the WPAR is restarted. Sounds simple enough? But
think about a system that has a large amount of memory, like a WebSphere
Application Server (Java) or DB2 Database Server. Even with a fast Gigabit
network, the time required to write the state file may be a lot longer than
youíd expect and could cause an outage of several minutes or more. But I guess
if you are moving WPARs after hours in order to perform some maintenance activity
that could take several hours, then a 3, 5 or 10 minute ďfreezeĒ of an
application may be acceptable (not unlike some HACMP failovers)?

My third concern is that if
I choose to deploy WPAR mobility in itís current form and use NFS for my WPAR
file systems, I would need to ensure that they were highly available. Wouldnít
I? Could you imagine if I only had one NFS server and it suddenly stopped
responding and my 200 WPARs also stopped working. What a nightmare! So Iíd
probably need to deploy new hardware, in a cluster, so that my NFS server was
always available to my WPARs. This is expensive and adds complexity. Iím hoping
IBM has a better solution in the works?

At the moment the WPAR
mobility solution feels a little kludgy to me and Iíd be reluctant to use it
for any real production workload. I had a similar feeling about VIO when it was
first introduced and I still wonít use it for production systems as it does not
offer the best performance. However, Iím confident that IBM can work on these
issues and after listening to customer feedback from the Beta program will, by
the time AIX 6.1 is released, deploy more robust mechanisms for implementing WPAR
Mobility.

Chris Gibson is a Senior AIX Systems Administrator for Insurance Australia Group
(IAG), Australasia's leading general insurance
provider. Chris has been working with UNIX systems for more than 14 years.
Since 1999 he has been involved with RS/6000, SP and pSeries systems running
AIX. His areas of expertise include AIX 5L and NIM. He has also worked with
HACMP, CSM and PSSP. He is an IBM Certified Advanced Technical Expert - pSeries
and AIX 5L. He is also a co-author of the IBM Redbook, NIM from A to Z in AIX
5L. He can be reached at cg@gibsonnet.net.