But [language warning] boy, oh boy, do I find it confusing remembering whether I'm using the restricted shell (logged in as the user padmin) or switching to the full-blooded, no decaf, root access. You do that using the command oem_setup_env (as anyone would have guessed).

Jumping the Gate of the Restricted Shell

Now maybe, like me, you avoid damaging your VIOS environment by avoiding that oem_setup_env command. But just between you and me (and just about every other AIXer I've ever met), sometimes it's nice to have, isn't it? Like when you're playing with disk device drivers that you simply can't install or configure in the restricted shell.

It can be very frustrating remembering if you're padmin (restricted) or effectively root (oem_setup_env), so when you're logged in as padmin and run a typical root command, and then get an error, you're likely to set up an alias like this in /home/padmin/.profile

alias aix="ioscli oem_setup_env"

UPDATE:

or, in recent versions of the VIOS (as of August 2015), simply this:

alias aix="oem_setup_env"

Not sure which one to use?

Type the following "type" command to identify which version of oem_setup_env is valid:

type oem_setup_env

Thanks to lillracksingen for the comment.

And then log in as padmin and run aix every time. Full shell, here we come!

Except you're not supposed to do that!

If you really, really, really have to go to the full shell, just for one command, there's another way. It's insanely obvious, which is probably why it took me 9 years even to think about it and then I didn't think about it at all. It was sent to me by John Tucker, an AIX enthusiast in the US and a reader of this blog. So a tip of the Akubra hat to you, mate.

Here's the tip:

when you want to run a command as root, but you're logged in the restricted shell as padmin, run the command as padmin but pipe it to oem_setup_env

So, if you want to run a command such as uptime to see how long since the last reboot, there's probably a few ways of doing this in the restricted shell, but good luck working out what they are.

You run uptime and you get a friendly message to tell you that you made a mess of it.

Sure, you could do the oem_setup_env thing and then run uptime, then exit (of course) to take you back to the restricted shell, but why not just try using the pipe?

Isn't that nice? It still doesn't exactly protect you from jumping the wall of the restricted shell and then going wild wiping out your VIOS, but at least it's keeping you more or less on the right side of the law.

Thanks again to John Tucker. I couldn't afford to send you an Akubra, John, but here's a link to the Wikipedia article on this iconic Australian headwear.

Most IBM Power Systems these days have a Virtual I/O Server, commonly known as the VIOS.

The VIOS facilitates the sharing of physical I/O resources. In other words, you don't need to assign an adapter for disk for each logical partition, and then another adapter for the network traffic. Instead, the VIOS owns the adapters and then you can share the resources (disk and network) by creating virtual adapters.

The benefit? This greatly reduces the amount of I/O drawers, cabling and configuration, so it makes it easy for you to have multiple logical partitions on the same physical Power System.

Now there are a lot of people who are really strong at AIX or IBM i but perhaps not so confident about setting up or even looking at the VIOS. In fact, I wonder whether there may be one or two people in the industry who positively avoid it, perhaps preferring to stay in their own OS comfort zone. I perfectly understand that. However, not working on the VIOS can be really holding you back if you want to manage your Power Systems more effectively.

The good news is that the Virtual I/O Server is not at all hard to learn, as it's a tiny subset of the commands or functions you need to know compared to an AIX or IBM i logical partition, so if you know of anyone who is afraid of the VIOS, or suffering of VIOS phobia, they would do well to watch this video.

Update: Although the original video is no longer available, this other one from "Mr NMon", Nigel Griffiths, will walk you through the VIOS installation process.

Here are some helpful resources on setting up Shared Ethernet Adapters.

If you've worked with dual Virtual I/O Servers (VIOS), you've most likely got experience with Shared Ethernet Adapters (SEAs). You can set them up in failover mode or using the more recent load sharing. With failover, the Ethernet adapters on the backup SEA (for example, on VIO server 2) sit idle until there is a planned or unplanned failure on VIOS1's SEA. In other words, SEAs in failover mode is a binary configuration: if SEA on VIOS1 is up, then your physical Ethernet adapters in the SEA on VIOS2 are sitting idle. Clearly, with expensive 10 Gb physical Ethernet adapters, you don't want them parked in the garage until your main set of wheels breaks down.

With load sharing, you can set some VLANs to use the SEA on VIOS1, and other VLANs to use VIOS2, and still have both SEAs ready to take over should its cousin on the other VIOS fail.

Now if you're having trouble getting your head around all of this, there's a very helpful, hands-on document from our friends at chmod666.org called Optimizing network Part 1: VIO Server load sharing. It not only explains what load sharing is, and how it differs from failover mode. It also provides some great screen shots along with a step-by-step guide on setting up load sharing. There are many very helpful commands you can run to look at what a running configuration looks like.

This is the epic of the upgrade for the Virtual I/O server (VIOS) from version 1.4 to version 2.2. It's not a pleasant read, so if you're of a weak stomach, just skip the whole blog post, but note two things when it comes to VIOS and firmware upgrades:

Always read the README

Consider a fresh install of the VIOS

The Epic Begins

Much as we'd like all our systems to be running
the newest and latest of everything, most of us are not so lucky.
There are still some very old environments out there, running on old
hardware. I suppose it's a testimony to the systems that after 8 or
10 years, they can still run a business on them. There's nothing
like complete neglect to keep a reliable system running on inertia.
Just hope nobody needs to change anything.

I recently was asked to look at a system which was running dual Virtual I/O Servers (VIOS), both at version 1.4.0.0,
which dates back at least five years. The aim was to get them running
a recent VIOS version (2.2.0.10) so we could connect to new V7000 storage and a new
Fibre Channel switch. The ability to use
Shared Storage Pools on the VIOS was also an attraction.

The site had been running stably for some
years, but the firmware, VIOS and AIX were all seriously out of date.
A plan had to be worked out to upgrade.

For full disclosure, I have to say that the upgrade has not yet been done. This is just a walk through of the planning for the upgrade. So, take a deep breath, and here we go:

VIOS Hardware and Firmware Prereqs

The VIOS ties in so closely with the
hypervisor, that it's really critical to ensure your system firmware
is up to date before you consider any upgrades to your VIOS. If
you're using a Hardware Management Console (HMC), then that should be
at a compatible level. The upgrade order is:

HMC

System Firmware

VIOS

AIX

I might point out that in most documentation "Firmware" refers only to the System firmware. Device firmware, such as for Fibre Channel adapters, doesn't rate a mention. So my plan was to include that, since it undoubtedly would not have been updated since the system was first installed.

Firmware alert: setback #1

As the system I was working on was a Power5 server which was managed by a Hardware Management Console (HMC), I had to know if the system firmware would be Concurrent (no downtime) or Disruptive.

I was going from firmware version SF240_358 to SF240_417. In other words, the major release was SF240, so it should ordinarily be a concurrent update when your system is managed by the HMC.

HMC managed systems

Service packs within a release are updates and can be installed concurrently in most cases. Levels above your current release are release upgrades and are disruptive.

Non-HMC managed systems

Installing firmware is always disruptive.

However, this was not one of those "most cases" in which the installation could not be done concurrently:

The firmware update requested can not be installed concurrently because the update path contains a disruptive service pack.

In other words, complete downtime for all virtual clients and the VIO servers.

Firmware alert: setback #2

I also found that the adapter firmware had a prerequisite which I needed to be aware of:

VIOS Requirements

If the adapter is in a partition running VIOS, it is required that VIOS v1.5.2.0 or later be installed. If VIOS is not at this level, install Fix Pack 11.1 prior to installing the new microcode. Failure to do so may result in damaging the adapter. Installing Fix Pack 11.1 will ensure that the required AIX APARs are installed.

Memory and Disk requirements

VIOS version 2.2 requires 4 GB of RAM and at
least 30 GB of disk for rootvg. (If you're using software mirroring
for rootvg, you need at least 60 GB).

updateios Won't Do Everything!

When you log
into the VIOS restricted shell as the user padmin,
you have a simplified command set. The commands are very easy to
learn, and once you get used to them, you can almost guess your way
through.

Now there's a
great command for running updates called updateios.
The syntax is very easy. For example, if you have your updates in a
directory called /home/padmin/update, then the command you need is:

updateios
-dev /home/padmin/update

Now you might think that moving from VIOS 1.4
to VIOS 2.2 only requires these steps:

Download the latest VIOS updates from the IBM
Support Portal

Copy them to a directory on my VIOS

Run updateios

Not So Fast!

It would be nice, but I wasn't so lucky.

First, since
I was going from version 1.X (in my case 1.4.0.0) to version 2.X, I
need to use the migration media. As you may know, the current VIOS
2.X is running AIX 6.1 under the covers. And by “under the covers”
I mean using that sneaky oem_setup_env
command which gives you full root access to the underlying AIX. When
you're logged in as padmin, on the other hand, you're in a restricted
shell, with a very limited command set available.

Migration to VIOS 2.X

First Stop: IBM Support Portal

Of course,
the first port of call was the IBM Support Portal. You can just go to
www.ibm.com
and then select Support & Downloads or take the shortcut to IBM
Fix Central.
You'll need an IBM ID to log in. Registration is free and it only
takes a minute to do.

When you
select the product, in the Product Group, click on the drop down box
and go to Software >
Virtualization software. Here's a screenshot:

in the box Select from Virtualization
software, use the drop down to
go to PowerVM Virtual I/O Server:

From there you can choose the Installed
Version. You can find that out by logging onto the Virtual I/O Server
as the user padmin and then running the command ioslevel.

Notice that 1.4.0.0 doesn't even make the list? The oldest version starts at 1.4.1.2. I'd have to make do.

1.4.0.0 is old. My beloved Virtual Media Repository was introduced with version 1.5. That alone was reason for me to upgrade.

Next Stop Roadblock: Getting to version 2.X

I was aware that moving from version 1.X to version 2.X required booting off a Migration DVD (or NIM resource). That made sense, because it's a similar procedure when you're migrating from AIX 5.3 to AIX 6.1 or 7.1. This jump is not just a list of patches (similar to the AIX update_all command). It was a migration. So, I thought the best approach would be to use the latest available Migration DVD. The Support Portal pointed me to

Migration DVD 2.1.3.10

However, using that DVD had a prerequisite: the last 1.5.2.6 Service Pack. Could I update from 1.4.0.0 to 1.5.2.6 (latest Service Pack)?

I went to the readme and found that (alas):

VIOS 1.5.2.6-FP 11.1 SP-02 should be applied to systems that are currently at IOS Level 1.5.2.1-FP-11.1. If your IOS level is lower than 1.5.2.1-FP11.1, you must install Fix Pack 11.1 before you can install the Service Pack.

Oh great!

So my upgrade path was now looking like this:

Use updateios to get from 1.4.0.0 to 1.5.2.1-FP11.1

updateios again from 1.5.2.1-FP11.1 to 1.5.2.6-FP 11.1 SP-02

Boot off Migration DVD 2.1.3.10 and continue migration

updateios to get to the latest available VIOS (2.2.0.10-FP24 at the time of writing this blog)

I haven't mentioned the time taken to download, back up, reboot etc. which are essential to each of the above steps.

A Small Breakthrough

I found on the IBM Support Portal another, older, VIOS Migration DVD, which would take me from 1.X to 2.1.

Now my upgrade path looked like this:

Boot off Migration DVD 2.1.0.0 and continue migration

updateios to bring me to the latest available VIOS

That's much simpler.

I went to the IBM
Support Portal, selected Downloads and then downloaded the latest VIOS packs. At last, my plan was complete, and I should be able to run updateios, which is all I ever wanted to run in the first place.

An Alternate Approach

If you think this was jumping too many hoops, consider another option for your own environment: doing a fresh install of the VIOS. Sure, you'd need to map and recreate Virtual Target Devices, and Shared Ethernet Adapters, but if you're organised and have all of that well-documented, it may be a better option, especially if you only only have a small number of virtual disks that get mapped through to clients.

For network redundancy, you can set up two Shared Ethernet Adapters (SEAs) - one on each Virtual I/O Server (VIOS). The two SEAs work together with one active, the other idle, so the configuration is called failover mode. This is done by a control channel - a heartbeat that works through a virtual ethernet adapter on each SEA. The failover mode is very effective, because it allows you to shut down a single VIOS and automatically have the other SEA on the alternate VIOS take the traffic. With failover mode you can use VLAN tagging, something not available on the Network Interface Backup configuration.

Idle adapter

The failover does have one big drawback. The non-active adapter sits idle while all the traffic passes through the primary adapter. With a relatively expensive 10-GB adapter sitting idle, it seems a bit of a waste.

The Virtual Switch solution required a Network Interface Backup (NIB) configuration on the VIO clients. This would require a primary and a backup virtual Ethernet adapter on each logical partition. This solution was inventive, but I wondered at the time whether it might get a little too complex, especially when you get to a larger number of logical partitions.

Load Sharing

Now there is an option for configuring SEA failover (not Netwrok Interface Backup) with Load sharing mode. This means you can have both physical network adapters active, instead of one sitting idle in the marketplace, waiting to be called up for duty. This loadsharing configuration leaves any complexity on the VIOS and away from the VIO clients. You have to be running VIOS version 2.2.1.0 or later (check your version by logging in as padmin and running the ioslevel command).This saves the manual load balancing by hand which happens when you use NIB. The load sharing option looks to me to be a much more attractive option than the earlier solutions.

Nigel Griffiths (of IBM Power Systems Advanced Technology Support, Europe - but he is also known as Mr Nmon) gave a great presentation on Shared Storage Pools at Copenhagen. This was also presented in the excellent Webinar Series on Power Systems Virtualisation.

Here's a small extract on why you would use Shared Storage Pools:

In smaller companies, there is no SAN team. The server guys are the SAN guys. The idea of Shared Storage Pools is to allocated LUNs to the VIOS(s) and a single VIOS command can allocate space to a Virtual Server (an LPAR). You can run a single command using cfgassist (VIOS's smitty) to allocate disk space. You can even use the Virtual Storage Management feature in the HMC GUI.

In the two years since this post was first written, there have been a number of other presentations on Shared Storage Pools. You can get hold of these from the Webinar series linked to in this post. (Links have been updated).

A vanilla installation of the VIO server is very easy - once you can get
the LPAR to access the installation files. But that's the part which can be unnecessarily cumbersome and time consuming.

Sure, you can use a physical Virtual I/O server installation DVD, but
that means getting the DVD to a data centre - and probably you with it.
Now if you're in a lab setting, or have a data centre across the floor from where you sit, that may not be any issue at all. But
for many of us, getting into a data centre is harder than getting into a
high security prison (other than the normal procedure, which
in both cases unfortunately involves criminal activity). By the time you get security
clearance and are physically present, you can write off half a day.
What's more, it's half a day of your dedicated time, because you won't get
anything else useful done. And that's not taking into account the time and energy involved in a long
drive or a flight as well if the data centre is not in your
neighbourhood. All for the sake of a virtualised system, with lower TCO and a greener solution than using dedicated hardware for each AIX system.

Getting past the media

Suppose you
do get the media, allocate your physical adapter with the DVD-ROM to the
VIO Server LPAR and then boot off it. Fairly straightforward from that
point. But then, after a successful first install of the VIO server,
you have to remove the slot containing the adapter for the DVD-ROM, assign it to the second VIO server, and then build that VIO server, and then eject the
DVD. Then you can leave the data centre and do the rest remotely.

When you've made it to finding the VIOS installation source, you can pretty easily find your way through the SMS menu options, especially
if you've got experience building AIX LPARs, but getting to the VIOS
installation "media" - is still a big challenge.

Now if you don't use physical media, you still have to download the installation files and find yourself an FTP server or NFS server, but that is not dedicated time. You can set off the download ahead of time, and you're able to devote yourself to something else (or nothing else, for that matter). Anyway, even counting the time it takes to arrange downloads and copy files to FTP or NFS servers, it still doesn't compare with a trip to a data centre.

"Hasn't this loser heard of NIM?"

Now if you're thinking: "why doesn't that whingeing Aussie just set up NIM?", the answer is in the documentation for installing a VIOS via NIM. It's not the quality of the doco I'm complaining about: it's the quantity. There's just too much to learn, and it's complex. For someone who has never installed NIM before it's a huge amount to digest when you just want to do a basic installation of a VIO server. Even for people who have worked in AIX for a long time, if they haven't
used NIM for a while (or at all), it's not something they can pick up
again in 5 minutes.

Nigel Griffiths presented a webinar this week on VIOS installation. In that he says about installing the VIO Server:

"Could use NIM - only for the NIM experts - IMHO."

I couldn't agree more.

I challenge anyone who hasn't a lot of NIM experience to set up NIM
unassisted and install the VIO server unassisted (apart from the documentation). You might trace your way back to the
excellent NIM from A to Z Redbook for AIX 5L, but then again if you're
installing Power7, with AIX 7, you may not be too confident trawling
through an AIX 5 Redbook (even though it's almost all helpful and relevant). You might not even think of it.A remote, vanilla installation doesn't have to be that hard!

HMC installios command

There's another
option for installing the VIO server - using the HMC command installios. That requires command line
access to the HMC, but at least you can do it remotely by downloading
the VIOS installation file in ISO format and mounting it via NFS. Much
less to set up than a fresh NIM installation, and Rob McNelly explains how it can be done. Doing it this way, you can say good-bye to installation media and build a Power system without physical access to a data centre.

Yes, installios is an option, but pretty rarely used, I'm told, and it's only really for an initial VIOS installation. If you want to do a migration, or for some other reason boot the VIOS off media, you can't do it via the HMC.

Virtual Media Repository

As
you know, I use the Virtual Media Repository (also called Virtual Media Library) a
lot, and it's quick and easy to master. I think you can learn pretty
much all you need to know about it through a short article or by
using help on the padmin VIOS restricted shell.

You can use the
Virtual Media Repository to download images in ISO format and then use them to mount onto
virtual clients. That makes it a simple task to build a new AIX system
from scratch, or from a mksysb backup of another AIX logical partition,
or even create a file of anything into ISO format and make it available, all without using physical media.

But
how do you mount an ISO image for the VIO Server to use? You can boot a
virtual client off a Virtual SCSI (VSCSI) adapter and have an operating
system built in 20 minutes or less. Why can't you boot a VIO server off
a VSCSI adapter? After all, VIO servers can have VSCSI server adapters, and VIO clients can have client adapters. Wouldn't it be good the other way around?

"But ... but ... but it's a VIO Server!"

Yes, yes. I know the VIO server is a server. But you can still give it client serial adapters. Why not client VSCSI
adapters? How do you get the VIO server image to map to that VSCSI
client adapter? I don't know. Perhaps the HMC (or SDMC) would be a
natural fit for it, similar to file-backed devices.

Now, you
might be wondering, dear reader, whether that AIX guy Down Under is
still on the same planet as you. But bear with me a moment more, if only
for old time's sake.

Virtual ethernet, but not VSCSI

Alright, I'll grant that giving the yet-to-be-installed VIO server its
own VSCSI client adapter to boot from is a bit of a pipe dream. It can
boot from an ethernet adapter, but you still have to have something like
NIM or the HMC to present the installation media across the network. So
I'll resign myself to the fact that at the moment, I have to install
VIO server 1 the current way.

Wouldn't it be good at least if you could install VIO server 1 (with NIM, or
physical media, or via the HMC installios command), and then do the other VIO server
via the Virtual Media Library? And if you ever need to migrate your VIO
server (the underlying code for VIO server version 2 is still AIX 6, but one
day it will go to AIX 7), it's back to physical media or upgrading NIM
and then learning how to do the migration steps. Same difficulty if you
happen to lose your padmin password (or, more likely, lose the sys admin
who knew it).VIO Server as a Virtual Media client

Imagine being able to do a migration (upgrade) of two VIO servers where they could
load ISO images from each other's Virtual Media Repository. You could
then upgrade each VIO server in turn by booting off the new VIOS server installation image presented via the other VIO
server's VM Repository.

If you were doing that with NIM, you'd have to
commit yourself to migrating NIM first, so that it is at least at the
same level as the new VIOS' underlying AIX level.

If only you
could boot the VIO server off the VSCSI adapter - maybe loading the installation media
from another LPAR somehow. And you could even do it through a nice GUI
interface, such as the HMC or SDMC.

ISO image access for the VIO server

Now you can load ISO images for use by a VIO server (as opposed to the
VIO clients) another way: with the AIX loopmount command on the VIO server.
To do that, you'd need to log in as padmin on the VIO server and then go to the full root shell by running oem_setup_env. It's not too hard to do but you can't boot off the image,
as when you reboot the VIO server, you lose access to the mounted file
systems, including the loopmount-ed ones.

NIM has its usesI don't mean to say that NIM is of no use at all. It can come in very handy for installing software across multiple LPARs, and for migrations using nimadm - which can reduce outage time. NIM is also used by IBM Systems Director. What I'm trying to point out here is that the Virtual Media Repository is so easy to learn, and easy to use, letting the VIO server be able to boot from or load an ISO image via a VSCSI client would be an altogether Jolly Good Thing.

A strange "insufficient memory" error on the VIO server took a little bit of untangling.

The error occurred when I logged into the VIO server command line as padmin shortly after running a backup. I was attempting to display the Virtual Media Library* contents. When I entered the lsrep command, it reported the following error:

Insufficient memory, not able to complete command.

I tried to list the virtual resources using the HMC GUI (Server > Virtual Resources > Virtual Storage Management) but it seemed to take an eternity. It eventually responded.

Could I really have been out of memory? The VIOS nmon command reported plenty of free memory. Oddly enough, this out of memory error occurred shortly after I had done a backup of the VIO server.

"What has changed?"

As it turned out, my problem was very simple and entirely self-inflicted: I had unmounted the Virtual Media Library file system, /var/vio/VMLibrary before running the backup and neglected to mount it again after the backup. That leaves you, dear reader, with two questions:

Why did I unmount the Virtual Media Library file system?

How on earth did I find the problem?

Second question first.

Turn on debugging

You can look under the covers of your VIO commands using:

export CLI_DEBUG=33

Doing this showed that the command called by lsrep was lsfs (list a file system), and here's how it looked:

AIX: "lsfs /var/vio/VMLibrary >/dev/null 2>&1"Insufficient memory, not able to complete command.Aha! So I naturally tried a df to display the file system and it wasn't mounted!

Why unmount the media library?

My media library was in rootvg and I didn't want to include a whole swag of large .iso files from the VM Library as part of my backup. I could have used the mkrep command to create the library in a separate storage pool (the VIOS equivalent of the AIX Volume Group), but I wanted to keep it in a large, but largely unused, rootvg. Having the VM Library in rootvg saves assigning extra disk for it, and it's not likely to be a performance bottleneck.

The drawback with having the VM Library in rootvg is that your backup may become significantly larger and take a truckload of time. That's why I unmounted it. As I neglected to mount that file system again, I hit the rather misleading "Insufficient memory" message.

Shutting out the media

So if you do have your media library in rootvg, it will be helpful to know how to exclude it from the VIOS backup.

If you're using the backupios command to back up the VIOS, and you're on VIOS version 2.1.3 or later (check using the ioslevel command), then you can use the -nomedialib flag. But if you're using another backup method, such as viosbr, or your VIOS is older than 2.1.3, you've really got to exclude your VMLibrary some other way. Such as unmounting it. From the VIOS restricted shell, that's the unmount command (u-n-mount not umount).

unmount /var/vio/VMLibrary

Just remember to mount it again.

* The Virtual Media Library is also known as the Virtual Media Repository or VMR.

No matter
how long I work on AIX, I never get excited when I get a "Method
error". Fortunately, this one was very easy to fix. I was simply missing a VSCSI map on the VIO server.

A little
bit of background will help explain the problem and the solution.

Dual VIOS disk configuration

On a dual VIOS configuration with MPIO on the VIO Server, the LUNs are presented through two virtual SCSI adapters, one on each VIO server. (For more information on MPIO redundancy, see Janel Barfield's Advanced Power6 VIO Configurations.)

On
the VIO client (the LPAR running AIX), you can run cfgmgr to discover
the new disk and then set the parameters you want such as health check
interval, queue depth and max_transfer size using chdev.

Priority 2

Then you
run the chpath command to specify which VSCSI path you want for
the priority. With two VSCSI adapters, both are set to priority 1, so
you only need to change one of them to priority 2.

In this example, I thought I had two vscsi paths. vscsi0 (presented via VIOS 1) and vscsi1 (via VIOS2).

But when I went to change the path, I got that horrid method error.

The problem was that I had masked the LUN from Storage Subsystem to both VIO servers, but I hadn't run the mkvdev command on VIOS 2 to assign the LUN to the vscsi adapter on VIOS 2.

Leading up the path

Here's how I found that out:

A quick look at the paths for hdisk4 showed there wasn't one for vscsi1. I used the lspath command without any flags or arguments. This showed that there was a mapping for every hdisk through two VSCSI adapters (vscsi0 and vscsi1) ... every disk, that is, except for hdisk4.

No entry for vscsi1, which is why the method error reported that "Invalid routine argument" error.

Map maker

I logged onto the VIOS 2 as the user padmin and was able to identify the only LUN which hadn't been mapped using lspv -free. (It's one of those handy options on the VIOS lspv command. For more details about this, check out the earlier post vscsi disks on the loose: map 'em or scrap 'em.)

After creating the map using the mkvdev command, I logged onto the VIO client (the AIX LPAR) and ran cfgmgr. Now the new path showed up on vscsi1, and the chpath command worked, this time without any Method error.

I noticed that the VIO server (VIOS) error log was reporting some failed paths for LUNs connecting to the SAN. The VIOS command errlog -ls (the equivalent of the AIX command errpt -a) showed errors on the Fibre Channel adapter fscsi2:

Each LUN that had a failed path still had other paths on this VIOS functioning correctly. In addition, each of the LUNs is presented to the VIO client via MPIO through this and another VIO server. That makes for a lot of redundancy, which gave us some breathing space to sort out the real cause of one of the many paths being lost. In the meantime, it's quite easy to remove the failing path on the VIOS using the rmpath command.

View paths for a LUN

First, I used the VIOS lspath command from the VIOS restricted shell to look at a single PV. This showed that there were multiple paths from the VIOS through to the SAN (in this case going to SVC).

lspath
-dev hdisk63

Or via the AIX shell after logging in to the VIOS as padmin and running oem_setup_env:

lspath -l hdisk63

Whichever version of the lspath command you use, here's the output showing several paths for the same disk.

Removing all the paths for hdisk63 via fscsi2 would work, but it would remove the successful paths to fscsi2 at the same time. A bit drastic, but let's face it, sledgehammers had to be invented for a reason. Anyway, as there are several other paths to the same LUN - four via fscsi0 and another four via fscsi1, removing three good paths from fscsi2, as well as the one that has failed isn't really a problem. After all four fscsi2 paths are exterminated, you can rediscover the three good paths using the VIOS cfgdev command or the AIX command cfgmgr.

Here are the steps I took to remove all four paths for fscsi2 from hdisk63:

Here's a little aside for the benefit of readers not overly familiar with Australian slang. A "dummy" is a pacifier / comforter sometimes given to babies to, well, pacify them. On occasion some babies have been known to expunge the said dummy with speed and skill of Olympian standards.

Well, the rmpath command didn't actually remove the paths. It kept them Defined in the ODM. When I ran cfgdev (or cfgmgr), the command spat the dummy.

We would have been better to remove the fscsi2 paths from the ODM altogether, using the rmpath command with the -rm flag. This is similar to the -d flag on the rmdev command, as it deletes the references from the ODM.

rmpath command

Purpose

Syntax

Once again, I'll use the -rm flag to remove the path from the ODM. Otherwise it would simply go from Available to Defined and still report a problem when running cfgmgr. But this time, I can narrow the path down to a single connection using the -conn flag:

I recently saw an usual configuration for two Shared Ethernet Adapters and it took me a while to understand why either of the SEAs was working at all. First, a little background.

(Actually, a LOT of background)

When you have Shared Ethernet Adapters set up in failover mode, all external network traffic goes through one SEA. The second SEA on the other VIO server is there for redundancy, in case the first one should go down for some reason (network disconnected, VIOS reboot). The one with the highest trunk priority usually takes the traffic. For example, on VIOS1 you may have a virtual adapter with trunk priority 1, and VIOS2 has trunk priority 2. Let's call the SEA on VIOS1 "SEA1". SEA2 is on VIOS2 with trunk priority 2. There is a control channel adapter on each SEA which allows SEA2 to check that SEA1 still has external network access. If it does, SEA2 just sits there patiently, just waiting for SEA1 to fail so it can take over (have you ever worked with someone like that?)

Ordinarily, you'd have SEA1 as the active adapter, as shown by some of the output of the VIOS entstat command run as the user padmin:

entstat -all ent6 | more

High
Availability Mode: Auto

Priority:
1 Active: True

The same command on SEA2, shows that the adapter on VIOS2 is not Active for external traffic:

High
Availability Mode: Auto

Priority:
2 Active: False

Now to test that SEA2 would actually work in the event of a reboot of VIOS1, or a network failure on SEA1, you could unplug SEA1's network cables, disable its switch ports or reboot VIOS1. Or you could simply set the High Availability Mode on SEA1 to standby (once again, as the padmin user):

chdev -dev ent6 -attr ha_mode=standby

A new entstat on SEA1 now ought to show this:

High
Availability Mode: Standby

Priority:
1 Active: False

and SEA2 would automatically become active, even though it's priority 2.

High
Availability Mode: Auto

Priority:
2 Active: True

And you'd see that in the VIOS2' error report via the VIOS command errlog (use errlog -ls for a long listing):

LABEL:
SEAHA_PRIMARY

Probable CausesBECOME PRIMARY

Failure
CausesBECOME PRIMARY

Recommended Actions
BECOME PRIMARY

Detail DataBecome the Primary SEA

On VIOS1 you'd see an equivalent "BECOME BACKUP" message in the error report. (If you didn't use the errlog command, and chose oem_setup_env followed by the AIX errpt command, take five marks off your scorecard for today's class.)

That's how it's supposed to work. And when the VIOS1 comes up again (if you'd rebooted it), or you set the SEA1 high availability mode on SEA1 back to auto, it ought to become the primary ethernet adapter once again:

Typically, aShared Ethernet Adapterin a failover setup is operating inautomode, and the primary adapter is decided based on which adapter has the highest priority (lowest numerical value).

(The "lowest numerical value" means the highest priority. So priority 1 is higher priority, because 1 is a lower number than 2. I hope that's clear, class.)

If they're both in auto mode - as they usually would be - then changing
the priority 2 SEA to standby ought to have no effect, since priority 1
should already have the traffic as indicated by Active: True from the
entstat command.

Now, what if both SEAs had their ha_mode set to standby? What would happen?

I've seen exactly this situation, and it puzzled me for a while. SEA1 was set to standby, effectively relinquishing its Active status to SEA2, but SEA2 was in no position to take over, since it, too, was set to standby. Clearly, this was not an SEA failover configuration that was going to work, and yet traffic was going through SEA1. In fact, the output of the entstat command on SEA1 showed this:

High
Availability Mode: Standby

Priority:
1 Active: True

I was puzzled at first, because I thought that neither SEA was in a position to stake a claim for hosting external network traffic. But the ha_mode=standby really means this:

A shared Ethernet device can be forced into the standby mode, where it will behave as the backup device as long as it can detect the presence of a functional primary.

It was easy enough to figure out what was happening. The SEA1 had been created first (assuming no reboots of either VIOS) and it, by default became the Active SEA. When SEA2 was set up with a valid control channel, it never attempted to take over from SEA1, even though SEA1 was in standby mode. Why? Because SEA2 was also sitting there like an loaded gun, but with no "ha_mode=auto" to pull its trigger.

"After you". "No, I insist, after you."

Basically, the two SEAs were too polite to take over control from the other one, and the first one to be set up remained the boss by default.

I don't know why they were both set up to standby, but the configuration worked effectively as if there was only a single SEA - SEA1. Failover was never going to happen.

Fixing it would be very easy: set the ha_mode on each SEA to auto. If you do it on SEA2 first, then it ought to take control and become the primary adapter, but as soon as you make the change on SEA1, that will become the primary (active) adapter because it is priority 1.Resources

For ... ahem .. undisclosed reasons, an organisation's AIX administrator has gone Missing in Action. The empty shoes are to be filled by you. You've never heard of the company "Clean Doubt" and never logged into their systems. You don't know what applications or databases they run, so you're full of questions for the handover meeting. You are introduced to the boss who has an iron handshake and a gravelly voice. His name is Mr. Clean "call me Squeaky" - and he explains that the previous sysadmin "had to be let go". "Accidents do happen", he adds with a steely smile. He hands you a piece of paper with the root passwords on it. He wants to know how his systems are running. You've got two hours to report back.

Somehow you both know that the handover session is over.

Incentive

You don't have local knowledge, but after your meeting with Squeaky, you do have motivation to get that knowledge. Quickly.

If you've worked on IBM Power Systems before, you're actually a good deal ahead of the game. After all, you can check a few basic things which will give you half a clue about what the systems are running:

Check file systems, volume groups

File system names can give a good indication of the sort of applications that might be running

Check processes

the output of ps -ef will soon tell you what applications or databases are running

Host names

Most places seem to change their host name policy every 18 months to two years (coincidentally about the same shelf life as the management). Do the host names contain some clues as to what is running? Consider these hostnames. Some shed more light than others.

oraprod (Oracle production)

saptst (SAP test)

jupiter05 (too many LPARs - wish they'd discovered some more planets)

server01

rs12erp (ERP system migrated from our RS/6000 server)

brat (Brisbane Disaster Recovery AIX Test)

Cron jobs and scripts

Are there regularly scheduled jobs running in the cron or initiated by some other scheduler? Are there some scripts which run frequently and may be central to the healthy flow of data?

Backups and Disaster Recovery

Are backups being taken? At all? How often? What's the backup utility? Is there a backup checklist or some documentation?

Is there a DR plan? Has it been tested recently? Has there ever been a disaster? Was there a recovery?

Documentation

Is there documentation which can help you get a feel for the environment? Is it reasonably up to date? Are there project managers, colleagues, anyone who can shed some light on what the system does?

Do you have access to your predecessor's bookmarks or favourites? This alone can tell you a lot about how organised, up to date and ready to learn the person was. Did your unmentionable forerunner follow Rob McNelly's AIXchange blog? What about Chris Gibson's AIX blog? Or AIXpert? Did the unfortunate sysadmin click on the link to follow AIX Down Under on Twitter? Was that decision the right one? Was it the last one?

NIM and management servers

Can you find a NIM server? How about an HMC or IVM? Is there an IBM Systems Director server or maybe some other server which does system monitoring? These can give you a quick snapshot of the lay of the land, so you can get a quick (albeit superficial) grasp of what the systems are all about.

Listen.

Is there a DBA screaming for something urgent? See if you can empathise and then discreetly ask what kind of database the system is running. Are there developers who have an urgent release and users demanding to be able to log in? This sort of buzz can give you a little bit of a feel about what's important, what's hot, what's not.

does anyone else want to cry on your shoulder about the guy who got the chop disappeared in mysterious circumstances? It's amazing what you learn over a cup of coffee.

Check versions

Checking the versions which are running will give you an idea of how up to date the system is. It may indicate if your predecessor (whose name is never to be mentioned) was keen to keep things crisp and clean or was more interested in counting down the days and planning an early escape retirement. There's nothing to be gained from bad-mouthing the absent, but it may give you an idea of how much work you've got cut out for you. You'll also get a feel for how well peoples' expectations have been managed, which is a very important part of an administrator's job.

Are there 30 LPARs, maybe some WPARs? Are there any VIO servers or do all the hosts still run with dedicated adapters? For that matter, the hardware vintage will help you know whether there's a collection of old standalone servers accrued over the centuries or a Power7 that's on the latest firmware and running PowerVM with all the bells and whistles. The prtconf command will help you here. What version is the HMC? What does the output of the VIOS command ioslevel tell you? What about the AIX levels using oslevel -s?Happy handover

Much as you'd like to walk in and pick up the pieces of a system running AIX without any assistance, there is some local knowledge that would save you an immense amount of time. Well, spare a thought for your successor.

This time of year is a chance to tidy up a little, document, and provide some sort of framework for a quick handover, in the unlikely event that you're the one who goes M.I.A, courtesy of Mr. S. Clean.

If you want to get the size of a disk (virtual or physical), you can use the AIX getconf command.

getconf DISK_SIZE /dev/hdisk0140013

The size is reported in MB, so the disk above is a 140 GB disk.

You could use the bootinfo -s, but that command is

deprecated!

so I'm not even going to give you a link to its man pages.

Maybe you're not exactly sure what "deprecated" means, but it sounds bad, doesn't it? According to Wikipedia, "deprecated" derives from theLatinverbdeprecare, meaning "to ward off (adisaster) by prayer". So don't use bootinfo, OK?

Back to AIX and disk sizes.

To display disk sizes you could, of course, use the AIX lspv command, but that only works if the disk is in a volume group.

If the disk (be it physical or virtual) is on the VIOS, you can display it using the VIOS command lspv -size. Unlike the AIX lspv command, this VIOS command should show the size, even if the disk is not assigned to a volume group on the VIOS.

If you want to create a DVD image on AIX, you can do it right from the command line, without even touching physical media, using the mkdvd command.

It was a wonder back in ... well, a long time ago ... when these new-fangled CDs came out and you could play music and use them as coffee cup holders (not at the same time). But the novelty of having a button on your computer that spits out a CD or DVD at you has somewhat faded. We want an image, but not a spitting image.

On AIX you can create an ISO format image of a directory without actually having to hit that eject button.

Directory services

The mkdvd command, as its documentation explains, lets you create a DVD (or DVDs) from amksysborsavevgbackup image.

It doesn't need to be restricted only to mksysb backups. It also lets you:

create
a DVD or DVD that duplicates an existing directory structure such as
the following:

What's the -S for? It actually stands for stop. Here's why, The mkdvd command creates a file system before it copies the images onto your physical DVD, and then, being a well-brought-up AIX command, it cleans up after itself. Since we don't want to use the physical DVD, removing the image would somewhat defeat the purpose of the exercise. So no clean up, please, be a slob (maybe that's what -S really stands for).

I SO want to remember your name

The file in the /mkcd/cd_images directory includes the process ID in its name. If it spans over multiple DVDs, you get a .vol1 at the end of the first ISO image and I'll let you guess what the second one has appended to it.If you're going to use this file for longer than three picoseconds, you'll probably want to rename it to something memorable as you copy it to somewhere useful.

Once the DVD image is created, you can scp it to the VIO server and load it into the VM Library using mkvopt (or if you're cheeky, copy it straight into the file system /var/vio/VMLibrary without using mkvopt. You can always change its permissions to read only using the VIO server chvopt command:

chvopt
-name memorable_file.iso -access ro

That allows you to share it with several virtual optical devices via the loadopt command. Or if you're too young to remember what CDs look like, use the HMC GUI or IVM.

Who needs the media?

The mkdvd command has some other options, such as specifying an alternate directory for the CD image with the -I flag, and creating the new file system in a different volume group for the file systems with the -V flag. And of course you can use it to create a new mksysb (bootable via the VM Library), or simply burn an existing mksysb image.Why
your boss likes youThere may have been 100 people apply for your job, but you got it. Why do you think that happened? I'll give you two possibilities:

you're smart and don't use physical media when you could use virtualOR

the boss randomly threw out the other 99 applications on the grounds that the company doesn't like employing unlucky people.

When you create an LPAR (Logical Partition), you have to create a profile to specify the processors, memory and I/O slots you want to allocate. I’m going to focus on memory right now, and specifically what theminimum, desired and maximummemory settings mean. You’ll see that the memory allocation can change while the LPAR is up and running using Dynamic LPAR (DLPAR) and DLPAR's can only go as low as min and as high as max. Makes sense, doesn't it?

This is about setting up LPARs that use dedicated memory. I’m not touching on Active Memory Sharing because sharing is not dedicated. I’m also not going to go into the Power7 magic called Active Memory Expansion. Just the vanilla answer to “how much memory am I assigning to my LPAR and can I shoot anyone who tries to steal it?”

MINIMUM MEMORY

The minimum memory is the amount which is required for the LPAR to start. If you don’t have the minimum, you can’t activate the LPAR. And for as long as that LPAR is up, nothing will take away that memory. You can’t even dynamically remove it using DLPAR. If you really don’t have the minimum available when it comes time to activate your LPAR, there is only one option: change the profile settings and attempt to activate it again. By the way, if you change the settings in a profile once an LPAR is activated, you need to shut it down and activate the new profile for the settings to take effect. A software reboot ( for example, using shutdown –r) won’t activate a new LPAR profile.

Minimum memory and Required adapters

There’s a useful comparison between the minimum requirements of memory and setting adapters as required. Supposing you have a physical adapter that will give you access to disks or the network. You want that adapter, because if you don’t have the disks you don’t have the LPAR. So you’d make that adapter required. If that adapter was also set to required on a second LPAR, the first LPAR to get activated would get the adapter, and the second LPAR to get activated … wouldn’t get activated. Required means business. It’s a demand, not a wish and if the LPAR doesn't get it, it's going to take its bat and go home.

Now just as an adapter can be set to required, an LPAR can have a minimum amount of memory. Once the LPAR is activated with that profile, the minimum memory, like the adapter, can’t be taken away until the LPAR is shut down. If you try it with DLPAR, you’ll get told off very quickly.

DESIRED MEMORY

The desired memory generally is the amount of memory you actually get when you activate an LPAR. At least, you’ll get that much if it’s available (not used by other activated LPARs or the hypervisor). As you’d expect, the desired memory can’t be less than the minimum but it can be the same. It’s usually higher, because it’s your memory wish list, and because if you play your cards right (and set your values properly), you usually get it. When you activate your LPAR, your memory allocation will be as close to the desired as is available at the time. So you could have a minimum of 1 GB, a desired of 24 GB and the actual allocation of only 10 GB because that's all that was available when the LPAR got activated.

Whatever you end up with when you activate the LPAR (somewhere between min and desired), if someone happens to run a Dynamic LPAR operation to remove some of your allocated memory, then you could go below your desired amount. Your memory could go right down till your LPAR has the minimum set in the activated profile. No one can take it below your minimum without a shut down of your LPAR.

Once again, let’s consider the adapter comparison. Supposing you had an adapter which had a DVD-ROM connected to it. You’d be happy to have it moved around between LPARs without having to shut either one down. So you make the adapter desired on both LPARs. The first one to get activated gets it, but not for keeps. You can still move the adapter (and the DVD-ROM that hangs off it) to the second LPAR, or for that matter, even to another LPAR which doesn’t have it as desired at all.

MAXIMUM MEMORY

Next we get to maximum memory. Actually, we don’t get to it next. You really only get to see the max when you assign extra memory using Dynamic LPAR. Maximum memory is the high water mark for the LPAR. You won’t ever exceed this amount unless you shut down, adjust your profile settings and activate the LPAR with the new profile. The maximum memory can't be less than desired but it can be the same. It's usually greater. And guess what? The maximum memory can't be less than the minimum. Just thought I'd mention it for some readers who needed to hear it (not you, of course).

So, supposing your system has 512 GB of physical memory, and you set your min, desired and max to 1 GB, 2 GB and 500 GB respectively. For the life of the LPAR, the OS will have 2 GB of memory. If it needs more, it will go to paging space, unless you dynamically allocate more or do the shutdown / edit profile / activate thing. IBM Power Systems are good, but they’re not mind readers. If you want more memory, you have to say so! Change the Desired amount before you activate your profile, or adjust it using Dynamic LPAR, OK?

Now if you find in the course of running your partition that it needs more memory, you can assign more dynamically up to the maximum memory setting in the profile which you have activated. Sounds like a good reason to set the max for all LPARs to the highest possible value, doesn’t it? Except that there’s a little man inside the hypervisor who’s on alert, just in case you want to grow the memory on demand. He’s a bit expensive, keeping him on call with all that spare memory in his back pocket, so if your max memory is absurdly high, you’re setting aside some memory in the hypervisor for him to be able to grow your LPAR’s memory just in case. He doesn't reserve everything you might possibly need to allocate, but it is an overhead if you're sailing close to the wind with your total system memory.

WATCH OUT FOR MAX: CUSTOMER SERVICE WITH A FROWN

When I was a young system admin and file systems were 2 GB, if someone asked for an extra 2 GB, I told them that my motto was “the customer is always wrong” and that if they thought I was going to throw away good disk after bad just because they were too lazy to clean up their files they had another thing coming. So they cleaned up and I gave them some extra disk, and told them to come back with their begging bowl in a month if they dared.

The purpose of this little lesson in customer service is to show that assigning extra resources isn’t always a smart solution. Expensive, yes, but not always wise. In setting the memory requirements for your LPARs, it’s a bit hit and miss, but you hopefully will, in the end, hit the right amount of min, desired and max without breaking the budget or losing all your friends.

There's nothing easier than setting up your VIO
servers and AIX LPARs to point to the right DNS servers. Yes,

I know!

I
know!

I know what you're
thinking:

You don't say “DNS Servers”. That's redundant. You just say “DNS”.

Why should you point to a badly configured DNS? It could bring your system
to its knees (metaphorically) and you to your knees (literally).

Regarding the abovementioned point 1, I have been (self-)diagnosed with a
mild case of RAS
syndrome. I'm receiving treatment for it with the help of the internet.

As for point 2, if you don't have responsibility for the DNS, you can at
least find out who does, and ask what name servers they'd like you to use. You can hardly blame others if you're referring to a name server which was retired when your Grandma was just learning to read. On the other hand, if you correctly point to that other guy's name servers and performance heads south after that, you've got no one to blame but Someone Else. Isn't that where you want to be?

Checking the DNS you
use

Domain name servers, and domain names, seem to have a very short shelf life.
Chances are the first LPARs you built are pointing to DNS (I won't say DNS
Servers) which have long since shuffled off this mortal coil. You'll need to
check you're pointing to the latest valid name servers, both on the VIOS and the
AIX LPARs.

DNS from the VIO server

You can list the name servers and domain that your VIO servers are
pointing to using the VIOS command cfgnamesrv.Use
the -ls flag to list:, as the command documentation explains:

# cfgnamesrv -ls

domain xyz.aus.century.comname server 192.9.201.1

You can
add a name server using (you guessed it) -add

To add a name server entry with IP address 192.9.201.1,
type:

cfgnamesrv -add -ipaddr 192.9.201.1

And it's -rm to remove an entry. Use -ipaddr to remove a name server
or –domain to remove the domain. There's also a -ch option which lets you change the domain, rather than delete the old and add the new.

Shell escape: make peace with the VIO
server

Allow me a little aside about VIO server commands.

I know you could edit /etc/resolv.conf
(note, there's no "e' at the end of resolv) which contains the list of name servers you
want to use. Still, it's really worth learning a few of the VIO server commands
with their syntax. Sure, they're a bit longer than AIX, and slightly different
from AIX commands, which can be frustrating, but there is a remarkable
consistency and guessability to the VIO commands in the restricted
shell (the one with padmin permissions).

List is done using –ls

Delete (remove) uses -rm

Devices
are referred to with -dev

Adding entries is generally done
via –add

and so on.

Of course, you could always edit /etc/resolv.conf. After all, it’s a
short and simple file. But many other files are not so simple, so it is
worth
learning the small subset of VIO server commands which are written
specifically
to save you remembering the format of all the files you need to
change.

What about AIX and editing
/etc/resolv.conf?

AIX has its own command to change name servers: namerslv.
Its syntax is slightly different to that of the VIOS cfgnamesrv command. Look it up. Or you could edit /etc/resolv.conf, and I expect that's how you'll go about it. For that matter, you can use SMIT and get through to the network menus to list, add or delete name servers and domains that way.

LED 581 during boot

If your system appears to hang at LED 581 during a system reboot, chances are
it's having trouble connecting to the DNS server (there I go again) in
/etc/resolv.conf. There is an IBM technote regarding Understanding
LED 581. Although the technote was written for AIX 4 and 5, I expect it's
still valid for AIX 6 and 7. To tell you the truth I haven't read the whole
document. I'm a little busy giving myself some therapy for my newly discovered
language disorder.

We're really spoiled with virtualisation. It's so easy for us to map a LUN or a logical volume and build an LPAR, then load an operating system onto it. This makes it easy for temporary, test systems to be built and never get cleaned up.

Why is that? Simple. It's because

There's
nothing so permanent as a temporary solution.

Removing an LPAR that you no longer want to use is easy enough, but somehow those LUNs or logical volumes on the VIO server just never seem to get cleaned up. Which disks are they again?

Or maybe you need to identify new LUNs which you have masked to the VIO server so you can map them to make them available on the VIO client. Time to

Round 'em up, cowboy!

Here, at the usual price to AIX Down Under readers, are three commands which will help you to FIND THAT DISK TO NOWHERE as well as give you an idea of just what size disks (physical or virtual) you're using.

There are three commands:

lspv -free lets you see which disks are not mapped to a vscsi device

lslv -free shows the logical volumes which aren't mapped

lspv -size shows all disk with their sizes in megabytes

List DISKS which are not mapped:

You can list LUNs mapped to a VIO server but not associated with a vscsi device. This also will show you any internal disks which haven't been mapped through to a VIO client. Use the VIO server lspv command with the -free flag:

Lists only physical volumes that are available for use as a backing device. If the physical volume is already used as a backing device or is assigned to a shared memory pool (to be used as a paging space device by a shared memory partition), it is not available and is not listed.

Notice that it shows the size of the disk in meg? Pretty nice.

List LOGICAL VOLUMES which are not mapped:

In the same way, you can list logical volumes which are not mapped through to a vsci adapter for allocating to a VIO client. Can you guess what that command will be?

lslv
-free

#
lslv -free

LV
NAME SIZE(megabytes) VOLUME GROUP

client1_hd0
2048 rootvg

That was just a test lv I created. Shouldn't really clutter the VIOS rootvg, should I?

List SIZES of physical volumes

You can see the sizes of your disks on the VIO server using the lspv -size command. I really like this one.
In AIX it's not so easy to work out the size of a disk (either physical or a LUN) unless it belongs to a volume group. This command on the VIO server lets you see the sizes of them all,

whether they are in a VG or not

whether they have been mapped as backing devices or not.

lspv
-size

NAME
PVID SIZE(megabytes)

hdisk0
00c5a47ed8b627a6 70006

hdisk1
00c5a47ed9e58b2b 70006

hdisk51
none 25600

hdisk52 none 512000

hdisk5
none 25600

hdisk6 00c5a47eed76a134 40960

hdisk7 none 51200

The sizes of these LUNs are all over the shop. Who woulda known?

Deploy or destroy

Those three little commands can save you a lot of time and help you to trace naughty little (or big) disks which someone has let loose on the system. They're just slacking off, pretending to be working but actually just leaning on their shovel. Once you know they exist, you're half way there to fixing them. You could map them to a vscsi adapter using the mkvdev command, or you could get rid of them, first on the VIO server and then - for LUNs - on the SAN.

Three points which jumped out from the page of the installation instructions:

From now on, theVIO Server rootvg requires at least 30 GB. If you're mirroring your rootvg, note that's 30 GB of useable disk. Two LUNs of 15 GB wouldn't suffice, but one of 30 GB would. (Of course, if you're using LUNs you should have redundancy anyway, but if you're on local disk with software mirroring, you need the 30 GB for each disk).

If you're using the File Backed Media Repositories (and you would have given up reading AIX Down Under long ago if you're not), you need to unload images first. That doesn't mean you remove the ISO images from the Virtual Media Library or the VM Library itself. It's just a matter of not having the images loaded onto the virtual optical devices. Check here for details.

It is MANDATORY that you use the -install flag when running the updateios command to apply Fix Pack 24. Failure to use the -install flag will leave your VIO server in an unusable state.

Don't say I didn't warn you.

One VIOS at a ttime

If you have a dual VIOS setup with all your redundancy in place (MPIO, SEA failover and active memory sharing paging space) you should be able to do one VIO server at a time and keep your VIO clients up and running. If you've got a single VIO server (e.g. when using IVM instead of an HMC), or redundancy not quite where it should be, you'll have to take the hit of a scheduled outage.Resources

Now it's time to change the padmin .profile for shell history and shell prompt. I'll call on my colleague, Mark Chandler's new aixplode blog for that. Here's a link to that post on the padmin .profile: http://bit.ly/d7WpQU

I copied the two ISO images to the Virtual Media Library on the VIO server, and I broadly followed the procedure outlined by my compatriot Chris Gibson in his article AIX Migration with File-Backed VIOS Devices. I gave the two ISO images shorter names when I copied them into /var/vio/VMLibrary. You can also rename them using the VIO server mkvopt or chvopt commands.The fine print

All up, the migration took about 35 minutes from the SMS menus to the login prompt following the reboot. I migrated from AIX 6.1 on a virtualised LPAR (no dedicated physical adapters) and there were 555 filesets to install. Most of my time was spent speed reading copyright notices.

Hardware requirements

Given my reluctance to open data centre doors, and the fact that it's spring in Sydney so I didn't bring my winter woollies with me, I decided to do it all from the comfort of my desk.

Here is the list of hardware I was in physical contact with:

a keyboard

a mouse

a chair

a coffee cup

df shows free space(see update below)After the reboot, I noticed a small but significant change to the df command in AIX 7. It now displays free space by default, without using the -I flag. I still like to have the sizes in Gigabytes, so I used df -g

The default output of the df command hasn't changed from AIX 6 to AIX 7. It just took me to do a migration to notice!

AIX 7 migration - a positive experience

It was a very straightforward migration with no surprises, apart from some warnings about missing filesets to do with virtual devices. I ignored them and they went away (the warnings, not the devices).

I am tempted to use the VM Library rather than NIM for the other LPARs I have to do, as most of the LPARs are on the one physical server and are clients of the same VIO server.

Installing AIX 7.1 was a very positive experience.

From my desk in the remote antipodes, I'm reaping the benefits of virtualisation on IBM Power Systems. Looks to me that IBM has done a great job with AIX 7.1.

If you’re looking forward to some of the new features of AIX 7, but find migrating to the new release a little daunting, then it's time for you to put into practice the underlying principle which drives Australia:

relax!

Why? Because some AIX 7 features will be backported to AIX 6.1 TL 6!

IBM is getting really smart at listening to the needs of the AIX community. We've seen (and participated in) the open beta program for AIX 6 and now for AIX 7. And now, we're seeing more and more where new features which are really helpful are backported to the most recent TLs of earlier releases. This is

jolly
good news

Here are two AIX 7 (and AIX 6.1 TL 6) features you might be interested in:

volume groups just for Solid State Disks

and VIO scsi support for WPARs

Solid State Disk Volume Groups

From AIX 7, and in AIX 6.1 TL 6, there is the ability to create a volume group restricted to a certain PV type, such as Solid State Disks. You might use this for very high usage disks which you typically would like to isolate from other disks. Oracle redo logs would be a prime candidate.

When you run the AIX 7 mkvg command, you can set a PV condition using the –X flag. The default is “none”, which means you can mix and match PV types in the same VG. Or you can use –X SSD to guarantee that slower, non-SSD, disks can never be added to your VG. Here's what the syntax of mkvg looks like under AIX 7.1:

You'll also be able to report per-file statistics to identify the hot files which might be good candidates for SSD disks. The filemon command has been enhanced for reporting these hot files.

vscsi disk support for WPARs

From AIX 7.1 (and back ported to 6.1 TL 6) you can export a disk from the VIO server to a WPAR. You might like to do this to map a rootvg or a data disk for the WPAR.

Although WPARs haven't taken off quite as they might have, it's probably worth looking at them now, especially if your environment has a large number of LPARs.

Not such a jump from 6 to 7

The jump from AIX 6 to 7 doesn’t seem to be anywhere near as big as from 5.3 to 6.1. Even so, there are people who prefer to wait a few months before going to a new release. If you want to dip your toes in, you can upgrade to AIX 6.1 Technology Level 6 and still get some of the new AIX 7 features.