October 2014

April 2012

April 24, 2012

Today is a major announcement day for IBM i with Technology Refresh 4 (TR4). Steve's “You and i” blog reviews the content of this announcement.

Last year, IBM provided support to suspend an i partition, which I previously wrote about in “Suspend my i”.

The most significant function in IBM i TR4 is Live Partition Mobility. With Live Partition Mobility, you can take an active, running IBM i partition and move it from one frame to another frame, without disruption to your applications. I'm not going to discuss more about what mobility is and why you might want to use it, as Steve covered that in his blog. What I do want to write about is a little more detail on just what you need to accomplish this.

There are both hardware and software prerequisites:

POWER7 tower/rack system for both the source and destination systems

Firmware service pack 730_xx, 740_xx, or later

IBM Hardware Management Console V7R7.5.0M0, or later

All I/O must be virtual and supported by VIOS; it can be Virtual SCSI (VSCSI), Virtual Fibre Channel (NPIV) or Virtual Ethernet

The configuration must use only external storage and that storage must be accessible to both the source and destination systems

Both source and destination systems must be on the same Ethernet network

IBM i 7.1 TR4 PTF group – SF99707 level 4

PowerVM Enterprise Edition on both the source and target system

VIOS 2.2.1.5 (FP25 SP3) on both the source and target system [[CORRECTION: It's VIOS 2.2.1.4]]

There are a few restrictions that are too detailed to review in this blog; the Live Partition Mobility documentation in the IBM i developerWorks site reviews the complete set of prerequisites, requirements and restrictions.

The actual move of a partition is initiated from the HMC. IBM Systems Director VMControl has relocation, which you can imagine will ultimately support IBM i as well.

In general, applications and the operating system are unaware that the partition is moved from one system to another. There are some exceptions to this, such as Collection Services; when the partition is starting to run on the target system, the Collection Services collector job will cycle the collection so correct hardware information is recorded on the target system.

There are two new work management exit points that are part of the IBM i suspend and mobility support. These exit points allow to you run a program before and after a partition is suspended or moved. (These exit points are the same, regardless of whether you are suspending a partition or moving a partition).

Work with Registration Information

Type options, press Enter.

5=Display exit point 8=Work with exit programs

Exit

Exit Point

Opt Point Format Registered Text

_ QIBM_QWC_RESUME RSMS0100 *YES Resume system

_ QIBM_QWC_SUSPEND SSPS0100 *YES Suspend system

The suspend exit is called before the partition is suspended or moved and is called two twice. The first time it is called to check to see if the operation is allowed. The second call is to prepare for the move or suspend operation.

The resume exit is called after the move is complete or after the system has been restarted from a suspended state. Applications can use these exit points to monitor and accept or refuse move or suspend requests. Running an exit program after a move or suspend also allows an application to refresh any copies of any system-specific information, log information about mobility requests for licensing or other purposes, adjust heart beat rates or whatever actions an application may require.

For more information on these exit points, refer to the “What's New”for APIs in the IBM i 7.1 Information Center.

The following messages may be logged to the QSYSOPR message queue as part of the suspend or move request:

CPI09A5 - Partition suspend request in progress.

CPI09A6 - Partition not ready for suspend request.

CPI09A7 - Partition suspend request canceled.

CPI09A8 - Partition resumed after migration.

CPI09A9 - Partition resumed from hibernation.

The system serial number will change when a partition has been moved. The model and processor feature may also change.

The documentation on enhancements added to IBM i via Technology Refreshes can be found on the IBM developerWorks web site.

Other sources for documentation on Live Partition Mobility for IBM i include:

The handling of inactive job time out (QINACTITV) is being changed with a 7.1 PTF (the other system values are unaffected) to address two long-standing problems.

Accuracy. Prior to this change, a job can be inactive for up to twice the specified interval before an action is taken. For inactivity intervals measured in hours, this level of uncertainty is simply too high.

Performance. A large number of jobs can be ended at once when the subsystems check for inactivity. This can affect overall system performance at the time the jobs are being ended.

The change is available only for the 7.1 release with PTF SI46398. This is a delayed PTF, so it requires an IPL to put in on or to take it off.

First of us, let's review this change as it pertains to accuracy.

Prior to this change, the subsystem would wait for the QINACTITV number of minutes between the times it checked for inactive interactive jobs. Jobs can be inactive for nearly two intervals before the inactivity is detected; if a job runs just after the system checks for inactivity, at the next check it will not be inactive for the full interval, so the timeout will not occur until the following interval. For example, if you have QINACTITV set to 60 minutes, jobs may need to be inactive for nearly 120 minutes for the inactivity to be detected.

The PTF improves the timeliness in which inactivity is identified.

The QINACTITV function is still enforced by the subsystem code periodically checking for inactivity, but with the change, the subsystem tries to check more often to improve the accuracy. In most cases, the subsystem will now check for inactivity every 10 minutes and will keep track of how long a job has been inactive. If the QINACTITV system value is less than 15 minutes, the subsystem will continue to use the value specified by QINACTITV. (Checking for inactivity every 10 minutes only helps for the cases where QINACTITV is greater than or equal to 15 minutes).

A job can still be inactive for longer than the QINACTITV system value. For any given QINACTITV, you can work out an exact value for how long a job can be inactive before an action is taken. The job becomes inactive sometime during the 10 minutes between checks. So, in the simple case where the QINACTITV value is evenly divisible by 10, a job can be inactive for up to 10 minutes plus the QINACTITV value. But the QINACTITV value does not have to be evenly divisible by 10 and the subsystem cannot take action until the next time it checks for inactivity. For an QINACTITV value such as 101, a job can be inactive for at most the QINACTITV time plus 19 minutes.

Consider an example where QINACTITV is set to 90 minutes. With the old code, the subsystem would take the action defined by the QINACTMSGQ system value if a job was inactive anywhere from 90 minutes to 180 minutes. With the new code, the subsystem will take action if a job has been inactive anywhere from 90 minutes to 100 minutes.

Consider another example. In this case QINACTITV is set to 45 minutes. With the old code, the subsystem would take action if a job was inactive anywhere from 45 minutes to 90 minutes. With the new code, the subsystem will take action if a job has been inactive anywhere from 50 minutes to 60 minutes.

A job will be inactive for at least the QINACTITV value before action is taken. But sometimes you might be interested in "at most" rather than "at least". If you have a requirement that jobs must not be left inactive for longer than some specified value, you need to set the QINACTITV value to something less than the limit that you want to enforce. With the old code, the QINACTITV value had to be half of the limit that you wanted to enforce. With the new code, the QINACTITV value can be much closer to the limit that you want to enforce.

If system value QINACTMSGQ is set to the name of a message queue and the job identified by the CPI1126 message remains inactive and is not ended or disconnected, another CPI1126 is sent the next time the subsystem checks for inactivity. This behavior is unchanged, but the subsystem may be checking for inactivity much more often. This could affect the logic of whatever program is handling messages sent to that message queue. The program will have a lot less time before it gets another CPI1126 message for a given job.

It can still be useful to set the QINACTMSGQ (Inactive job message queue) system value to the name of a message queue so that you can control which jobs get ended or disconnected and how many jobs get processed at a time. Activity can be uneven and can be influenced by external factors such as lunchtime. A significant number of jobs can become inactive within the same 10-minute interval. This is less of a problem with a 10-minute interval than it is with an interval measured in hours. Also, due to other changes in the code, this is less of a problem when the interactive work is spread across multiple subsystems.

Now on to the performance improvements provided by this PTF.

Subsystems that are started at the same time will no longer check for inactivity at the same time. When the subsystems start or when the QINACTITV system value is changed, the subsystems now use a small but variable delay when setting up the checking for inactivity. This helps reduce the number of jobs that are disconnected or ended at a single time. With the old code, you could introduce delays between starting subsystems and these delays would translate more or less directly into delays between the times the different subsystems would check for inactivity. With the new code, every subsystem has to check every 10 minutes, but the system takes care of spreading out the work. Now, the only reason to have delays between starting subsystems would be to manage the amount of work done when starting the subsystems.

When a subsystem sets up its checking for inactivity, it has no idea how many other subsystems will need to do this same type of work. In addition to checking for inactive interactive jobs, subsystems also check the number of prestart jobs that are not being used. PTF SI46398 spreads out the work for ending of inactive interactive jobs and for ending of unused prestart jobs.

Because it is still the subsystem that checks for inactivity, much remains the same. But the system now reacts to inactivity sooner and spreads out the work better. This makes the enforcement of QINACTITV more predictable and reduces the impact on overall system performance.

I'd like to thank Dan Tarara for this blog; Dan is a member of the IBM i work management team. Thanks, Dan!

April 10, 2012

Hopefully most of you now know that in the 6.1 release, IBM moved the Work with System Activity (WRKSYSACT) command from the Performance Tools licensed program product into the base operating system. WRKSYSACT has been around for a long time, but it has a few features that you may not be aware of.

Work with System Activity is a great tool for getting a basic understanding of what is happening on your partition in terms of performance. WRKSYSACT will show you jobs, threads, and tasks that are using system resources – CPU utilization, synchronous and asynchronous I/O, storage allocation information, etc. WRKSYSACT is the only green-screen command interface in the operating system where you will find information about tasks. Other command interfaces – such as WRKACTJOB – only show information about jobs and threads.

It would take far too much space to write about all of the things you can see through WRKSYSACT; rather, there are a few features that I want to discuss in more detail.

WRKSYSACT in Batch Mode

First of all, did you know that you can run WRKSYSACT in a batch monitoring mode and write the performance data to a file? You enable the batch mode with the OUTPUT parameter; by default this is * and the information is presented on your display. But there are two additional options, *FILE and *BOTH, which will write the performance data to a database file (QAITMON in QPFRDATA by default). You can specify the sample intervals with the INTERVAL parameter (which also controls the auto-refresh intervals as well) and how many intervals are collected (NBRITV parameter). When you take the option to write the data to a file, a job named WRKSYSACT is submitted to QBATCH to collect the performance data.

The Print Activity Report (PRTACTRPT) command can then be used to print out a report from the sample data collected by WRKSYSACT. The PRTACTRPT command can generate a summary report as well as a detailed report and has options to allow you to specify a timeframe of interest, specific jobs you may want in the report, as well as what criteria the report should use in ordering.

Wait Information

Wait information will tell you what a particular job, thread or task is waiting on when it is waiting to do more work. I wrote about wait accounting in “i Can Tell You Why You're Waiting” some time ago. You can also see wait information using WRKSYSACT; you simply need to determine which job/thread/task you want to look at, and use option 6, Wait detail, for that job or task. Option 6 will bring up a display that will list the wait categories for that job/thread/task. Be aware that this is a high-level summary of the kinds of wait for that job or task. However, it make give you some clues about what job or task you need more detailed information for; you can then use the Performance Data Investigator with Collection Services or Job Watcher data to view detailed wait information.

Storage Allocated/Deallocated

WRKSYSACT will also show the storage allocated or deallocated by a job/thread/task by pressing the F11 key to get to View 4. I wrote about “Understanding Disk Space Usage” and in that blog reviewed WRKSYSACT and the storage allocated/deallocated information.

Scaled CPU

Scaled CPU time is a metric that can be used to determine if energy management features, such as power capping or power savings, are in effect on your IBM i partition. These energy management capabilities can be turned on through Active Energy Manager, which is a plug-in to IBM Systems Director. When you use these energy management features, the system's processor may run at lower or higher frequencies, which can affect your partition's performance. I wrote about this topic at length in the blog “i Can Understand Scaled CPU Time.”

On the WRKSYSACT display, the value “Average CPU rate” will be displayed if you are using energy management capabilities. This rate is an easy way to determine if your processor has been slowed down due to power capping or power savings.

One final note – one thing that has not changed with WRKSYSACT is the limit of one user at any given time. WRKSYSACT collects a lot of information from critical interfaces in the operating system and the limit of one user ensures limited contention on those critical interfaces.

April 03, 2012

A few weeks ago, I wrote about IBM Systems Director 6.3 and IBM i, summarizing the major enhancements in the latest release of Systems Director that are applicable to i. I briefly mentioned that you can now get information about your IOA cache batteries through Systems Director inventory. This week, I want to expand on the support within Systems Director for IOA cache batteries and tell you how you can monitor their status with Director. This blog expands upon all the other ways you can Display the Status of your IOA cache batteries.

First of all, the support for IOA cache batteries information in the Systems Director inventory requires that you have the latest IBM Universal Manageability Enablement for i licensed program product (5770UME) installed on each IBM i partition you will be working with. This program product is part of IBM i at no additional charge, but it isn't installed by default. The process of installing the underlying Systems Director agents required for IBM i is most succinctly documented in the PDF for Installing Systems Director Agents on IBM i.

When you collect inventory for IBM i with Systems Director 6.3, you will now be able to view information about your cache batteries as part of that inventory. The default columns aren't the most useful however; if you click on the resource for the cache battery inventory (e.g., DC01), you can see the properties that are available, as the following image shows:

You can customize the columns that are displayed to directly view more useful information--such as days to warning and days to error--as you can see in the following screen capture:

However, you can do much more than simply view the status of your IOA cache batteries through Systems Director. You can now monitor the status of your cache batteries and receive CIM indications for the “Days to Error” value.

You create your own monitor for IOA cache batteries through the Create Monitor wizard. Start with the IBM i you wish to monitor and drill into the CIM monitoring metrics. The monitor path you need to follow is IBMi/CIM Agent/root/cimv2/IBMi_IOACacheBattery->device (where device is one of the device selections you can see). The following screen capture shows how you find this within theCreate Monitor wizard:

You will see a list of possible metrics to monitor, but not all of them are implemented for IOA cache batteries. Of particular importance for monitoring are the Days to Warning and Days to Error metrics. The following screen capture shows you the battery metrics that are applicable.

Once you have created a monitor with the desired metrics, you can then set thresholds and automation plans just like you can for other monitors within IBM Systems Director. In my contrived example, I set low level thresholds on the Days to Error metric for warning and critical levels. Because my IOA cache battery has many days left until it gets into an error state, I had to set my levels to artificially high levels to force the threshold to trigger.

When an IOA Cache Battery event occurs, it's logged in the event log just like any other event in IBM Systems Director. You can create event filters and enable automation plans to easily get notified about a pending cache battery low warning or error. And remember, all of the kinds of things that you can do with a single resource can generally be done with groups of resources, so using IBM Systems Director, you can set up thresholds, event filters, and automation plans to monitor ALL of your IBM i partitions for potential cache battery issues.

IBM Systems Magazine is a trademark of International Business Machines Corporation. The editorial content of IBM Systems Magazine is placed on this website by MSP TechMedia under license from International Business Machines Corporation.