08/29/95
7135 RAIDiant Array --Common Installation Issues
SPECIAL NOTICES
Information in this document is correct to the best of our
knowledge at the time of this writing. Please send feedback
by fax to "AIXServ Information" at (512) 823-4009.
Please use this information with care. IBM will not be
responsible for damages of any kind resulting from its use.
The use of this information is the sole responsibility of
the customer and depends on the customer's ability to eval-
uate and integrate this information into the customer's
operational environment.
+----------------------------------------------------------+
| |
| NOTE: The information in this document has NOT been |
| verified for AIX 4.1. |
| |
+----------------------------------------------------------+
ABOUT THIS DOCUMENT
This document contains setup tips for the IBM RAIDiant Array
7135 model 110. This document applies to AIX 3.2.5.
INSTALL THE RAIDIANT ARRAY SOFTWARE FIRST
You should install the RAIDiant Array software before
attaching the 7135 to your RISC System/6000.
If you have already booted with the 7135 attached before you
installed the RAIDiant Array software, extra SCSI disk
drives will have been configured into the system. Remove
them by doing the following:
1. List all configured 7135 hdisks with "lsdev -Ct array".
2. Remove disks from the configuration with "rmdev -l
-d".
3. Power off the 7135 after removing the drive definitions
from the system and leave it off until you have
installed the RAIDiant Array software.
After installing the software you can attach the 7135, power
it on and boot the RISC System/6000.
RAIDIANT ARRAY SOFTWARE VERSIONS AND UPDATES
Once the software is installed, you can determine which
version you have with "lslpp -l scarray.obj".
If you get a line of output showing "scarray.obj
03.02.00.00" (the 3.2 level) you have the code that supports
active-passive controllers. If you have "scarray.obj
7135 RAIDiant Array --Common Installation Issues 1
08/29/95
03.03.00.00" or "scarray.obj 03.03.00.01" (the 3.3 level)
you have the code that supports dual-active controllers.
3.3.0.1 level is required to support the 2.2 GB or 4.5 GB
disk drives. We recommend that you get the dual-active code
as it provides better performance if you have two control-
lers and has design changes to prevent problems that can
occur in the active-passive code. If you do not have
installation media for scarray.obj, or you have active-
passive and would like dual-active code, you can order it
free by calling IBM Direct (800-IBM-CALL) and asking for
feature code (FC) 5059 for AIX. To get to 3.3.0.1 requires
that you install the dual-active code and several fixes
listed below.
For all 7135 installations, we recommend these fixes (note
that the latest superceding fix is in parenthesis):
U433447 (U435254) SMIT support for 7135
U434467 (U437507) Diagnostic support for 7135
U435115 Device Driver/Diagnostics enhancements
U436790 (U437507) Device Driver/Diagnostics enhancements
U436971 mkvg fix for PVs with > 1016 PPs/PV
For 7135s connected via the SCSI-2 Differential F/W adapter
the following fixes are required:
U436450 (U437739) SCSI-2 Fast/Wide Device Driver
U429285 SCSI-2 Fast/Wide Differential Enhancement
For the scarray.obj 3.2 active-passive code:
U432241 (U436353) 7135 fixes
For 2.2 GB and 4.5 GB disk drive support (scarray.obj 3.3
only):
U437007 7135 Maintenance Package 3.3.0.1
U437287 7135 and iostat fixes
You can check to see if you have these fixes with "lslpp -lB
...". Either a state of APPLIED or COMMITTED
means that it is installed, and a PTF number in parenthesis
indicates a superceding PTF. If you ordered AIX with FC
5059 after 5/26/95, you will have these fixes on your
install tape.
Be sure you have a copy of the correct version of the doc-
umentation. For scarray.obj 3.2, you should have the "READI
and REACT Software Guide for the IBM 7135 RAIDiant Array
Model 110" SC23-2572-01. For scarray.obj 3.3, you should
have "7135 RAIDiant Array for AIX, Installation and Refer-
ence" SC23-2728-01.
7135 RAIDiant Array --Common Installation Issues 2
08/29/95
VERIFY THE RISC SYSTEM/6000 SEES THE 7135 CORRECTLY
If your RISC System/6000 configures the 7135 correctly, you
will have a disk array router (dar) device for each 7135.
For each disk array controllers (dac), you will have a dac
device. And for each LUN, you will have an hdisk device.
Each bank of 5 physical drives in the 7135 comes preconfig-
ured as a RAID-5 LUN. For example, if you have a 7135 with
two controllers and 15 disk drives, and you have one
internal disk drive in your RS/6000 (hdisk0), you will have
a dar0, dac0, dac1, hdisk1, hdisk2 and hdisk3 as follows:
dac0 Available 00-03-00-00 7135 Disk Array Controller
dac1 Available 00-04-00-10 7135 Disk Array Controller
dar0 Available 7135 Disk Array Router
hdisk1 Available 00-03-00-00 7135 Disk Array Device
hdisk2 Available 00-03-00-01 7135 Disk Array Device
hdisk3 Available 00-03-00-02 7135 Disk Array Device
Use the following commands to list these devices:
lsdev -C | grep Array (lists all the 7135 devices)
lsdev -Ct dar (lists only the dar)
lsdev -Ct dac7135 (lists only the dacs)
lsdev -Ct array (lists only the 7135 hdisks)
All of these devices should be in the "Available" state
(this assumes that the devices are powered on and connected
to the RISC System/6000). If you have devices in the
"Defined" state, you may have inaccurate information in the
customized device database and need to correct it. For
example, if the 7135 was attached to the RISC System/6000
before the RAIDiant Array software was installed, you will
have additional "Defined" hdisks. Phantom or ghost disks
can also be created when the 7135 is originally attached to
an adapter in one slot and is later attached to another
slot. Use the following procedure to clean up the database
(if you all the devices do not show up in the Available
state, you may have a hardware problem):
1. Remove all the 7135 information from the device database
using one of the following methods:
o With the following script:
lsdev -Ct array | cut -d ' ' -f1 | xargs -n1 rmdev -d -l
lsdev -Ct dar | cut -d ' ' -f1 | xargs -n1 rmdev -d -l
lsdev -Ct dac7135 | cut -d ' ' -f1 | xargs -n1 rmdev -d -l
o Manually:
a. Remove all 7135 and non-existent hdisks using
"rmdev -l -d" for each hdisk where
is one of the disks you want to remove
from the database.
b. Remove all dars using "rmdev -l -d" for
each dar.
7135 RAIDiant Array --Common Installation Issues 3
08/29/95
c. Remove all dacs using "rmdev -l -d" for
each dac.
o Using smit (with the 3.3 code only) follow the menu:
# smit array_menu
Disk Array Controller Configuration
Remove a Disk Array Subsystem
dar0 Available 7135 Disk Array Router
KEEP definition in database [no]
2. Reconfigure the devices using "cfgmgr" or reboot the
system. The system will "walk the bus" and configure
the devices it finds.
3. Verify that the devices configured correctly using
"lsdev -C | grep Array".
4. Update the boot logical volume using "savebase -v" so it
contains the correct device information.
Note that for scarray.obj 3.2, running "cfgmgr" or
rebooting does not always put the 7135 hdisks in the
Available state (e.g., after the hdisks have become
Defined due to a hardware problem or after a reinstall),
and you must run "ace -z dac0" to get them into the
Available state.
VERIFY SCSI CABLING
The SCSI-2 Fast/Wide Differential Adapter has two SCSI
busses: internal and external. You cannot have drives that
are part of rootvg attached to the same adapter as a 7135.
If the system does not see all the array controllers, or
they come up in the Defined state, you may have a cabling
problem. Use the following procedure to make sure the SCSI
adapter can communicate to the array controller (this trans-
mits a SCSI inquiry command to a selected SCSI address and
listens for a response).
1. Determine the SCSI adapter location address and the
array controller SCSI IDs. If the array controller is
configured, you can use the "lsdev -Cc array" command.
For the example above:
dac0 Available 00-03-00-00 7135 Disk Array Controller
dac1 Available 00-04-00-10 7135 Disk Array Controller
dac0 is connected to the SCSI adapter with location code
00-03-00, and dac0's SCSI ID is 0 (the 7th digit).
2. If you can't list the dacs, you must physically check
it's SCSI ID (you must lower the fans on the back of the
7135 controller and then you can see the windows on the
left showing the SCSI IDs) and determine which adapter
it is cabled to. The default and preferred SCSI ID
switch settings are 0 and 1.
7135 RAIDiant Array --Common Installation Issues 4
08/29/95
3. Varyoff all 7135 volume groups using "varyoffvg
".
4. Put the 7135 hdisks and dars in the Defined state with
"rmdev -l ".
5. Run "diag" and choose the following:
Service Aids
SCSI Bus (press Enter twice)
6. Select the SCSI adapter attached to one of the dacs.
7. Press F4 and choose the SCSI ID of the dac.
8. Press F7 to commit and send the SCSI inquiry.
If the command fails, there is a problem with either the
SCSI cabling, the SCSI adapter, or the dac.
RAID LEVELS
The 7135 comes preconfigured with each bank of five disks
configured as a RAID 5 LUN; thus, from AIX's point of view
it looks like a 7.67 GB hdisk (assuming you have 2 GB drives
in the 7135). We expect most customers will use the default
configuration. Due to the RAID 5 write penalty, you should
expect slower performance than you would have with standard
SCSI disk drives. The write penalty is a result of the
fault tolerance RAID 5 provides.
We don't expect many customers to use RAID 3. It is not
possible to create a volume group with RAID 3 hdisks on a
7135; consequently, you cannot create a file system and
regular AIX files and your application will have to use raw
I/O to the hdisk using RAID 3.
RAID 0 stripes the data across the drives in the LUN, thus
providing higher performance, but no fault tolerance.
RAID 1 mirrors the data across two drives. Note that the
AIX LVM provides mirroring capability without using a 7135.
Mirroring with the 7135 offloads the CPU and AIX from doing
the mirroring.
VOLUME GROUPS AND LOGICAL VOLUME GUIDELINES
When you create a volume group (VG), generally you need to
use a physical partition size greater than the default. The
LVM limits the number of physical partitions (PPs) to 1016
in a physical volume (PV). The number of default 4 MB PPs
required for the 8 GB default hdisk in a 7135 would exceed
the 1016 limit. No warning will be given for this condition
unless you have U436971 installed in which case it will not
allow you to create a VG violating this limit.
You cannot change the PP size in a volume group without
deleting the volume group and recreating it. You can check
how many PPs are in your 7135 hdisks by running "lspv
7135 RAIDiant Array --Common Installation Issues 5
08/29/95
" and looking at the "TOTAL PPs:" field. This field
should not exceed 1016.
In general, you should not mix hdisks with different RAID
characteristics in the same volume group. If you mix RAID 0
and RAID 5 hdisks, and a file system has partitions on both
hdisks, and you lose a drive in the RAID 0 LUN, you will
have lost all your data in the file system.
Be aware that changing many parameters of the LUN will cause
you to lose all the data on the LUN. AIX does not know how
to handle a hdisk that changes size. Consequentially, you
cannot easily change a LUN once it is loaded with data. For
example, if you want to increase a LUN's size, you will have
to backup all the data, unmount and remove the file systems,
remove the hdisk from the volume group, change the LUN, add
the hdisk back to the volume group, recreate and mount the
file systems and restore the data.
If performance is important, you should plan your volume
groups and logical volume placement before you create them
because recreating them and restoring your data can be time
consuming. Intra-disk and inter-disk allocation policies
still apply for RAID hdisks. However, if you implement sub-
luns, creating a filesystem in the "center" of the disk is
actually creating a filesystem in the center of the sub-lun
which may be on the edge of the physical disks. You can get
performance planning assistance from the AIX Support Fami-
ly's Consult Line service.
MICRCODE
Microcode is code that is run by hardware other than a CPU,
and for the 7135, we are concerned with microcode for the
SCSI adapters, disk array controllers (dac), physical disks
in the 7135, and for the EEPROMs in the 7135. This micro-
code is typically loaded as a file on the RISC System/6000
and is downloaded to the device. For the SCSI 2 differen-
tial adapters, the code is downloaded when the system boots.
For the Fast & Wide SCSI adapter, the dacs and the 7135
disks, you download the code by using diagnostic service
aids from AIX.
Incompatible levels of microcode are a common source of
problems. Ensure that the latest microcode you have avail-
able on your system is installed for all the 7135 compo-
nents.
After changing any microcode, or the EEPROM, you must shut-
down the RISC System/6000, power off the 7135, and reboot
for changes to take effect. You can change the microcode
for all the devices and reboot once. Generally, the order
of assuring and updating the microcode is as follows:
1. SCSI Adapters (scsi or ascsi)
2. Disk Array Controller (dac)
3. EEPROM
7135 RAIDiant Array --Common Installation Issues 6
08/29/95
4. Physical Disks
WARNING: Do not run diagnostics on the 7135 until all micro-
code has been loaded and the 7135 has been powered off and
on.
The microcode files are described below. Keep in mind that
the file names may change (generally to higher numbered file
names) as improvements are made.
SCSI ADAPTER MICROCODE
The 7135 is attached to a SCSI-2 Differential (narrow)
adapter or to a SCSI-2 Differential Fast & Wide adapter.
Each dac can be connected to a separate adapter on separate
SCSI buses, or both dacs can be connected to the same
adapter on the same SCSI bus.
You can list the SCSI adapters on your system with "lsdev
-Cc adapter | grep scsi".
NARROW ADAPTER
For the narrow adapter (type 4-2) determine the level of
microcode with "lsattr -El -a ucode" where
is the scsi adapter attached to the 7135, e.g., "lsattr -El
scsi1 -a ucode". You will see the microcode file listed.
The latest supported file is /etc/microcode/8d77.a0.16.
FAST & WIDE ADAPTER
For the Fast & Wide adapter(s) (type 4-6) use "lscfg -vl
ascsi*" and check the ROS Level and ID field for a value of
72, which is the microcode level. We recommend that you do
not attach internal disks to this adapter if it is also con-
nected to a 7135. If you do not have a microcode level of
72, and you have the 7135 connected to more than one
RS/6000, you will need ECA 192 (ordered by the CE), or for
the SP2 ECA 008, that provides the microcode on a diskette.
If your 7135 is attached to only one RS/6000, a microcode
level of 70 or 72 is OK. Use the documentation that comes
with the ECA to install the microcode. The microcode has a
file name of /etc/microcode/8EFC3011.72M. You can use the
following procedure if (i) you have the dosread command
installed on the system, (ii) you have PTF U429285 or
U429287 installed, AND (iii) the adapter is not used as a
boot adapter:
1. Varyoff any volume groups using the 7135.
2. Put the child devices of the ascsi in the Defined state
using "rmdev -l " including any 7135 hdisk, dar,
dac, vscsi and tmscsi devices.
3. Run "diag".
4. Choose "Service Aids".
5. Choose "Microcode Download".
7135 RAIDiant Array --Common Installation Issues 7
08/29/95
6. Choose the adapter to which you want to download the
microcode.
7. Choose "Functional Microcode".
8. Choose "Download the latest level of microcode".
9. Select "Yes" to the question "do you have the microcode
diskette?"
10. Insert the diskette into the diskette drive and press
Enter.
11. After the message indicating success, you should also
run advanced diagnostics, system verification on the
adapter.
12. Shutdown and reboot the system for the new microcode to
take effect.
DISK ARRAY CONTROLLER MICROCODE
To determine the level of microcode on the dac, run
lscfg -vl dac0 | grep Z2
You will see a line similar to
Device Specific.(Z2)........010010
For scarray.obj 3.2, the numbers should be 010010, and, for
scarray.obj 3.3, the numbers should be 020604 or 020605.
The 020604 microcode file is stored on the RISC System/6000
as /etc/array/fw/7135.02.01. The 020605 file is
/etc/array/fw/7135.02.06. We recommend the latest level of
microcode. If you have installed U437007, have 2.2 or 4.5
GB drives in your 7135, you must have the 020605 level of
microcode installed on the dac.
If your microcode is not correct, you can download the
latest level you have on disk with the following procedure:
1. Varyoff any volume groups using the 7135.
2. Run "diag".
3. Choose "Service Aids".
4. Choose "7135 RAIDiant Array Service Aids".
5. Choose the active array controller (usually dac0).
6. Choose "Array Controller Microcode Download".
7. Choose "Download microcode from the hard disk".
8. Choose "Download the latest level of microcode".
7135 RAIDiant Array --Common Installation Issues 8
08/29/95
If you get an error, try removing the hdisks, dar, and dacs
with "rmdev -l -d". Then run cfgmgr and try
again.
7135 EEPROM
SCARRAY.OBJ 3.2
To update the EEPROM with scarray.obj 3.2:
1. First, varyoff any 7135 volume groups.
2. Run the following command:
/usr/lpp/scarray/seteeprom /dev/
where is one of the 7135 hdisks.
3. You should download the microcode to the EEPROMs on both
dacs. If you have two array controllers, you need to
switch the active controller and run the command again.
You can switch the active controller with "smit react",
or, e.g., run the following command to switch from dac0
to dac1. (This command does not apply for scarray.obj
3.3.)
/etc/methods/chgdar -l dar0 -a pri_controller=dac1
SCARRAY.OBJ 3.3
To set the EEPROM with scarray.obj 3.3:
1. Varyoff any 7135 volume groups.
2. Run "diag" and choose
Service Aids
7135 RAIDiant Array Service Aids
Update EEPROM
3. Then choose the active dac and choose "YES" to download
it.
This will download the /etc/array/fw/7135eeprom.01 file
to the EEPROM.
4. Be sure to update both array controllers with this pro-
cedure if you have two of them. If you are installing
scarray.obj 3.3 and have not installed scarray.obj 3.2,
to update the EEPROM file on the second controller you
will have to run the following command.
chdev -l dar0
This will make both controllers active (i.e., dual-active).
At this point, you can update the EEPROM on the second con-
troller.
7135 RAIDiant Array --Common Installation Issues 9
08/29/95
7135 PHYSICAL DISK MICRCODE
For scarray.obj 3.2, to download the disk microcode, run
"diag", and choose "Service Aids", "7135 RAIDiant Array
Service Aids", "dac0" or "dac1" whichever is the active con-
troller, and "Spare Physical Disk Microcode Download". In
scarray.obj 3.3, the procedure is the same except on the
last menu the choice is titled "Physical Disk Microcode
Download". You should download the disk array controller
microcode and update the EEPROM before downloading the phys-
ical disk microcode.
There is no method to determine what microcode is downloaded
on the disks. You have to download the microcode to each
disk in the 7135 separately. It is OK to download this code
more than once. The microcode for 3.2 is in the file:
/etc/array/fw/0664M1H.A0900617.00020520
For scarray.obj 3.3, two files exist:
0664M1H.A0900617.00020540
0664M1S.A0900617.00020540
These are compatible with the 020604 level of dac microcode.
The new microcode will work with scarray.obj 3.2, but 3.3
requires the new microcode. The 020605 level of dac micro-
code is compatible with physical disk microcode with
filenames ending in 5A0 rather than the 540 listed previ-
ously.
For the 2.2 GB and 4.5 GB drives, no physical disk microcode
files exist on the system; consequentially, no microcode
download is required for these drives.
BACKUPS
Even though the 7135 provides fault tolerance, you should
still backup your data. Note that a "mksysb" backup will
not backup data that is not in the root volume group.
Sysback/6000's full system backup will backup data in all
volume groups. However, since the 7135 device drivers do
not run in maintenance mode, you can only restore data from
the 7135s after the rootvg has been restored and the system
has booted. If you have to reinstall a system with a 7135,
Sysback or mksysb, power off the 7135 during the boot from
tape, otherwise you may have extra "Other SCSI Disk" drives
defined in the system database, and your hdisk names may be
out of order after the reinstall.
7135 RAIDiant Array --Common Installation Issues 10
08/29/95
READER'S COMMENTS
Please fax this form to (512) 823-4009, attention "AIXServ Informa-
tion". You may also e-mail comments to: elizabet@austin.ibm.com.
These comments should include the same customer information requested
below.
Use this form to tell us what you think about this document. If you
have found errors in it, or if you want to express your opinion about
it (such as organization, subject matter, appearance) or make sug-
gestions for improvement, this is the form to use.
If you need technical assistance, contact your local branch office,
point of sale, or 1-800-CALL-AIX (for information about support offer-
ings). These services may be billable. Faxes on a variety of sub-
jects may be ordered free of charge from 1-800-IBM-4FAX. Outside the
U.S. call 415-855-4329 using a fax machine phone.
When you send comments to IBM, you grant IBM a nonexclusive right to
use or distribute your comments in any way it believes appropriate
without incurring any obligation to you.
NOTE: If you have a problem report or item number, supplying that
number may help us determine why a procedure did or did not work in
your specific situation.
Problem Report or Item #: Branch Office or Customer #:
Be sure to print your name and fax number below if you would like a
reply:
Name: Fax Number:
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
END OF DOCUMENT (7135.inst.tips.325.ha_, 4FAX# 3140)
7135 RAIDiant Array --Common Installation Issues 11