05/22/96
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2
SPECIAL NOTICES
Information in this document is correct to the best of our
knowledge at the time of this writing. Please send feedback
by fax to "AIXServ Information" at (512) 823-4009.
Please use this information with care. IBM will not be
responsible for damages of any kind resulting from its use.
The use of this information is the sole responsibility of
the customer and depends on the customer's ability to eval-
uate and integrate this information into the customer's
operational environment.
+----------------------------------------------------------+
| |
| NOTE: The information in this document has NOT been |
| verified for AIX 4.1. |
| |
+----------------------------------------------------------+
WARNING: Please do not use this document if the system is a
/usr client, diskless client, or dataless client.
WARNING: Do not use this procedure on an AFS fileserver.
Doing so may remove all AFS fileserver data because fsck
would be run on the AFS volumes.
CAUSES OF AN LED 552, 554, OR 556
The known causes of an LED 552, 554, or 556 during IPL on a
RISC System/6000 are:
o A corrupted file system.
o A corrupted journaled-file-system (JFS) log device.
o A bad IPL-device record or bad IPL-device magic number.
(The magic number indicates the device type.)
o A corrupted copy of the ODM (Object Data Manager) data-
base on the boot logical volume.
o A fixed disk (hard disk) in the state of inactive in the
root volume group.
SUMMARY OF THE RECOVERY PROCEDURE
To diagnose and fix the problem, you will need to boot from
BOSboot diskettes or tape and run fsck (file-system check)
on each file system. If the file-system check fails, you
may need to perform other steps.
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 1
05/22/96
STEPS
WARNING: Please do not use this document if the system is a
/usr client, diskless client, or dataless client.
1. Turn the key to the Service position.
2. With BOSboot diskettes or tape OF THE SAME VERSION AND
LEVEL AS THE OPERATING SYSTEM, boot the system.
+----------------------------------------------------------+
| |
| WARNING: If you boot a 3.2 system with 3.1 media, or |
| boot a 3.1 system with 3.2 media, then you will not be |
| able to use the standard scripts (getrootfs or |
| /etc/continue) to bring your workstation into full main- |
| tenance mode. |
| |
| Moreover, performing the scripts on a 3.1 system with |
| 3.2 boot media may actually remove some files and |
| prevent your system from booting successfully in normal |
| mode until missing files (/etc/mount and /etc/umount) |
| are replaced on the disk. |
| |
+----------------------------------------------------------+
NOTES:
a. For information on BOSboot diskettes, refer to
InfoExplorer or order fax #2462 from 1-800-IBM-4FAX
(or 415-855-4FAX outside the U.S. from a fax machine
phone).
b. If booting from diskettes, when you see LED c07,
insert the next diskette. This may be the optional
display extensions diskette (required for AIX 3.2.5)
or the display diskette.
c. If you have AIX 3.2.5, used a fddi network install,
and selected that option for the install device, you
will need the communications extensions diskette.
If this diskette is not available, you can build it
from another system with fddi installed or call your
branch office for assistance.
NOTE: If you get a 552, 554, or 556 on this step, the
diskette or tape is bad, and the machine is trying to
boot off the fixed disk. Try it again with new BOSboot
diskettes or tape.
Follow the prompts to the installation/maintenance menu.
3. Choose the maintenance shell (option 5 for AIX 3.1,
option 4 for AIX 3.2).
4. Determine the hdisk# to use with the getrootfs or
/etc/continue command. If you have only one disk, then
"hdisk0" is the proper hdisk# to use. If you have more
than one disk, do the following:
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 2
05/22/96
o FOR AIX 3.2.4 OR LATER:
Run
getrootfs
The output indicates the disk that should be used
with getrootfs in the next step.
o FOR AIX 3.1 TO 3.2.3E:
Run
lqueryvg -Atp hdisk# | grep hd5
for each hdisk# (hdisk0, hdisk1, etc.) until you get
output that looks like:
00005264feb3631c.2 hd5 1
The exact output you get will be different but will
follow the form
large_number.x hd5 1
You may find more than one disk has this output.
These will all be disks which belong to the rootvg
volume group. You may use any of the disks identi-
fied to be in rootvg in the following step.
5. Now access the rootvg volume group by running
/etc/continue (for AIX 3.1) or getrootfs (for AIX 3.2).
('#' is the number of the fixed disk, determined in step
4.)
For AIX 3.1 only, run
/etc/continue hdisk# sh
For AIX 3.2 only, run
getrootfs hdisk# sh
If you get errors from /etc/continue or getrootfs, do
not continue with the rest of this procedure. Correct
the problem causing the error. If you need assistance
correcting the problem causing the error, contact one of
the following:
o local branch office
o your point of sale
o 1-800-CALL-AIX (to register for fee-based services)
All of the above avenues for assistance may be billable.
| 6. Run the following commands to check and repair file
systems. (The "-y" option gives fsck permission to
repair file systems when necessary.)
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 3
05/22/96
fsck -y /dev/hd1
fsck -y /dev/hd2
fsck -y /dev/hd3
fsck -y /dev/hd4
For AIX 3.2 only, also run
fsck -y /dev/hd9var
If fsck indicates that
o block 8 could not be read, the file system is prob-
| ably unrecoverable. See step 7. for information on
unrecoverable file systems.
o block 8 could be read, but one of the following
errors is given:
"fsck: Not an AIXV3 file system."
"fsck: Not a recognized file system type."
| then go to step 8.
o a file system has an unknown log record type, or
| fsck fails in the logredo process, then go to step
| 9.
| o THE FILE SYSTEM CHECKS WERE SUCCESSFUL, SKIP TO STEP
| 11.
7. The easiest way to fix an unrecoverable file system is
to recreate it. This involves deleting it from the
system and restoring it from a backup. Note that hd2
and hd3 can be recreated but hd4 cannot be recreated.
If hd4 is unrecoverable, you must reinstall AIX. For
assistance with unrecoverable file systems, contact your
local branch office or point of sale, or call
1-800-CALL-AIX to register for fee-based support ser-
vices. DO NOT FOLLOW THE REST OF THE STEPS IN THIS FAX.
8. Attempt to repair the file system with this command:
(replace "hd#" with the appropriate file system)
dd count=1 bs=4k skip=31 seek=1 if=/dev/hd# of=/dev/hd#
| Now skip to step 10.
9. A corruption of the JFS log logical volume has been
detected. Use the logform command to reformat it.
On AIX 3.1 only, run
/etc/aix/logform /dev/hd8
On AIX 3.2 only, run
/usr/sbin/logform /dev/hd8
Answer YES when asked if you want to destroy the log.
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 4
05/22/96
| 10. Repeat step 6 for all file systems that did not success-
| fully complete fsck the first time. If step 6 fails a
| second time, the file system is almost always unrecover-
| able -- see step 7 for an explanation of the options at
| this point. In most cases, step 6 will be successful.
| In that case, continue to step 11.
11. With the key in Normal position, run the following com-
mands to reboot the system:
exit
sync;sync;sync
shutdown -Fr
As you reboot in Normal mode, notice how many times LED
551 (that's right, 551, not 552) appears. If LED 551
appears twice, fsck is probably failing because of a bad
fshelper file. If this is the case and you are running
| AFS, see step 15.
The majority of LED 552s, 554s, and 556s will be
resolved at this point. If you still have an LED 552,
554, or 556, try the following steps:
| 12. Repeat steps 1 through 4.
| 13. For AIX 3.2, skip to step 14.
For AIX 3.1 only, run the following commands, which will
remove much of the system's configuration, saving it to
a backup directory.
exit
mkdir /etc/objrepos/bak
cp /etc/objrepos/Cu* /etc/objrepos/bak
cp /etc/objrepos/boot/Cu* /etc/objrepos
Determine which disk is the boot disk with the lslv
command. The boot disk will be shown in the PV1 column
of the lslv output.
lslv -m hd5
Save the clean ODM database to the boot logical volume.
('#' is the number of the fixed disk, determined with
the previous command.)
savebase -d /dev/hdisk#
| If you are running AFS, go to step 15. Otherwise, go to
| step 16.
14. For AIX 3.2 only, run the following commands, which will
remove much of the system's configuration, saving it to
a backup directory.
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 5
05/22/96
mount /dev/hd4 /mnt
mount /dev/hd2 /usr
mkdir /mnt/etc/objrepos/bak
cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
cp /etc/objrepos/Cu* /mnt/etc/objrepos
/etc/umount all
exit
Determine which disk is the boot disk with the lslv
command. The boot disk will be shown in the PV1 column
of the lslv output.
lslv -m hd5
Save the clean ODM database to the boot logical volume.
('#' is the number of the fixed disk, determined with
the previous command.)
savebase -d /dev/hdisk#
| If you are running AFS, go to step 15. Otherwise, go to
| step 16.
15. If you are running the Andrew File System (AFS), use the
following commands to find out whether you have more
than one version of the v3fshelper file.
For AIX 3.1:
cd /etc/helpers
For AIX 3.2:
cd /sbin/helpers
Then in both AIX 3.1 and 3.2,
ls -l v3fshelper*
If you have only one version of the v3fshelper file (for
example, "v3fshelper"), proceed to step 17.
If there is a version of v3fshelper marked as original
(for example, "v3fshelper.orig"), run the following com-
mands:
cp v3fshelper v3fshelper.afs
cp v3fshelper.orig v3fshelper
16. WARNING: Please do not proceed further if the system is
a /usr client, diskless client, or dataless client.
| Recreate the boot image (hdisk# is the fixed disk deter-
| mined in step 13 or 14.):
bosboot -a -d /dev/hdisk#
| 17. If you copied files in step 15, copy the AFS file-system
helper back to v3fshelper:
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 6
05/22/96
cp v3fshelper.afs v3fshelper
18. Turn the key to Normal position and run
shutdown -Fr
If you followed all of the above steps and the system still
stops at an LED 552, 554, or 556 during a reboot in Normal
mode, you may want to pursue further system recovery assist-
ance from one of the following:
o local branch office
o your point of sale
o 1-800-CALL-AIX (to register for fee-based services)
All of the above avenues for assistance may be billable.
For reasons of time and the integrity of your AIX operating
system, the best alternative at this point may be to
reinstall AIX.
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 7
05/22/96
READER'S COMMENTS
Please fax this form to (512) 823-4009, attention "AIXServ Informa-
tion". You may also e-mail comments to: elizabet@austin.ibm.com.
These comments should include the same customer information requested
below.
Use this form to tell us what you think about this document. If you
have found errors in it, or if you want to express your opinion about
it (such as organization, subject matter, appearance) or make sug-
gestions for improvement, this is the form to use.
If you need technical assistance, contact your local branch office,
point of sale, or 1-800-CALL-AIX (for information about support offer-
ings). These services may be billable. Faxes on a variety of sub-
jects may be ordered free of charge from 1-800-IBM-4FAX. Outside the
U.S. call 415-855-4329 using a fax machine phone.
When you send comments to IBM, you grant IBM a nonexclusive right to
use or distribute your comments in any way it believes appropriate
without incurring any obligation to you.
NOTE: If you have a problem report or item number, supplying that
number may help us determine why a procedure did or did not work in
your specific situation.
Problem Report or Item #: Branch Office or Customer #:
Be sure to print your name and fax number below if you would like a
reply:
Name: Fax Number:
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
END OF DOCUMENT (led552.krn, 4FAX# 1457)
Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 8