Failed upgrade due to old installation files found in upgrade path

Publication Date: 2012-07-25Views: 65Downloads: 0

Issue Description

Immediately after upgrading U2000 from V100R005C00SPC601 to V100R006C00SPC200, customer tried to load the new U2000 license file provided by Huawei for the new version. This operation, performed using the U2000 client, failed. The upgrade was performed entirely by customer himself, using the upgrade guide provided by Huawei.

Alarm Information

Loading the new license failed.

Handling Process

1. At first, we resend the V100R006 license file to customer, to ensure he was using the right file. At the same time, we emphasized that this should be transferred in ASCII mode, according the the upgrade guide. This, however, did not resolve the problem.

2. Then we asked customer to run the ESN tool on the server and send the ESN number. We matched this against the U2000 license file we provided for version V100R006C00SPC200, and found them to be equal. We could thereby rule out that the problem was caused by the license file itself.

3. Next, we tried to update the license file using the command line tool on the server. The result was the same as before, “invalid license”. We concluded the problem was also not related to the U2000 client, but must be related to the U2000 server.

4. We contacted R&D to help troubleshoot the problem. R&D suspected the problem could be related to the lic_agent process. We checked the database according to instructions from R&D, and found that in one table, the U2000 version is described as V100R005, not V100R006 as expected. R&D recommended to update the database manually, and restart the license process, but after this operation, the U2000 client and system monitor are unable to login.

5. R&D analyzed logs, and outputs from the database, and finally provided the solution to solve the license problem - but only to find other problems related to the nemgr processes deployed on this system. At this time, it is clear that the upgrade is not successful. We investigated the install files and logs files related to the upgrade, and found that the /opt/upgrade/upgrade/project/NMS folder contains both the upgrade files from V100R003 to V100R005 and from V100R005 to V100R006. The upgrade files from V100R003 to V100R005 are from a previous upgrade, and should have been deleted by customer, according to the upgrade guide. The /opt/HWENGR/upgrade/project/NMS/ directory also contained the upgrade files for V100R003 to V100R005.

6. At this point, we performed a roll-back to V100R005. Then we removed all the upgrade files, and extracted only the files related to the V100R006 upgrade. After performing the upgrade again, the system worked without any issues.

Root Cause

After receiving the first information from customer, we suspected the problem could be caused by either of the following:
1. Customer is using the wrong license file (i.e. the license file for V100R005 and not V100R006).
2. Mismatch between ESN of the license file and the server.
3. “Corrupted” license file (i.e. transferred by FTP in binary mode instead of ASCII mode).
4. Problem with the U2000 client.
5. Problem on the U2000 server after upgrade.

Suggestions

What initially seemed to be a simple problem turned out to be critial. The problem was caused by customer not following the upgrade guide carefully, and did not delete the old installation files before upgrading. When customers decide to perform this kind of important upgrade on his own hands, we need to emphasize that the instructions provided by Huawei should be followed closely, and customer should not perform any other operations without proper guidance.