Pages

Monday

We upgraded our SOA install from 11.1.1.3 to 11.1.1.6 and hit a few issues. One of the main issues was that after upgrading DEV our inflight processes disappeared. We talked back and forth with Oracle and it was supposed to be supported but for some reason it wasn't working for us. I know that from SOA 10g to SOA 11g in-flight processes are not supported as part of the upgrade.

One of the ways we use BPEL is workflow for a custom application. It could take months for a workflow to complete and there are always in-flight processes. So there is never a quiet time, or a point at which all workflows are complete to do the upgrade.

We put a plan in place to manually re-place all items back in the proper stage. It seems like it was going to be a fair amount of effort so for DEV we only re-placed a subset of items to make sure we had the process correct.

However, once we upgrade TEST the in-flight processes somehow magically survived. We aren't sure why, there could be some small inconsistencies between DEV and TEST which caused it. Another reason could be that in DEV we upgraded to 11.1.1.6 first, then at a later point upgraded to 11.1.1.6.6. In TEST we went directly to 11.1.1.6.6.

The SOA upgrade itself isn't a very resilient process. If the PSA (Patch Set Assistant) gets interrupted for some reason its not smart enough to recover. In DEV it wasn't a big issue, I had an export of the MDS and SOAINFRA schemas and I just dropped and recreated. For PROD tho, the amount of data, thus downtime, would prevent us from doing this. We only hit an issue once in DEV and TEST was smooth, so for PROD we all had our fingers cross.

The PROD install was going smoothly until I had to run the PSA. Shortly after I started the schema upgrade it failed and a dreaded error appeared. In the $FMW_HOME/oracle_common/upgrade/logs directory I found:

The problem was a database session holding a lock in the SOAINFRA schema. We hit this problem in DEV and the solution was to shutdown applications that access SOAINFRA tables and change the schema password. However, I didn't notice a hung database session by one of our custom applications.

If you try to run the PSA again it says the SOAINFRA schema isn't valid and won't let you continue with the upgrade. So I tried manually updating the registry to state the schema is valid.

update schema_version_registry set status='VALID' where mr_name='SOAINFRA';

Great! The installer started up again but it quickly failed. Checking the logs:

Since this was production, I opened a P1 SR with Oracle right away. In these situations I try to get Oracle involved as quickly as possible. I'll still continue to do research on my end. Sometimes I find the solution quicker, sometimes Oracle Support does.

It provided a set of SQL statements to help rollback the schema upgrade. Only problem was my upgrade was from 11.1.1.3, not 11.1.1.5. Knowing that Oracle now supported manually rolling back the upgrade, then I looked at the script that was failing:

After a few trial and errors I managed to write a script to undo all the changes. Unfortunately if you missed one, which was easy to do as the script is a few thousand lines, you had to start over. Finally I managed to find all updated objects and continued with the upgrade.

Another upgrade issue I encountered showed itself in the startup logs for soa_server1:

After the startup we waited anxiously to hear from the superusers to let us know the status of in-flight processes. To much celebration (more so from them than us since they had alot of potential work to do) in-flight processes did not disappear.

13 comments:

Hi,It is difficult for company formation in Qatar to terminate a registered agency; in addition compensation is payable upon the termination of the agency, including upon the expiry of a fixed term agency.Thanks...