1. Work with Oracle Support.2. Rebuild the environment since I don't have any backups. Developers lose 1 day.3. Back it up, research the error and see if I can find a solution.

I worked with Oracle support up until 6:30am, at which point I believe my analyst was finished for the day and went home. This is a UAT environment, so the SR is a P2. I wasn't expecting 24x7 support, so this was just bad timing.

I searched metalink and didn't find any solid hits. So I tried google. Not many hits, but I received one exact match, in chinese. Google graciously offered to translate the page for me and about the only useful text on the page was:

"Who came across the internal bug?我用了alter database clear logfile group 后再重启动数据库就发生这个错误了。 I used the alter database clear logfile group restart the database after this error has occurred. "

At this point I had an hour left before testers started and only 2 options remaining. I figure I was going to have to rebuild the database anyways, so i'd try clearing the log files. I cleared the inactive groups without issue. The only group remaining was the current and if the clear logfile group command failed i'd have to proceed with the rebuild.

Thats my 6 hr simple change (most of that time was working with Oracle Support, taking a backup [only 5MB/s IO??!] of the environment before I played around with it, etc.) Now I need some sleep. You know your tired when you almost use dish washing liquid for cream in your coffee. Luckily an attentive coworker prevented disaster! Normally an all nighter doesn't leave me this drained but the previous night I only had 3 hrs sleep.

Wednesday

The solution depends on if you are using Oracle Internet Directory (OID) or not. If you are using OID you can login to the OID Console or the oidadmin GUI to reset the password easily.

But what if your application server is stand alone? If your trying to reset the oc4jadmin superuser account then the only solution is manually modify the $ORACLE_HOME/j2ee/home/config/system-jazn-data.xml file.

Search the file for your the username and place the new password, prefixed with an exclamation point "!" between the <credentials> </credentials> delimiters. I am assuming that you have to prefix it with the ! so that Oracle knows how to handle the password. ie, that its not encrypted.

Restart the application server and verify that you can login.

How about regular user accounts tho? Surprisingly you have to follow the same process. I must have poked around in the Application Server Console for half an hour in disbelief that I couldn't reset a user password without knowing the old password.

If I missed something or if you know of another way to reset user accounts, please post a comment.

Monday

Here is some interesting news from Collaborate today. Oracle has waived some Extended Support fees for some product lines including E-Business Suite 11i10 and 10gR2. Good news for those that can't afford to upgrade at the moment due to the economy.