I was running through the labs for the workshop when I came across an error I had never seen before. One of the very first steps in the first lab is to migrate a local PDB to the public cloud CDB using standard unplug and plug. The migration of the PDB worked fine and I was able to open the migrated PDB in the public cloud CDB. However, I forgot to create the extra tablespace in the PDB for the next step in the lab while the PDB was still plugged into my local CDB.

It didn’t seem like a big deal to try to create a standard tablespace in the PDB post public cloud migration. But, when I executed the command, I received the error, ORA-28374: typed master key not found in wallet.

Since I didn’t take an screen shots while it was happening, I decided to recreate the error without having to redo the migration. In fact, any PDB created in an Oracle Public Cloud CDB (other than the one created with the CDB creation) will have the same behavior.

For this demonstration, I have already created a new 12c multitenant database in the Oracle Public Cloud Database Service. Included with the database creation is one PDB called PDB1.

The wallet is open, but the status of OPEN_NO_MASTER_KEY tells me that the master key hasn’t yet been created for this pluggable database.

The next step then is to try to create the master key from PDB2.

SQL> connect sys/oracle_4U@localhost/pdb2.smiller.oraclecloud.internal as sysdba
Connected.
SQL> administer key management set key identified by oracle_4U;
administer key management set key identified by oracle_4U
*
ERROR at line 1:
ORA-46658: keystore not open in the container

The error message seems to indicate that the keystore (another term for the wallet) is not open but this error is actually a bit misleading. We know the keystore is open from the previous query. What it actually means is that keys cannot be modified in an autologin wallet.

So, we need to close the autologin wallet (cwallet.sso) and explicity open the file wallet (ewallet.p12) from PDB2.

SQL> administer key management set keystore close;
administer key management set keystore close
*
ERROR at line 1:
ORA-28365: wallet is not open

This is misleading as well because we know the wallet is open. However, the wallet is controlled from the root container and needs to be closed from there.

The error message and the query show that the wallet is still closed in PDB2 even after opening the wallet from the root container. Opening the wallet in the root container does not open the wallet in the pluggable containers.

Almost there. Although the wallet is open on PDB2, we still don’t have a master key but the WALLET_TYPE and WRL_TYPE columns indicate this is a file wallet, not an autologin wallet which is what want. Now, we should be able to create the master key for PDB2.

SQL> administer key management set key identified by Oracle_4U;
administer key management set key identified by Oracle_4U
*
ERROR at line 1:
ORA-46631: keystore needs to be backed up

It is a requirement to back up the wallet before creating a new master key to keep you from losing your current master key (even if one does not yet exist).

Why might you need your old master keys? If you encrypt your RMAN backups with transparent or dual mode encryption (which both use keys from the wallet), you need the keys you used to encrypt those backups in order to decrypt them if you need to restore those backups. In order to keep those keys around, the wallet needs to be backed up.

Fortunately, you can create the new key and create the backup with a single command.

Now that we have an open wallet and a fresh new master key in PDB2, we should be able to create a tablespace.

SQL> create tablespace test;
Tablespace created.

Success!

I don’t want to have to open the wallet every time I restart the instance, so I want to put the autologin wallet back in place. To do that, we simply create a new autologin wallet from the file wallet.

I found out later that all of the steps shown above can be done from the root container, which makes it a lot easier if you have more (especially if you have a lot more) than one PDB to re-key. Just make sure you are aware that you are re-keying all of the PDBs, regardless of whether they currently have a master key.

To demonstrate, I’ll follow all of the steps to fix the ORA-28374 issue from only the root container. I’m using PDB3 as the new pluggable database instead of PDB2 from the previous demonstration.

The day after going through this waste of time valuable learning experience, I saw a tweet from one of my favorite Germans (don’t worry @Brost, you’re still number one) about a blog post dealing with this very same issue.

It turns out that this is intentional behavior and unique to Oracle Public Cloud databases. Read Mike Dietrich’s blog post for more details.