The views expressed on this blog are my own and do not necessarily reflect the views of Oracle

March 26, 2016

Quorum disks in Exadata

An Exadata quarter rack has two database servers and three storage cells. In a typical setup, such a system would have three ASM disk groups, say DATA, RECO and DBFS_DG. Usually the disk group DATA would be high redundancy and the other two disk groups would be normal redundancy. The high redundancy disk group guards against the simultaneous failure of two partner disks or the complete failure of two storage cells.

There is a high availability problem with this setup. The loss of two storage cells would bring down the clusterware, which would in turn bring down all databases in the cluster; even those with datafiles in the high redundancy disk group. This is because the clusterware voting disks would be in a normal redundancy disk group, and the loss of two storage cells would mean the loss of the majority of the voting disks. The voting disks cannot be in the high redundancy disk group because that disk group would need five failgroups, and we cannot have a disk group with five failgroups in a quarter rack as we only have three storage cells.

The Exadata software version 12.1.2.3.0 introduces the functionality to create quorum disks on database servers. Those quorum disks can then be added to the high redundancy disk group to make it suitable for voting disks. With such a setup, we would have true high availability storage in a quarter rack, or any elastic rack with less than five storage cells.

In this post, I will show how to create quorum disks, add them to a high redundancy disk group, and migrate the clusterware voting disks to that high redundancy disk group.

No high redundancy disk groups

As the high availability setup in a quarter rack is questionable, I don't even have a high redundancy disk group, and my voting disks are in the normal redundancy disk group DBFS_DG.

[grid@exadb01 ~]$ asmcmd lsdg

State Type ... Voting_files Name

MOUNTED NORMAL ... N DATA/

MOUNTED NORMAL ... N RECO/

MOUNTED NORMAL ... Y DBFS_DG/

[grid@exadb01 ~]$

To achieve my goal, I will recreate DBFS_DG as a high redundancy disk group and add quorum disks to it. Let's see what else is in that disk group.

[grid@exadb01 ~]$ asmcmd find DBFS_DG "*"

+DBFS_DG/ASM/

+DBFS_DG/ASM/PASSWORD/

+DBFS_DG/ASM/PASSWORD/pwdasm.256.885726993

+DBFS_DG/ENT1/

+DBFS_DG/ENT1/DATAFILE/

+DBFS_DG/ENT1/DATAFILE/DBFS_TS.278.904476591

+DBFS_DG/EXACLUSTER/

+DBFS_DG/EXACLUSTER/ASMPARAMETERFILE/

+DBFS_DG/EXACLUSTER/ASMPARAMETERFILE/REGISTRY.253.885726993

+DBFS_DG/EXACLUSTER/OCRFILE/

+DBFS_DG/EXACLUSTER/OCRFILE/REGISTRY.255.885726995

+DBFS_DG/_MGMTDB/

...

+DBFS_DG/_MGMTDB/CONTROLFILE/

+DBFS_DG/_MGMTDB/CONTROLFILE/Current.260.885727877

+DBFS_DG/_MGMTDB/DATAFILE/

+DBFS_DG/_MGMTDB/DATAFILE/SYSAUX.257.885727813

+DBFS_DG/_MGMTDB/DATAFILE/SYSTEM.258.885727823

+DBFS_DG/_MGMTDB/DATAFILE/UNDOTBS1.259.885727839

...

+DBFS_DG/orapwasm

[grid@exadb01 ~]$

From the above, disk group DBFS_DG has the ASM password file, ASM spfile, the Oracle Cluster Registry (OCR) and the Grid Infrastructure management repository database (MGMTDB). As I would like to recreate the disk group, I need to move those files out to a temporary location, recreate the disk group, and put the files back into the disk group. Also, I would like to achieve all this without downtime or interruption to any of the services in the cluster.

Disk group DBFS_DG might also have the Database Based File System (DBFS) tablespace, and the volume group for the ASM Cluster File System (ACFS). If that is your case, and you would like to follow my example, you would need to take care of those as well.

Move the management repository database out of DBFS_DG

To move the MGMTDB database out of DBFS_DG I will just drop it and recreate it later.

As noted at the start of this post, all my database files are in a normal redundancy disk group. The last step in making this cluster truly highly available, would be to convert disk group DATA from normal to high redundancy. As this post is about the quorum disks, I will show that in a separate post.

Conclusion

With the introduction of the quorum disk feature in Exadata software version 12.1.2.3.0, we can now have the proper high availability setup in Exadata quarter racks or any Exadata elastic configuration with less than five storage cells.

In addition to Exadata software version 12.1.2.3.0, you will need the Grid Infrastructure version 12.1.0.2 + bundle 12.1.0.2.160119 + patch for 22682752 + patch for 22722476 or 12.1.0.2 + 12.1.0.2.160419.