The focus of the paper is on databases and zones. On SuperCluster, customers have the choice of running their databases in logical domains that are dedicated to running Oracle Database 11g R2. With exclusive access to Exadata Storage Servers, those domains are aptly called "Database" domains. If the requirement mandates, it is possible to create and use all logical domains as "database domains" or "application domains" or a mix of those. Since the focus is on databases, the paper talks only about the database domains and how zones can be created, configured and used within each database domain for fine grained control over multiple databases consolidated in a SuperCluster environment.

When multiple databases are being consolidated (including RAC databases) in database logical domains, zones are one of the options that fulfill requirements such as the fault, operation, network, security and resource isolation, multiple RAC instances in a single logical domain, separate identity and independent manageability for database instances.

The best practices cover the following topics. Some of those are applicable to standalone, non-engineered environments as well.

Solaris Zones

CPU, memory and disk space allocation

Zone Root on Sun ZFS Storage Appliance

Network configuration

Use of DISM

Use of ZFS filesystem

SuperCluster specific zone deployment tool, ssc_exavm

ssctuner utility

Oracle Database

Exadata Storage Grid (Disk Group) Configuration

Disk Group Isolation

Shared Storage approach

Dedicated Storage Server approach

Resizing Grid Disks

Oracle RAC ConfigurationSecuring the Databases, and

Example Database Consolidation Scenarios

Consolidation example using Half-Rack SuperCluster

Consolidation example using Full-Rack SuperCluster

Acknowledgements

A large group of experts reviewed the material and provided quality feedback. Hence they deserve credit for their work and time. Listed below are some of those reviewers (sincere apologies if I missed listing any major contributors).

In its simple form, a zone is a virtual operating system environment created within a single instance of the Solaris operating system. Efficient resource utilization is the main goal of this technology.

Solaris 10's zone partitioning technology can be used to create local zones that behave like virtual servers. All local zones are controlled from the system's global zone. Processes running in a zone are completely isolated from the rest of the system. This isolation prevents processes that are running in one zone from monitoring or affecting processes that are running in other zones. Note that processes running in a local zone can be monitored from global zone; but the processes running in a global zone or even in another local zone cannot be monitored from a local zone.

As of now, the upper limit for the number of zones that can be created/run on a system is 8192; of course, depending on the resource availability, a single system may or may not run all the configured zones effectively.

Global Zone

When we install Solaris 10, a global zone gets installed automatically; and the core operating system runs under global zone. To list all the configured zones, we can use zoneadm command:

% zoneadm list -v
ID NAME STATUS PATH
0 global running /

Global zone is the only one:

bootable from the system hardware

to be used for system-wide administrative control, such as physical devices, routing, or dynamic reconfiguration (DR). ie., global zone is the only zone that is aware of all devices and all file systems

from which a non-global zone can be configured, installed, managed, or uninstalled. ie., global zone is the only zone that is aware of the existence of non-global (local) zones and their configurations. It is not possible to create local zones, within a local zone

Steps to create a Local Zone

Prerequisites:

Plenty of disk space to hold the newly installed zone. It needs at least 2G space to copy the essential files to the local zone, and of course the disk space needed by the application(s) you are planning to run, in this zone; and

Since there is more than 5G free space, I've decided to install a local zone under /zones.

% mkdir /zones

Next step is to define/create the zone root. This is the path to zone's root directory that is relative to the global zone's root directory. Zone root must be owned by root user with the mode 700. This will be used in setting the zonepath property, during the zone creation process

In a Sparse Root Zone, the directories /usr, /sbin, /lib and /platform will be mounted as loopback file systems. That is, although all those directories appear as normal directories under the sparse root zone, they will be mounted as read-only file systems. Any change to those directories in the global zone can be seen from the sparse root zone.

However if you need the ability to write into any of those directories listed above, you may need to configure a Whole Root Zone. For example, softwares like ClearCase need write permissions to /usr directory. In that case configuring a Whole Root Zone is the way to go. The steps for creating and configuring a new 'Whole Root' local zone are as follows:

192.168.175.126 is the IP address. It must have all necessary DNS entries

[Added 08/25/08] The whole add fs section adds the file system to the zone. In this example, the file system that is being exported to the zone is an existing UFS file system.

\* set dir=/repo2

/repo2 is the mount point in the local zone

\* set special=/dev/dsk/c2t40d1s6
set raw=/dev/rdsk/c2t40d1s6

Grant access to the block (/dev/dsk/c2t40d1s6) and raw (/dev/rdsk/c2t40d1s6) devices so the file system can be mounted in the non-global zone. Make sure the block device is not mounted anywhere right before installing the non-global zone. Otherwise, the zone installation may fail with ERROR: file system check </usr/lib/fs/ufs/fsck> of </dev/rdsk/c2t40d1s6> failed: exit status <33>: run fsck manually. In that case, unmount the file system that is being exported, uninstall the partially installed zone (zoneadm -z <zone> uninstall) then install the zone from the scratch (no need to re-configure the zone, just do a re-install).

\* set type=ufs

The file system is of type UFS

\* set options noforcedirectio

Mount the file system with the option noforcedirectio[/Added 08/25/08]

\* dir=/opt/csw

read-only path, will be lofs'd (loop back mounted) from global zone. Note: it works for sparse root zone only -- whole root zone cannot have any shared file systems

zonecfg commands verify and commit, verifies and commits the zone configuration for the zone, respectively. Note that it is not necessary to commit the zone configuration; it will be done automatically when we exit from zonecfg tool. info displays information about the current configuration

Observe that the zone appserv has it's own virtual instance of lo0, the system's loopback interface and the zone's IP address is also being served by the eri0 network interface

Login to the Zone {console} and performing the internal zone configuration. zlogin utility can be used to enter a zone. The first time we log in to the console, we get a chance to answer a series of questions for the desired zone configuraton. -C option of zlogin can be used to log in to the Zone console.

That is all there is in the creation of a local zone. Now simply login to the newly created zone, just like connecting to any other system in the network.

[New 08/27/2008] Mounting file systems in a non-global zone

Sometimes it might be necessary to export file systems or create new file systems when the zone is already running. This section's focus is on exporting block devices and the raw devices in such situations i.e., when the local zone is already configured.

Exporting the Raw Device(s) to a non-global zone

If the file system does not exist on the device, raw devices can be exported as they are, so the file system can be created inside the non-global zone using the normal newfs command.

The following example shows how to export the raw device to a non-global zone when the zone is already configured.

After the zonecfg step, reboot the non-global zone to make the block device visible inside the non-global zone. After the reboot, check the existence of the block device; and mount the file system within the non-global zone.

Sometimes it is desirable to have the flexibility of mounting a file system in the global zone or non-global zone on-demand. In such situations, rather than exporting the file systems or block devices into the non-global zone, create the file system in the global zone and mount the file system directly from the global zone into the non-global zone. Make sure to unmount that file system in the global zone if mounted, before attempting to mount it in the non-global zone.

To unmount the file system from the non-global zone, run the following command from the global zone.

# umount /zones/appserv/root/repo1

Removing the file system from the non-global zone

eg.,

Earlier in the zone creation step, the block device /dev/dsk/c2t40d1s6 was exported and mounted on the mount point /repo2 inside the non-global zone. To remove the file system completely from the non-global zone, run the following in the global zone.

The following instructions are for cloning a non-global zone on the same system. The example shown below clones the siebeldb zone. After the cloning process, a brand new zone oraclebi emerges as a replica of siebeldb zone.

Create the equivalent zone orabi on the new host -- use the zonecfg command with the -a option and the zonepath on the new host. Make any required adjustments to the configuration and commit the configuration.

It is possible to force the attach operation without performing the validation. You can do so with the help of -F option

# zoneadm -z orabi attach -F

Be careful when using this option because it could lead to an incorrect configuration; and an incorrect configuration could result in undefined behavior

[New: 07/19/2009]

Tip: How to find out whether connected to the primary OS instance or the virtual instance?

If the command zonename returns global, then you are connected to the OS instance that was booted from the physical hardware. If you see any string other than global, you might have connected to the virtual OS instance.

Alternatively try running prstat -Z or zoneadm list -cv commands. If you see exactly one non-zero Zone ID, it is an indication that you are connected to a non-global zone.