Oracle Database Support FAQ

What is Oracle CDM? How does it help solve my challenges?

Database administrators working extensively with Oracle are challenged when faced with mission-critical use cases such as Backup, Recovery, DevOps, and Business Analytics. This is especially true given that their Oracle databases have expanded in size and number over time, and that the databases need to be up and running 24x7x365.

Oracle DBAs struggle with the following:

Backups are slow, complex and need constant management

Backup process slows down the production servers

Recoveries are slow and complex

Repurposing App consistent backups (clones) for DevOps and Business Analytics is slow, complex and storage inefficient

ECX simplifies Oracle database copy management by enabling administrators to orchestrate application-consistent copy creation, cloning and recovery in minutes, instead of hours or days. ECX copy management leverages the advanced snapshot and replication features of the underlying storage platform to rapidly create, replicate, clone, and restore copies of Oracle databases in the most efficient way possible, in both time and space. ECX enables you to focus on the backup and recovery requirements of your business rather than the technical details of the underlying storage platforms.

ECX is an intelligent copy data management solution that delivers end-to-end automation, orchestration, and self-service functionality for your Oracle environment through a comprehensive and scalable Inventory. With the self-service features of ECX, your users are empowered to create clones on demand, freeing DBAs, while at the same time offering the advanced recovery features needed for Oracle environments.

Do I need to deploy any additional agents to protect Oracle standalone or Oracle RAC servers?

ECX for Oracle is delivered as a VMware OVA that is easily deployed on demand in a matter of minutes. Once deployed, you simply register your Oracle servers with appropriate credentials and then let ECX discover the rest. ECX eliminates the complexity of manually deploying and maintaining application agents on Oracle servers. A lightweight application-aware component is automatically injected to the required Oracle servers on demand and automatically updated to the latest version if required.

ECX auto-discovers databases and enables copies only of eligible databases. To be eligible for ECX backup, the Oracle database needs to be residing on a supported storage platform. With ECX, application owners do not need to be concerned about storage infrastructure.

ECX creates and uses in-place copies, so no data is physically moved. Replication to off-host storage is performed by leveraging storage replication, which reduces the amount of impact on Oracle Servers and Databases. ECX generated application-consistent copies are both space and time efficient. With the same ease, a DBA can automate the creation of remote copies for DR use cases.

Does ECX Oracle solution leverage the Storage Consistency Group feature?

The storage consistency group feature allows storage administrators to take a snapshot of database applications where the data is spread across multiple volumes to maintain consistency across all volumes.

In a typical Oracle Database, the data is spread across different volumes for better IO performance and availability. ECX Oracle application-consistent backup creation ensures that appropriate consistency groups are automatically created to maintain consistency across all related volumes.

What level of selection granularity is supported for Oracle backup workflows?

ECX Oracle backup workflows support copy selection at the following levels:

One or more Oracle home locations

One or more databases

One or more container databases for Oracle 12c

Why can I not select some of the databases for protection in an Oracle backup workflow?

You cannot select a database if it is not eligible for protection. Hover your cursor over the database name to view the reasons the database is ineligible, such as that the database files, control files, or redo log files are stored on unsupported storage.

If you select the parent Oracle Home in a Backup job definition, all databases under it are protected. If a new database is added under the home, it will be automatically protected once it is cataloged. Discovery and cataloging of new Oracle databases occurs as part of regularly scheduled Oracle Inventory job.

Does the Oracle database need to be on supported storage for ECX?

Yes, the ECX Oracle solution leverages storage snapshots for database protection. All databases, database files, control files and redo logs must be on supported storage systems for it to be eligible for protection.

Does ECX leverage the Oracle 12c Storage Snapshot feature?

This new feature of Oracle 12c enables you to take a storage snapshot of your database without needing the database to enter BACKUP mode. In Oracle, when you need to recover, you can use a point in time of the snapshot. You can roll forward by using the database archive logs, and use this snapshot feature to recover part or all of the database. ECX fully supports this feature starting in ECX 2.5.1.

Does ECX support protection of Offline Databases?

Databases in offline mode are not automatically included during backup workflows, unless they share storage volumes with a selected database that is active. ECX marks the offline databases and does not present them for Oracle Restore Workflows. Users may be able to retrieve these database files as flat files and perform application mount outside of ECX.

Does ECX support protection of Oracle databases not running in Archive Log mode?

Yes, protection of Oracle Databases running in NOARCHIVELOG mode are now supported for both Inventory and Backup use cases. You can recover database to the point of the most recent snapshot. PIT recoveries are not supported for databases running in NOARCHIVELOG mode.

Does ECX support protection of Oracle databases using pFile (text initialization parameter files)?

Yes, ECX now supports ECX backup of databases started through pFile in addition to spFile.

Oracle Archive log management

Does ECX support archive log backup and log management?

Oracle DBMS creates database transaction logs as part of its operation. Oracle databases can run in the following logging modes:

NOARCHIVELOG mode – In NOARCHIVELOG mode, no transaction logs are created, and there is no capacity to run point-in-time recovery or online backups. This is the default.

ARCHIVELOG mode – In ARCHIVELOG mode, the database makes copies of all online redo logs after they are filled. These copies are called archived redo logs. The archived redo logs are created via the ARCH process. The ARCH process copies the archived redo log files to one or more archive log destination directories. These saved archived logs are used for point-in-time recovery.

ECX automatically discovers the location where Oracle writes archived logs. If this location resides on storage from a supported vendor, ECX can protect it. If the existing location is not on supported storage, or if you wish to create an additional backup of database logs, enable the Create Additional Archive Log Destination option in the Oracle Backup job definition, then specify a path that resides on supported storage. When enabled, ECX configures the database to start writing archived logs to this new location in addition to any existing locations where the database is already writing logs. If multiple databases are selected for backup, then each of the servers hosting the database must have their destination directories set individually.

Can I specify a retention period for backed up archive logs within ECX?

If the Create Additional Archive Log Destination option is selected, ECX automatically manages the retention of only those archived logs that are under the new destination specified in the job definition. After a successful backup, logs older than the backup are automatically deleted from the ECX-managed destination.

If the Use Existing Archive Log Destination(s) option is selected in Oracle Backup job definition, ECX does not automatically purge any archived logs. The retention of archived logs must be managed externally, for example using RMAN. In order to support point-in-time recovery, ensure that the retention period is at least large enough to retain all archived logs between successive runs of the Oracle Backup job.

Oracle RMAN integration

Does the ECX Oracle solution integrate with RMAN?

Oracle Recovery Manager (RMAN), a command-line and Enterprise Manager-based tool, is the method preferred by Oracle DBAs for backup and recovery of Oracle databases, including maintaining an RMAN repository.

ECX creates application-consistent Oracle database copies simply – with no need to build and maintain complex RMAN protection scripts. At the same time, ECX automates cataloging of Oracle database copies in the RMAN recovery catalog. This enables DBAs to leverage RMAN for:

Verification - ECX-created Oracle database copies can be instantly mounted so they can be easily verified through the RMAN verify command

Job-level prescripts and postscripts are scripts that can be run before or after a job runs.

Snapshot prescripts and postscripts are scripts that can be run before or after a storage-based snapshot subpolicy runs. (Please refer to pre/post script topic in the ECX User’s Guide for details.)

Data Masking

Does the ECX Oracle solution offer Data Masking integration with third party masking tools?

A concern for security officers in any organization is to keep confidential information locked down, even internally. Data masking is used to hide confidential data, by replacing it with fictitious data, when making data copies for DevTest or other use cases. It prevents leakage of sensitive data in non-production databases via static data masking [SDM], and production data in transit via dynamic data masking [DDM].

ECX includes integrated data masking workflows with the ability to leverage third party masking tools. Traditionally, data masking is difficult, slow, and storage-consuming, but with ECX it is easily integrated into the Oracle Backup workflow, allowing creation of masked copies at a specified frequency. Masked copies are automatically marked in the Inventory. Access to secure copies is managed by the administrator by leveraging the application-level RBAC.

Then, when your TestDev, DevOps, or research/analytics work is completed, you can save the clone to more permanent storage or simply tear it down.

Can I create multiple clones from the same database snapshot?

Yes, multiple copies can be provisioned by ECX from the same snapshot. These copies can be either mounted with new DB names on the same Oracle Server or with same/different names on different Oracle servers.

How much space is used by my Oracle clones?

ECX utilizes zero-footprint instant clone capabilities provided by storage arrays. These clones do not use any additional space (except for maintaining pointers). Only new changes/writes to the read/writable clones utilize space.

What is the granularity of Database recovery supported by ECX?

Supported recoveries for standalone or RAC configurations

Instant recovery of Oracle databases regardless of the size of the database

Recover database(s) to original or to a new server (physical or virtual) simply with few clicks

Recover database(s) with new names simply with few clicks

Recover at DR site from replicated copies simply with few clicks

Database can be recovered to point of snapshots to original or new location

Perform PIT recoveries to original or new location simply with few clicks

Recover RAC databases to any node on RAC simply with few clicks

Database running on older versions can be recovered to an instance running same or newer version (ECX inherits any limitations defined by Oracle)

Each selected database in a Restore job can have a separate destination specification. Databases can be recovered using a new name to an original or new instance

You can select one or more databases in a single Restore job definition

Recoveries are supported across the same storage type (i.e. ASM to ASM and standalone to standalone)

Databases are always recovered in online mode

Users can additionally perform advanced fine-grained recoveries via RMAN integration. You can use ECX to instantly mount required snapshot copies to a specified Oracle server and use RMAN to recover. All RMAN-supported recoveries are available to users.

What granularity of Point-in-Time recovery is supported?

ECX enables you to recover databases to a specific point-in-time to enable you to:

Restore to the state just before the point of failure.

Restore multiple databases to a consistent time.

During PIT (point-in-time) Oracle database recoveries, if no log file snapshot exists that is newer than the chosen recovery PIT, the job creates a fresh log snapshot and uses it for recovery.

What happens to Database SID during recovery?

The Oracle System ID (SID) is used to uniquely identify a particular database on a server. One cannot have more than one database with the same SID on a single server. When using RAC, all instances belonging to the same database must have unique SIDs. A Restore job definition provides the user with an option to define a new name for a database during recovery. ECX will rename the SID if the user chooses a new database name during recovery, otherwise the original SID will be used.

How is the Oracle Initialization parameter file (init.ora) processed during Database recovery?

The Oracle initialization parameter file (init.ora) is created by the DBA and defines the overall instance configuration, such as how much memory should be allocated to the instance, the file locations, and internal optimization parameters.

ECX catalogs the initialization parameters during a job and uses them during recovery. ECX provides advanced options to control the behavior of processing initialization parameters used to start up the recovered database in Instant Database Recovery and DevOps workflows.

Customizable options allow you to use the same parameters as the source or specify a template pfile file to use.

Additionally, cluster-related parameters like instance_number, thread, and cluster_database are set automatically by ECX depending on the appropriate values for the destination.

ECX provides advanced options for Instant Database Recovery and DevOps workflows to override the behavior of mount point handling during recovery. The Mount Point Rename option provides these selections:

Append a timestamp: ECX appends a timestamp to the original mount point.

Do not rename: ECX uses the same path/name for mount points or ASM diskgroups as the source.

Add a custom prefix: Select this option to specify a custom prefix to be prepended to the source paths/names. The prefix value may contain leading or trailing slashes. In the case of ASM diskgroup names, the slashes are removed.

Add a custom suffix: Select this option and specify a custom suffix to be appended the source paths/names.

Replace a substring: Select this option to specify a custom string of characters in the old mount point to be replaced with another string of characters.

Can I recover an Oracle Database running on a Linux physical server to Oracle running on a Linux VM?

Yes, the recovered database will be recovered as Physical RDM (pRDM).

Can I recover an Oracle Database running in Linux VM to Oracle running on a Linux Physical Server?

Yes, the pRDM configured database will be recovered as LUNs on the Physical Linux Oracle server. Oracle configured with VMDK can only be recovered to VM.

Can I set up an Oracle Database to automatically be refreshed on a schedule?

The database "refresh" can be achieved by checking the "Allow overwrite and force cleanup of old session" option in the Restore job definition. You can run a job that spins up a database and the session will go into a Pending state awaiting cleanup. You can then run the same job again at a later time (either manually or scheduled) and it will automatically clean up the previous session before starting the new one. The cleanup process automatically stops and dismounts the database from the previous session. The new session will then mount and start the database again from the chosen snapshot (which is typically the latest as of runtime).

Where are Oracle specific and ECX specific logs if errors occur?

All required logs (ECX and Oracle application) are collected as part of the current log collection functionality. There should be no need to manually obtain Oracle application logs from within Oracle Servers.

Can a persistent agent be installed?

All required plugins are automatically injected on demand and automatically updated to the latest version if required.

Transparent Data Encryption (TDE) stops would-be attackers from bypassing the database and reading sensitive information from storage by enforcing data-at-rest encryption in the database layer. This is application-layer encryption that wouldn’t typically impact ECX. The encryption keys live outside the database in a "wallet" that is managed separately by the administrator. ECX will create copies of the data on server A and mount them on server B, for example. The database software on server B should be able to read the encrypted data as long as the necessary keys are installed in the wallet there.

How do I refresh? How do I promote to Production?

All database recovery operations can leverage Instant Recovery mode (Test) and can then either be deleted or promoted to permanent mode via workflow control. This behavior is controlled via the job definition option Make Permanent.

Enabled - Always make permanent

Disabled - Never make permanent

User election - Allows the user to select Make Permanent or Cleanup when the job session is pending

Does ECX provide Oracle specific reporting?

Reports are offered to assure that your Oracle databases are sound and that your ECX jobs are verified. Reports provided by ECX specifically for application support include:

Application Configuration Report, which describes valuable system information about your Oracle Database Servers, and affirms that Oracle is configured correctly to be eligible for backup creation.

Application RPO Compliance Report, which determines which of your Oracle database servers are not in compliance with your RPO parameters, and displays the reasons for their non-compliance.

[1] For Oracle 12c multitenant databases, ECX supports protection and recovery of the container database, including all pluggable databases under it. Granular recovery of specific PDBs can be performed via Instant Disk Restore recovery combined with RMAN.

[2] Standalone databases protected by ECX can be recovered to the same or another standalone server as well as to a RAC installation. When recovering from standalone to RAC, if the source database uses Automatic Storage Management, then it will be successfully recovered to all nodes in the destination cluster. If the source database uses non-ASM storage, the database will be mounted only on the first node in the destination RAC. Source disks for standalone databases restoring to a virtual RAC environment must be thick provisioned.

RAC databases protected by ECX can be recovered to the same or another RAC installation as well as to a standalone server. In order to recover a RAC database to a standalone server, the destination server must have Grid Infrastructure installed and an ASM instance must be running.

[5] RAC database recoveries are not server pool-aware. ECX can recover databases to a RAC, but not to specific server pools.

[6] ECX supports recovering databases from a source physical server to a destination virtual server by provisioning disks as physical RDMs. Similarly, ECX can recover databases from a source virtual server that uses physical RDM to a destination physical server. However, source databases on VMDK virtual disks can only be recovered to another virtual server and not to a physical server.

[7] ECX does not support VADP-based protection of virtual Oracle servers. Oracle data must reside directly on one of the supported storage systems listed above.

[8] On AIX LPAR/VIO servers, Oracle data must reside on disks attached to the server using NPIV. Virtual SCSI disks are not supported.

[9] Linux LVM volumes containing Oracle data must use LVM version 2.02.118 or above. On SLES 11 SP4, this version of LVM may not be available through the official repositories, in which case, databases running on or recovered to SLES 11 SP4 systems must use ASM or non-LVM filesystems only.

[10] Masking and DevOps recoveries are not supported on virtual servers.

[12] Oracle servers registered as virtual must have VMware Tools installed and running.

[13] Oracle 12c multithreaded configurations are not supported.

[14] Data masking is not supported for Oracle in NetApp storage environments. Masking is not supported on source database on NFS (a copy of which will be cloned and masked) or source databases on replica copies. Instead of the default mirror copy, you must select a snapshot copy as a replication source.

[15] NetApp systems running in 7-Mode are not supported.

Note: Oracle database data and the flash recovery area (FRA) must reside on supported storage systems. ECX can back up archived logs to a supported storage system if they are not already on one.

For Oracle RAC clustered nodes running prior to vSphere 6.0, virtual machines cannot use a virtual SCSI controller whose SCSI Bus Sharing option is set to None. This is required to ensure that ECX can hot-add shared virtual disks to the cluster nodes. For vSphere 6.0 and above, this requirement does not apply. Instead, if an existing shared SCSI controller is not found on vSphere 6.0 and above, ECX automatically enables the "multi-writer" sharing option for each shared virtual disk.

Does ECX support Oracle running on a VMware VM?

Oracle support for VMware virtual machines requires Oracle data/logs to be stored on VMDK virtual disks or Physical RDMs (pRDM). Virtual RDM disks are not supported. The VMDKs must reside on a datastore created on LUNs from supported storage systems. Similarly, the Physical RDMs must be backed by LUNs from supported storage systems.

Granular recovery of specific Pluggable Databases (PDBs) can be performed via Instant Disk Restore recovery combined with RMAN. To do so:

Perform an Instant Disk Restore of the Container Database (CDB) by using an Oracle Restore job in ECX. The Oracle CDB backup may already have been cataloged into RMAN when the backup was created, or you can opt to do this during the Instant Disk Restore.

Login to RMAN. List the copies in the catalog and identify the tag from which you want to recover. Tags for ECX-created entries are generally of the form "ECX_<timestamp>".

Close the existing PDB by running the following command:

alter pluggable database <PDB_name> close;

Recover the PDB by running the following command:

run {

restore pluggable database <name> from tag '<tag_name>';

recover pluggable database <name>;

}

Open the recovered PDB by running the following command:

alter pluggable database <PDB_name> open;

Can I back up Oracle running on any storage to a supported storage system via VADP (VM Replication job)?

No, not through ECX Oracle Backup workflow. You can leverage a VMware Backup job with pre/post script to protect Oracle database in such configuration.

Oracle Requirements

Review the following requirements and pre-requisites for registering an Oracle provider in ECX.

Software

The bash and sudo packages must be installed. Sudo must be version 1.7.6p2 or above. Run sudo -V to check the version.

To achieve this, the ECX agent user must belong to the OSASM operating system group, typically named asmadmin.

Shell user limits for the ECX agent user must be the same as those for the user that owns the Oracle home, typically named oracle. Refer to Oracle documentation for requirements and instructions on setting shell limits. Run ulimit -a as both the oracle user and the ECX agent user and ensure their settings are identical.

ECX discovers Oracle installations and databases by looking through the files /etc/oraInst.loc and /etc/oratab, as well as the list of running Oracle processes. If the files are not present in their default location, the "locate" utility must be installed on the system so that ECX can search for alternate locations of these files.

ECX discovers databases and their storage layouts by connecting to running instances and querying the locations of their datafiles, log files, etc. In order for ECX to correctly discover databases during cataloging and copy operations, databases must be in "MOUNTED," "READ ONLY," or "READ WRITE" mode. ECX cannot discover or protect database instances that are shut down.

Databases must be started using a server parameter file (spfile). ECX does not support copy operations for databases that are started using a text-based parameter file (pfile).

ASM Disk Discovery

When ECX mounts snapshots/clones of ASM disks, it configures the disks to set the appropriate permissions required to make them discoverable by ASM:

The disk owner and group are set to the owner of the Grid installation and the OSASM group respectively. These are typically grid and asmadmin. ECX automatically discovers the appropriate owner and group information on each server.

The disk permissions are set to 660.

Additionally, ECX creates aliases/symbolic links with names that follow a consistent pattern. To ensure that ASM is able to discover the disks mapped by ECX, you must update the ASM_DISKSTRING parameter to add this pattern.

Linux:

ECX creates udev rules for each disk to set the appropriate ownership and permissions. The udev rules also create symbolic links of the form /dev/ecx-asmdisk/<diskId> that point to the appropriate device under /dev.

To ensure the disks are discoverable by ASM, add the following pattern to your existing ASM_DISKSTRING: /dev/ecx-asmdisk/*

AIX:

ECX creates a device node (using mknod) of the form /dev/ecx_asm<diskId> that points to the appropriate hdisk under /dev. ECX also sets the appropriate ownership and permissions for this new device.

To ensure that the disks are discoverable by ASM, add the following pattern to your existing ASM_DISKSTRING: /dev/ecx_asm*

Notes:

If the existing value of the ASM_DISKSTRING is empty, you may have to first set it to an appropriate value that matches all existing disks, then append the value above.

If the existing value of the ASM_DISKSTRING is broad enough to discover all disks (for example, /dev/*), you may not need to update it.

Refer to Oracle documentation for details about retrieving and modifying the ASM_DISKSTRING parameter.

Sample Configuration of an ECX Agent User

The commands below are examples for creating and configuring an operating system user that ECX will use to log in to the Oracle server. The command syntax may vary depending on your operating system type and version.

Create the user that will be designated as the ECX agent user: useradd -m ecxagent

Set a password if using password-based authentication: passwd ecxagent

If using key-based authentication, place the public key in /home/ecxagent/.ssh/authorized_keys, or the appropriate file depending on your sshd configuration, and ensure the correct ownership and permissions are set, such as:

If ASM is in use, also add the user to the OSASM group: usermod -a -G asmadmin ecxagent

Place the following lines at the end of your sudoers configuration file, typically /etc/sudoers. If your existing sudoers file is configured to import configuration from another directory (for example, /etc/sudoers.d), you can also place the lines in a new file in that directory: