Please note, in version 5.0 the database storage for the RM application changed. RM has now its own database and index files that need to be taken care of in the backup. This has been added to below.This article has been preliminary updated to include new applications delivered as part of 6.0 as well as the Liberty application server replacement of Tomcat.

After you start using the Rational solution for Collaborative Lifecycle Management (CLM), it will contain more and more important data. Therefore, it is essential to make regular backups of the data to avoid losing it. The Rational solution for Collaborative Lifecycle Management Information Center contains information on how to back up the databases. To minimize down time, you can increase the reliability of the Rational solution for CLM by using hardware that provides high reliability. However, failure can still happen. Since whole development teams depend on the infrastructure, it is desirable to be able to recover it as quickly as possible. This article provides specifics about what to consider when backing up the Rational solution for CLM.

Deployment topology to back up

For the purposes of this article, assume that the Rational solution for CLM is deployed in a fully distributed topology.

Jazz Team Server is installed on its own server; for example, JTS_SERVER.

The Change and Configuration Management (CCM) application (Rational Team Concert) is installed on its own server: CCM_SERVER.

The Quality Management (QM) application (Rational Quality Manager) is installed in its own server: QM_SERVER.

The Requirements Management (RM) application (Rational Requirements Composer) is installed in its own server: RM_SERVER.

For simplicity, assume that the installation is new and without a migration from prior versions. The context roots are /jts, /ccm, /rm, /qm.

The databases are hosted on one or more servers.

As described in Building products from applications, the Rational solution for CLM is now built around a central Jazz Team Server and several applications that share this Jazz Team Server. In this topic, the following abbreviations are used:

Jazz Team Server is referred to as JTS.

The Change and Configuration Management application is referred to as the CCM application.

The Quality Management application is referred to as the QM application.

The Requirements Management application is referred to as the RM application.

The assumptions above will result in a deployment topology as shown below. Each application node has an application server installed and the application is deployed on it. JTS, the CCM application, and the QM application have their own own database to store data. The RM application below Version 5.x uses JTS to store its data. Since version 5.x RM has its own database. In the example topology, the databases hosted on the database nodes are referred to as JTS_DB, which are used by JTS and the RM application; CCM_DB, which is used by the CCM application; and QM_DB, which is used by the QM application.

Note: Since 5.0 the RM Application has its own database RM_DB that needs to be backed up.

Additional CLM applications such as Design Manager require backup like any other Jazz application, if deployed.

In case a data warehouse such as Rational Reporting for Development Intelligence or Rational Insight is set up, keep in mind that setting up a separate backup procedure for its database and configuration data is also needed.

Relevant data for backup operations

This section answers the question, "What could happen?" In the topology described above, you must plan for the following potential failure scenarios:

Loss of one or more database server nodes

Loss of one or more application server nodes

The loss of the database server or the application server node would each require a substitute or replacement. In the worst case, this could mean reinstalling or reconfiguring a machine as a replacement. You can minimize downtime also by providing replacement systems already configured to take over as described in setting up a basic high availability configuration for the application server node. In any case, it is necessary to have all required data available.

The data needed to restore a complete deployment as described above consists of:

Practicing your backup and restore procedure

A backup procedure is worthless if data is missing, inconsistent, unreadable, or not working for other reasons. Practice and test your backup and restore procedure on an isolated test infrastructure on a regular basis, to be sure that it works.

Synchronizing server clocks

For normal operations as well as for backup and restore, it is important that the clocks of your machines that run the Rational solution for CLM are closely synchronized. The applications that require the synchronization provide setting up a connection to a NTP server to monitor the machine clocks in the advanced server properties.

Backing up the Jazz application configuration files

While installing the products, you choose install directories. For example, if you installed by using IBM Installation Manager on Linux, the default server installation directory is /opt/IBM/JazzTeamServer. If you installed on Windows, the default server installation directory is C:\Program Files(X86)\IBM\JazzTeamServer. This installation directory will be referred to as server-install-dir in the following text. The image below shows a typical folder structure after installing all applications on Apache Tomcat.

During the setup and configuration of Jazz Team Server and the other applications, several files are changed either automatically or manually. Some files that are updated during administration operations and should therefore be included in a regular backup as well.

In this article, the backup of the Jazz application configuration files will be represented by the following pseudocode:

backup APP_CONFIG

In the pseudocode, APP, represents the application node that must be backed up. In the pseudocode for a specific application, APP must be replaced with the acronym that represents the application, which will be JTS, CCM, QM, or RM. This pattern is kept in the article. The pseudocode below would do the backup for all applications:

backup CCM_CONFIG
backup QM_CONFIG
backup RM_CONFIG
backup JTS_CONFIG

Files to include in the Jazz application configuration files backup

All the Jazz application configuration files are stored in one or more folders underneath the configuration data folder:

Full backup

for each application. This option will safely back up all data that might be necessary to restore the Jazz application configuration. On the other hand, it backs up more data than is actually needed.

Note: By default, the index files for the JFS index and the Fulltext index are also stored in this location. If you use a full backup for the folder, make sure that the backup of the index files is successful or change the server configuration and place the index files into a location outside of the server-install-dir/server/conf folder. See below for more information.

Selective backup

The configuration folder also contains program code which is often not desired to be included in backups. To be more selective, it is possible to only include the most important files and folders into the back up. This can especially exclude all data that is installed with the product.

Each application has its own configuration folder. An application can use more than one folder. The folders must be backed up together.

JTS uses the following folders for JTS and the Lifecycle Project Application (LPA):

For each application, at least back up the following files from the configuration folder:

teamserver.properties
log4j.properties

In general, back up all files of this format:

*.properties

Some applications store data in RDF format. Check for files of the following format and back them up in case there are any:

*.rdf

Other files created during the installation are typically not modified. If in doubt, check the modification date and time, and include files that changed after the installation finished.

Configuration folder

As mentioned above, if you use this as a general advice for Jazz based solutions; for example, to back up Design Manager, or for newer versions of the tool, back up all of the relevant data in all available configuration folders in:

server-install-dir/server/conf

Like JTS, Design Manager uses more than one configuration folder; in this particular case, dm and vvc. Be sure to catch all relevant data from these folders.

Do not add any folders to the

server-install-dir/server/conf

folder. Especially do not create copies of the existing folders and store them in this folder. This will cause issues with your application; for example, if you run setup again in an upgrade scenario.

Backing up custom server extensions

A full backup contains all the required data. If you deployed any custom server extensions, and do a selective backup, either back up the corresponding files or at least have the files available for rapid redeployment. The files that are needed to deploy custom server extensions are typically stored in the folder:

server-install-dir/server/conf/APP/provision_profiles

The files that must be backed up are the provision profile .ini file and the corresponding update site folder.

Backing up the deployed patches

A full backup contains all the required data. If you deployed any patches, and do a selective backup, either back up the corresponding files or at least have the files available for rapid redeployment. See the patch description for the files that need to be included in a backup.

Backing up the application server configuration

Back up the application server configuration data for all application servers, in order to be able to restore it later. How to back up this data and which data needs to be included in the backup depends on the application server in use. The application server configuration changes only due to administrative tasks. It should be sufficient to use an incremental backup once a day.

Backing up the application server configuration files for the APP application will be be represented below by the following pseudocode:

backup APP_APPSERVER_CONFIG

The following pseudocode shows the steps to back up all application server configuration files for all nodes:

If you provide a server certificate, include it in your backup. You can find information about the storage location of the used certificate in the server.xml file. The Connector element contains a keystoreFile property that contains the file location.

If you edited the application's web.xml files; for instance, when using Tomcat with LDAP, include these files in your backup. The web.xml files to backup are stored in this location:

If you provide a server certificate, include it in your backup. You can find information about the storage location of the used certificate in the server.xml file. The element keyStore contains a location property that contains the file location. If this shows a file or a relative path the default location for the file is in

Backing up the databases

It is necessary to backup all databases used by the applications. The RM application since version 5.0 has its own database. In versions before 5.0 RM used the JTS database to store its data and therefore did not have a separate database to back up. If you use a data warehouse, also set up a backup procedure for its database.

Backing up the application database for the APP application will be be represented below by the following pseudocode:

backup APP_DB

The pseudocode below represents all steps and the sequence needed to back up for all databases for all applications:

backup CCM_DB
backup QM_DB
backup RM_DB
backup JTS_DB

Ensuring a consistent database backup

When backing up a database, make sure that the database backup is complete and consistent from a transaction perspective. A backup is consistent if taken offline, while the database is inactive, or when it is locked for write operations. Some enterprise databases allow consistent online backups with transaction logging or vendor specific procedures and tools.

There are several databases to back up and the applications interact. JTS creates elements in the registered applications. The JTS cannot create an element if it is already there because the database of the registered application is ahead of the JTS. To avoid conflicts and ensure consistency, the databases for all registered applications must be backed up to a point in time equal or earlier than the JTS database. In the example topology, this holds for the CCM and the QM databases. This has an impact on the database restore operation, especially for online backups.

Derby databases

If a Derby database is used, the application server must be shut down to perform the database backup.

To back up the databases, compress and back up the directory and its content used by Derby as repository. By default, the directory is called repositoryDB. The default location for this directory is the derby folder in the configuration folder of each application:

Verify the location of your setup by checking the com.ibm.team.repository.db.jdbc.location property in the teamserver.properties file of each application.

Enterprise databases

See the system requirements on the Getting Started tab of the individual product download page for the supported database management systems. Also see the information center and the library for information about how to backup other databases. For detailed information on how to backup your database, see the documentation for the database management system you are using.

Offline backup

It is always possible to do an offline backup. Before you back up the database, stop the application server so that no user interaction occurs during the backup. This ensures the backup is consistent. You can also consistently back up all other data related to the server while it is shut down.

Online backup

Some database management systems, such as DB2, support online backup. This can be used to continuously back up the databases. For more details about DB2 online backup, see this article.

If using online backup, you still want to back up the Jazz application configuration files as well as the application server configuration files. You will most likely not be able to back up and restore the configuration files consistently with the online backup. However, these files are only changed when configuring the application server or administrating the capabilities of the server configuration. Back up these files after performing any of these tasks. Otherwise you must redo changes that did not make it into the last backup.

Ensuring a consistent backup of database and index files

To create a consistent backup of the JFS index files and restore from it, back up the JFS index files before or during the backup of the database. The indexing service will re-create index information for database content not found in the index over time. Backing up the indexes will be be represented below by the following pseudocode:

backup APP_JFSINDEXES

In the pseudocode above, APP represents the application node that must backed up. The following code backs up all nodes:

Note: Up to version 5.0 the RM application did not have its own JFS index files and these indexes where maintained by JTS.

JFS index file locations

The location of the JFS index files can be found in the teamserver.properties file for each application as value of the following properties:

com.ibm.team.jfs.index.root.directory

The location of the index files is, by default, relative to the folder where the teamserver.properties file is located. This should be changed to an absolute path.

Backup of the JFS index files in version 3.x products

To avoid a corrupted backup of the JFS index files in 3.x products, stop the indexing service during the backup period or shut down the server. Stopping the indexing services can increase the server up time when running backups. It is necessary to resume the indexing services after the backup has been performed.

The indexing service can be stopped by using the -suspendIndexer repotools command:

repotools-APP -suspendIndexer APP_PARAMETER

The indexing service can be restarted by using the -resumeIndexer repotools command:

repotools-APP -resumeIndexer APP_PARAMETER

In the above pseudocode, repotools-APP needs to be replaced by the repotools command for the application that is backed up. For example, repotools-ccm for the CCM application. In addition, APP_PARAMETER represents the required parameters for the command, such as the repository URL of the server and the login information for an administrative user. For the syntax and options, see the repotools command help. As an example for JTS, the command looks like this:

Backing up the JFS index files in 4.0 - 4.0.4

In version 4.0, a new repotools command was introduced that does the backup of the JFS index files for an application, with the server online.

Backing up the JFS index in version 4.0. and later is described by using the same pseudocode:

backup APP_JFSINDEXES

Where repotools-APP must be replaced by the repotools command for the application that is backed up. For example, repotools-ccm for the CCM application. In addition, APP_PARAMETER represents the required parameters for the command, such as the repository URL of the server and the login information for an administrative user. For the syntax and options, see the repotools command help. As an example for JTS, the command looks like this:

Using this command maintains the consistency of the JFS index files and does not require to stop and restart the indexer.

Note: In versions before 4.0.5 the defect Defect 245648 prevents from suspending indexing the Fulltext index. In these versions it is necessary to do a consistent backup of these indexes, while the server is running.

Backing up the JFS index files in 4.0.5 and later

Since version 4.0.5 the JFS index provides a capability to back up the work item index along with the JFS index. See Fulltext index for considerations, especially for online backup.

Backing up the Fulltext index files

Some applications, such as CCM and QM, also use a Fulltext index to support various search and query operations. You must back up the Fulltext index files to be able to restore the applications consistently. The Fulltext index cannot be backed up independent of the database.

Backing up the indexes is represented by the following pseudocode:

backup APP_FULLTEXTINDEX

In the pseudocode above, APP represents the application node that needs to be backed up. The code below backs up all nodes. JTS does not have a Fulltext index:

backup CCM_FULLTEXTINDEX
backup QM_FULLTEXTINDEX

Ensuring a consistent backup of database and Fulltext index

Beginning with version 4.0.5 the work item index backup is now included in the back up of the JFS index.

However, the work item index is still built up during work item operations. If restored from an online backup of the JFS index, it does not index operations that are not included in the backup. So other than the rest of the JFS index files, the work item index does not catch up with later changes and will remain inconsistent with later work item operations. Note as of 5.0.1, the fulltext work item index does catch up with the database similarly to how the JFS indices catching up.

To get a consistent backup of the work item index files a recreation of the index might still be necessary. If a consistent online backup of the work index is the primary concern, a complete offline backup should then be performed in order to get a fully consistent backup of the databases and the other files.

Fulltext index file locations

The location of the Fulltext index files can be found in the teamserver.properties file for each application as value of the following properties:

com.ibm.team.fulltext.indexLocation

The location of the Fulltext index files is, by default a relative path. It will be based on the application relative runtime location. If your CLM application is running on Tomcat this will be the same as the application configuration directory; if your CLM application is running on WebSphere Application Server, this will be the profile path, for example: //profiles/AppSrv01/

This should be changed to an absolute path.

Backup scenarios

With all relevant backup operations described above, it is now possible to look at different scenarios and provide the sequence of backup operations.

Offline backup

Shutting down and starting up CLM

If you shut down a CLM environment that uses multiple distributed servers, make sure to

Start up JTS first

Shutdown JTS last

You can start up and shut down the other applications in any order, if you make sure the JTS is handled as described above.

Offline backup sequence for a single application on a single server

A complete offline backup procedure for one application on a single server machine would perform the following sequence of activities:

In the scenario above the backup of the JFS index files assumes suspending the indexer in 3.x and using the backupJFSIndexes command in 4.x.
During startup in a distributed environment, start JTS first, then start the other servers.

Offline backup with reduced server downtime

The example above has the longest downtime for offline backup scenarios. There are several options to reduce downtime, dependent on the systems and infrastructure in use. These include:

Backing up the application configuration files while the server is up

Backing up the application server configuration while the server is up

Stopping indexing for backing up the JFS index and keeping the server operational during the backup

Isolating the server from users while backing up the database

Online backup of the database

The following example only takes the server offline to backup the databases and the Fulltext index files:

Online backup

As of 5.0.1, a continuous online backup is possible which means both the JFS index and Fulltext index files can be backed up online. However, for versions prior to 5.0.1, it is necessary to take the server offline at least to back up the Fulltext index. In that case, to restore from an online backup without a consistent backup of the Fulltext index, you must re-create the Fulltext index files from the restored databases.

Restoring from a backup

Replace backup by restore in the pseudocode to get a restore strategy.

Temporal database consistency during a restore

Keep a valid temporal sequence of the database states during backup and restore. Make sure to restore the databases for all applications registered to a JTS; for example, the CCM and the QM database, from a point in time shortly before or equal to the JTS database backup last transaction. This is especially important for using online backup for the database, but also holds for offline backup. In offline backups, you can create a consistent backup if all servers are down at the same time. This requirement is due to the JTS creating elements in the registered applications and potential conflicts in case the database of the registered application is ahead of the JTS database.

When restoring the JFS index files from a backup, make sure that the time of the backup of the index files is from before the application's database backup. If you restore a backup of the JFS index files that is more recent than the backup of the database of that application, error messages will indicate that the index is ahead of the database. In this case, you must re-create the index files.

You also must restore a backup of the Fulltext index files that is consistent with the databases or you must re-create the Fulltext index. A complete restore procedure will perform the following sequence of activities:

Complex scenarios

With CLM 2011, you can install more than one CCM and QM application on different servers and run them against a common JTS. In that case, you must add these additional servers to the backup and perform all backup steps for each individual server. It is necessary to keep the temporal consistency for the JTS and all the applications registered to it. If you use more than one JTS, there is no requirement to keep the temporal consistency between the different groups of JTS and its registered applications.

Summary

This article described the data and steps required to back up a deployment of the Rational solution for CLM. The article provides several options for backing up and also provides information about how to back up more complex Jazz deployments.

Appendix

Backing up applications that were upgraded from a release of version 2

In the case of an installation that was upgraded to a version of the 3.0 release, the upgraded applications use the /jazz path instead of /ccm or /qm. Therefore, all occurrences of these strings in this article must be replaced with /jazz.

Re-creating the index files

You can re-create the index files from a restored database backup. Be aware that re-creation of the index files is an expensive operation and must performed with the server shut down.

Re-creating the JFS index

Oracle backups

See this technote about Oracle backup, especially if you intend to use datapump.

There is a known issue with Oracle databases that are restored from a backup in 3.0.x. For details, see work item 180771. If this issue occurs, Support can provide a solution.

Backing up products that were upgraded from a release of version 2

In the case of an installation that was upgraded from a release of version 2, the context root will be jazz instead of ccm or qm. In these cases, all occurrences of the context root in the URLs and paths above must be replaced with jazz.