As members of the Fusion Middleware Architecture Group (a.k.a the A-Team), we get exposed to a wide range of challenging technical issues around security and Oracle Fusion Middleware. We're using this blog to answer common questions and provide interesting solutions to the real-world scenarios that our customers encounter every day.
NOTICE: All our post and much more can now be found at http://www.ateam-oracle.com/category/identity-management/

Tuesday, November 18, 2014

Introduction

Since the introduction of MDC support in OAM 11g, Customers have been asking for Automated Synchronization between Master and Clone OAM Environments. It is supported in OAM R2PS2. Thanks to the development team! Before R2PS2, It required T2P process to keep all the data centers in synch which is manual process or customer had to write crone jobs to run T2P process at frequent intervals. Please note that T2P process is still supported with R2PS2 if that is the preference for some reason.

How does APS function?

APS is client side pull model. Each client (clone environment) pulls changes from the server (Master environment). A Replication thread runs after every pre-configured time Interval on clone environment and fetches changes from replication service (REST Service) running on master environment.
For every cloned environment, Master environment keeps track of a change sequence number which indicates when cloned environment was last synched. Master environment also keeps track of change log which is list of changes (Create/Update/Delete) that have happened in the environment. Along with that, Master environment keeps track of mapping between cloned environment specific change sequence number and change log.
Cloned environment while updating configuration can transform configuration parameter values to environment specific values depending on transformation rules. This is very interesting and determines success/failure of APS in an environment. Unfortunately it is not well documented. No worries. I will talk about transformation rules in little more details in this blog.

Step-by-step process to configure APS

While setting up clone environment, make sure latest configuration is copied from master environment to clone environment. That is running copyConfig.sh to generate new oamt2pConfig.jar file every time (As documented in MDC Implementation blog post by Vinay Kalra from A Team) you start setting up a new clone environment instead of using old oamt2pConfig.jar.

For APS to work, partnerInfo.properties file used for MDC configuration (While running addPartnerForMultiDataCentre WLST command) should have an additional REST endpoint property defined as described below where NYC-CLUSTER (Master/Primary environment) and LON-CLUSTER (Clone environment) are two cluster IDs. You may already have run addPartnerForMultiDataCentre before; however you need to run that again with RESTEndpoint parameter if that was not added in partnerInfo.properties file when you had last run that. You need to run addPartnerForMultiDataCentre commands as below on both Master and Clone environments.

Now you have to add "-Doracle.oam.EnableMDCReplication=true" Java Property in setDomainEnv.sh file on both master and clone environments as shown below to enable APS service. After adding that Java property, restart admin servers

After enabling APS service, execute following curl command to test if APS is enabled on both the data centers. If it is enabled, you will see {"ok":"true"} JSON response.

Now you need to setup replication agreement between master and clone environment by running following curl command. Service responds back with JSON object that has identifier for the replication agreement. Save that identifier because you have to use that identifier to update or delete replication agreement.

I have highlighted some of the important parameters from JSON response.

First one is identifier. As mentioned above, it is a reference to replication agreement setup between NYC-CLUSTER and LON-CLUSTER data centers.

Second is polling Interval. Clone data center will poll for changes against master data center after every polling interval. Typically policy and configuration changes are not done very often and so this number can be as high as 900 seconds as setup by default by OAM. However you can change that if needed as we will do in next step.

Third one is startingSequenceNumber. That is the change sequence number that I talked about earlier in the blog post. Initial agreed upon sequence number is 101 in this example. Clone data center will start polling for changes that have happened after 101 sequence number. To understand more about sequence number and how it is used, refer to sequence diagram below.

Now using replication agreement identifier number, you can make changes in the replication agreement configuration. Before doing that, Run first command as below to get current status of replication agreement. As shown below, we are changing poll Interval to 60 seconds. Service responds back with JSON object that is the status of replication agreement before making the change. You need to fetch replication agreement status again to see updated configuration.

It updates PrimaryServerList for all the WebGates and replaces them with OAM server host from Clone environment. The best way to understand transformation rule is to open oam-config.xml file in web browser and browse through the tree. Above mentioned transformation rule replaces PrimaryServerList
with ${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost} value. If I browse that value in oam-config.xml, I get oam1-lon.oracle.com value as shown in the screen below so APS thread will update PrimaryServerList host for every webgate agent to oam1-lon.oracle.com.

The drawback or limitation of this transformation rule is; it updates all the servers/hosts in the PrimaryList with oam1-lon.oracle.com. You can update transformation rule as below so that it updates servers in PrimaryServerList with respective servers from Cloned data center.
We are updating 0th PrimaryServerList (oam1-nyc.oracle.com) with oam_server1 host (oam1-lon.oracle.com) from clone environment and 1st PrimaryServerList (oam2-nyc.oracle.com) with oam_server2 host (oam2-lon.oracle.com) from clone environment as shown and highlighted in the screen below.

Note: This transformation rule assumes that you have two OAM server instances in both Primary and Cloned environment. Similar to this transformation rule, you can choose to update SecondaryServerList as well for all the WebGates.

It is recommended to use Load Balancer (LB) between WebGate and OAM server. If you are using it right, you do not have to update PrimaryServerList across data centers and so can remove this transformation rule from the XML. However you do need to update PrimaryServerList for IAMSuiteAgent and accessgate-oic unless you have configured those agents as well to talk to LB. Below change will make sure that PrimaryServerList is updated only for IAMSuiteAgent and accessgate-oic and not updated for any other webgate agent.

If you used global LB to define logout URL for all the WebGate agents in master environment, then you don't need to replace logoutURL in clone environment and can remove this transformation rule.
If you are using DCC authentication scheme and using global LB hostname to define DCC login and logout URL, then also you don't need to replace login and logout URL in cloned environment and can remove the transformation rule.
Now that you understand how to write custom transformation rules, it’s time to configure OAM to use custom transformation rules which unfortunately is not documented. You need to update setDomainEnv.xml file as shown in the below screen shot on clone environment. Each clone environment can use different transformation rules. Restart admin server in clone environment after changing transformation rule.

APS as I said in the beginning was a long pending feature and I hope this blog will help you setup MDC environment with APS. If you create environment specific transformation rules, please share it here so that everyone will benefit from it.