3 Before You Perform Tasks in Data Integration Platform Cloud

To perform tasks in Data Integration Platform Cloud, you must first download, install, and configure your agent, and then create Connections. Agents enable the Data Integration Platform Cloud instance to connect with your data sources. Agents also orchestrate the tasks that you run on your data sources from the Data Integration Platform Cloud console. Connections enable you to define source, staging, or target data sources for your tasks.

What is an Agent?

The DIPC agent connects the DIPC host to the on-premises and cloud data
sources. The DIPC agents also orchestrate the jobs that you run on the data sources from the
DIPC console.

You can download the DIPC agent from the DIPC console and install it on any machine,
on-premises, or cloud that has access to your data sources. For example, you can install
the DIPC agent on premises, or on cloud, such as Oracle Cloud VMs or Third-Party Cloud
VMs.

DIPC Classic also provides the DIPC host agent. The DIPC host agent is a pre-configured
ready-to-use agent that is located on the DIPC host. This agent works in the same manner
as the DIPC agent, but only with the cloud data sources and components. If you want to
use on-premises data sources or components, you must either download and use the DIPC
agent, or set up a VPN connection so that the DIPC host agent can access the on-premises
data sources.

For all the certified data sources that you connect to, you must download
and run an agent with a component for that data source, so that the agent can
communicate with the Data Integration Platform Cloud server. For example, for Oracle 12c, you can either download the Linux or
Windows version of the Oracle 12c component, and then set up the agent.properties file
with connection information, register it and run it.

An agent:

Exchanges heartbeats with the server and reacts based on server
availability

Captures server health records of the host process and sends them to
the server for monitoring

Reduces installation requirements

Makes maintenance and patching easier

Reduces the number of processes running on the Virtual Machine
(VM)

Agent Certifications

For all the tasks that you create in Data Integration Platform Cloud, you assign an agent. Agents must run only on their certified platforms.

Available Components

The Components comprise a set of binary files and agent properties file that you need to set up to run the DIPC tasks. Based on the DIPC tasks you want to run, you need to select the appropriate components, when you download the DIPC agent using the DIPC console. The components that you select are included in the agent package.You can select from the following components when you download the agent:

Linux

Big Data (OGG)

Data Integrator (ODI)

Data Preparation

Data Lake

Oracle 11g (OGG)

Oracle 12c (OGG)

Windows

Big Data (OGG) (This component is not certified with any of the tasks. You can use it to perform data integration through the hosted VMs.)

Oracle 12c (OGG)

SQL Server (OGG)

All data sources must have x86_64, the 64 bit version of x86 operating systems, with the latest upgrade.

Run your remote agents only on operating systems that are certified for the agents.

Set up the remote or host agents for data sources certified for your task.

Operating Systems Certified for DIPC Agents

The following tables list the operating systems your agents can run on. If your certified data sources have the same version of operating system, then you can run the DIPC agent on the same machine as the data source. Otherwise, run the agents on their certified operating systems and ensure that they have access to your data sources. For example, if your Oracle 11g database is on a Windows machine, then you can't run the Oracle 11g component that you downloaded with the agent on the same machine as there is no Windows component for Oracle 11g. Instead, you must either download a DIPC agent's 11g component on a Linux machine or use the DIPC host agent. Then run your agent remotely with a setup that has access to the database. Alternatively, you can have a DIPC agent on Windows with the 12c component, connecting to an Oracle Database Cloud Service 12c Classic.

If you want to use MySQL for the ODI Execution task, then you must modify the
agent.properties file to use GoldenGate 12.2. The default version is GoldenGate
12.3.

Import a Client Certificate to the Agent

The Data Integration Platform Cloud
agent and server communicates via the Transport Layer Security (TLS) protocol. If the remote
agent has to trust the Data Integration Platform Cloud
server, the public key (certificate) used by Data Integration Platform Cloud server should be recognized/trusted by the on-premises
agent client.

To import a client certificate:

The Data Integration Platform Cloud client side
default certificate store (trust store) already contains the root certificate of the
accredited certifying authority, so the client can verify and trust the signed
certificate. Hence, it’s not mandatory to import a client certificate, before
starting the agent. Certificate import is required, if any error occurs. Otherwise,
it’s optional to import a certificate to the Agent’s JAVA_HOME.

If the following
error occurs, you should manually import the Data Integration Platform Cloud server's certificate to the on-premises agent client
side JRE trust store. This error occurs only when the client side default Java
certificate store fails to verify or recognize the server's
certificate.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found

Log in to Data Integration Platform Cloud.

In the address bar of your web browser, click View/Show site information (the security padlock icon).

If you’re using Chrome, select Certificate and then in the Details tab of the Certificate dialog, select Subject and click Copy to File. Follow the prompts in the Certificate Export Wizard to save the certificate.

If you’re using Firefox, click Show connection details (the right arrow next to the URL), and then click More information. In the Page Info dialog, click View Certificate, and then click Export in the Details tab. Save the certificate.

Navigate to $JAVA_HOME, using the keytool present under $JAVA_HOME/bin/keytool

Do one of the following

Import the client side certificate into the cacerts present under
$JAVA_HOME

Connectors are representative of the check boxes that appear on the
DIPC download UI page. So possible values are:

For linux:

GGMON_ORACLE_12c

GGMON_ORACLE_11g

GGMON_BIGDATA

DATALAKE

DATAPREPARATION

ODI

For Windows:

GGMON_ORACLE_12c

GGMON_SQLSERVER

Connectors:

"GGMON_ORACLE_12c":"GoldenGate for Oracle 12c"

"GGMON_ORACLE_11g":"GoldenGate for Oracle 11g"

"GGMON_SQLSERVER":"GoldenGate for SQL Server"

"GGMON_BIGDATA":"GoldenGate for Big Data"

"ODI":"Oracle Data Integrator"

"DATALAKE":"DataLake"

"DATAPREPARATION":"DataPreparation"

Before You Register Your DIPC Agent

Adding your DIPC Agent as a confidential application enables you to get certain parameters (agentIdcsScope, agentClientId, agentClientSecret) required to register your DIPC Agent. However, you must perform some additional steps to obtain the values of the other parameters required for the registration.

Make sure that you have an IDCS account before you register the remote agent. DIPC does not support federated Single Sign-On (SSO).

You need the values of the following parameters to register your DIPC agent, and connect to DIPC in OAuth mode:

When you create an Oracle Data Integration Platform Cloud instance, you'll automatically get the idcsServerUrl. As a part of provisioning, you’ll get access to the Identity Cloud Service (IDCS) console (https://<idcs_Service_ID>.identity.examplecloud.com/ui/v1/adminconsole). For details on how to provision a Data Integration Platform Cloud instance, see Create Instances for Data Integration Platform Cloud.

In the IDCS console, go to My Services page, and click View Details for Identity Cloud.

If the Identity Cloud service tile doesn’t appear on your My Service Dashboard, click Customize Dashboard and then click Show for Identity Cloud.

Under Additional Information, You’ll find the Identity Service Id (idcs_Service_ID).

Obtain the value of dipchost

Get the value of dipchost from the address bar of the DIPC console.

The dipcport value

Use the default dipcport 443. If your on-premises system doesn't allow any outgoing internet traffic, you must make an exception rule for port 443 of the DIPC host machine.

Add Your DIPC Agent as a
Confidential Application

DIPC application server is a protected server. You must add your DIPC Agent
as a confidential application to get the values for the parameters required to register your
Agent. Adding the DIPC Agent as a confidential application ensures secure interaction
between the DIPC Agent and the DIPC application server. After you add the DIPC Agent as a
confidential application, you can connect it to the DIPC application server in OAuth
mode.

When you add the DIPC Agent as a confidential application, you get the values of the
agentIdcsScope, agentClientId, and
agentClientSecret parameters. Make sure that you note
down the values of these parameters, as they are required to register the DIPC
agent. Also, take note of your Data Integration Platform Cloud instance.

Add DIPC Agent as a Confidential
Application for DIPC Classic

DIPC Agent must be added as a confidential application to establish secure
interaction between the DIPC agent and the DIPC application server. After you add
the DIPC agent as a confidential application, you can connect it to the DIPC
application server in OAuth mode.

To add DIPC Agent as a confidential application for DIPC Classic:

Log in to the IDCS console as the application administrator.

From the navigation menu, select Applications.

Click Add, then select Confidential Application.

On the Add Confidential Application Details page, enter a Name in the App Details, and then click Next.

On the Client page, select Configure this application as a client now.

Take note of the Client ID and Client Secret before closing this message.

If you need to regenerate the Client ID and Client Secret again later, return to the Identity Console Applications page, select your application, and under the Configuration tab, click Show Secret and Regenerate.

Click Activate.

In the Activate Application dialog, click Activate Application.

Add DIPC Agent as a Confidential
Application for DIPC

DIPC does not provide an Agent on its host. You can deploy an agent to any
type of infrastructure where you need to execute jobs, either on-premises, Database
Cloud Service, Oracle Cloud Infrastructure as a Service, and so on.

To add DIPC Agent as a Confidential Application for DIPC

Log in to the IDCS console as the application administrator.

From the navigation menu, select Applications.

Click Add, then select Confidential Application.

On the Add Confidential Application Details page, enter a Name and Description in the App Details section.

Leave the other fields as is, and then click Next.

On the Client page, select Configure this application as a client now.

Note down the Client ID and Client Secret before
closing this message.

Now, you've all the parameters ready to go
to the agent instance location and register your
agent. See Register Your Agent.

If you need to regenerate the Client ID and Client Secret again later, return to the
Identity Console Applications page, select your
application, and under the Configuration tab, click
Show Secret and Regenerate.

Click Activate.

In the Activate Application dialog, click Activate Application.

Register Your Agent

After downloading the agent package, you must unzip and run the register script to register it with Data Integration Platform Cloud.

Make sure that the time you set on the machine where the DIPC agent resides and the DIPC
server are the same and correct/current time. If the time on both the machines is not in
sync, the DIPC server doesn’t process the messages send by the DIPC agent, and shows the
following error:

If a value is not provided for the parameters for which you don't have defaults, you will be prompted to enter it in the command line . For example , "user" , "dipcPort", "dipcHost", "idcsServerURL," and so on are critical parameters. If you provide some defaults to these, it will not work.

Parameter

Description

<agentInstanceDirectory>

Indicates the name of the agent instance directory.

This is optional.

All Agent instances are always created in :
${agent_unzip_loc}/dicloud/agent

The default value is dipcagent001.

-user=<diuser>

You must provide the username with which the agent will connect to the Data
Integration Platform Cloud host/IP.

This is mandatory.

For example:

-user=dipcOperator

-password=<dipassword>

You must provide the password with which the agent will connect to Data Integration
Platform Cloud host/IP.

-password=password

-recreate

Specifies that you're recreating the agent.

This is optional.

The default value is false.

-dipcport=<port>

You must provide the port to which this agent will connect and register.

This is mandatory.

For example:

-dipcport=443

-dipchost=<dipc.example.host.com>

You must provide the Data Integration Platform Cloud host/IP to which this
agent will connect and register.

This is mandatory.

Note: Do not prefix with https://

For example:

-dipchost=dipc.example.host.com

For Data Integration Platform Cloud, it looks like this:

dipcinstancename-dispaas.dipc.ocp.examplecloud.com

-debug

Specifies that you're running this script in debug mode and will also generate a debug log file.

This is optional.

The default value is false.

-authType=<OAUTH2>

This property determines the authentication mode with the server. It is OAUTH2. This is optional. Default value is OAUTH2.

When the script is run, it reads all the parameters you set in that file. You can
also see your JDK Trust Store
/home/oracle/public/jdc/jre/lib/security/cacerts . If
you’ve not set up your $Java_Home correctly, it will show an
error here.

Go to the agents directory by running the script $ cd agent, and then run $ cd agent directory name.

You can see a bin directory, and a conf directory. And inside the conf directory,
you’ve the agent properties file. Proceed with Set Your Agent Properties.

You'll find the registration script log file, dicloudRegisterAgent.log, in the same directory where the script is run.

Set Your Agent Properties

After you’ve registered your Agent with Data Integration Platform Cloud, you can set your agent properties.

The agent.properties file for each agent created is located in the conf directory. For example, ${agent_unzip_loc}/dicloud/agent/<agentInstanceName>/conf/agent.properties. This file enables you to perform advanced configuration of the agent. It contains the configuration properties, their default values, and a detailed explanation of each property. If it’s a clean agent, and GoldenGate wasn't running before, you can use the default values, without any changes. If you want to run multiple GoldenGate instances or agents, make sure that all the ports are pointing to different agents.

If required, you can set proxy details in agent.properties to connect to Object Storage Connection.

Go to dicloud/agent/dipcagent/conf/agent.properties.

Add proxy details in this format:

agentUseProxy=

proxyHost=

proxyPort=

The Agent ports should be configured as unique port of the system, especially when there are more than one agent on the same operating system.

Make sure to set gghome in the path to access ggsci prompt:

PATH=%12CGGHOME%;%12CGGHOME%\lib12;%12CGGHOME%\crypto;%PATH%

For Windows , you must set it as PATH=%12CGGHOME%;%12CGGHOME%\crypto;%PATH%

Configure Agent Secure Sockets
Layer (SSL)

An advanced configuration requires configuring Agent Secure Sockets Layer
(SSL). You must configure the SSL settings in the agent.properties file to establish an
encrypted link between the DIPC agent and the DIPC server.

Set the following in the agent.properties file:

agentUseSSL = true

agentTrustStorePath= <path to SSL trust store>

Execute and set the unlock password for the trust store configured above.

To create a SSL Trust-Store unlock password:

.${AGENT_INSTANCE_HOME}/bin/createTrustStoreUnlockPassword.sh

OR

To update a SSL Trust-Store unlock password:

.${AGENT_INSTANCE_HOME}/bin/updateTrustStoreUnlockPassword.sh

Updating or viewing the configured server (DIPC) credentials

To view the configured username for the agent to talk to the DIPC
server:

.${AGENT_INSTANCE_HOME}/bin/viewServerCred.sh

To update the configured username for the agent to talk to DIPC:

.${AGENT_INSTANCE_HOME}/bin/updateServerCred.sh

Viewing and updating IDCS client configuration

To
update the IDCS Client ID and Client Secret configured for this agent:

.${AGENT_INSTANCE_HOME}/bin/updateAgentIdcsClientDetails.sh

To view the IDCS Client ID and Client Secret configured for this
agent:

.${AGENT_INSTANCE_HOME}/bin/viewAgentIdcsClientDetails.sh

Configuring Agent through Proxy

Set the
following in the agent.properties file:

agentUseProxy =
true

proxyHost=<dicloudProxy.example.com>

proxyPort=<80>

If proxy needs authentication, execute the following commands as
required:

Creating the credentials for the agent to talk to
proxy

.${AGENT_INSTANCE_HOME}/bin/createProxyCred.sh

Updating the credentials for the agent to talk to proxy

.${AGENT_INSTANCE_HOME}/bin/updateProxyCred.sh

Viewing the proxy username configured for this agent to talk to a
proxy

.${AGENT_INSTANCE_HOME}/bin/viewProxyCred.sh

Start and Stop the Agent

You can use these commands to start and stop the Agent.

After you install and configure your DIPC agent, run the following command to start it:

Monitor Your Agents

The Agents page displays a list of all agents registered with your Data Integration Platform Cloud instance along with the current status of each agent. You can select an agent to view its details, as well as the agent’s Drill-down Status that displays the components of the agent and whether they are running or not.

Monitoring the status of your agent enables you to know the current state of your agent, and act on it accordingly.

Status

Description

RUNNING

When the Data Integration Platform Cloud instance receives heart-beat messages from the remote Agent at a regular interval, as specified by the Agent's property "agentHeartBeatInterval" in the [agent-home]/conf/agent.properties file.

By default, the agent sends a heart-beat message every 10 seconds.

ERROR

When the Data Integration Platform Cloud instance receives heart-beat messages from the remote Agent at a regular interval, and the Agent sends error messages, indicating errors at its end.

UNREACHABLE

When the Data Integration Platform Cloud instance has not received a heart-beat message from the remote Agent for a duration exceeding twice the agent's hear-beat interval.

STOPPED

When the remote Agent has been stopped.

Create a Connection

Connections enable you to specify the connection details to your source, staging, or target data sources. You then select the source and target connections to use as part of your tasks in Data Integration Platform Cloud.

To create a new Connection:

Click Create from the Getting Started section of the Home page, or select Connection from the Create menu in the Catalog. You can also create Connections from any Create Task screen.

View and Edit Connections

After creating your connections, you can find them in the Catalog, along with the data entities harvested from the Connection. From there, you can view, edit, test, or delete them.

You can select any connection to view its details. The details are grouped under following categories:

Summary - contains details of the connection such as its name, description, type, user name, URI, Schema, and tags. You can create new tags on this page. To create a new tag, enter tag name in the Tags field and press the enter key.

History - contains details of the actions performed on the connection, so you have a record of how the connection has changed over time.

Set up Default Connections

Setting default connections is a one-time task and necessary to perform certain tasks in Data Integration Platform Cloud. You set Default Connections on the Admin page.

Select an Object Storage Classic Connection

You need to select an Object Storage Classic Connection to:

To run a Data Preparation Task on a remote file accessible from Data Integration Platform Cloud

To connect to an Oracle Autonomous Data Warehouse connection

To connect to an Oracle Object Storage connection

Before you can select a default Connection for Object Storage Classic, you must first create an Object Storage Classic Connection.

Understand the Catalog

The Catalog enables you to access components such as Tasks, Connections, Data Entities, Data Assets, and Execution Environments, that exist in Data Integration Platform Cloud. You can also create and edit these components from the Catalog.

View a history of actions performed on all the components in the Catalog.

There are several ways to filter the list of components in the Catalog. To view only Connections, click the menu next to Catalog and select Connections. Likewise, you can select Tasks, Data Entities, Data Assets, or Execution Environments to see only Tasks, Data Entities, Data Assets, or Execution Environments respectively. Click Filter Favorites to view all Connections, Tasks, Data Entities, and Data Assets that you marked as Favorite. You can use the Sort By menu to sort the list by Most Recent (default), Alphabetical, and Popularity. Click Show/Hide Filters to toggle the filter selectors. Filter selectors change depending on the view you’ve set (either All, Connections, Tasks, Data Entities, Data Assets, or Execution Environments).

Search the Components in the Catalog

When you enter a text string into the Search field, the search engine returns any Connections, Tasks, and Data Entities that meet your criteria, searching all fields of these components for a match. As you type in your text string, the search engine actively returns components, and lists them underneath the search bar for you to quickly access what you’re looking for.

Search Components by Properties

You can also enter property name and value pairs into the Search field to limit your search to specific fields and values. You can search on the following properties:

id

name

description

tags

createdBy

connectionType

entityType

taskType

technology

connection

schema

attributes

The property names are case sensitive, so be sure to enter the property names in the Search field as you see them in Data Integration Platform Cloud.

Enter a search string in the following format to perform a property search:

<property-name>:<value>.

You can also use operators such as OR (||) and AND (&&) to build on your search query. The default operation is OR. For example, category:task name:dis_view is the same as category:task OR name:dis_view.

Use parentheses around property name and value pairs to indicate the order of operations to be performed. For example,

(category:task || name:dis_view) && createdBy:Joe Smith

View Component History

You can find past actions, who performed them, and when in the History tab of any component in the Catalog.

All components listed in the Catalog such as Tasks, Data Entities, or Connections have the following actions in common: First you create them, then you can update or delete them. When you select any item listed in the Catalog, you view its Summary page. From here, you can access the component’s History. The History tab is a read-only page that displays a table with three columns:

Action: Displays either Create, Update, or Delete.

User: Displays the name of the user who performed the action.

Last Updated: Displays the date when the user performed the action.

Data Entities

Data Entities are the data objects you can use as your sources or targets.

Data Entities are listed in the Catalog along with other components such as Connections and Tasks. To view only Data Entities in the Catalog, select Data Entities from the Type filter menu. You can select any data entity to view its details. The details are grouped under the following categories:

Summary: Contains details of the data entity, such as its name, identifier, description, type, and popularity.

Metadata: Contains details of the data in the data entity along with examples. The examples are seen in the Sample Values column with five sample values.

You can also select each metadata object to see its profiling metrics. By examining the data, you can:

Determine how easily the existing data can be used.

Evaluate data quality.

Review the risks of integrating data in new applications, including joins.