3.2.1 General Characteristics and Goals of the Exalogic Network Configuration

When you initially set up your Exalogic system, the IP over Infiniband (IPoIB) is configured by default. In addition to IPoIB, you must manually configure Ethernet over InfiniBand (EoIB) network access for those components that are going to be exposed over ethernet out of the Exalogic rack.

An optimized Oracle Fusion Middleware system constrains communication between the various elements of the topology so it is performed over the Exalogic Infiniband network as much as possible. For example, components should listen in Infiniband interfaces to eliminate overhead in accessing the appropriate Gateways and to make use of the optimized Inifiband network.

Additionally, when the same Exalogic rack is shared with other Oracle Fusion Middleware systems, such as WebCenter and Fusion Middleware SOA, or even with other type of deployments, such as test or development, then EoIB access might require isolated VLAN connectors for SOA. VLANs can be used for this logical division of workload and for enforcing security isolation. However, the definition of such VLANs is outside the scope of this guide.

3.2.2 Map of the Network Interfaces Used by the Components of the SOA Topology on Exalogic

Figure 3-1 describes the components of an Oracle Fusion Middleware SOA Exalogic enterprise deployment on Exalogic, and the type of interfaces and communication protocols they use.

The IP addresses used in Figure 3-1and the sections in this chapter (both internal IP addresses, such as 192.168.10.1, and external IP addresses, such as 10.10.10.1) are just examples and are used for consistency throughout this document. However, other IPs are valid. It is a good practice to follow an order and separate types of servers in IP ranges. For example, 192.168.50.x for Oracle Traffic Director, and 192.168.20.x for SOA. These ranges depend on the subnet masks they use, but will facilitate IP tracking.

For more information about the network map diagram, see the following:

The OWSM Managed Servers are accessed using the default bond0 interface by SOA servers and other consumers that are expected to be internal to the InfiniBand fabric. There is no need for the OWSM servers to be accessed from outside the Exalogic compute node, so there is no need for the OWSM servers to use an EoIB interface.

3.2.3.2 Communication with the SOA Managed Servers

The SOA servers listen on both EoIB and IPoIB interfaces.

Their default channel uses an IPoIB listen address. This is for optimized server-to-server invocations as well as for Coherence dehydration and deployment optimizations.

3.2.3.4 Communication with the Oracle Traffic Director Instances

The Oracle Traffic Director configuration is deployed to two instances with listeners listening on ANY/*. For increased isolation, you can associate the Oracle Traffic Director EoIB interfaces (used by the front end load balancer to distributed requests between the two Oracle Traffic Director nodes) to a separate VLAN from the rest of the EoIB addresses. Listeners can be accessed both externally over the EoIB network by the front end load balancer, and internally over the IPoIB network by the application tier components.

For load balancing and high availability, four virtual IPs are mapped to Oracle Traffic Director failover groups: two for external access, one for SOA internal access, and one for OSB internal access.

The Oracle Traffic Director administration server uses an EoIB virtual IP address so it can be failed over to a different node (this failover node cannot host an Oracle Traffic Director administration node already). Note that the OTD administration server virtual IP address is not part of any failover group. To failover the OTD Admin Server, you must perform a backup and restore of the Administration Server instance (or use a shared mount) in a different node that does not contain an OTD Administration node.

3.2.3.5 Communication with the WebLogic Server Administration Server

The WebLogic Server Administration Server can listen on an EoIB or on IPoIB and be exposed to the external world by Oracle Traffic Director. The suitable approach is determined by the type of operations performed on the Administration Server. For example, if JMX management and metrics are accessed externally, as occurs in most cases, EoIB is required. Given the typical lifecycle of a SOA system and the use of the Administration Server by deployment operations from external clients, the default channel for the Administration Server should use an EoIB IP address.

Ideally, different types of traffic, such as management and deployments as opposed to runtime invocations, should be isolated from each other. For this, the Administration Server's and SOA/OSB server's EoIB addresses may be placed on a different VLANs. Whether this is required depends also on the type of management and deployment operations performed in each system. For simplicity, this guide uses a model where the SOA/OSB servers and Admin Server EoIB exist in the same VLAN and partition.

3.3.1 Summary of the Required IPoIB Virtual IP Addresses

For all communications over the IPoIB network, the WEBHOST compute nodes and WSM managed servers use the default bond0 interface and the IP address assigned to this interface by default when you set up your Exalogic hardware and software.

Footnote 1 These two EoIB virtual IP addresses are optional and are used for a VLAN separation of the Oracle Traffic Director access points, and also for faster failure detection by using VRRP OTD.

3.4.2 Configuring the EoIB Network for the SOA Enterprise Topology

Information about configuring the Ethernet over Infiniband (EoIB) network on Exalogic is available in the Oracle Exalogic Elastic Cloud Machine Owner's Guide. The instructions here are specific to configuring the EoIB network for the SOA Exalogic enterprise deployment.

Create the appropriate VNIC and VLAN associations, as well as EoIB Virtual IPs on SOAHOST1 and SOAHOST2.

Note:

For VLAN separation of the Oracle Traffic Director access addresses, the appropriate VNICS and EoIB virtual IP addresses must be configured as access points for the Oracle Traffic Director listeners on WEBHOST1 and WEBHOST2. In addition, a separate VLAN must be created for them.

Use an SSH client, such as PuTTY, to log in to a Sun Network QDR InfiniBand Gateway Switch. Oracle recommends signing in as ilom-admin and running the commands through the /SYS/Fabric_Mgmt Linux shell target of the Oracle ILOM CLI.

On the compute node, run the following command to display the list of VNICs available on the compute node:

el01cn01# mlx4_vnic_info -l

This command displays the name of the new interface, as seen on the compute node, such as eth4. Note this ID.

Create another VNIC for the same compute node, but using a connector on a different gateway switch. Note the ethX ID of this VNIC too.

It is recommended that you configure the two EoIB interfaces as a bonded interface, such as bond1.

Create interface files for the VNICs on the compute node.

To ensure correct failover behavior, the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel-assigned ethX interface name (eth4, eth5, and so on). Instead, Oracle recommends that the interface file name and value of the DEVICE directive in the interface file be derived from the EPORT_ID and IOA_PORT values:

Note:

Any other unique naming scheme is also acceptable.

Run the following command to find the EPORT_ID:

#mlx4_vnic_info -i ethX | grep EPORT_ID

For example:

e101cn01#mlx4_vnic_info -i eth4 | grep EPORT_ID EPORT_ID 331

Note the EPORT_ID that is displayed, 331 in this example.

Run the following command to find the IOA_PORT:

#mlx4_vnic_info -i ethX | grep IOA_PORT

For example:

e101cn01#mlx4_vnic_info -i eth4 | grep IOA_PORT
IOA_PORT mlx4_0:1

Note the number after the colon (:) is the IOA_PORT value that is displayed, in this case 1.

Build the interface file name and device name using the following convention:

Interface file name: ifcfg-ethA_B

Device name: ethA_B

A is the EPORT_ID, and B is the number after the colon (:) in the IOA_PORT value.

For example:

Interface file name: ifcfg-eth331_1

Device name: eth331_1

In this example, 331 is the EPORT_ID, and 1 is the value derived from the IOA_PORT.

Create the interface file for the first VNIC, eth4 in the example, by using a text editor, such as VI, and save the file in the following directory:

Make sure that the name of the interface file (ifcfg-eth331_1 in the example) is the name derived in step 12.

For the DEVICE directive, specify the device name (eth331_1 in the example) derived in step 12.

For the HWADDR directive, specify the dummy MAC address created in step 5.

Create an interface file for the second VNIC, for example, eth5.

Be sure to name the interface file and specify the DEVICE directive by using a derived interface name and not the kernel-assigned name, as described earlier. In addition, be sure to specify the relevant dummy MAC address for the HWADDR directive.

After creating the interface files, create the ifcfg-bond1 file. If the file already exists, verify its contents

3.4.3 Creating the EoIB Virtual IPs for the WEBHOST1 and WEBHOST2 Compute Nodes

Be sure to associate the virtual IP addresses with the subinterfaces (bond1:n) listed in the table.

3.4.4 Verifying Connectivity Between Virtual IP Addresses

Verify the connectivity between the different virtual IP addresses: Make sure that all the added virtual IP addresses are accessible externally (specially the load balancer host), and also from the nodes themselves using the bond1 interface.

The ranges 10.10.20.X, 10.10.30.x, 10.10.40.X are used to facilitate scale up or out assignments, as shown in the NetMask example. As long as virtual IPs are reachable to each other and in the same subnet, other values may be used.

3.5 Defining the Required Hostname Resolution

Appropriate hostname resolution is critical to topology designs that can sustain network changes, system relocation and disaster recovery scenarios. It is important that the required DNS (either /etc/hosts or central DNS server) definitions are in place and that WebLogic Servers use hostnames and virtual hostnames instead of using IPs and virtual IPs directly. Additionally, the Exalogic enterprise deployment requires a set of virtual server names for routing requests to the proper server or service within the topology through the external load balancer and the Oracle Traffic Director servers.

These virtual server names must be enabled in the corporate network. IPoIB addresses must be resolved only inside the rack's name resolution system. If multiple racks are going to be connected, to elude possible IP conflict, it is good practice to place these also in a central DNS server. Network administrators at the corporate level should enable this. Alternatively hostnames may be resolved through appropriate /etc/hosts file propagated through the different nodes. Table 3-4 provides an example of names for the different floating IP addresses used by servers in the SOA system.

In Table 3-4, virtual host names that include the suffix, "PRIV-Vn," are those mapped to IPoIB virtual IP addresses that are routed to network interfaces on the internal, Infiniband fabric. Those without the "-PRIV-Vn" suffix are mapped to EoIB virtual IP addresses that are routed to network interfaces on the external EoIB network.

Table 3-4 Hostname and Virtual IP information

Hostname Example for This Guide

IP Example and Interface

Type

Host

Bound By

Details

ADMINVHN

10.10.30.1/bond1:1

EoIB /Floating

ComputeNode3/SOAHOST1

Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the Administration Server from ComputeNode3 to ComputeNode4.

OTDADMINVHN

10.10.20.1/bond1:1

EoIb /Floating

ComputeNode1/WEBHOST1

OTD Administration Server

A floating IP address for the Administration Server is recommended, if you want to manually migrate the OTD Administration Server from ComputeNode1 to ComputeNode2.

WEBHOST1-PRIV

192.168.10.1/bond0

IPoIB/ Fixed

ComputeNode1/WEBHOST1

NA

WEBHOST2-PRIV

192.168.10.2/bond0

IPoIB/ Fixed

ComputeNode2/WEBHOST2

NA

WEBHOST1-PRIV-V1

192.168.50.1/bond0

IPoIB/Floating

ComputeNode1/WEBHOST1

OTD SOA Internal Failover group

The IP for this VHN is managed by OTD. It is used for IPoIB routing to the WLS_SOAn servers

WEBHOST2-PRIV-V2

192.168.50.2/bond0

IPoIB/Floating

ComputeNode2/WEBHOST2

OTD OSB Internal Failover group

The IP for this VHN is managed by OTD. It is used for IPoIB routing to the WLS_OSBn servers

WEBHOST1VHN1

10.10.40.1/bond1

EoIB/Floating

ComputeNode1/WEBHOST1

OTD SOA/OSB External Routing Failover group

The IP for this VHN is managed by OTD. It is used as external-facing EoIB. This is the VHN that the front end load balancer adds to its pool

WEBHOST2VHN1

10.10.40.2/bond1

EoIB/Floating

ComputeNode2/WEBHOST2

OTD SOA/OSB External Routing Failover group

The IP for this VHN is managed by OTD. It is used as external-facing EoIB. This is the VHN that the front end load balancer adds to its pool

SOAHOST1-PRIV

192.168.10.3/bond0

IPoIB/ Fixed

ComputeNode3/SOAHOST1

Node Manager and WLS_WSM1

BOND0 IP used by Node Manager and WSM1 running on ComputeNode3.

SOAHOST2-PRIV

192.168.10.4/bond0

IPoIB/ Fixed

ComputeNode4/SOAHOST2

Node Manager and WLS_WSM2

BOND0 IP used by the Node Manager and WSM2 running on ComputeNode4.

WEBHOST1-PRIV-V1

192.168.50.1/bond0:1

IPoIB/ Floating

ComputeNode1/WEBHOST1

OTD Instance 1 Failover Group

Initially enabled in ComputeNode1 can be failed over by OTD to ComputeNode2.

WEBHOST2-PRIV-V1

192.168.50.2/bond0:1

IPoIB/ Floating

ComputeNode2/WEBHOST2

OTD Instance 2 Failover Group

Initially enabled in ComputeNode1 can be failed over by OTD to ComputeNode2.

SOAHOST1-PRIV-V1

192.168.20.3/bond0:1

IPoIB/ Floating

ComputeNode3/SOAHOST1

WLS_SOA1 default channel

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOAHOST2-PRIV-V1

192.168.20.4/bond0:1

IPoIB/ Floating

ComputeNode4/SOAHOST2

WLS_SOA2 default channel

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

SOAHOST1-PRIV-V2

192.168.40.3/bond0:3

IPoIB/ Floating

ComputeNode3/SOAHOST1

WLS_OSB1 DEFAULT CHANNEL

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOAHOST2-PRIV-V2

192.168.40.4/bond0:3

IPoIB/ Floating

ComputeNode4/SOAHOST2

WLS_OSB2 DEFAULT CHANNEL

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

SOAHOST1VHN1

10.10.20.3/bond1:2

EoIB/ Floating

ComputeNode3/SOAHOST1

WLS_SOA1 External HTTP/T3 channel

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOAHOST2VHN1

10.10.20.4/bond1:2

EoIB/ Floating

ComputeNode4/SOAHOST2

WLS_SOA2 External HTTP/T3 channel

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

SOAHOST1VHN2

10.10.40.3/bond1:3

EoIB/ Floating

ComputeNode3/SOAHOST1

WLS_OSB1 External HTTP/T3 channel

Initially enabled in ComputeNode3 can be failed over by server migration to ComputeNode4.

SOSHOST2VHN2

10.10.40.4/bond1:3

EoIB/ Floating

ComputeNode4/SOAHOST2

WLS_OSB2 External HTTP/T3 channel

Initially enabled in ComputeNode4 can be failed over by server migration to ComputeNode3.

The virtual server names here pertain to real addresses that are available in the corporate network (and some also externally to the internet). Although these virtual servers use the same name, they are different entities from the virtual servers defined in Oracle Traffic Director. The virtual server names described here map to real IPs. The virtual server names used in OTD are just management entities defined in OTD for appropriate routing.

3.6.1 soa.mycompany.com

This virtual server name acts as the access point for all HTTP traffic to the runtime SOA components, such as soa-infra, and Workflow. Redirection of non-SSL/HTTP traffic to SSL/HTTPS is configured. Clients access this service using the address soa.mycompany.com:443.

3.6.2 admin.mycompany.com

This virtual server name acts as the access point for all internal HTTP traffic that is directed to administration services such as WebLogic Administration Server Console and Oracle Enterprise Manager.

The incoming traffic from clients is not SSL-enabled. Clients access this service using the address admin.mycompany.com:80 and the requests are forwarded to port 7777 on WEBHOST1 and WEBHOST2.

3.6.3 osb.mycompany.com

This virtual server name acts as the access point for all HTTP traffic to the runtime Oracle Service Bus resources and proxy services. Redirection of non-SSL/HTTP traffic to SSL/HTTPS is configured. Clients access this service using the address osb.mycompany.com:443.

3.6.4 soainternal.mycompany.com

This virtual server name is used for internal invocations of SOA services. This URL is not exposed to the internet and is only accessible from the intranet. For SOA systems, users can set this while modeling composites or at runtime with the appropriate EM/MBeans, as the URL to be used for internal services invocations. Invocations like callbacks and internal WebServices, however, use an address in OTD for optimized performance on infiniband. For more information see Section 7.7, "Defining Oracle Traffic Director Virtual Servers for an Exalogic Enterprise Deployment."

The incoming traffic from clients is not SSL-enabled. Clients access this service using the address soainternal.mycompany.com:80 which is enabled as Oracle Traffic Director Failover Groups.

3.7 Configuring the Load Balancer

Several virtual servers and associated ports must be configured on the load balancer for different types of network traffic and monitoring. These should be configured to the appropriate real hosts and ports for the services running. Also, the load balancer should be configured to monitor the real host and ports for availability so that the traffic to these is stopped as soon as possible when a service is down. This ensures that incoming traffic on a given virtual host is not directed to an unavailable service in the other tiers.

3.7.1 Load Balancer Requirements

The enterprise topologies use an external load balancer. The external load balancer must have the following features:

Ability to load-balance traffic to a pool of real servers through a virtual host name: Clients access services using the virtual host name (instead of using actual host names). The load balancer can then load balance requests to the servers in the pool.

Port translation configuration.

Monitoring of ports (HTTP and HTTPS).

The ability to configure virtual server names and ports on your external load balancer. The virtual server names and ports must meet the following requirements:

The load balancer should allow configuration of multiple virtual servers. For each virtual server, the load balancer should allow configuration of traffic management on more than one port. For example, for Oracle WebLogic Clusters, the load balancer must be configured with a virtual server and ports for HTTP and HTTPS traffic.

The virtual server names must be associated with IP addresses and be part of your DNS. Clients must be able to access the external load balancer through the virtual server names.

Resource monitoring / port monitoring / process failure detection: The load balancer must be able to detect URL, service, and node failures (through notification or some other means) and to stop directing non-Oracle Net traffic to the failed node. If your external load balancer has the ability to automatically detect failures, you should use it.

Oracle highly recommends configuring the load balancer virtual server to return immediately to the calling client when the back-end services to which it forwards traffic are unavailable. This is preferred over the client disconnecting on its own after a timeout based on the TCP/IP settings on the client compute node.

SSL acceleration (this feature is recommended, but not required).

Configure the virtual server(s) in the load balancer for the directory tier with a high value for the connection timeout for TCP connections. This value should be more than the maximum expected time over which no traffic is expected between Oracle Access Management Access Manager and the directory tier.

Ability to preserve the Client IP Addresses: The load balancer must have the capability to insert the original client IP address of a request in an X-Forwarded-For HTTP header to preserve the Client IP Address.

3.7.2 Load Balancer Configuration Procedures

The procedures for configuring a load balancer differ, depending on the specific type of load balancer. Refer to the vendor supplied documentation for actual steps. The following steps outline the general configuration flow:

Create a pool of servers. This pool contains a list of servers and the ports that are included in the load balancing definition. For example, for load balancing between the web hosts you create a pool of servers which would direct requests to hosts WEBHOST1 and WEBHOST2 on port 7777.

Create rules to determine whether or not a given host and service is available and assign it to the pool of servers described in Step 1.

Create a Virtual Server on the load balancer. This is the address and port that receives requests used by the application. For example, to load balance Web Tier requests you would create a virtual host for https://soa.mycompany.com:443.

If your load balancer supports it, specify whether or not the virtual server is available internally, externally or both. Ensure that internal addresses are only resolvable from inside the network.

Use your system's frontend address as the virtual server address (for example, soa.mycompany.com). The frontend address is the externally facing host name used by your system and that will be exposed in the Internet.

Use port 80 and port 443. Any request that goes to port 80 (non-ssl protocol) should be redirected to port 443 (ssl protocol).

Enable address and port translation.

Enable reset of connections when services and/or nodes are down.

Assign the pool created in step 1 to the virtual server.

Create rules to filter out access to /console and /em on this virtual server.

soainternal.mycompany.com:80

WEBHOST1VHN1.mycompany.com:7777 WEBHOST2VHN1.mycompany.com:7777

HTTP

No

No

Use your internal administration address as the virtual server address (for example, soainternal.mycompany.com). This address is not externalized. It is used by OTD.

Specify HTTP as the protocol

Enable address and port translation.

Enable reset of connections when services and/or nodes are down.

Assign the pool created in step 1 to the virtual server.

Optionally, create rules to filter out access to /console and /em on this virtual server.

osb.mycompany.com:443

WEBHOST1VHN1.mycompany.com:7777 WEBHOST2VHN1.mycompany.com:7777

HTTP

No

Yes

Use port 80 and port 443. Any request that goes to port 80 (non-ssl protocol) should be redirected to port 443 (ssl protocol).

Enable address and port translation.

Enable reset of connections when services and/or nodes are down.

Assign the pool created in step 1 to the virtual server.

3.8 Configuring Firewall Ports

Many Oracle Fusion Middleware components and services use ports. As an administrator, you must know the port numbers used by these services, and to ensure that the same port number is not used by two services on a host.

Most port numbers are assigned during installation.

Table 3-6 lists the ports used in the Oracle Exalogic deployment reference topology, including the ports that you must open on the firewalls in the topology.

Firewall notation:

FW0 refers to the outermost firewall.

VLAN Partition refers to the partition between the web tier and the application tier.

FW2 refers to the firewall between the application tier and the data tier.

Table 3-6 Ports Used for the SOA Enterprise Deployment on Exalogic

Type

Firewall

Port and Port Range

Protocol / Application

Inbound / Outbound

Other Considerations and Timeout Guidelines

Browser request

FW0

80

HTTP / Load Balancer

Inbound

Timeout depends on all HTML content and the type of process model used for SOA.

Browser request

FW0

443

HTTPS / Load Balancer

Inbound

Timeout depends on all HTML content and the type of process model used for SOA.

Callbacks and Outbound invocations

VLAN Partition

80

HTTPS / Load Balancer

Outbound

Timeout depends on all HTML content and the type of process model used for SOA.

Callbacks and Outbound invocations

VLAN Partition

443

HTTPS / Load Balancer

Outbound

Timeout depends on all HTML content and the type of process model used for SOA.

Timeout depends on the type of RMI/T3 invocation and also on the time the longest remote invocation operation may take.

Oracle Service Bus Access HTTP

VLAN Partition

8011

Range: 8011-8021

HTTP / WLS_OSBn

Inbound

Set the timeout to a short period (5-10 seconds).

Oracle Service Bus Access RMI/T3

F0/VLAN Partition

8003

RMI/T3/WLS_OSBn

Both

Timeout depends on the type of RMI/T3 invocation and also on the time the longest remote invocation operation may take.

Administration Console access

VLAN Partition

7001

HTTP / Administration Server and Enterprise Manager

T3

Both

You should tune this timeout based on the type of access to the admin console (whether it is planned to use the Oracle WebLogic Server Administration Console from application tier clients or clients external to the application tier).

Database access

FW2

1521

SQL*Net

Both

Timeout depends on all database content and on the type of process model used for SOA.

Oracle Notification Server (ONS)

FW2

6200

ONS

Both

Required for Gridlink. An ONS server runs on each database server.

Oracle Traffic Director Administration Server Console access

VLAN Partition

8989

HTTP

Both

Tune this timeout based on the type of access to the OTD Admin Console (whether you plan to use the Console from application tier clients or clients external to the application tier)