Main menu

Category Archives: Network

Post navigation

Many applications requires to disable firewall on Linux. The most common used commands are as follows:

Stop the ipchains service.
# service ipchains stop
Stop the iptables service.
# service iptables stop
Stop the ipchains service after reboot.
# chkconfig ipchains off
Stop the iptables service after reboot.
# chkconfig iptables off

Another popular one is to set SELINUX=disabled in the /etc/selinux/config file to disable some extra security restrictions.

The above usually works fine with me when turning off firewall. Recently I run into a situation that makes me to add extra check for firewall stuff. The consultant tried to install Oracle Big Data Discovery on a Red Hat Linux VM and connect it to an Oracle Big Data Appliance (BDA) X6-2 Starter Rack. He used similar approaches as above to turn off the firewall and Linux security between this Red Hat VM and BDA. But still run into a weird issue when BDD application on BDA nodes try to pull a request from a web service on this Red Hat VM. The result has never come back.

I tried ping and ssh. Both worked. Hmm, it does show the connectivity between both. Looks like
firewall issue. Check with network infrastructure team. It has firewall rules between the two, but not enabled yet.

I noticed the OS is Red Hat 7.1 Linux. Could be some new firewall feature in 7.1? After some investigation, yes, it does. On Redhat 7 Linux, the firewall run as firewalld daemon. So let me find out what it does.

Usually there is no need to replace Cisco switch on Exadata. However, certain enterprises might have their own standards to use different switch as part of enterprise standard. In this case, the Cisco ethernet switch on Exadata will be replaced. Oracle Support has a nice document about the process to replace Cisco switch, How to replace a Cisco ethernet switch in an Engineered Systems rack (Doc ID 1531203.1). This document is a good one, but focuses only on steps of replacing Cisco Switch, not enough to specify whether additional steps that need to be performed.

At first, I thought Cisco switch only affected the traffic on Management Network on Exadata and don’t have to shut down database and cell nodes. After discussing with my colleague, Andy Colvin, he brought some good points. Although it is not required to shut down the system, there will be no way to get into any of the components via SSH. Furthermore the storage servers will lose connectivity to DNS, which will have adverse consequences on performance. With so many network cables moving around, it would definitely be easier to shut down the entire system and replace the switch. Yes, that makes sense. Here are the high level steps to replace Cisco Switch.1. Shutdown database nodes 2. Shutdown cell nodes 3. Flip off the switches on PDUs to make sure everything is down. 4. Replace the Cisco switch 5. Turn on PDUs and verify new Ethernet switch 6. Start cell nodes 7. Start database nodes.

Here are the detail steps.Step 1. Shudown database nodes

1) Logon as oracle user to db node 1 and source one db env, get the status of the database.

crsctl status res -­t | more

check status of oracle instances.ps -ef |grep pmon

The above steps are optional. Just to make sure all databases are running normal. If seeing issues in database, you might want to resolve it first before replacing Cisco switch. You don’t add the complexity of issues in the middle of switch changes.

2) Stop all the database currently running on Exadata by using srvctl command.srvctl stop database -d yourdbname

3) Logon as root user to db node 1 and stop crs on the current node./u01/app/11.2/grid/bin/crsctl stop crs

During the shutdown process of CRS, run the following command regularly to check the number of oracle processes. It should reduce to 0 when CRS is stopped.ps -ef|grep d.bin|grep -v grep|wc -l

4) Verify all oracle databases are shut down.ps -­ef | grep pmon

5) Power off the node
Logon to ILOM to Power Off Server
orshutdown -h -y now

1) Turn on PDUs
After turning on PDU, the IB switches are automatically starts. Make sure to give a few minutes to allow IB switches fully boot up before doing anything.

2) Verify the IB switch
To verify IB switch is ok, run the following command as root user on IB switchenv_test

3) Verify the network connectivity to/from the IB switch. You don’t want start cell nodes if you know you have connectivity issues from/to IB switches. There is no nslookup command on IB switch. So you have to use ping command to figure out whether DNS is working or not on IB switches.
a. First ping IB switch and ssh to it as root user

b. After login, ping a server outside Exadata by hostname. It should work.

At least at the time I wrote this blog, there is no oracle support document showing how to change DNS on Exadata. So it might be a good idea to show how to do it. Similar to my previous post, Change Time Zone Configuration on Exadata, changing DNS also involves the changes in the four components on Exadata.

DB nodes

Cell nodes

IB Switches

Ethernet Switches

The following example, we assume the current DNS servers are using the following two IPs, 192.168.10.12 and 192.168.10.13 and we would like to change nameserver to 192.168.10.14 and 192.168.10.15

Step 1. Change at InfiniBand Switches

1. Logon to the first IB switch as root user.ssh root@enkx3sw-ib2.enkitec.com

2. Edit file /etc/resolv.conf

cp -p /etc/resolv.conf /etc/resolv.conf.yyyymmddvi /etc/resolv.conf

Change the line ofnameserver 192.168.10.12nameserver 192.168.10.13
tonameserver 192.168.10.14nameserver 192.168.10.15
3. Verify the change
Note: Interestingly, there is no nslookup program at CentOS on InfiniBand Switch. So have to use ping a hostname to see whether it can translate hostname to an IP.

Many times I need to check out the network traffic on Exadata. 12c OEM Cloud Control is good way to monitor system performance on Exadata. However, sometime I need something quick and want to see the network traffic result from command line. Here are the commands I usually use to check out network traffic for ethernet network and infiniband network.

The first command is dstat.dstat -dnyc -N eth0,bondeth0,bondib0 -C total -f

If add three options lms, it will also shows load, memory usage and swap usage.dstat -dnyclms -N eth0,bondeth0,bondib0 -C total -f

Another command is sarsar -n DEV 3 100|egrep ‘bondib0|bondeth0|eth0’

The above command does not show the heading for the sar command. Here the one with heading:

In the previous post, iDB vs RDS vs SDP on Exadata, I discussed high level overview about these three different concepts used in Oracle Exadata and related Oracle Engineered Systems. I will show a few more useful commands to illustrate these protocols.

The first command I would like to talk about is ibhost. This InfiniBand command discovers the InfiniBand fabric topology or uses the existing topology file to extract the channel adapter nodes. The followings is the output from our x3 1/8 rack Exadata.

For a 1/8 rack, it has 2 database nodes, 3 cell nodes and 2 IB switches. You might notice we have many more nodes than supposed to be on the InfiniBand fabric. From the naming, you might figure out we have our X3 Exadata, Oracle Big Data Appliance, and Oracle Exlatics connected together all within the same InfiniBand network.

In TCP/IP network, we use ping command to verify whether a host can be accessed or not. Similarly, in InfiniBand network, we use rds-ping command to ping another IB node in the network. The following example shows we could do rds-ping from Exlatics node to the ibvip on the first database node.

Then, run the following command on source IB node.rds-stress -s enkx3db01-ibvip.enkitec.com -p 4000 -t 1 -D 600000
-s specify the hostname
-p specify the port number
-t specify the number of tasks
-D specify the total bytes in the RDMA message
After it starts on the source node, the target node will also show the progress.

Like netstat command to check out TCP connections, there is corresponding command, sdpnetstat, for SDP connections. Unfortunately, by default, this command does not exist on Exadata for now, but it does exist on Oracle Big Data Appliance or Oracle Exlatics. Here is one example of the output from Exlatics.

If you have done work on Exadata, you probably hear many buzz words, like Storage Index, Smart Scan, Offloading, and etc. Many of these features are based on InfiniBand Architecture, which is high-speed interconnect architecture with high throughput and low latency. Talking about InfiniBand, many of us know iDB and RDS on Exadata. But not many people know about SDP. In this post, I discuss more in detail about among iDB, RDS and SDP.

Oracle Exadata uses the Intelligent Database protocol (iDB) to transfer data between Database Node and Storage Cell Node. It is implemented in the database kernel and work as funtion shipping architecture to transparently maps database operations to Exadata operations. iDB can be used to transfer SQL operation from Database Node to Cell node, and get query result back or full data blocks back from Cell Node.

iDB is built on Reliable Datagram Sockets (RDS v3) protocol and runs over InfiniBand ZDP (Zero-loss Zero-copy Datagram Protocol). The objective of ZDP is to eliminate unnessary copying of blocks. RDS is based on Socket API with low overhead, low latency, high bandwidth. Exadata Cell Node can send/receive large transfer using Remote Direct Memory Access (RDMA).

RDMA is a direct memory access from the memory of one computer into another computer without involving either’s operating system. The transfer require no work to be done by CPUs, caches, or context switches, and transfers continue in parallel with other system operations. It is quite useful in massively parallel processing environment.

RDS is highly used on Oracle Exadata. RDS can deliver high available and low overhead of datagrams, which is like UDP but more reliable and zero copy. It accesses to InfiniBand via the Socket API. RDS v3 supports both RDMA read and write and can allow large data transfer up to 8MB. It also supports the control messages for asynchronous operation for submit and completion notifications.

If you want to optimize communications between Oracle Engineered System, like Exadata, Big Data Appliance, and Exlatics, you can use Sockets Direct Protocol (SDP) networking protocol. SDP only deals with stream sockets.

SDP allows high-performance zero-copy data transfers via RDMA network fabrics and uses a standard wire protocol over an RDMA fabric to support stream sockets (SOCK_STREAM). The goal of SDP is to provide an RDMA-accelerated alternative to the TCP protocol on IP, at the same time transparent to the application.

It bypasses the OS resident TCP stack for stream connections between any endpoints on the RDMA fabric. All other socket types (such as datagram, raw, packet, etc.) are supported by the IP stack and operate over standard IP interfaces (i.e., IPoIB on InfiniBand fabrics). The IP stack has no dependency on the SDP stack; however, the SDP stack depends on IP drivers for local IP assignments and for IP address resolution for endpoint identifications.

In a future post, I will discuss about some commands useful to check out Infiniband traffic, RDS and SDP.