Changing IP Addresses on InfiniBand Network with 12.2 GI on Exadata

I was tasked a while back with modifying the IP addresses assigned to the InfiniBand network on an X6-2 rack. We were configuring a ZFS appliance to share between 2 Exadata clusters, and we wanted to keep them on separate network ranges. In this case, we were moving the interfaces from 192.168.10.x/22 (the Exadata default network range) to 192.168.14.x/22. The process is documented very well by Oracle in the Exadata Maintenance Guide, but after going through the procedure, I hit an issue. I could start CRS on one node, but any additional nodes would fail to start after a 10 minute timeout:

It was strange – I could start CRS on any node on the cluster, but it would only run on a single node. As soon as I tried to start CRS on a second node, we’d get the same error every time. I checked the CRS alert log, and didn’t get much:

After digging and cross-referencing with some MOS notes, I figured out that this was due to the new listeners created in 12c for the ASM network. When I ran srvctl, I saw 2 listeners that were registered with the old InfiniBand network:

I found MOS note #283684.1, which describes the process for modifying the cluster interconnect when using flex ASM. It turns out that my issue was related to nodes not being able to communicate on the ASM network. Even though the new network shows up in oifcfg, the nodes are trying to communicate with these listeners. I had to add a new ASM network listener and remove the old ones: