Exadata ovm creation: Second argument is not defined: bondeth_mode

There are some days which start as exatastic days and end up at Oracle Support. This blogpost is (hopefully) one with temporal validity. I know that’s a 12c feature as well, but that’s another story. The error which I’ll discuss here, will tell you about the “Second argument is not defined: bondeth_mode” error in the domu maker logfile.

A while ago, one of my customers asked me to extend one of their exadata’s with 2 extra compute nodes, make sure they could do ovm on it. Oh by the way, make sure that they split the admin network from the existing one and while we’re busy, we’ll put the new vm’s in other public networks as well. Maybe I should put the “how did I manage to get this done” in another blog post, but it went actually surprisingly well. Bit creativity, some escalations at the network/firewall people (this is an installation in a secure environment) later, I found this problem which is quite interesting.

Step 2 of the OEDA is “creating virtual machines”, if you’re not using capacity on demand. So in my case … i had to create vm’s. I took the new configfile and he created one vm. Failed on the other one. This is strange because he managed to create the bridge, so he should be able to use it.
I tried it with the vm’s separately, same result. Basically everything I tried (even older OEDA, mind this sentence) turned out to provide me with this error.

First of all, I want to stress that the output is anonymised in order to keep the customers’ environment secure. So here we go.

First things first. This are the steps I need to execute for install.sh:

The first validation was already ok, so not going to show that one here, but the next output is interesting. So the actual vm creation failed like this. I used a separated xml in order not to disturb the other succeeded vm.

Pretty cryptic huh? The strangest thing is that by creating the second vm (long story about that one, but I had to create them separately and 2nd one must be done first. Just don’t ask 🙂 ) the bridge was created successfully:

So I did that and that, as this first vm will only be used for monitoring, I actually don’t need the rest, just a plain linux, so this might be a good workaround/solution, right?

[sourcecode][root@demoexa01db01 linux-x64]# /opt/exadata_ovm/exadata.img.domu_maker start-domain /EXAVMIMAGES/conf/final-demoexa01adm01vm01.example.com-vm.xml
[INFO ] Verify run time values are consistent with the configuration from file: /opt/oracle.cellos/cell.conf
[INFO ] Start network discovery in DOM0 to check and to create bridges for DOMU
[root@demoexa01db01 linux-x64]#[/sourcecode]

Exadata says no … all logfiles show the same. Almost desperate! BUT! There is another place to look. Take a look at the /var/log/cellos/exadata.img.domu_maker.log that is the logfile domu maker uses to store what he is doing. There is a log and a tracefile and that is exactly what we are looking for now. Let’s investigate.

[sourcecode][1496740473][2017-06-06 09:16:36 +0000][EXCEPTION][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_check_bondeth_mode][1527] Second argument is not defined: bondeth_mode
[/sourcecode]

I checked whatever I could check and eventually ended up opening a SR with oracle. They did a very good job and they confirmed my suspicions. This behaviour matches the description and behaviour in bug# 26184688. Unfortunately it isn’t a public one.

The solution is very simple. We’re at the beginning of june now and this issue should be covered in the OEDA of june 2017. So they provided me a password protected download link for the june2017 OEDA and that should work. So I downloaded the patched version (basically just a release) and tried again. So you’d think all good. Close but no cigar. So I went back to development and 2 days later I got a new Ocmd version via the SR. This one did the job wonderfully.

I’m still puzzled why I received the bondeth message on invoking the domumaker manually. This question is still on the table at Oracle Support, so if I have an answer, I’ll update this blogpost.