Oracle Nodemanager connection issue (BEA-300033)

19Aug

This blogpost started with a simple BEA-error regarding NodeManager communication. So the first parts documents the solution for this. Later the error occured again and the Weblogic domain configuration was changed to what we expected it to be in the first place (see update 24-08-2011). Later on we tried to pinpoint where this non-expected configuration occured from, and this resulted in the last update (see update 25-08-2011).

This is where it all started (19-08-2011):

Strange situation in a multiple hosts Weblogic domain:

AdminServer is running

Managed Servers are all running (on remote hosts)

Network communication between Managed Servers and Admin works, because configuration changes (the config.xml file) is being replicated to the Managed Servers.

However the Managed Servers can not be restarted due to an unreachable remote Node Manager.

At first I was wondering where this username originated from since it was not part of the embedded LDAP. Then I discovered the credentials were generated during the domain creation and can be found in the Weblogic console at <domain> -> Security -> General -> Advanced -> Nodemanager Username

The first solution was found on the Oracle Forums where it is mentioned that we could nmEnroll the remote machines NodeManager again to the AdminServer. During the installation of the Domain we already did this, so apparently for some reason unknown the NodeManager user and password token got conflicted between the 2 entities.

So for each remote host we performed the following task using WLST:

connect('weblogic','welcome1','t3://myserver:7001')
nmEnroll('C:/myDomain','C:/myOraHome/wlserver_10.3/common/nodemanager')
Enrolling this machine with the domain directory at C:/myDomain ...
Successfully enrolled this machine with the domain directory at C:/myDomain.

Syntax for the nmEnroll command is: nmEnroll([domainDir], [nmHome])

Update 24-08-2011:

However the problem kept reoccuring in the domain so we needed to find a permanent solution.
Instead of the default Weblogic generated Nodemanager credentials we wanted to try to configure a fixed name/password in the Weblogic domain and on each host.

In Weblogic console:

Click on domainname in Domain Structure (left menu)

Select Security -> General -> Advanced

Configure the NodeManager Username with weblogic

Configure the NodeManager Password with welcome1 (2x)

On each remote host:

Navigate to the folder %DOMAINHOME%\config\nodemanager

Edit the file: nm_password.properties

Content should be set to:
username=weblogic
password=welcome1

Save

Restart the Weblogic Node Manager

Update 25-08-2011:

Discussing this behaviour with Jacco Landlust our conclusion was quickly that we seen different configuration results over time. After a few tests we came to the conclusion:

The Weblogic Config Wizard generates nodemanager credentials for your domain when you choose production mode. I can’t remember reading about this in the Weblogic documentation regarding DEV vs PRD differences. So I’m not sure at the moment if this is working-as-expected and if there is some specific security reason for it.