Hi,

Jay SenSharma

Jay SenSharma

Actual Post Location: http://middlewaremagic.com/weblogic/?page_id=241

Many times we face a very common issue while configuring a Domain which has some remote Managed Servers asssigned to a Machine. The issue we might face is “NodeManager is Inactive”, “SSLException”, “Hostname Verification failed”, “domain salt not found”…etc

=========Issue-1).While Starting the Nodemanager if you see the following Exception …

<Fatal error in node manager server>
weblogic.nodemanager.common.ConfigException: Native version is enabled but node manager native library could not be loaded
at weblogic.nodemanager.server.NMServerConfig.initProcessControl(NMServerConfig.java:212)
at weblogic.nodemanager.server.NMServerConfig.<init>(NMServerConfig.java:172)
at weblogic.nodemanager.server.NMServer.init(NMServer.java:174)
at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:139)
at weblogic.nodemanager.server.NMServer.main(NMServer.java:286)
at weblogic.NodeManager.main(NodeManager.java:31)
Caused by: java.lang.UnsatisfiedLinkError: no nodemanager in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1682)

.

Debugging—->Then please make sure that the Library Path is set properly. There are some variables available like “LD_LIBRARY_PATH (UNIX only)” and “SHLIB_PATH” (For HP-UX operating system) to the Correct library:
Example:
(For Solaris/Linux)
LD_LIBRARY_PATH:$WL_HOME/server/lib/solaris:$WL_HOME/server/lib/solaris/ociXXX_X
(For HP-UX)
SHLIB_PATH=$SHLIB_PATH:$WL_HOME/server/lib/hpux11:$WL_HOME/server/lib/hpux11/ociXXX_X
.

=========Issue-2).If you see the SSL issues in the NodeManager Logs:

javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from aaa.bbb.com – 44.46.5.15. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.

.
Debugging—-> Usually we get this kind of error while starting the Server if the Certificates or SSL configuration is not correct.In this case we need to check the certificates are correct or not which is being used by Admin Server and the Node Manager.

If Admin and the Node Manager are using demo certificates which comes with WebLogic by default, then then we need to check the DNS name is correct or not. Just for Testing better to Disable the Host Name Varification for some time to see if the issue is with the incorrect HostName presence in the Certificate or not …We can use the following JAVA_OPTION to disable the HostName Varification:   -Dweblogic.security.SSL.ignoreHostnameVerification=true
If we are starting the Server using NodeManager then we must disable the HostName Varification in the “startNodeManager.sh” script as well…
-Dweblogic.nodemanager.sslHostNameVerificationEnabled=false
.
It is always a best practice to enable the following JAVA_OPTIONS as well whenever we get the SSL Exception in the Server Logs:   -Dweblogic.security.SSL.enforceConstraints=off        -Dssl.debug=true
.

Usually we get this kind of error while starting the Server if the Certificates or SSL configuration is not correct.In this case we need to check the certificates are correct or not which is being used by Admin Server and the Node Manager. Scenario-1). If Admin and the Node Manager are using demo certificates which comes with WebLogic by default, then then we need to check the DNS name is correct or not. Just for Testing better to Disable the Host Name Varification for some time to see if the issue is with the incorrect HostName presence in the Certificate or not …We can use the following JAVA_OPTION to disable the HostName Varification:  -Dweblogic.security.SSL.ignoreHostnameVerification=true

.

If we are starting the Server using NodeManager then we must disable the HostName Varification in the “startNodeManager.sh” script as well…

-Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

.
It is always a best practice to enable the following JAVA_OPTIONS as well whenever we get the SSL Exception in the Server Logs:   -Dweblogic.security.SSL.enforceConstraints=off -Dssl.debug=true

.

Debugging—-> Specify the following flag in startWeblogic.sh
-Dssl.debug=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.enforceConstraints=off
————–And the following flag in startNodeManager.sh
-Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

.

Debugging—-> Just for confirming that the actual issue is with SSL or not …Please disable the SSL Communication between AdminServer and NodeManager first:
NodeManager to listen over SSL by default, you can set
SecureListener=false (in <WL_HOME>commonnodemamanernodmanager.properties)
And also under machines, change the Listen Type to “Plain” from Admin Console.
Machine->Machine1->nodemanager—>type (Plain)
Then the communication between Admin Server and NodeManger will not be over SSL.
.

=========Issue-3).When you go to AdminConsole Nodemanager monitoring you see NodeManager is “Not Reachable/ Inactive”

Machine->Machine1->Monitoring:
Status: Inactive
Version: (not available)
Debugging—-> Please make sure that you have done the nmEnroll() using WLST. Nodemanager must be Enrolled to the Domain. Please follow the 7th and 8th Steps from the below link to enroll the Nodemanager to the WLS Domain.
http://middlewaremagic.com/weblogic/2010/04/28/weblogic-clustering-in-remote-boxes/
.

=========Issue-4).If you see the following Error While starting your Nodemanager:

<Warning> <I/O error while reading domain directory: java.io.FileNotFoundException: Domain directory ‘F:beawlserver_10.3commonnodemanager’ invalid (domain salt file not found)>
java.io.FileNotFoundException: Domain directory ‘F:beawlserver_10.3commonnodemanager’ invalid (domain salt file not found)
at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:81)
at weblogic.nodemanager.server.DomainManager.<init>(DomainManager.java:53)
at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:252)
at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:218)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:109)
at weblogic.nodemanager.server.Handler.run(Handler.java:66)
at java.lang.Thread.run(Thread.java:619)

Debugging—->Whenever u use WLST never use Back Slash as a Path Separator….Use the following (Note I changed the BackSlash as Forward Slash):
wls:/base_domain/serverConfig> nmEnroll(‘F:/bea/user_projects/domains/YOUR_DOMAIN_NAME’,’F:/bea/wlserver_10.3/common/nodemanager’)

.

Debugging—->Alternative…Still If you want to use the BackSlash only then do the Following (Note i added a preceeding character ‘r’ before the path)
wls:/base_domain/serverConfig> nmEnroll(r’F:beauser_projectsdomainsYOUR_DOMAIN_NAME’,r’F:beawlserver_10.3commonnodemanager’)
Name of the Domain was also missing in your command. You need to use own Domain name where i used “YOUR_DOMAIN_NAME” in my edited Command above.

.

Debugging—->If you still face the Issue then Please Copy the “SerializedSystemIni.dat” file from “F:/bea/user_projects/domains/base_domain/security” Location and Paste it inside “F:/bea/wlserver_10.3/common/nodemanager/security” Location. Then retsart your NodeManager. Or Better Enroll The NodeManager and restart it again.

Practical Implementation Of Above Theory

http://forums.oracle.com/forums/thread.jspa?messageID=4497751

http://forums.oracle.com/forums/thread.jspa?messageID=4483537&tstart=0

http://forums.oracle.com/forums/thread.jspa?messageID=4466417

http://forums.oracle.com/forums/thread.jspa?threadID=1030788

http://forums.oracle.com/forums/thread.jspa?messageID=4272137

http://forums.oracle.com/forums/thread.jspa?messageID=4277611

http://forums.oracle.com/forums/thread.jspa?messageID=4526851

.
.
Thanks
Jay SenSharma

If you enjoyed this post, please considerleaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.