Ravish Mody

Today, Chris Giddings one of our subscriber asked us by commenting if we have any automated script which can start the Admin Server as they have this type of requirement and the script should not take much load too on there system. So seeing the requirement we took out sometime and wrote a WLST script which can do that job for Chris and hope that this script would also help our other subscribers too.

The logic behind this WLST script is that first this script would start the NodeMnager, then the script would connect to NodeMnager and then it would also start the Admin Server. So in short this script would do three things in no time without much load and just by changing few things in the property file called domains.properties as per your environment which has all the details about the domains and the server which has to started.

Note:

– Make sure you associate a Machine for the Admin Server
– Make sure you keep both this files (i.e. domains.properties and Start_Admin.py) in the same folder when you are running the script or else you would have to give the path.
– Also this script would create nm_data.properties, nodemanager.log and nodemanager.properties if they are not present in the directory from where you are running the script.
– This script does not use SSL, so you would have to first start the adminserver using startWebLogic.sh first, then go to the Admin Console path

Environment –> Machines –> [YOUR_MACHINE_NAME] –> Configuration [tab] –> Node Manager [sub-tab]

Type = Plain

– Following is the nodemanager.properties which we had used you can notice that SecureListener=false as we not using SSL and StartScriptEnabled=true because we had got OutOfMemory: PermGen error so by doing this the memory parameters would be picked up by  startWebLogic.sh

#Fri Apr 29 20:12:43 IST 2011
DomainsFile=/WLS/Start_Admin/nodemanager.domains
LogLimit=0
PropertiesVersion=10.3
AuthenticationEnabled=true
NodeManagerHome=/WLS/Start_Admin
JavaHome=/Jdk/jdk1.6.0_21/jre
LogLevel=INFO
DomainsFileEnabled=true
StartScriptName=startWebLogic.sh
ListenAddress=localhost
NativeVersionEnabled=true
ListenPort=5556
LogToStderr=true
SecureListener=false
LogCount=1
StopScriptEnabled=false
QuitEnabled=true
LogAppend=true
StateCheckInterval=500
CrashRecoveryEnabled=false
StartScriptEnabled=true
LogFile=/WLS/Start_Admin/nodemanager.log
NodemanagerHome=/WLS103/wlserver_10.3/common/nodemanager
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50

Steps to Start AdminServer And NodeManager Using WLST

1. Below is the details which domains.properties would have

username=weblogic
password=weblogic
host=localhost
nm.port=5556
domain.name=Domain_7001
domain.dir=/WLS103/user_projects/domains/Domain_7001
nm.type=plain
server.name=AdminServer

2. And below is the WLST python script Start_Admin.py which would call the above properties.

#############################################################################
#
# @author Copyright (c) 2010 - 2011 by Middleware Magic, All Rights Reserved.
#
#############################################################################
from java.io import FileInputStream
import java.lang
import os
import string

propInputStream = FileInputStream("domain.properties")
configProps = Properties()
configProps.load(propInputStream)

Username = configProps.get("username")
Password = configProps.get("password")
Host = configProps.get("host")
nmPort = configProps.get("nm.port")
domainName = configProps.get("domain.name")
domainDir = configProps.get("domain.dir")
nmType = configProps.get("nm.type")

startNodeManager()
print ''
print '============================================='
print ' NODE MANAGER started Successfully...!!!'
print '============================================='
print ''
nmConnect(Username,Password,Host,nmPort,domainName,domainDir,nmType)
print ''
print '============================================='
print 'Connected to NODE MANAGER Successfully...!!!'
print '============================================='
print ''

serverName = configProps.get("server.name")
print '###### serverName = ', serverName
nmStart(serverName)
serverName = configProps.get("server.name")
print ''
print '============================================='
print '===> Successfully started ', serverName, '  <==='
print '============================================='
print ''

3 . Following would be the output as soon as you run the Start_Admin.py using the below command

Command:

java weblogic.WLST Start_Admin.py

Output:

D:OracleTest AppDomain>java weblogic.WLST Start_Admin.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Launching NodeManager ...
 Properties: {}
 Command: /Jdk/jdk1.6.0_21/jre/bin/java -d64 -classpath /Jdk/jdk1.6.0_21/jre/lib/rt.jar:/Jdk/jdk1.6.0_21/jre/lib/i18n.jar:/WLS103/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/WLS103/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/Jdk/jdk1.6.0_21/lib/tools.jar:/WLS103/wlserver_10.3/server/lib/weblogic_sp.jar:/WLS103/wlserver_10.3/server/lib/weblogic.jar:/WLS103/modules/features/weblogic.server.modules_10.3.0.0.jar:/WLS103/wlserver_10.3/server/lib/webservices.jar:/WLS103/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/WLS103/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar:/WLS103/wlserver_10.3/common/eval/pointbase/lib/pbclient57.jar:/WLS103/wlserver_10.3/server/lib/xqrl.jar weblogic.NodeManager -v
NMProcess: <Apr 29, 2011 8:32:53 PM> <INFO> <Loading domains file: /WLS/Start_Admin/nodemanager.domains>
NMProcess: <Apr 29, 2011 8:32:53 PM> <WARNING> <Domains file not found: /WLS/Start_Admin/nodemanager.domains>
NMProcess: <Apr 29, 2011 8:32:53 PM> <INFO> <Loaded node manager configuration properties from '/WLS/Start_Admin/nodemanager.properties'>
NMProcess: Node manager v10.3
NMProcess:
NMProcess: Configuration settings:
NMProcess:
NMProcess: NodeManagerHome=/WLS/Start_Admin
NMProcess: ListenAddress=localhost
NMProcess: ListenPort=5556
NMProcess: ListenBacklog=50
NMProcess: SecureListener=false
NMProcess: AuthenticationEnabled=true
NMProcess: NativeVersionEnabled=true
NMProcess: CrashRecoveryEnabled=false
NMProcess: JavaHome=/Jdk/jdk1.6.0_21/jre
NMProcess: StartScriptEnabled=true
NMProcess: StopScriptEnabled=false
NMProcess: StartScriptName=startWebLogic.sh
NMProcess: StopScriptName=
NMProcess: LogFile=/WLS/Start_Admin/nodemanager.log
NMProcess: LogLevel=INFO
NMProcess: LogLimit=0
NMProcess: LogCount=1
NMProcess: LogAppend=true
NMProcess: LogToStderr=true
NMProcess: LogFormatter=weblogic.nodemanager.server.LogFormatter
NMProcess: DomainsFile=/WLS/Start_Admin/nodemanager.domains
NMProcess: DomainsFileEnabled=true
NMProcess: StateCheckInterval=500
NMProcess: QuitEnabled=true
NMProcess:
NMProcess: Domain name mappings:
NMProcess:
NMProcess:
NMProcess: <Apr 29, 2011 8:32:54 PM> <INFO> <Plain socket listener started on port 5556, host localhost>
Successfully launched the Node Manager.
The Node Manager process is running independent of the WLST process.
Exiting WLST will not stop the Node Manager process. Please refer
to the Node Manager logs for more information.
The Node Manager logs will be under /WLS/Start_Admin/.

=============================================
 NODE MANAGER started Successfully...!!!
=============================================

Connecting to Node Manager ...
NMProcess: <Apr 29, 2011 8:33:03 PM> <Info> <Domain_7001> <MS-1> <Startup configuration properties loaded from "/WLS103/user_projects/domains/Domain_7001/servers/MS-1/data/nodemanager/startup.properties">
Successfully Connected to Node Manager.

=============================================
Connected to NODE MANAGER Successfully...!!!
=============================================

###### serverName =  AdminServer
Starting server AdminServer ...
NMProcess: <Apr 29, 2011 8:33:03 PM> <Info> <Domain_7001> <AdminServer> <Rotated server output log to "/WLS103/user_projects/domains/Domain_7001/servers/AdminServer/logs/AdminServer.out00007">
NMProcess: <Apr 29, 2011 8:33:03 PM> <Info> <Domain_7001> <AdminServer> <Server error log also redirected to server log>
NMProcess: <Apr 29, 2011 8:33:03 PM> <Info> <Domain_7001> <AdminServer> <Starting WebLogic server with command line: /WLS103/user_projects/domains/Domain_7001/bin/startWebLogic.sh >
NMProcess: <Apr 29, 2011 8:33:03 PM> <Info> <Domain_7001> <AdminServer> <Working directory is "/WLS103/user_projects/domains/Domain_7001">
NMProcess: <Apr 29, 2011 8:33:03 PM> <Info> <Domain_7001> <AdminServer> <Server output log file is "/WLS103/user_projects/domains/Domain_7001/servers/AdminServer/logs/AdminServer.out">
Successfully started server AdminServer ...

=============================================
===> Successfully started  AdminServer   <===
=============================================

.

.

Regards,
Ravish Mody

Ravish Mody

Many times we come across NodeManager is not reachable issue and its one of the conman issues which lot of people face it, in many cases we have found that this issue happens due to missing step of nmEnroll. It means you might have not Enrolled your NodeManager with the Admin Server and thus the NodeManager is not reachable. Hence we have come up  with this post so that anyone can resolve the same issue if the below steps are been followed properly:

.

.

Steps to Resolve NodeManager is not reachable issue

Step1) Start the AdminServer using startScript “startWebLogic.sh/cmd”

Step2) Start the NodeManager.

Step3) Login to admin-console and then see if the NodeManager is Reachable or not at the below console path

Console Path:

Machines -> <YOUR_MACHINE_NAME>  ->  Monitoring (tab)  ->  Node Manager Status (sub-tab)  -> Status: Reachable

If its NOT REACHABLE then please do the following from the Admin Server Box to Enroll the NodeManager

a) Open a command prompt and run "setWLSEnv.sh/cmd" in it then run the following commands

   java weblogic.WLST
   connect('weblogic','weblogic','t3://localhost:7001')
   nmEnroll('C:/bea103/user_projects/domains/7001_Domain','C:/bea103/wlserver_10.3/common/nodemanager')

b)Then again check in the console path if NodeManager is Reachable.

Step4). Once NodeManager is Enrlolled and Reachable in the admin console then continue the following steps.

Step5). Kill the AdminServer because now we need to start it using WLST command nmStart() and NOT using the start scripts.

Step6). Open a command prompt run “setWLSEnv.sh/cmd” and then run the following command

java   weblogic.WLST

Step7). Now connect to the NodeManager using the following command (7001_Domain is the Domain Name for example.)

nmConnect('weblogic','weblogic','localhost','5556','7001_Domain')

Step8). Once you are successfully connected to the Nodemanager please start the AdminServer using the following command

nmStart('AdminServer')

Step9). Once the Server is started run the following command to get the AdminServer Status

nmServerStatus('AdminServer')

Now if you can see the Admin Server status that means now your AdminServer is now reachable. Hope these steps solves your NodeManager is not reachable.

Regards,

Ravish Mody

Hi,

Jay SenSharma

Actual Post Location: http://middlewaremagic.com/weblogic/?page_id=241

Many times we face a very common issue while configuring a Domain which has some remote Managed Servers asssigned to a Machine. The issue we might face is “NodeManager is Inactive”, “SSLException”, “Hostname Verification failed”, “domain salt not found”…etc

=========Issue-1).While Starting the Nodemanager if you see the following Exception …

<Fatal error in node manager server>
weblogic.nodemanager.common.ConfigException: Native version is enabled but node manager native library could not be loaded
at weblogic.nodemanager.server.NMServerConfig.initProcessControl(NMServerConfig.java:212)
at weblogic.nodemanager.server.NMServerConfig.<init>(NMServerConfig.java:172)
at weblogic.nodemanager.server.NMServer.init(NMServer.java:174)
at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:139)
at weblogic.nodemanager.server.NMServer.main(NMServer.java:286)
at weblogic.NodeManager.main(NodeManager.java:31)
Caused by: java.lang.UnsatisfiedLinkError: no nodemanager in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1682)

.

Debugging—->Then please make sure that the Library Path is set properly. There are some variables available like “LD_LIBRARY_PATH (UNIX only)” and “SHLIB_PATH” (For HP-UX operating system) to the Correct library:
Example:
(For Solaris/Linux)
LD_LIBRARY_PATH:$WL_HOME/server/lib/solaris:$WL_HOME/server/lib/solaris/ociXXX_X
(For HP-UX)
SHLIB_PATH=$SHLIB_PATH:$WL_HOME/server/lib/hpux11:$WL_HOME/server/lib/hpux11/ociXXX_X
.

=========Issue-2).If you see the SSL issues in the NodeManager Logs:

javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from aaa.bbb.com – 44.46.5.15. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.

.
Debugging—-> Usually we get this kind of error while starting the Server if the Certificates or SSL configuration is not correct.In this case we need to check the certificates are correct or not which is being used by Admin Server and the Node Manager.

Usually we get this kind of error while starting the Server if the Certificates or SSL configuration is not correct.In this case we need to check the certificates are correct or not which is being used by Admin Server and the Node Manager. Scenario-1). If Admin and the Node Manager are using demo certificates which comes with WebLogic by default, then then we need to check the DNS name is correct or not. Just for Testing better to Disable the Host Name Varification for some time to see if the issue is with the incorrect HostName presence in the Certificate or not …We can use the following JAVA_OPTION to disable the HostName Varification:  -Dweblogic.security.SSL.ignoreHostnameVerification=true

.

If we are starting the Server using NodeManager then we must disable the HostName Varification in the “startNodeManager.sh” script as well…

-Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

.
It is always a best practice to enable the following JAVA_OPTIONS as well whenever we get the SSL Exception in the Server Logs:   -Dweblogic.security.SSL.enforceConstraints=off -Dssl.debug=true

.

Debugging—-> Specify the following flag in startWeblogic.sh
-Dssl.debug=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.enforceConstraints=off
————–And the following flag in startNodeManager.sh
-Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

.

Debugging—-> Just for confirming that the actual issue is with SSL or not …Please disable the SSL Communication between AdminServer and NodeManager first:
NodeManager to listen over SSL by default, you can set
SecureListener=false (in <WL_HOME>commonnodemamanernodmanager.properties)
And also under machines, change the Listen Type to “Plain” from Admin Console.
Machine->Machine1->nodemanager—>type (Plain)
Then the communication between Admin Server and NodeManger will not be over SSL.
.

=========Issue-3).When you go to AdminConsole Nodemanager monitoring you see NodeManager is “Not Reachable/ Inactive”

Machine->Machine1->Monitoring:
Status: Inactive
Version: (not available)
Debugging—-> Please make sure that you have done the nmEnroll() using WLST. Nodemanager must be Enrolled to the Domain. Please follow the 7th and 8th Steps from the below link to enroll the Nodemanager to the WLS Domain.
http://middlewaremagic.com/weblogic/2010/04/28/weblogic-clustering-in-remote-boxes/
.

=========Issue-4).If you see the following Error While starting your Nodemanager:

<Warning> <I/O error while reading domain directory: java.io.FileNotFoundException: Domain directory ‘F:beawlserver_10.3commonnodemanager’ invalid (domain salt file not found)>
java.io.FileNotFoundException: Domain directory ‘F:beawlserver_10.3commonnodemanager’ invalid (domain salt file not found)
at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:81)
at weblogic.nodemanager.server.DomainManager.<init>(DomainManager.java:53)
at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:252)
at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:218)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:109)
at weblogic.nodemanager.server.Handler.run(Handler.java:66)
at java.lang.Thread.run(Thread.java:619)

Debugging—->Whenever u use WLST never use Back Slash as a Path Separator….Use the following (Note I changed the BackSlash as Forward Slash):
wls:/base_domain/serverConfig> nmEnroll(‘F:/bea/user_projects/domains/YOUR_DOMAIN_NAME’,’F:/bea/wlserver_10.3/common/nodemanager’)

.

Debugging—->Alternative…Still If you want to use the BackSlash only then do the Following (Note i added a preceeding character ‘r’ before the path)
wls:/base_domain/serverConfig> nmEnroll(r’F:beauser_projectsdomainsYOUR_DOMAIN_NAME’,r’F:beawlserver_10.3commonnodemanager’)
Name of the Domain was also missing in your command. You need to use own Domain name where i used “YOUR_DOMAIN_NAME” in my edited Command above.

.

Debugging—->If you still face the Issue then Please Copy the “SerializedSystemIni.dat” file from “F:/bea/user_projects/domains/base_domain/security” Location and Paste it inside “F:/bea/wlserver_10.3/common/nodemanager/security” Location. Then retsart your NodeManager. Or Better Enroll The NodeManager and restart it again.

Practical Implementation Of Above Theory

http://forums.oracle.com/forums/thread.jspa?messageID=4497751

http://forums.oracle.com/forums/thread.jspa?messageID=4483537&tstart=0

http://forums.oracle.com/forums/thread.jspa?messageID=4466417

http://forums.oracle.com/forums/thread.jspa?threadID=1030788

http://forums.oracle.com/forums/thread.jspa?messageID=4272137

http://forums.oracle.com/forums/thread.jspa?messageID=4277611

http://forums.oracle.com/forums/thread.jspa?messageID=4526851

.
.
Thanks
Jay SenSharma