Hi,
In Response to Mr. Chris Giddings.. Comment/Query on ADMIN State.
Here we are going to see a Scenario which is very common but troublesome. Many times we observe that while trying to restart the Managed Servers the Servers move to ADMIN State rather than moving to the RUNNING State. This usually happens If it is unable to activate some of the Modules which are deployed on this Server. usually it happens If any Application is Not getting activated properly or a Datasource or JTA recovery or a JMS System / SubSystem is not getting activated properly then also it moves into the Admin State. So To find out the Root cause we need to keenly observer the Logs. Specially we need to check what configuration changes we made on the Server recently.
More Alerts On the same Monitoring: http://middlewaremagic.com/weblogic/?p=5838
Most Probable Cause: Most of the cases it happens if any of the Database is down and WebLogic tries to create the Connection Pool at the boot up time. One way to avoid this kind of scenario is to set the InitialCapacity of the DataSource to 0 (Zero) so that weblogic will not try to create any JDBC Connection object at the start up time so like this we can avoid Connection creation failure conditions and avoid moving our server to ADMIN State.
NOTE: We have an option to forcibily move our WLS Server to the RUNNING State (from ADMIN State) which works almost 80% cases….Still it is must that we find the root cause of Why the Server is moving to ADMIN State. So please consider this article as a Workaround …And not as a solution. Because even if we can bring our server in RUNNING state …still we have to find out why the Server Moved to ADMIN State.
In this Demonstration we will see a simple WLST script which will check if any of the server present inside Domain is in ADMIN State or not? If YES then it will try to force them to move to RUNNING State.
Step1). Create a Directory somewhere in your file system like : “C:WLST_AdminStateCheck”
Step2). Write a Properties file “domain.properties” inside “C:WLST_AdminStateCheck” like following:
domain.name=7001_Plain_Domain admin.url=t3://localhost:7001 admin.userName=weblogic admin.password=weblogic1 totalServersToMonitor=2 server.1.url=t3://localhost:7001 server.2.url=t3://localhost:7003
Step3). Write the WLST Script “serverAdminState.py” inside “C:WLST_AdminStateCheck” directory.
############################################################################# # # @author Copyright (c) 2010 - 2011 by Middleware Magic, All Rights Reserved. # ############################################################################# from java.io import FileInputStream propInputStream = FileInputStream("domain.properties") configProps = Properties() configProps.load(propInputStream) domainName=configProps.get("domain.name") adminURL=configProps.get("admin.url") adminUserName=configProps.get("admin.userName") adminPassword=configProps.get("admin.password") totalServerToMonitor=configProps.get("totalServersToMonitor") i=1 while (i <= int(totalServerToMonitor)) : url=configProps.get("server."+ str(i)+".url") connect(adminUserName,adminPassword,url) serverRuntime() state=cmo.getState() name=cmo.getName() if state == 'ADMIN' : print "ALERT::::::::Server Name: " + name + " Is currently in State: " + state try: print 'Resuming Server: .....' cmo.resume() print "Server: "+name +"Moved to State : " + cmo.getState() except: print "NOTE:::::::::Unable to Move Server: " + name + " To good State" else: print '' print '' print "GOOD::::::::> Server Name: " + name + " Is currently in State: " + state + ' :)' i = i + 1
Step4). Run the “. ./setWLSEnv.sh” by adding two DOTs separated by a single space …..before the actual script like following : (use ‘cd’ command to move inside the <BEA_HOME>/wlserver_10.3/server/bin) then run the following command….
. ./setWLSEnv.sh
Note: the first DOT represents that set the Environment in the current Shell, AND the second ./ represents execute the script from the current directory.
Step5). Now run the WLS Script like following:
java weblogic.WLST serverAdminState.py
Following would be the output:
java weblogic.WLST serverAdminState.py Initializing WebLogic Scripting Tool (WLST) ... Welcome to WebLogic Server Administration Scripting Shell Type help() for help on available commands Connecting to t3://localhost:7001 with userid weblogic ... Successfully connected to Admin Server 'AdminServer' that belongs to domain '7001_Plain_Domain'. Warning: An insecure protocol was used to connect to the server. To ensure on-the-wire security, the SSL port or Admin port should be used instead. Location changed to serverRuntime tree. This is a read-only tree with ServerRuntimeMBean as the root. For more help, use help(serverRuntime) GOOD::::::::> Server Name: AdminServer Is currently in State: RUNNING 🙂 Connecting to t3://localhost:7003 with userid weblogic ... Successfully connected to managed Server 'ManagedServer-1' that belongs to domain '7001_Plain_Domain'. Warning: An insecure protocol was used to connect to the server. To ensure on-the-wire security, the SSL port or Admin port should be used instead. ALERT::::::::Server Name: ManagedServer-1 Is currently in State: ADMIN Resuming Server: ..... Server: ManagedServer-1Moved to State : RUNNING
.
.
Regards,
Jay SenSharma
April 17th, 2011 on 4:43 am
Hi Jay,
I have observed situation where after deployment of a Application on Cluster, few nodes of the cluster have the application deployed and few nodes doesn’t have the application Deployed.
Those nodes which doesn’t have the application deployed goes to Admin State and using Admin console we bring back the Node to Running State, but that doesn’t solve the deployment issue.
We need to Bounce the Node to over come deployment Issue.
So, I was expecting to call WLST commands to bounce that Managed Server as well.
Thanks
Sathya
April 17th, 2011 on 9:52 am
Hi Sathya,
http://download.oracle.com/docs/cd/E13222_01/wls/docs92/config_scripting/reference.html#wp1060188
Above link has the command to start the Server as well as to shutdown them.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
April 19th, 2011 on 3:18 am
JaySenSharma,
Thank you very much for posting this! We’ll give it a try in the next couple days!
We know the root cause of why our servers come up in Admin state, and while they are few… they will remain for some time.
A separate script is to be written to check on the deployment state of our data sources across those servers which start in Admin state. That script will then repair the deployment of the data source on those servers by untargeting and retargeting the data source(s).
It’s not a pretty solution, but it’s what we’ve found works with our 160 data sources!
April 19th, 2011 on 9:57 am
Hi JAy ,
Very good article indeed . But this works only when the server going to ADMIN state which is not so critical .
– For instance when a server going to ADMIN becuase of a data source issue and when we try to resume it , It doesnt go to running state even if goes to Running state the application doesnt work . SO just by resuming the manage servers to Running mode doesnt fix the issue .
-Some instances where the resuming doesnt work and the server will go back to ADMIN state and even in that case this scenario wont work right .
-Kiran
April 19th, 2011 on 11:39 am
Hi Kiran,
Yes, The above article is a workaround only which works in 80% cases as mentioned in the article. But still if it does not work in 20% cases then the problem is very serious and we must contact Oracle WebLogic Server Support in such scenarios.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
July 9th, 2011 on 12:13 am
Hi Jay,
Setting the initial capacity to 0 from the admin console will not let the managed server go in ADMIN STATE in case the data source is not available for a application war file?
July 9th, 2011 on 12:19 am
Hi Jay,
Does setting the initial capacity to 0 from the admin console will not let the managed server go in ADMIN STATE in case the data source is not available for a application war file? Is it that simple ?? Everytime there was a database refresh our managed servers would go down since there wasnt any data source available….
July 9th, 2011 on 9:27 am
Hi Middleware_guy,
If the Database is down at the time of WebLogic Startup then it will not be able to make the DataSources conections so there are very high chances that due to this the WebLogic will move to ADMIN State. So if you think that Databases might be down when you are trying to start your WebLogic then it is good to set the DataSource Connection Pool Initial Capacity to 0.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
July 11th, 2011 on 10:00 pm
So will the managed server not move to admin state if i set this initial capacity to 0 ? Is this a solution?
July 12th, 2011 on 9:34 am
Hi Middleware_guy,
Setting initial capacity =0 Just guarantee that the WebLogic Server will not move to Admin State even when the Database is not available. Means if the Server was moving to the Admin State Just because the Unavailability of the Database at the time of WebLogic Start then setting InitialCapacity to 0 will help to reduce the chances.
But if the WebLogic is moving to the Admin State due to some other reasons then we need to find out the root cause of “what kinf os services are unable to be activated properly by WebLogic at the time of startup”? and then we will need to work on that.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
October 2nd, 2017 on 1:15 am
Hijay,
I am facing a issue which makes managed server to failed state after every 2 or 3 hours.I have to restart the BI services every time to start.I cant find a solution could you provide me a solution to overcome this problem.
May 25th, 2012 on 7:08 am
Great Article
February 20th, 2013 on 9:08 pm
Jay, what if the ADMIN state comes up only on the adminserver? To elaborate further, the adminserver has NO deployments or data sources, JMS, etc… bound to it.
February 21st, 2013 on 2:21 am
Just updating my last post….I found the issue. The admin server was targeted in a diag module,untargeted it and admin server came up fine.
January 17th, 2018 on 5:47 pm
Hi,
I have an EAR deployed on cluster(weblogic12c). This application is JMS based and singleton services are activated on managed server1 for order preservation of messages from AQ. However this leads to the deployment showing warning state on other managed servers as it is not able to connect to foreign servers on these nodes (This is the expected behavior.) Is there a way to stop the MDB from trying to connect to foreign server if its not the master server and hence not show warning. Any insights will be highly useful. Thanks!
August 13th, 2018 on 8:30 pm
Greetings,
I have a question, in one of our deployments, one of the two targeted servers goes to Admin from Active, The occurrence of this random and nothing appears in the logs. I have checked the logs for any stopped shutdown of modules/services during startup or during that time period when it goes to admin state.
Can you please give a suggestion?
Regards
SKB5