Monitoring

Getting All Server’s Lifecycle State in a Domain

Hi,

Jay SenSharma

Jay SenSharma

Based on the query of one of our Magic User  Mr. Vibhor Rastogi (http://middlewaremagic.com/weblogic/?p=2110#comment-4287) We are developing this article.   It was asked that how we can get the Server’s actual lifecycle state using JMX Code when the server is not in Running state. Earlier we developed a JMX Code “Getting ServerState (Enhanced Version) for WLS9.x & Above” but that code was able to just fetch the Servers state which already started listening on it’s Listen Port so it was easy to get it’s State using ServerRuntimeMBean.

But in the current demonstration we will see how to fetch the Server’s different lifecycle states like “Starting”, “Stand By”, “Admin” , “Resuming”, “Running” etc while the Server is coming up and the Listen port is not actually opened so far.

.
Just use the following JMX Code and compile and run it after running the “$WLS_HOME/server/bin/setWLSEnv.sh” or “setWLSEnv.cmd” script so that the CLASSPATH and PATH will contain the required set of Jars and executables.
ServerLifecycleState.java

import weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean;
import weblogic.management.runtime.*;
import javax.management.MBeanServerConnection;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.naming.Context;
import weblogic.management.jmx.MBeanServerInvocationHandler;
import java.util.Hashtable;
import java.io.IOException;
import java.net.MalformedURLException;
import weblogic.management.runtime.JDBCDataSourceRuntimeMBean;
import javax.management.*;
import javax.naming.*;

public class ServerLifecycleState {

private static MBeanServerConnection connection;
private static JMXConnector connector;
private static final ObjectName service;
private static String combea = "com.bea:Name=";
private static String service1 = "DomainRuntimeService,Type=weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean";
private static String service2 = "RuntimeService,Type=weblogic.management.mbeanservers.runtime.RuntimeServiceMBean";

   static
      {
        try {
               service =new ObjectName(combea + service1);
            } catch (MalformedObjectNameException e) {
              throw new AssertionError(e.getMessage());
            }
      }

   public static void initConnection(String hostname, String portString, String username, String password) throws IOException,MalformedURLException
     {
	String protocol = "t3";
	Integer portInteger = Integer.valueOf(portString);
	int port = portInteger.intValue();
	String jndiroot = "/jndi/";
	String mserver = "weblogic.management.mbeanservers.domainruntime";
	JMXServiceURL serviceURL = new JMXServiceURL(protocol, hostname, port, jndiroot + mserver);
	Hashtable h = new Hashtable();
	h.put(Context.SECURITY_PRINCIPAL, username);
	h.put(Context.SECURITY_CREDENTIALS, password);
	h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES,"weblogic.management.remote");
	connector = JMXConnectorFactory.connect(serviceURL, h);
	connection = connector.getMBeanServerConnection();
     }

   public static ObjectName[] getServerRuntimes() throws Exception
    {
        return (ObjectName[]) connection.getAttribute(service,"ServerRuntimes");
    }

   public void printNameAndState() throws Exception
    {
	ObjectName arr[]=getServerRuntimes();
	ObjectName domain = (ObjectName) connection.getAttribute(service,"DomainConfiguration");
	System.out.println("t Domain: " + domain.toString());
	ObjectName[] servers = (ObjectName[]) connection.getAttribute(domain,"Servers");
    String serverState="UNKNOWN";
	for (ObjectName server : servers)
              {
		        String aName = (String) connection.getAttribute(server,"Name");
       		    try{
 			         ObjectName ser= new ObjectName("com.bea:Name="+aName+",Location="+aName+",Type=ServerRuntime");
			         serverState=(String) connection.getAttribute(ser,"State");
			         System.out.println("nt Server: "+aName+"t State: "+serverState );
		           }
		        catch(Exception e)
		           {
					 DomainRuntimeServiceMBean domainRuntimeService = (DomainRuntimeServiceMBean) MBeanServerInvocationHandler.newProxyInstance(connection, new ObjectName(DomainRuntimeServiceMBean.OBJECT_NAME));
					 DomainRuntimeMBean domainRuntime = domainRuntimeService.getDomainRuntime();
					 ServerLifeCycleRuntimeMBean serverLifeCycleRuntimeMBean = domainRuntime.lookupServerLifeCycleRuntime(aName);
			         System.out.println("t Server: "+aName+"t In LifeCycleState : "+ serverLifeCycleRuntimeMBean.getState());
		           }
              }
      System.out.println("t -------------------------- data/time:t"+new java.util.Date()+" ----------------------------");
    }

  public static void main(String[] args) throws Exception
   {
	String hostname = args[0];
	String portString = args[1];
	String username = args[2];
	String password = args[3];
	ServerLifecycleState s = new ServerLifecycleState();
	initConnection(hostname, portString, username, password);
	for(int i=0;i<1000;i++)
	   {
	       s.printNameAndState();
		   Thread.sleep(500);
	   }
	connector.close();
   }
 }

Compile and run the following program and for testing in parallel start any of your Managed Server.
you will see following kind of output Notice the Managed Server LifeCycle state is changing:

  -------------------------- data/time:  Mon Jun 20 00:16:36 IST 2011 ----------------------------
  Domain: com.bea:Name=9001_domain,Location=9001_domain,Type=Domain

  Server: AdminServer     State: RUNNING
  Server: MS1     In LifeCycleState : SHUTDOWN
  -------------------------- data/time:  Mon Jun 20 00:16:37 IST 2011 ----------------------------
  Domain: com.bea:Name=9001_domain,Location=9001_domain,Type=Domain

  Server: AdminServer     State: RUNNING
  Server: MS1     In LifeCycleState : STARTING
  -------------------------- data/time:  Mon Jun 20 00:16:38 IST 2011 ----------------------------
  Domain: com.bea:Name=9001_domain,Location=9001_domain,Type=Domain

  Server: AdminServer     State: RUNNING
  Server: MS1     In LifeCycleState : STARTING
  -------------------------- data/time:  Mon Jun 20 00:16:38 IST 2011 ----------------------------
  Domain: com.bea:Name=9001_domain,Location=9001_domain,Type=Domain

  Server: AdminServer     State: RUNNING
  Server: MS1     In LifeCycleState : STARTING
  -------------------------- data/time:  Mon Jun 20 00:16:39 IST 2011 ----------------------------
  Domain: com.bea:Name=9001_domain,Location=9001_domain,Type=Domain

  Server: AdminServer     State: RUNNING
  Server: MS1     In LifeCycleState : RESUMING
  -------------------------- data/time:  Mon Jun 20 00:16:39 IST 2011 ----------------------------
  Domain: com.bea:Name=9001_domain,Location=9001_domain,Type=Domain

  Server: AdminServer     State: RUNNING

  Server: MS1     State: RUNNING
Server State Using ServerLifeCycleRuntimeMBean

Server State Using ServerLifeCycleRuntimeMBean

.
.
Thanks
Jay SenSharma


How to Generate HeapDump in case of Windows Service ?

Hi,

Jay SenSharma

Jay SenSharma

Based on the query of Magic Subscriber “fipnova51we are writing this article.  This article is very very important in case of Windows Service.  Because if we run any Server like JBoss or WebLogic like a Windows Service then NONE of the JVM utilities like “jmap”, “jconsole” , “jps” …etc will work. So it becomes very difficult to collect the Thread Dump or Heap Dump of the Running Service.

We wrote an article using JMX to collect all the JVM details using JMX code “Remote JVM Analysis (Mini JConsole)“. Here we are extending the same program to make the JMX program to even collect the HeapDump whenever we want.

Step1). Make a Windows Service for example you can use the following Script to make your WebLogic Server as Windows Service…

echo off
SETLOCAL
set DOMAIN_NAME=Test_Domain
set USERDOMAIN_HOME=C:bea103user_projectsdomainsTest_Domain
set SERVER_NAME=AdminServer
set WLS_USER=weblogic
set WLS_PW=weblogic
set PRODUCTION_MODE=true
set JAVA_VENDOR=Sun
set JAVA_HOME=C:bea103jdk160_05
set MEM_ARGS=-Xms1024m -Xmx1024m  -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1

set JAVA_OPTIONS=-Dweblogic.Stdout=C:bea103user_projectsdomainsTest_DomainASstdout.txt -Dweblogic.Stderr=C:bea103user_projectsdomainsTest_DomainASstderr.txt %MEM_ARGS%
call "C:bea103wlserver_10.3serverbininstallSvc.cmd"
ENDLOCAL

NOTE: Make sure that the -Dcom.sun.management.* flags are added properly in the JAVA_OPTIONS of the script.

Step2). Now You can also use the following Script to Uninstall your WebLogic Server Windows Service …

SETLOCAL
set WL_HOME=C:bea103wlserver_10.3
rem *** Uninstall the service
"%WL_HOME%serverbinbeasvc" -remove -svcname:"beasvc Test_Domain_AdminServer"
ENDLOCAL

Step3). Now Write the following JMX code “GenerateHeapDump.java”  inside some directory.

import java.util.Hashtable;
import java.io.*;
import javax.management.MBeanServerConnection;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.naming.Context;
import java.lang.management.MemoryMXBean;
import java.lang.management.ManagementFactory;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
import javax.management.*;
import javax.management.openmbean.CompositeDataSupport;

public class GenerateHeapDump
{
private static MBeanServerConnection connection;
private static JMXConnector connector;
public static void Connection(String hostname, String port) throws IOException
{
Integer portInteger = Integer.valueOf(port);
Hashtable h = new Hashtable();
JMXServiceURL address = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://"+hostname+":"+port+"/jmxrmi");
connector = JMXConnectorFactory.connect(address,null);
connection = connector.getMBeanServerConnection();
System.out.println("GOT THE MBeanServerConnection---SUCCESSFULLY");
}

private static void doGarbageCollection() throws Exception
{
ObjectName memoryMXBean=new ObjectName("java.lang:type=Memory");
connection.invoke(memoryMXBean,"gc", null, null);
System.out.println("nnt------Garbage Collection Done Successfully-----");
}

private static void getHeapDump(String fileName) throws Exception
   {
      ObjectName memoryMXBean=new ObjectName("com.sun.management:type=HotSpotDiagnostic");
      Object[] params = new Object[] { "C:/WindowsService_Script/"+fileName, Boolean.TRUE };
      String[] signature = new String[] { String.class.getName(), boolean.class.getName() };
      Object result = connection.invoke(memoryMXBean, "dumpHeap" , params, signature);
      System.out.println("nt Heap Dump Generated to " +fileName);
}

public static void main(String[] args) throws Exception
{
String hostname = args[0];
String port = args[1];
Connection(hostname, port);
System.out.println("nt----------Generating Heap Dump---------");
getHeapDump("HeapDump.hprof");
connector.close();
}
}

Step4). Run your Windows Service and then compile and run your Program to generate the HeapDump.

C:WindowsService_Script>set PATH=c:bea103jdk160_05bin;%PATH%;

C:WindowsService_Script>javac GenerateHeapDump.java

C:WindowsService_Script>java GenerateHeapDump localhost 9999
GOT THE MBeanServerConnection---SUCCESSFULLY

        ----------Generating Heap Dump---------

         Heap Dump Generated to HeapDump.hprof

Step5). Now you can see that the Heap Dump is generated as desired … 🙂

.

.

Thanks

Jay SenSharma


WebLogic Server is in ADMIN State ?

Hi,

Jay SenSharma

Jay SenSharma

In Response to Mr. Chris Giddings.. Comment/Query on ADMIN State.
Here we are going to see a Scenario which is very common but troublesome. Many times we observe that while trying to restart the Managed Servers the Servers move to ADMIN State rather than moving to the RUNNING State.  This usually happens  If it is unable to activate some of the Modules which are deployed on this Server. usually it happens If any Application is Not getting activated properly or a Datasource or JTA recovery or a JMS System / SubSystem is not getting activated properly then also it moves into the Admin State. So To find out the Root cause we need to keenly observer the Logs. Specially we need to check what configuration changes we made on the Server recently.

More Alerts On the same Monitoring:  http://middlewaremagic.com/weblogic/?p=5838

Most Probable Cause: Most of the cases it happens if any of the Database is down and WebLogic tries to create the Connection Pool at the boot up time.  One way to avoid this kind of scenario is to set the InitialCapacity of the DataSource to  0 (Zero) so that weblogic will not try to create any JDBC Connection object at the start up time so like this we can avoid Connection creation failure conditions and avoid moving our server to ADMIN State.

NOTE: We have an option to forcibily move our WLS Server to the RUNNING State (from ADMIN State) which works almost 80% cases….Still it is must that we find the root cause of Why the Server is moving to ADMIN State.  So please consider this article as a Workaround …And not as a solution. Because even if we can bring our server in RUNNING state …still we have to find out why the Server Moved to ADMIN State.

In this Demonstration we will see a simple WLST script which will check if any of the server present inside Domain is in ADMIN State or not?  If YES then it will try to force them to move to RUNNING State.

Step1). Create a Directory somewhere in your file system like :  “C:WLST_AdminStateCheck”

Step2). Write a Properties file “domain.properties” inside “C:WLST_AdminStateCheck” like following:

domain.name=7001_Plain_Domain
admin.url=t3://localhost:7001
admin.userName=weblogic
admin.password=weblogic1

totalServersToMonitor=2
server.1.url=t3://localhost:7001
server.2.url=t3://localhost:7003

Step3). Write the  WLST Script “serverAdminState.py” inside “C:WLST_AdminStateCheck” directory.


#############################################################################
#
# @author Copyright (c) 2010 - 2011 by Middleware Magic, All Rights Reserved.
#
#############################################################################

from java.io import FileInputStream

propInputStream = FileInputStream("domain.properties")
configProps = Properties()
configProps.load(propInputStream)

domainName=configProps.get("domain.name")
adminURL=configProps.get("admin.url")
adminUserName=configProps.get("admin.userName")
adminPassword=configProps.get("admin.password")
totalServerToMonitor=configProps.get("totalServersToMonitor")

i=1
while (i <= int(totalServerToMonitor)) :
	url=configProps.get("server."+ str(i)+".url")
	connect(adminUserName,adminPassword,url)
	serverRuntime()
	state=cmo.getState()
	name=cmo.getName()
	if state == 'ADMIN' :
		print "ALERT::::::::Server Name: " + name + " Is currently in State: " + state
		try:
			print 'Resuming Server: .....'
			cmo.resume()
			print "Server: "+name +"Moved to State : " + cmo.getState()
		except:
			print "NOTE:::::::::Unable to Move Server: " + name + " To good State"
	else:
		print ''
		print ''
		print "GOOD::::::::> Server Name: " + name + " Is currently in State: " + state + '                     :)'
	i = i + 1

Step4). Run the “. ./setWLSEnv.sh” by adding two DOTs separated by a single space …..before the actual script like following : (use ‘cd’ command to move inside the <BEA_HOME>/wlserver_10.3/server/bin) then run the following command….
.  ./setWLSEnv.sh

Note: the first DOT represents that set the Environment in the current Shell, AND the second ./ represents execute the script from the current directory.

Step5). Now run the WLS Script like following:

java        weblogic.WLST        serverAdminState.py

Following would be the output:

java weblogic.WLST serverAdminState.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain '7001_Plain_Domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Location changed to serverRuntime tree. This is a read-only tree with ServerRuntimeMBean as the root.
For more help, use help(serverRuntime)

GOOD::::::::> Server Name: AdminServer Is currently in State: RUNNING                     🙂
Connecting to t3://localhost:7003 with userid weblogic ...
Successfully connected to managed Server 'ManagedServer-1' that belongs to domain '7001_Plain_Domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

ALERT::::::::Server Name: ManagedServer-1 Is currently in State: ADMIN
Resuming Server: .....
Server: ManagedServer-1Moved to State : RUNNING

.
.
Regards,
Jay SenSharma


Copyright © 2010-2012 Middleware Magic. All rights reserved. |