JVM Tuning

How to Generate HeapDump in case of Windows Service ?

Hi,

Jay SenSharma

Jay SenSharma

Based on the query of Magic Subscriber “fipnova51we are writing this article.  This article is very very important in case of Windows Service.  Because if we run any Server like JBoss or WebLogic like a Windows Service then NONE of the JVM utilities like “jmap”, “jconsole” , “jps” …etc will work. So it becomes very difficult to collect the Thread Dump or Heap Dump of the Running Service.

We wrote an article using JMX to collect all the JVM details using JMX code “Remote JVM Analysis (Mini JConsole)“. Here we are extending the same program to make the JMX program to even collect the HeapDump whenever we want.

Step1). Make a Windows Service for example you can use the following Script to make your WebLogic Server as Windows Service…

echo off
SETLOCAL
set DOMAIN_NAME=Test_Domain
set USERDOMAIN_HOME=C:bea103user_projectsdomainsTest_Domain
set SERVER_NAME=AdminServer
set WLS_USER=weblogic
set WLS_PW=weblogic
set PRODUCTION_MODE=true
set JAVA_VENDOR=Sun
set JAVA_HOME=C:bea103jdk160_05
set MEM_ARGS=-Xms1024m -Xmx1024m  -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1

set JAVA_OPTIONS=-Dweblogic.Stdout=C:bea103user_projectsdomainsTest_DomainASstdout.txt -Dweblogic.Stderr=C:bea103user_projectsdomainsTest_DomainASstderr.txt %MEM_ARGS%
call "C:bea103wlserver_10.3serverbininstallSvc.cmd"
ENDLOCAL

NOTE: Make sure that the -Dcom.sun.management.* flags are added properly in the JAVA_OPTIONS of the script.

Step2). Now You can also use the following Script to Uninstall your WebLogic Server Windows Service …

SETLOCAL
set WL_HOME=C:bea103wlserver_10.3
rem *** Uninstall the service
"%WL_HOME%serverbinbeasvc" -remove -svcname:"beasvc Test_Domain_AdminServer"
ENDLOCAL

Step3). Now Write the following JMX code “GenerateHeapDump.java”  inside some directory.

import java.util.Hashtable;
import java.io.*;
import javax.management.MBeanServerConnection;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.naming.Context;
import java.lang.management.MemoryMXBean;
import java.lang.management.ManagementFactory;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
import javax.management.*;
import javax.management.openmbean.CompositeDataSupport;

public class GenerateHeapDump
{
private static MBeanServerConnection connection;
private static JMXConnector connector;
public static void Connection(String hostname, String port) throws IOException
{
Integer portInteger = Integer.valueOf(port);
Hashtable h = new Hashtable();
JMXServiceURL address = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://"+hostname+":"+port+"/jmxrmi");
connector = JMXConnectorFactory.connect(address,null);
connection = connector.getMBeanServerConnection();
System.out.println("GOT THE MBeanServerConnection---SUCCESSFULLY");
}

private static void doGarbageCollection() throws Exception
{
ObjectName memoryMXBean=new ObjectName("java.lang:type=Memory");
connection.invoke(memoryMXBean,"gc", null, null);
System.out.println("nnt------Garbage Collection Done Successfully-----");
}

private static void getHeapDump(String fileName) throws Exception
   {
      ObjectName memoryMXBean=new ObjectName("com.sun.management:type=HotSpotDiagnostic");
      Object[] params = new Object[] { "C:/WindowsService_Script/"+fileName, Boolean.TRUE };
      String[] signature = new String[] { String.class.getName(), boolean.class.getName() };
      Object result = connection.invoke(memoryMXBean, "dumpHeap" , params, signature);
      System.out.println("nt Heap Dump Generated to " +fileName);
}

public static void main(String[] args) throws Exception
{
String hostname = args[0];
String port = args[1];
Connection(hostname, port);
System.out.println("nt----------Generating Heap Dump---------");
getHeapDump("HeapDump.hprof");
connector.close();
}
}

Step4). Run your Windows Service and then compile and run your Program to generate the HeapDump.

C:WindowsService_Script>set PATH=c:bea103jdk160_05bin;%PATH%;

C:WindowsService_Script>javac GenerateHeapDump.java

C:WindowsService_Script>java GenerateHeapDump localhost 9999
GOT THE MBeanServerConnection---SUCCESSFULLY

        ----------Generating Heap Dump---------

         Heap Dump Generated to HeapDump.hprof

Step5). Now you can see that the Heap Dump is generated as desired … 🙂

.

.

Thanks

Jay SenSharma


Security Breach And Attack For Java Based Application Servers

Hi,

Jay SenSharma

Jay SenSharma

DISCLAIMER:

In this article we may see an abnormal behaviors of weblogic. Which may not be necessarily a BUG but it is always to be aware of such behavior while using Weblogic. The idea behind making this page is just to make awareness among the WebLogic Admins to be alert specially when the some of these behaviors are related to WebLogic Security.

Some of the behaviors of WebLogic which may be due to inappropriate Security implementation in the Security system of WebLogic, Even if in some cases it is work as designed, Still it suggests to keep an eye on it and try to make those features more enhanced. Some of them are now fixed by the Application Server Vendor but still some need to be fixed or enhanced. The intentions here are not to point to the weak points of any Application Server but solely to make people aware about such strange or uncommon behaviors.

==========================================================================
Any WebServer or Application Server which runs on below mentioned JVM are not safe due to the security breach. For example if you just want to hang A server then just sent the following request to the Server using any HttpClient like JMeter or any other Utility which allows you to send the Http Header of your Choice.

Once you are able to send the following Http Request Header successfully to the Java based Application/Web Server …the Server will try to parse the Http Request Header and it will Hang  while processing this request.

“GET”,”/”,headers={“Accept-Language”: “en-us;q=2.2250738585072012e-308”}

Just for an example try to run the following simple Java program which just tries to parse a Double value 2.2250738585072012e-308. As soon as you will run this program you will see that your JVM will Hang….and the CPU Utilization will be around 100%   😉

class HangJVM
{
  public static void main(String[] args)
    {
      System.out.println("Test:");
      double d = Double.parseDouble("2.2250738585072012e-308");
      System.out.println("Value: " + d);
     }
}

The JVMs which are affected are as following:
Java SE
JDK and JRE 6 Update 23 and earlier for Windows, Solaris, and Linux
JDK 5.0 Update 27 and earlier for Solaris 9
SDK 1.4.2_29 and earlier for Solaris 8

Java for Business
JDK and JRE 6 Update 23 and earlier for Windows, Solaris and Linux
JDK and JRE 5.0 Update 27 and earlier for Windows, Solaris and Linux
SDK and JRE 1.4.2_29 and earlier for Windows, Solaris and Linux

Save your JVM and your Application Server From Attack or Contact your Support 😉

Or Please get a Fix from Support  Which Updates the “rt.jar” of the JVM The Fix Details are available in the following Link  http://middlewaremagic.com/weblogic/?p=5393#comment-2821

And

Regarding the “Oracle Security Alert for CVE-2010-4476″You can get the Temp Security Patches related to this issue from the following link:http://www.oracle.com/technetwork/topics/security/alert-cve-2010-4476-305811.html

.
Thanks
Jay SenSharma


Analyzing Garbage Collection Log

Hi,
Jay SenSharma

Jay SenSharma

It’s always best to enable the Garbage collection Logging in our production environment as well because it does not cause any resource overhead or any side effect on weblogic server or an other application server’s performance.  GC log helps us in investigating man issues. Apart from issues it helps us to find out if some tuning is required based on the statistics of the Garbage collection.
.
Garbage collection logging can be enable and collected in a separate log file by using the following JAVA_OPTIONS:
-Xloggc:D:/gcLogs/GCLogs.log         -XX:+PrintGCDetails        -XX:+PrintGCTimeStamps
As soon as you add these JAVA_OPTIONS which are JVM specific (above will work for Sun and Open JDKs fine) the JVM will start generating the garbage collection logging in the GCLog.log file. Now if you will open this file then you can
see something like following:
4.636: [GC [PSYoungGen: 230400K->19135K(268800K)] 230400K->19135K(2058752K), 0.0635710 secs] [Times: user=0.08 sys=0.01, real=0.06 secs]
7.302: [GC [PSYoungGen: 249535K->38396K(268800K)] 249535K->51158K(2058752K), 0.0777300 secs] [Times: user=0.21 sys=0.04, real=0.07 secs]
7.521: [GC [PSYoungGen: 49735K->38388K(268800K)] 62496K->51933K(2058752K), 0.0741680 secs] [Times: user=0.15 sys=0.04, real=0.07 secs]
7.595: [Full GC (System) [PSYoungGen: 38388K->0K(268800K)] [PSOldGen: 13545K->51794K(1789952K)] 51933K->51794K(2058752K) [PSPermGen: 19868K->19868K(39936K)], 0.3066610 secs] [Times: user=0.28 sys=0.02, real=0.31 secs]
9.752: [GC [PSYoungGen: 230400K->26206K(268800K)] 282194K->78000K(2058752K), 0.0728380 secs] [Times: user=0.15 sys=0.00, real=0.08 secs]
11.906: [GC [PSYoungGen: 256606K->38393K(268800K)] 308400K->94759K(2058752K), 0.1058920 secs] [Times: user=0.19 sys=0.00, real=0.10 secs]
13.480: [GC [PSYoungGen: 268793K->38394K(268800K)] 325159K->109054K(2058752K), 0.0762360 secs] [Times: user=0.20 sys=0.03, real=0.08 secs]
18.115: [GC [PSYoungGen: 268794K->38384K(268800K)] 339454K->179238K(2058752K), 0.1351350 secs] [Times: user=0.42 sys=0.10, real=0.14 secs]
20.860: [GC [PSYoungGen: 268784K->38394K(268800K)] 409638K->200343K(2058752K), 0.1063430 secs] [Times: user=0.29 sys=0.03, real=0.11 secs]
22.148: [GC [PSYoungGen: 268794K->38399K(268800K)] 430743K->221395K(2058752K), 0.1173980 secs] [Times: user=0.24 sys=0.02, real=0.12 secs]
23.357: [GC [PSYoungGen: 268799K->26775K(268800K)] 451795K->231618K(2058752K), 0.0714130 secs] [Times: user=0.15 sys=0.03, real=0.08 secs]
24.449: [GC [PSYoungGen: 257175K->29170K(268800K)] 462018K->239909K(2058752K), 0.0312400 secs] [Times: user=0.06 sys=0.01, real=0.04 secs]
You can notice something in the above output:
Point-1). [Full GC (System) [PSYoungGen: 38388K->0K(268800K)]    It means a Full GC is happening on the complete Heap Area including all the Areas of the Java Heap Space.
.
Point-2). [GC [PSYoungGen: 230400K->19135K(268800K)]   Indicates some small GCs which keep on happening in the young generation very frequently,This garbage collection cleans the Young Generation short living Objects.
.
Point-3). Meaning of the [GC [PSYoungGen: 230400K->19135K(268800K)]   line is around 256MB (268800K) is the Young Generation Size, Before Garbage Collection in young generation the heap utilization in Young Generation area was around  255MB (230400K)  and after garbage collection it reduced up to 18MB (19135K)
.
Point-4). Same thing we can see for Full Garbage collection as well….How effective the Garbage collection was…[Full GC (System) [PSYoungGen: 38388K->0K(268800K)] [PSOldGen: 13545K->51794K(1789952K)]  Here it says that around
[(old)1789952K +  young (268800K) ]  memory space means  OldGeneration is consuming 1.75GB space and Young Generation is consuming around 255 MB space  So it means total Heap size is around 2GB.
.
But analyzing the Garbage collection log like above technique Line by Line is very bad…so here we have an alternative was to analyze the Garbage Collection log in few Seconds to see how much time the Full Garbage collection is taking as an average and other reports…etc.
.
Step1). Download the “garbagecat-1.0.0.jar   (881 KB) ”  tool from the follwing link: http://garbagecat.eclipselabs.org.codespot.com/files/garbagecat-1.0.0.jar
.
Step2). Open a command prompt and then make sure that JAVA is set in the Path so that we can use “jar” utility of JDK to run the “garbagecat-1.0.0.jar”  tool.
.
Step3). Put the “garbagecat-1.0.0.jar”  file and the “GCLog.log” file in the same directory. then run the following command:
java      -jar      garbagecat-1.0.0.jar      GCLog.log
.
Step4). As soon as ou run the above command you will see that in your current directory following files are created:
garbagecat-1.0.0.jar
GCLog.log
gcdb.lck
gcdb.log
gcdb.properties
report.txt
.
Step5). Now open the “report.txt” file to see the Over all report of the Garbage Collection something like following:
========================================
SUMMARY:
========================================
# GC Events: 12
GC Event Types: PARALLEL_SCAVENGE, PARALLEL_SERIAL_OLD
Max Heap Space: 2058752K
Max Heap Occupancy: 462018K
Max Perm Space: 39936K
Max Perm Occupancy: 19868K
Throughput: 95%
Max Pause: 306 ms
Total Pause: 1233 ms
First Timestamp: 4636 ms
Last Timestamp: 24449 ms
========================================
.
If you see that the Garbage Collection Max Pause time is very high like  more than 5-7 Seconds for a 2 GB heap then you need to worry about it. 😉
NOTE: Garbagecat  is a best utility to generate the Garbage Collection Report for Sun JDK and Open JDK for other JDKs you should use other tools for accurate results.
.
.
Thanks
Jay SenSharma

Copyright © 2010-2012 Middleware Magic. All rights reserved. |