Tag: JVM

Heap Dump Using Jhat for WebLogic Server Heap Analysis

Hi,
Jay SenSharma

Jay SenSharma

Many times we want to Generate the Heap Dump to see what all objects are created inside the JVMs Heap space. The Heap Dump tells us that what all objects are created inside the JVM and what is the size of those Objects. Based on this analysis we can findout that what all objects (Application Objects/Platform Objects) are consuming more memory inside the heap.

Here is a Basic Step to enable the Jhat tool to generate the Heap Dump of our WebLogic Server and a brief intro of using this Great and wonderful tool.
Jhat tool comes free and available inside the “<JAVA_HOME>bin” directory (from JDK1.6 onwards only). If you are using Jdk 1.5 or Lower version then u need to separately download the Jhat tool.from “https://hat.dev.java.net/
.
.
Step1). Open the “startWebLogic.sh”(Unix Based OS) or “startWebLogic.cmd”(Windows OS)  and then add the JAVA_OPTIONS inside it as following:
@REM **********WINDOWS Operating Systems<(startWebLogic.cmd)******************
@REM
@REM Call setDomainEnv here.
set DOMAIN_HOME=C:bea103user_projectsdomainsWS_Security_Domain
for %%i in ("%DOMAIN_HOME%") do set DOMAIN_HOME=%%~fsi
call "%DOMAIN_HOME%binsetDomainEnv.cmd" %*
set SAVE_JAVA_OPTIONS=%JAVA_OPTIONS% -Xrunhprof:format=b,file=MyHeapDump.hprof
set SAVE_CLASSPATH=%CLASSPATH%
#************Unix Based Operating Systems<(startWebLogic.sh)***********
#
# Call setDomainEnv here.
DOMAIN_HOME="C:/bea103/user_projects/domains/WLST_Domain"
. ${DOMAIN_HOME}/bin/setDomainEnv.sh $*
SAVE_JAVA_OPTIONS="${JAVA_OPTIONS}"   -Xrunhprof:format=b,file=MyHeapDump.hprof
SAVE_CLASSPATH="${CLASSPATH}"
Step2). Now restart Your Server. As soon as you will start your Server you will find that “MyHeapDump.hprof” file is created inside your <DOMAIN_HOME> root directory:
Heap Dump Location

Heap Dump Location

Wait For Some time 4-5 Minutes to get the complete heap Dump

Step3). Open the HeapDump using the Jhat tool like following:
C:bea103jdk160_05bin>jhat -J-mx1024m  C:bea103user_projectsdomainsWS_Security_DomainMyHeapDump.hprof

OR place the file in some other location before opening it…
C:bea103jdk160_05bin>jhat      -J-mx1024m       C:myHeapDumpsMyHeapDump.hprof
.
Step4). As soon as u run the Obove command jhat tool starts a Http Server on default port 7000…Please open a browser with the following URL:
http://localhost:7000
Opening Jhat Console

Opening Jhat Console

jhat Histogram Analysis

jhat Histogram Analysis

Jhat Object Query Language Editor

Jhat Object Query Language Editor

Jhat Non Product Instance Counter

Jhat Non Product Instance Counter

Jhat Platform/Non-Platform Instance Counter

Jhat Platform/Non-Platform Instance Counter

.
.
***** **** *** **____Advanced Jhat Analysis____ ***** **** *** **
.
Step5). Object Query Language (OQL)
OQL is SQL-like query language to query Java heap. OQL allows to filter/select information wanted from Java heap. While pre-defined queries such as “show all instances of class X” are already supported by HAT, OQL adds more flexibility. OQL is based on JavaScript expression language.
OQL query is of the form
select <JavaScript expression to select>
[ from [instanceof] <class name> <identifier>
[ where <JavaScript boolean expression to filter> ] ]
==================================================
(OQL) Object Query Language Querie

(OQL) Object Query Language Querie

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

========================================================

Issues U May Face While Using Jhat for Analysis:

Issue-1). While reading Heap Dump using Jhat u may find following error:
.
Reading from C:bea103user_projectsdomainsTest_DomainMyHeapDump.hprof…
Dump file created Sun Jul 11 20:22:53 IST 2010
java.io.IOException: Unrecognized heap dump sub-record type:  0
at com.sun.tools.hat.internal.parser.HprofReader.readHeapDump(HprofReader.java:489)
at com.sun.tools.hat.internal.parser.HprofReader.read(HprofReader.java:226)
at com.sun.tools.hat.internal.parser.Reader.readFile(Reader.java:79)
at com.sun.tools.hat.Main.main(Main.java:143)
Solution: Most of the time this means that the VM did not write the heap dump properly. At some point in the file, everything is zeroed out, hence the unknown sub-record type 0. There is not much you can do but get another heap dump. Or try to download “Eclipse Memory Analyzer(http://www.eclipse.org/proposals/memory-analyzer/) and just say file—>open—>MyHeapDump.hprof “
.
Issue-2). While Killing your Server Profiler will try to generate the HeapDump (around 30 MB minimum size with WLS Server) you may get the following error in the WLS Server stdout:
.
HPROF ERROR: Missing jclass when fields needed [hprof_class.c:647]
HPROF ERROR: Trouble with unprepared classes [hprof_reference.c:384]
HPROF ERROR: Missing jclass when fields needed [hprof_class.c:647]
HPROF ERROR: Trouble with unprepared classes [hprof_reference.c:384]
allocation sites … done.
<Jul 11, 2010 8:11:06 PM IST> <Error> <> <BEA-000000> <THREAD DUMP from JVM taken at ‘Sun Jul 11 20:07:47 2010’>
Solution: Please collect another Heap Dump or use “Eclipse Memory Analyzer”

—————————-

WORKAROUND

——————————————————-

Use the “Jmap” utility (Which comes with JDK1.6 default or download it separately) to generate the Binary Heap Dump File and then read it using Jhat tool….    http://middlewaremagic.com/weblogic/?p=2241
.
Why “Jmap”?
Jmap is much better than Jhat utility because to start with Jmap utility u need not to apply any JAVA_OPTION on the WebLogic Server … you can directly start with any WebLogic Server which is  Running on Sun JDK …. All u need to do is …Just find the Process ID of that WLS Server…thats all.
.
.
Step1). Find the process ID of your Server.
Step2). Run the following command: (Suppose 5552 is the Process ID of your Server Process)
C:bea103jdk160_05bin>jmap -dump:format=b,file=heapJMap.bin 5552
Dumping heap to C:bea103jdk160_05binheapJMap.bin …
Heap dump file created
Step3). Make sure that while reading the Binary Heap Dump using JHat …u set the Heap Size to 1024 or according to the Heap dump size….
C:bea103jdk160_05bin>jhat -J-Xmx1024m heapJMap.bin
Reading from heapJMap.bin…
Dump file created Mon Jul 12 10:05:58 IST 2010
Snapshot read, resolving…
Chasing references, expect 239 dots………….
…………………………………………
Eliminating duplicate references…………….
…………………………………………
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.
=================================================
.
.
Thanks
Jay SenSharma

Taking Thread Dumps

Hi,

Jay SenSharma

Jay SenSharma

Thread Dumps are one of the very important JVM reports which we can use to analyze Server/JVM Hang Situations and the Root Cause of it… Here are few very common techniques which we can use to take Thread Dumps…

Note: Taking only one or two Thread Dumps is not very useful. Always make sure that you collect atleast 5-6 Thread Dumps taken in the interval of 10-12 Seconds….Ehich will be really helpful to investigate the Hang Situation.

————————————————–

Option 1). In Windows Machines:   Just Press (Ctrl + Break) Buttons together in the CommandWindow ehere your Server in running.

In Solaris/Linux …etc Unix Based Boxes you need to findout the Servers Process ID…by running the following command from the Same box where the Server is running:

A). ps -ef | grep java

qProcessbox 20650 1   0   Mar 26 ?         104:51 /opt/app/qProcessbox/java/jdk1.5.0_22/bin/java

B). Now use     kill -3 20650

Syntax:  kill -3 <ProcessID>

The OutPut of the Thread Dump will be generated in the Server STDOUT.

————————————————–

Option 2). We can Collect Thread Dump Using “weblogic.Admin” which is deprecated …but still available …and may be available in near future as well…As i think because it is one of the best debugging utility for Admins.

java weblogic.Admin -url t3://AdminHostName:7001 -username weblogic -password weblogic THREAD_DUMP

java weblogic.Admin -url t3://MS1HostName:7003 -username weblogic -password weblogic THREAD_DUMP

java weblogic.Admin -url t3://MS2HostName:7005 -username weblogic -password weblogic THREAD_DUMP

The THREAD_DUMPS will be generated in the Servers STDOUT file.  If you want to See the Thread Dump In your Same Command Window Where you are running the “weblogic.Admin” utility…then Enable the “redirect-stdout-to-serverlog” Or by adding the JAVA_OPTION in serverStartScript (-Dweblogic.log.RedirectStdoutToServerLogEnabled=true) flag from Admin Console…sothat the STDOUT entries also will be redirected to the Server Logs…then you can read The Server Logs using the following command remotely/locally.

java weblogic.Admin -url t3://MS2HostName:7005 -username weblogic -password weblogic SERVERLOG
————————————————–

Option 3). Through WLST you need to write the WLST script like “AdminThreadDump.py” In this case you will see the Thread Dumps on the Same Shell Prompt…. not need to look into the Server STDOUT

connect('weblogic','weblogic','t3://localhost:7001')
cd ('Servers')
ls()
cd ('AdminServer')
ls()
threadDump()
cd('..')
cd('ManagedServerOne')
threadDump()

Now open a command Shell and then run the “setWLSEnv.sh”

then run the WLST script like:

java weblogic.WLST AdminThreadDump.py
or
java weblogic.WLST /opt/bea/MyScripts/AdminThreadDump.py
=====================================

Option 4). Simplest option to take Thread  Dump is :

Login to AdminConsole—>Server —> Monitoring —> Threads

Option 5). You can even collect THREAD_DUMPS using JMX code like mentioned in the below link …using “JVMRuntimeMBean”     taking-threaddump-along-with-jrockitruntime-details/

Option 6). Taking ThreadDump Using T3ServicesDef

Option 7). Collecting Thread Dumps Using Jstack utility

Option 8). If you are using WebLogic Server as a Windows Service then you can collect the Thread dump as Described by Former GSS India Oracle Mentor SriDevi :http://middlewaremagic.com/weblogic/?p=823#comment-1011

Option 9). Taking the JVM Details (Heap size and Jdk Details including OS details) including Thread Dump:

java weblogic.Admin -url t3://ServerHostName:7001 -username weblogic -password weblogic GET -pretty -type JVMRuntime

Collecting Thread Dump Using JVMRuntime

Collecting Thread Dump Using JVMRuntime

.

.

Thanks

Jay SenSharma


JVM Crash And Native OutOfMemory

Hi,

Jay SenSharma

Jay SenSharma

*******JVM Crash Investigation*********

JVM Core Dump is the most important File to investigate the JVM Crash. By default the Core Dump will be genarated. But Just in case if JVM is not able to generate the Core Dump then there may be the following reasons:

If there is not enough disk space or quota to write the file in your File System.
If JVM is not having to create or write a file in the directory.
if another file exists in the same directory with that is read-only or write-protected.

Unix/Linux-specific: Use the limit or ulimit commands to determine if core dumps are disabled.
Example, on Linux, the command “ulimit -c unlimited” enables core dumps to be written, no matter what their size. Core dump sizes can be restricted if disk space limitations are a concern.

It may be possible to get a thread dump before the process exits.  HotSpot supports the Java_Option -XX:+ShowMessageBoxOnError; the corresponding JRockit option is -Djrockit.waitonerror.  When the JVM is crashing, it may prompt the user :::Do you want to debug the problem?::: This pauses the process, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.  However, this does not work in all cases (for eg., in case of stack overflow).

=========================

*******Crash Because of OutOfMemory:*******

Please apply the Following Flag in your JAVA_OPTIONS in the start Script of your Server: -XX:+HeapDumpOnOutOfMemoryError

Details:
1) -XX:+HeapDumpOnOutOfMemoryError option available in 1.5.0_07 and 1.4.2_12, producing an hprof binary format heap dump (By default the DUMP will be generated in your Systems TEMP directory….In Window machine we can easily findout the TEMP directory by running the command “echo %TEMP%”)

2) Analyse hprof heap dumps using HAT, or jhat or YourKit (has an hprof import option)
hprof heap dumps are platform independent and so you don’t need to analyze the dump on the same system that produced it

3) running with -XX:+HeapDumpOnOutOfMemoryError does not impact performance – it is simply a flag to indicate that a heap dump should be generated when the first thread throws OutOfMemoryError.

4) NOTE: -XX:+HeapDumpOnOutOfMemoryError does not work with -XX:+UseConcMarkSweepGC in 1.5.0_07

=========================
TestCase:

public class TestXss {
public static void main(final String[] args) throws Exception {
// start the given number of threads
for (int i = 1; i <= Integer.parseInt(args[0]); i++) {
System.out.println("Starting Thread " + i);
final Thread t = new Thread("T[" + i + "]") {
public void run() {
try {
while (true) {
Thread.sleep(1000);
}
} catch (Exception e) {
e.printStackTrace();
}
}
};
t.setDaemon(true);
t.start();
Thread.sleep(5);
}
// wait
Thread.sleep(1000000);
}
}

———————————OUTPUT————————————
NOTE:   java -Xmx1496m -Xss1m TestXss 5000
Starting Thread 411
Starting Thread 412
Starting Thread 413
Starting Thread 414
Starting Thread 415
Starting Thread 416
Starting Thread 417
Exception in thread “main” java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at TestXss.main(TestXss.java:18)

————-


Note: if you see OOM in the Native Space  ( unable to create new native thread) Then Please first of all Try to decrease the -Xss size. It’s default value is -Xss512K   ———>Reduce it to ——–>  -Xss256K

Like this you will see that JVm is able to create more Native Threads…But beware that if u will decrease it more than a certain limit …u may start getting “java.lang.StackOverflowError”.

———————————————-

Thread Related Definitions

  • CompletedRequestCount. Number of completed requests in the priority queue.
  • ExecuteThreadIdleCount. Number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives.
  • ExecuteThreadTotalCount. Total number of threads in the pool.
  • HoggingThreadCount. Number of threads that are being hogged by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
  • MinThreadsConstraintsCompleted. Number of requests with min threads constraint picked up out of order for execution immediately since their min threads requirement was not met. This does not include the case where threads are idle during schedule.
  • MinThreadsConstraintsPending. Number of requests that should be executed now to satisfy the min threads requirement.
  • PendingUserRequestCount. Number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
  • QueueLength. Number of pending requests in the priority queue. This is the total of internal system requests and user requests.
  • StandbyThreadCount. Number of threads in the standby pool. Surplus threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed.
  • Throughput. Mean number of requests completed per second

How to rotate .out ( stdout) log file in weblogic 9.2 (solaris/linux)

By default Weblogic wont have options to rotate the *.out file, If you are running in unix or Linux boxes, you can add the following snippet

under /etc/logrotate.conf file append a function to handle *.out files

<Location of logs directory>/*.out

{
copytruncate
rotate
size=5Mb
}

————————————————————-

Thanks

Jay SenSharma


Copyright © 2010-2012 Middleware Magic. All rights reserved. |