Tag: Crash

Causes and First Aid of JVM Crash Issues?

Hi All,
Jay SenSharma

Jay SenSharma

Java Virtual Machine is a Native engine which allows our Java Applications to run. It performs the code optimization to improve the performance. In correct tuning, Low memory allocation, extensive code optimization, bad garbage collection strategy, API code leaking…etc are some of the reasons which may cause the JVM to crash.
.
Analyzing a JVM Crash is one of the very interesting and little time taking process sometimes it is even little complex to find out the root cause of the JVM Crash. Here in this article we will see some of the common mistakes, first aid solutions/debugging techniques to find out what kind of informations we can get by looking into the Core Dump.

What is Core Dump & Where to Find It?

Code dump is usually a Binary file which gets generated by the Operating System when JVM or any other process crashes. Sometimes it also happens that the JVM will not be able to generate the Crash dump. In Windows Operating Systems it will be generated in the Directore where the “Dr. Watson” tool is installed. In Windows it will be usually:  “C:Documents and SettingsAll UsersApplication DataMicrosoftDr Watson
.
By default in Unix based Operating Systems the Core Dump files are created in the directory where the Java Program/Server was started even sometimes it is generated in the “/tmp” directory of the Operating System. But using the following Java Option we can change it’s the Crash Dump/Heap   Dump generation locations:  -XX:HeapDumpPath=/opt/app/someLocaton/ and  -XX:+HeapDump JVM Options.
.
NOTE: These Flags does not gurantee that always the Heap/Crash dump will be generated at the time of JVM Crash. There are some more reasons behind the Core Dump not gets generated…like Process Limitations or the Less Disk Quota or unavailability of the Free File Descriptors.

Who Generates the Crash/Core Dump?

JVM does not generate the Coe dump. Rather it is the Operating System which generates the Core Dump. Core Dump is a Binary file which may be several hundred Mega Bytes or Giga Bytes in size. The Operating systems just log the exception/error messages and the details of the Threads along with the Native libraries loaded with that java process.
.
Many times a brief Textual Crash file is also generated by the JVM itself sometimes during Crash. Usually the file name is “hs_err_pid<WebLogicPID>.log” in case of Sun JDK. Similarly JRockit JVM also generates a Textual file with name “*.dump” in case of JVM Crash.

Use of -XX:+ShowMessageBoxOnError?

The Thread Dump is also very helpful to analyze the Server Crash. Thread dump tells us what was the status and activities performed by the Threads at the time of crash. It may be possible to get a thread dump before the process exits. HotSpot supports the JAVA_OPTIONS -XX:+ShowMessageBoxOnError
.
The corresponding JRockit JVM Option is -Djrockit.waitonerror.  When the JVM is crashing, it may prompt the user ” Do you want to debug the problem? ” This pauses the process, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.  However, this does not work in all cases (for eg., in case of stack overflow).
.
Along with above there are various options available to get the Thread Dumps as described in : http://middlewaremagic.com/weblogic/?p=823

What May Cause JVM Crash?

Reason-1). Usually Native Code causes the JVM Crash. Native code is a code written in Languages like C/C++, Java Native Interface APIs (JNI).
.
Reason-2). JDBC Drivers specially the Native Drivers.
.
Reason-3). JVM Code Optimization.
.
Reason-4). Less Memory availabity for Native Area of a Java Process.
.
Reason-5). Application Servers Native Performance Pack Libraries.
.
Reason-6). JVMs library itself can cause the Crash.
.
Reason-7). High CPU Utilization by the Threads. As described in : http://middlewaremagic.com/weblogic/?p=4348
.
Reason-8). Presence of Wrong Native Libraries in the PATH or in the “-Djava.library.path
.
Reason-9). Presence of A Different Version of Libraries in “java.library.path” or in “SHLIB” or in “LD_LIBRARY_PATH” variables. Like setting a 64-bit version of Library in a 32-bit version of JVM’s library path or vise-versa.
.
.

Tools To Analyze the Core Dumps?

Core/Crash dump is Operating System specific, So to analyze these Dumps we must use the Tools provided by the same Operating System vendors. Various kind of tools are provided by the Operating Systems to analyze these Core Dumps like:
Tool-1). Dr. Watson Tool  in Windows OS. Windows OS Start (Button)—>Run…—>drwtsn32
.
Tool-2). “pstack” and “pmap” in Solaris Operating System.
.
Tool-3). “procstack” and “procmap” in AIX Operating System.
.
Tool-4). “lsstack” and “pmap” in Linux Operating System.
.
Tool-5). “pflag” if available in HP-UX Operating System.
.
.

What May Help To Avoid JVM Crash?

It totally depends on What Caused the Crash or What Libraries caused the JVM crash to avoid the occurance of the JVM crash for next time. But following things should be taken in consideration while analyzing and avoiding the Crash.
.
Point-1). If the Native Jdbc Driver is causing issues, If our appliation is using the Native JDBC Drivers then Switching from Pure Native Jdbc Driver (Type-2 Drivers) to the Pure Java JDBC Driver (Type-4 Driver) may help.
.
Point-2). If the JVM Libraries are causing the Crash then Upgrading to a Later Version of the JDK. If that Application Server has that new JDK in it;s Supported Configuration list.
.
Point-3). If the JVM Code Optimization is causing the Crash, then Disabling the Code Optimization of JVM by applying the JVM Options.
For JRockit JVM Code Optimization can be disabled using JAVA_OPTION  -Xnoopt
For Sun JDK Code Optimization can be disabled using   JAVA_OPTION  -Xint
.
Point-4). Some times the “Just In Time Compiler” code generation also causes the JVM Crash. In these scenarios In Case of Sun JDKs disabling the JIT Compiler can help. We can disable the JIT Compiler by adding the JVM Option:  “-Djava.compiler=none
.
Point-5). Presence of a different bit version (32 bit or 64-bit libraries) of Library in the “-Djava.library.path
.
Point-6). Disabling the Application Servers Native Performance Packs. In WebLogic The Native IO Can be disabled using “-Dweblogic.NativeIOEnabled=false” JVM Option.
.
.
Thanks
Jay SenSharma

JVM Crash And Native OutOfMemory

Hi,

Jay SenSharma

Jay SenSharma

*******JVM Crash Investigation*********

JVM Core Dump is the most important File to investigate the JVM Crash. By default the Core Dump will be genarated. But Just in case if JVM is not able to generate the Core Dump then there may be the following reasons:

If there is not enough disk space or quota to write the file in your File System.
If JVM is not having to create or write a file in the directory.
if another file exists in the same directory with that is read-only or write-protected.

Unix/Linux-specific: Use the limit or ulimit commands to determine if core dumps are disabled.
Example, on Linux, the command “ulimit -c unlimited” enables core dumps to be written, no matter what their size. Core dump sizes can be restricted if disk space limitations are a concern.

It may be possible to get a thread dump before the process exits.  HotSpot supports the Java_Option -XX:+ShowMessageBoxOnError; the corresponding JRockit option is -Djrockit.waitonerror.  When the JVM is crashing, it may prompt the user :::Do you want to debug the problem?::: This pauses the process, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.  However, this does not work in all cases (for eg., in case of stack overflow).

=========================

*******Crash Because of OutOfMemory:*******

Please apply the Following Flag in your JAVA_OPTIONS in the start Script of your Server: -XX:+HeapDumpOnOutOfMemoryError

Details:
1) -XX:+HeapDumpOnOutOfMemoryError option available in 1.5.0_07 and 1.4.2_12, producing an hprof binary format heap dump (By default the DUMP will be generated in your Systems TEMP directory….In Window machine we can easily findout the TEMP directory by running the command “echo %TEMP%”)

2) Analyse hprof heap dumps using HAT, or jhat or YourKit (has an hprof import option)
hprof heap dumps are platform independent and so you don’t need to analyze the dump on the same system that produced it

3) running with -XX:+HeapDumpOnOutOfMemoryError does not impact performance – it is simply a flag to indicate that a heap dump should be generated when the first thread throws OutOfMemoryError.

4) NOTE: -XX:+HeapDumpOnOutOfMemoryError does not work with -XX:+UseConcMarkSweepGC in 1.5.0_07

=========================
TestCase:

public class TestXss {
public static void main(final String[] args) throws Exception {
// start the given number of threads
for (int i = 1; i <= Integer.parseInt(args[0]); i++) {
System.out.println("Starting Thread " + i);
final Thread t = new Thread("T[" + i + "]") {
public void run() {
try {
while (true) {
Thread.sleep(1000);
}
} catch (Exception e) {
e.printStackTrace();
}
}
};
t.setDaemon(true);
t.start();
Thread.sleep(5);
}
// wait
Thread.sleep(1000000);
}
}

———————————OUTPUT————————————
NOTE:   java -Xmx1496m -Xss1m TestXss 5000
Starting Thread 411
Starting Thread 412
Starting Thread 413
Starting Thread 414
Starting Thread 415
Starting Thread 416
Starting Thread 417
Exception in thread “main” java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at TestXss.main(TestXss.java:18)

————-


Note: if you see OOM in the Native Space  ( unable to create new native thread) Then Please first of all Try to decrease the -Xss size. It’s default value is -Xss512K   ———>Reduce it to ——–>  -Xss256K

Like this you will see that JVm is able to create more Native Threads…But beware that if u will decrease it more than a certain limit …u may start getting “java.lang.StackOverflowError”.

———————————————-

Thread Related Definitions

  • CompletedRequestCount. Number of completed requests in the priority queue.
  • ExecuteThreadIdleCount. Number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives.
  • ExecuteThreadTotalCount. Total number of threads in the pool.
  • HoggingThreadCount. Number of threads that are being hogged by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
  • MinThreadsConstraintsCompleted. Number of requests with min threads constraint picked up out of order for execution immediately since their min threads requirement was not met. This does not include the case where threads are idle during schedule.
  • MinThreadsConstraintsPending. Number of requests that should be executed now to satisfy the min threads requirement.
  • PendingUserRequestCount. Number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
  • QueueLength. Number of pending requests in the priority queue. This is the total of internal system requests and user requests.
  • StandbyThreadCount. Number of threads in the standby pool. Surplus threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed.
  • Throughput. Mean number of requests completed per second

How to rotate .out ( stdout) log file in weblogic 9.2 (solaris/linux)

By default Weblogic wont have options to rotate the *.out file, If you are running in unix or Linux boxes, you can add the following snippet

under /etc/logrotate.conf file append a function to handle *.out files

<Location of logs directory>/*.out

{
copytruncate
rotate
size=5Mb
}

————————————————————-

Thanks

Jay SenSharma


Copyright © 2010-2012 Middleware Magic. All rights reserved. |