Hi,

Jay SenSharma

Jay SenSharma

*******JVM Crash Investigation*********

JVM Core Dump is the most important File to investigate the JVM Crash. By default the Core Dump will be genarated. But Just in case if JVM is not able to generate the Core Dump then there may be the following reasons:

If there is not enough disk space or quota to write the file in your File System.
If JVM is not having to create or write a file in the directory.
if another file exists in the same directory with that is read-only or write-protected.

Unix/Linux-specific: Use the limit or ulimit commands to determine if core dumps are disabled.
Example, on Linux, the command “ulimit -c unlimited” enables core dumps to be written, no matter what their size. Core dump sizes can be restricted if disk space limitations are a concern.

It may be possible to get a thread dump before the process exits.  HotSpot supports the Java_Option -XX:+ShowMessageBoxOnError; the corresponding JRockit option is -Djrockit.waitonerror.  When the JVM is crashing, it may prompt the user :::Do you want to debug the problem?::: This pauses the process, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.  However, this does not work in all cases (for eg., in case of stack overflow).

=========================

*******Crash Because of OutOfMemory:*******

Please apply the Following Flag in your JAVA_OPTIONS in the start Script of your Server: -XX:+HeapDumpOnOutOfMemoryError

Details:
1) -XX:+HeapDumpOnOutOfMemoryError option available in 1.5.0_07 and 1.4.2_12, producing an hprof binary format heap dump (By default the DUMP will be generated in your Systems TEMP directory….In Window machine we can easily findout the TEMP directory by running the command “echo %TEMP%”)

2) Analyse hprof heap dumps using HAT, or jhat or YourKit (has an hprof import option)
hprof heap dumps are platform independent and so you don’t need to analyze the dump on the same system that produced it

3) running with -XX:+HeapDumpOnOutOfMemoryError does not impact performance – it is simply a flag to indicate that a heap dump should be generated when the first thread throws OutOfMemoryError.

4) NOTE: -XX:+HeapDumpOnOutOfMemoryError does not work with -XX:+UseConcMarkSweepGC in 1.5.0_07

=========================
TestCase:

public class TestXss {
public static void main(final String[] args) throws Exception {
// start the given number of threads
for (int i = 1; i <= Integer.parseInt(args[0]); i++) {
System.out.println("Starting Thread " + i);
final Thread t = new Thread("T[" + i + "]") {
public void run() {
try {
while (true) {
Thread.sleep(1000);
}
} catch (Exception e) {
e.printStackTrace();
}
}
};
t.setDaemon(true);
t.start();
Thread.sleep(5);
}
// wait
Thread.sleep(1000000);
}
}

———————————OUTPUT————————————
NOTE:   java -Xmx1496m -Xss1m TestXss 5000
Starting Thread 411
Starting Thread 412
Starting Thread 413
Starting Thread 414
Starting Thread 415
Starting Thread 416
Starting Thread 417
Exception in thread “main” java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at TestXss.main(TestXss.java:18)

————-


Note: if you see OOM in the Native Space  ( unable to create new native thread) Then Please first of all Try to decrease the -Xss size. It’s default value is -Xss512K   ———>Reduce it to ——–>  -Xss256K

Like this you will see that JVm is able to create more Native Threads…But beware that if u will decrease it more than a certain limit …u may start getting “java.lang.StackOverflowError”.

———————————————-

Thread Related Definitions

  • CompletedRequestCount. Number of completed requests in the priority queue.
  • ExecuteThreadIdleCount. Number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives.
  • ExecuteThreadTotalCount. Total number of threads in the pool.
  • HoggingThreadCount. Number of threads that are being hogged by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
  • MinThreadsConstraintsCompleted. Number of requests with min threads constraint picked up out of order for execution immediately since their min threads requirement was not met. This does not include the case where threads are idle during schedule.
  • MinThreadsConstraintsPending. Number of requests that should be executed now to satisfy the min threads requirement.
  • PendingUserRequestCount. Number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
  • QueueLength. Number of pending requests in the priority queue. This is the total of internal system requests and user requests.
  • StandbyThreadCount. Number of threads in the standby pool. Surplus threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed.
  • Throughput. Mean number of requests completed per second

How to rotate .out ( stdout) log file in weblogic 9.2 (solaris/linux)

By default Weblogic wont have options to rotate the *.out file, If you are running in unix or Linux boxes, you can add the following snippet

under /etc/logrotate.conf file append a function to handle *.out files

<Location of logs directory>/*.out

{
copytruncate
rotate
size=5Mb
}

————————————————————-

Thanks

Jay SenSharma

If you enjoyed this post, please considerleaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.