JVM Tuning

High CPU Analysis in AIX

Hi All,
Jay SenSharma

Jay SenSharma

In response to the comment: http://middlewaremagic.com/weblogic/?p=958&cpage=1#comment-1971
I got an idea to develop a post on this. It is very interesting to start High CPU Analysis on AIX Operating system, because it doesnt have common and useful utilities like “prstat” or “top” unlike Solaris/Linux Operating Systems.
But if we want to analyze the High CPU in AIX operating system then we can use the following utilities:

1) dbx: The dbx utility can be used to examine corefiles as well as executable programs. By examining a core memory image file, you can use the dbx command to examine the state of the program when it faulted.

2) ps: The ps command writes the status of active processes and if the -m flag is given, displays the associated kernel threads to standard output. While the -m flag displays threads associated with processes using extra lines, you must use the -o flag with the THREAD field specifier to display extra thread-related columns.

.
So lets see how to how to proceed with the AIX High CPU analysis.
.

Step1). Open a AIX Shell Prompt and then find out the WebLogic Server Process ID(PID).

.

Step2). Now run the following command to findout the Thread information which is consuming maximum CPU cycles: (Here PID=weblogic Server Process ID)
ps  -mp <PID>  -o  THREAD
Suppose if the WebLogic Servers Process ID is = 23456 then please run the following command:
Example:
ps   -mp  23456  -o  THREAD
As soon as the above command lists the Thread details Notice the Note down TID value in the List which has high number of CPU. Suppose the TID is ‘11111’ then u note it down becasue u will need to use it in Step 4 for analysis.

.

Step3).Take 4-5 Thread Dumps of WebLogic Server in the interval of 10-12 seconds.
kill -3 <PID>
So far we have collected the raw dat, Now we will start Investigating 🙂

.

Step4). Run the dbx utility with -a with the sub command ‘thread’ to list down all the Threads of WLS. Example suppose if the WebLogic Servers Process ID is = 23456 then please run the following command:
$ dbx -a 23456
(dbx) thread
Somewhere in the output if the above command u will find a line like :
$t15 run running 11111 k no sys JVM_Send
In the above line t15 means the thread number 15  and 11111 is the TID of the thread which we noticed in Step2).

.

Step5). Now in the dbx window again run the following command to query for the thread 15…because Thread 15 information we got in previous command.
(dbx) th info 15

.

Step6). In the output of the above command please find the value of  ‘pthread_t’  attribute, which is actually Native Thread Number.
Suppose the ‘pthread_t’ attribute value is ‘3333’ The u need to find the value 3333 in the WebLogic Thread Dump to findout the Culprit Thread which is causing the High CPU.
In the thread dump u will find something like:
Execute Thread: '5' for queue 'default'  (TID:0x21df97a8, sys_thread_t:0x3a5bea108, state:R, native ID:0x3333) prio=5 at java.net.SocketOutputStream.socketWrite(Native Method)........

.

Step7). Remember to detach the ‘dbx’ utility with the WebLogic using the ‘detach’ command in the dbx prompt.
.
.
Thanks
Jay SenSharma

Using Jps.exe to distinguish WLS ProcessIDs

Hi,

Jay SenSharma

Jay SenSharma

Many times as a Weblogic Admin We find difficulty to identify the Process Id of our WebLogic Server …using (ps   -ef   |   grep   java) command Or the Step described in the Step2). of the following Link:  weblogic-heap-dump-genration-using-jmap/ …When There are more than one WebLogic Servers are running on the same Physical Box. So in this case we can use the “jps.exe” utility to easily get the process ID of each WebLogic Server.This utility can be seen inside ANY “<JAVA_HOME>binjps.exe”.

NOTE: jps.exe utility is available from JDK1.5 Onwards By Default.

Step1). Run Two or More than 2 WebLogic Server Instances in the same Physical Box.

Step2). Now open a command prompt and move inside the “<JAVA_HOME>bin” directory …And then run the command:

jps.exe     -v

Uniquely Finding Different WLS instances Process ID

Uniquely Finding Different WLS instances Process ID

In the above outout u can see that  when we run the “jps.exe    -v”  then it shows all the JAVA_OPTIONS of the Server including the WebLogic Property…  -Dweblogic.Name which represents the configured name of the Running WebLogic  Server.  This helps us to easily distinguish the correct process ID for our running Servers.

Not only the Process ID, Rather using the above tool we can see all the System property which has been taken by the WLS server as well as the Heap Settings…etc

Some other Options which can be used with jps.exe utility:

-m
Output the arguments passed to the main method. The output may be null for embedded JVMs.
-l
Output the full package name for the application’s main class or the full path name to the application’s JAR file.
-v
Output the arguments passed to the JVM.
-V
Output the arguments passed to the JVM through the flags file

.
.
Thanks
Jay SenSharma


Collecting Thread Dumps Using Jstack utility

Hi,

Jay SenSharma

Jay SenSharma

As we have seen many options to collect the Thread Dump of a Java Process specially in case of WebLogic. We have many options to collect the Thread Dumps as described in :   taking-thread-dumps/

Here we are going to see another additional option to collect the Thread Dump using “Jstack” utility which comes by default with Sun JDK 1.6 Onwards …It is easily down loadable as well. This utility can be seen inside the “<JAVA_HOME>/bin/jstack.exe”. Which requires the process ID (PID of any running Java process to collect and display the Thread Dumps.

NOTE: The “jstack.exe” which is available as part of JDK1.6 onwards…doesn’t have all the Options available inside it…Like  It doesn’t have

“-m” Option to generate the Thread Dump in Mixed mode. Means print native stack frames in addition to the Java stack.

“-F” Option to generate the Thread Dump Forcibly.

Step1). If you are using Unix Based Operating System then u can find out the Process ID (PID) by running the following Command:

ps -ef | grep java

OR

$JAVA_HOME/bin/jps -v

If you are using Windows then follow the below Steps to find out the Process ID (PID):

a) Press “Ctrl+Alt+Del” 2 buttons together.
b) Press the “Task Manager” Button
c) Click on “Processes” Tab.
d) Now Select the Menu Item “View” — > “Select Columns”
e) Check the “PID (Process Identifier)” Check box as well…
f) Now u can see the Process Id of “java.exe” in the “Processes” Tab.

Finding WLS Process ID in WINDOWS

Finding WLS Process ID in WINDOWS

Step2). Open a Command or Shell prompt and then Move inside the “%JAVA_HOME%/bin” directory and then run the “jstack.exe”  by passing the Process id of your Server Instance…

Syntax:

 jstack   -l   %PID%
Collecting Thread Dump Using JStack

Collecting Thread Dump Using JStack

.
.
Thanks
Jay SenSharma


Copyright © 2010-2012 Middleware Magic. All rights reserved. |