Hi All,

Jay SenSharma

High CPU utilization is a very very common thing for an Application Server Administrator. Almost every week or quarter or may be every day we can see some High CPU utilization by the Application Server. It is very important for an Administrator or System Admin or a Application Server Support Person to find out what is causing the High CPU.

.
High CPU may cause System Crash or Server Crash or Slow Responsiveness of Server. So we need to find out which Thread is actually consuming more CPU cycles this will help us to find out whether the Thread is processing some application code ….there may be a code bug or it may be Application Framework bug as well…etc

.
So here we are going to see a very simple demonstration of generating High CPU and then we will see How to Analyze High CPU utilization. For this purpose we are going to use 2 very basic utilities

.
1). jps: to know How to use JPS utility please refer to : http://middlewaremagic.com/weblogic/?p=2291
2). top: This is an Operating System Utility .. Please refer to “man” (Manual of your OS like Linux/Solaris)
3). jstack: to know how to use jstack please refer to : http://middlewaremagic.com/weblogic/?p=2281

Generating High CPU

Step1). Create a Simple Web Application with the following JSP code:
“HighCpuGenerator.jsp”

<%
for (int i=0; i < 3; i++)
    {
       Thread x=new Thread(new Runnable(){

                  public void run()
                    {
                       System.out.println("Thread " +Thread.currentThread().getName() + " started");
                       double val=10;
                       for (;;)
                         {
                            Math.atan(Math.sqrt(Math.pow(val, 10)));
                         }
                     }
              });
 
        x.start();
    }
%>

<h3>High CPU generation ....Please collect the 4-5 times "Top" command Output and the 4-5 ThreadDumps.</h3>

Step2). Now Deploy your Application which is having this JSP.
Step3). Hit the Application using “http://localhost:7001/HighCpuWebApp/HighCpuGenerator.jsp”

Analyzing High CPU:

Step4). Open a shell prompt and then get the Servers Process ID using “jps” utility which comes with Jdk1.5 and Higher inside the bin directory.

[jsenshar@jsenshar ~]$ export PATH=/home/jsenshar/MyJdks/jdk1.6.0_21/bin:$PATH
[jsenshar@jsenshar ~]$ jps -v

HighCpu_Collect5ing Process ID using JPS

Step5). Now In the same command prompt run the “JStack” utility to collect some Thread Dumps in a File… Suppose if the Application Servers Process ID is “5398” which we got in previous step.

HighCpu_ThreadDumps_JStack

Step6). Now In Parallel to collecting the Thread Dumps please collect the “top” commands output as well….like
(NOTE: Make sure that u collect the Thread Dumps and the Top commands results in Parallel whenever u observe any kind of High CPU utilization or Server Slow response)

[jsenshar@jsenshar HighCpu]$ top -H -b -p 5398

HighCPU_Top_Report

You will see some results like this :

top - 02:52:09 up 18:08,  4 users,  load average: 3.49, 3.25, 8.02
Tasks:  51 total,   6 running,  45 sleeping,   0 stopped,   0 zombie
Cpu(s): 96.6%us,  0.7%sy,  0.0%ni,  2.6%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   3848964k total,  3352224k used,   496740k free,    50940k buffers
Swap:  4194296k total,   400804k used,  3793492k free,  1116028k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5522 jsenshar  20   0 2884m 412m  19m R 77.8 11.0  40:00.46 java
 5521 jsenshar  20   0 2884m 412m  19m R 66.9 11.0  39:57.34 java
 6900 jsenshar  20   0 2884m 412m  19m R 65.2 11.0   0:07.82 java
 5520 jsenshar  20   0 2884m 412m  19m R 54.2 11.0  39:57.70 java

Step7). Now U will observe that there are some Thread IDs which are consuming more CPU Cycles like Child Thread PID=5522 is consuming 77.8% of CPU similarly Child Thread PID 5521 is consuming 66.9 % of CPU Cycles. These are very Hign CPU utilization Threads inside the Main Parent Process ID 5398.

Step8). Now convert those PIDs into Hexadecimal values…..Example     the Hex Decimal Value for     5522 will be      1592. So now please open the Thread Dump and then find out the word 1592

Decimal Value of the Thread ID provided by OS

Converted the Decimal Thread ID Value to Hexa Decimal Code

"Thread-17" daemon prio=10 tid=0x00007fe824439800 nid=0x1592 runnable [0x00007fe7fa9e2000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StrictMath.atan(Native Method)
        at java.lang.Math.atan(Math.java:187)
        at jsp_servlet.__highcpugenerator$1.run(__highcpugenerator.java:83)
        at java.lang.Thread.run(Thread.java:619)

Here you will see that The Thread with id=0x1592 is actually causing the High CPU …Now u can tell your Developers to check the Stack Trace of this Thread so that they can correct their Application Code or If you see any API Code or Framework code there then you know what u need to do and where the problem is.

Like in above case u can see that  My JSPs code is actually causing High CPU    “at jsp_servlet.__highcpugenerator$1.run(__highcpugenerator.java:83)”

.

.

Thanks

Jay SenSharma

Hi,

Jay SenSharma

Many times as a Weblogic Admin We find difficulty to identify the Process Id of our WebLogic Server …using (ps   -ef   |   grep   java) command Or the Step described in the Step2). of the following Link:  weblogic-heap-dump-genration-using-jmap/ …When There are more than one WebLogic Servers are running on the same Physical Box. So in this case we can use the “jps.exe” utility to easily get the process ID of each WebLogic Server.This utility can be seen inside ANY “<JAVA_HOME>binjps.exe”.

NOTE: jps.exe utility is available from JDK1.5 Onwards By Default.

Step1). Run Two or More than 2 WebLogic Server Instances in the same Physical Box.

Step2). Now open a command prompt and move inside the “<JAVA_HOME>bin” directory …And then run the command:

jps.exe     -v

Uniquely Finding Different WLS instances Process ID

In the above outout u can see that  when we run the “jps.exe    -v”  then it shows all the JAVA_OPTIONS of the Server including the WebLogic Property…  -Dweblogic.Name which represents the configured name of the Running WebLogic  Server.  This helps us to easily distinguish the correct process ID for our running Servers.

Not only the Process ID, Rather using the above tool we can see all the System property which has been taken by the WLS server as well as the Heap Settings…etc

Some other Options which can be used with jps.exe utility:

-m
Output the arguments passed to the main method. The output may be null for embedded JVMs.
-l
Output the full package name for the application’s main class or the full path name to the application’s JAR file.
-v
Output the arguments passed to the JVM.
-V
Output the arguments passed to the JVM through the flags file

.
.
Thanks
Jay SenSharma

Hi,

Jay SenSharma

As we have seen many options to collect the Thread Dump of a Java Process specially in case of WebLogic. We have many options to collect the Thread Dumps as described in :   taking-thread-dumps/

Here we are going to see another additional option to collect the Thread Dump using “Jstack” utility which comes by default with Sun JDK 1.6 Onwards …It is easily down loadable as well. This utility can be seen inside the “<JAVA_HOME>/bin/jstack.exe”. Which requires the process ID (PID of any running Java process to collect and display the Thread Dumps.

NOTE: The “jstack.exe” which is available as part of JDK1.6 onwards…doesn’t have all the Options available inside it…Like  It doesn’t have

“-m” Option to generate the Thread Dump in Mixed mode. Means print native stack frames in addition to the Java stack.

“-F” Option to generate the Thread Dump Forcibly.

Step1). If you are using Unix Based Operating System then u can find out the Process ID (PID) by running the following Command:

ps -ef | grep java

OR

$JAVA_HOME/bin/jps -v

If you are using Windows then follow the below Steps to find out the Process ID (PID):

a) Press “Ctrl+Alt+Del” 2 buttons together.
b) Press the “Task Manager” Button
c) Click on “Processes” Tab.
d) Now Select the Menu Item “View” — > “Select Columns”
e) Check the “PID (Process Identifier)” Check box as well…
f) Now u can see the Process Id of “java.exe” in the “Processes” Tab.

Finding WLS Process ID in WINDOWS

Step2). Open a Command or Shell prompt and then Move inside the “%JAVA_HOME%/bin” directory and then run the “jstack.exe”  by passing the Process id of your Server Instance…

Syntax:

 jstack   -l   %PID%

Collecting Thread Dump Using JStack

.
.
Thanks
Jay SenSharma