Buffer
Email
Print
Hi All,
Jay SenSharma

Jay SenSharma

In response to the comment: http://middlewaremagic.com/weblogic/?p=958&cpage=1#comment-1971
I got an idea to develop a post on this. It is very interesting to start High CPU Analysis on AIX Operating system, because it doesnt have common and useful utilities like “prstat” or “top” unlike Solaris/Linux Operating Systems.
But if we want to analyze the High CPU in AIX operating system then we can use the following utilities:

1) dbx: The dbx utility can be used to examine corefiles as well as executable programs. By examining a core memory image file, you can use the dbx command to examine the state of the program when it faulted.

2) ps: The ps command writes the status of active processes and if the -m flag is given, displays the associated kernel threads to standard output. While the -m flag displays threads associated with processes using extra lines, you must use the -o flag with the THREAD field specifier to display extra thread-related columns.

.
So lets see how to how to proceed with the AIX High CPU analysis.
.

Step1). Open a AIX Shell Prompt and then find out the WebLogic Server Process ID(PID).

.

Step2). Now run the following command to findout the Thread information which is consuming maximum CPU cycles: (Here PID=weblogic Server Process ID)
ps  -mp <PID>  -o  THREAD
Suppose if the WebLogic Servers Process ID is = 23456 then please run the following command:
Example:
ps   -mp  23456  -o  THREAD
As soon as the above command lists the Thread details Notice the Note down TID value in the List which has high number of CPU. Suppose the TID is ‘11111’ then u note it down becasue u will need to use it in Step 4 for analysis.

.

Step3).Take 4-5 Thread Dumps of WebLogic Server in the interval of 10-12 seconds.
kill -3 <PID>
So far we have collected the raw dat, Now we will start Investigating 🙂

.

Step4). Run the dbx utility with -a with the sub command ‘thread’ to list down all the Threads of WLS. Example suppose if the WebLogic Servers Process ID is = 23456 then please run the following command:
$ dbx -a 23456
(dbx) thread
Somewhere in the output if the above command u will find a line like :
$t15 run running 11111 k no sys JVM_Send
In the above line t15 means the thread number 15  and 11111 is the TID of the thread which we noticed in Step2).

.

Step5). Now in the dbx window again run the following command to query for the thread 15…because Thread 15 information we got in previous command.
(dbx) th info 15

.

Step6). In the output of the above command please find the value of  ‘pthread_t’  attribute, which is actually Native Thread Number.
Suppose the ‘pthread_t’ attribute value is ‘3333’ The u need to find the value 3333 in the WebLogic Thread Dump to findout the Culprit Thread which is causing the High CPU.
In the thread dump u will find something like:
Execute Thread: '5' for queue 'default'  (TID:0x21df97a8, sys_thread_t:0x3a5bea108, state:R, native ID:0x3333) prio=5 at java.net.SocketOutputStream.socketWrite(Native Method)........

.

Step7). Remember to detach the ‘dbx’ utility with the WebLogic using the ‘detach’ command in the dbx prompt.
.
.
Thanks
Jay SenSharma
Buffer
If you enjoyed this post, please considerleaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.