Hi All,
In response to the comment: http://middlewaremagic.com/weblogic/?p=958&cpage=1#comment-1971
I got an idea to develop a post on this. It is very interesting to start High CPU Analysis on AIX Operating system, because it doesnt have common and useful utilities like “prstat” or “top” unlike Solaris/Linux Operating Systems.
I got an idea to develop a post on this. It is very interesting to start High CPU Analysis on AIX Operating system, because it doesnt have common and useful utilities like “prstat” or “top” unlike Solaris/Linux Operating Systems.
But if we want to analyze the High CPU in AIX operating system then we can use the following utilities:
1) dbx: The dbx utility can be used to examine corefiles as well as executable programs. By examining a core memory image file, you can use the dbx command to examine the state of the program when it faulted.
2) ps: The ps command writes the status of active processes and if the -m flag is given, displays the associated kernel threads to standard output. While the -m flag displays threads associated with processes using extra lines, you must use the -o flag with the THREAD field specifier to display extra thread-related columns.
.
So lets see how to how to proceed with the AIX High CPU analysis.
.
Step1). Open a AIX Shell Prompt and then find out the WebLogic Server Process ID(PID).
.
Step2). Now run the following command to findout the Thread information which is consuming maximum CPU cycles: (Here PID=weblogic Server Process ID)
ps -mp <PID> -o THREAD
Suppose if the WebLogic Servers Process ID is = 23456 then please run the following command:
Example:
ps -mp 23456 -o THREAD
As soon as the above command lists the Thread details Notice the Note down TID value in the List which has high number of CPU. Suppose the TID is ‘11111’ then u note it down becasue u will need to use it in Step 4 for analysis.
.
Step3).Take 4-5 Thread Dumps of WebLogic Server in the interval of 10-12 seconds.
kill -3 <PID>
So far we have collected the raw dat, Now we will start Investigating 🙂
.
Step4). Run the dbx utility with -a with the sub command ‘thread’ to list down all the Threads of WLS. Example suppose if the WebLogic Servers Process ID is = 23456 then please run the following command:
$ dbx -a 23456 (dbx) thread
Somewhere in the output if the above command u will find a line like :
$t15 run running 11111 k no sys JVM_Send
In the above line t15 means the thread number 15 and 11111 is the TID of the thread which we noticed in Step2).
.
Step5). Now in the dbx window again run the following command to query for the thread 15…because Thread 15 information we got in previous command.
(dbx) th info 15
.
Step6). In the output of the above command please find the value of ‘pthread_t’ attribute, which is actually Native Thread Number.
Suppose the ‘pthread_t’ attribute value is ‘3333’ The u need to find the value 3333 in the WebLogic Thread Dump to findout the Culprit Thread which is causing the High CPU.
In the thread dump u will find something like:
Execute Thread: '5' for queue 'default' (TID:0x21df97a8, sys_thread_t:0x3a5bea108, state:R, native ID:0x3333) prio=5 at java.net.SocketOutputStream.socketWrite(Native Method)........
.
Step7). Remember to detach the ‘dbx’ utility with the WebLogic using the ‘detach’ command in the dbx prompt.
.
.
Thanks
Jay SenSharma
November 13th, 2010 on 11:25 am
Hi Jay,
Thats really wonderfull….
Now I think about some more customization on this which it will be suitable for all set of the app..
Thanks for providing me an Idea on this. Look forward from you.
Regards,
Kal.
November 15th, 2010 on 11:28 am
Hi Jay,
In my production environment We are using the Redhat OS but Now We newly buy IBM AIX 6.1 p7 server.We configure the weblgoic server 10.3.3 in IBM AIX 6.1 OS.But We deploy the ADF application in the newly configured weblogic console 10.3.3 in IBM AIX 6.1 OS the below mentioned error has come ” Caused By: oracle.adf.controller.ControllerException: ADFC-10001: cannot instantiate class ‘in.sis.ps.view.backing.common.PopupBean'” how to resolve it? We have to access the same application in IBM AIX 6.1 OS but the same application working in the Redhat OS & HP-UX OS but not working in the IBM-AIX Server.
Regards,
S.vinoth babu
November 15th, 2010 on 11:43 am
Hi Vinoth,
It looks like the problem is raised from your application code only: : ADFC-10001: cannot instantiate class ‘in.sis.ps.view.backing.common.PopupBean’, So please check the “PopupBean.java”. If the same application is working fine in RedHat Linux….But not working fine in AIX. Then better i will recommend you to open a Support ticket with Oracle ADF Support Team because change in OS should not directly impact negatively on the application access.
In the mean time please refer to some similar issues to get some workaround:
http://kr.forums.oracle.com/forums/thread.jspa?threadID=970889
And
http://cn.forums.oracle.com/forums/thread.jspa?threadID=1001624
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 22nd, 2010 on 7:21 pm
Hai Jay,
I saw some forums the ADF application is not working in the IBM-AIX OS version. Because the weblogic generic version 10.3.3 does not have the ADF library files for IBM-AIX OS. We have raised a ticket number in support.oracle website.
The weblogic 10.3.3 is working file in the IBM AIX 6.1 version but the ADF application is not working the BM AIX 6.1 OS.
Just six month before we are in IBM lab in Bangalore at that time we are using the weblogic 10.3.2 version is using in the IBM-AIX OS version 5.1. At that time also we have face the same problem. Now also the same problem is persist in the weblogic 10.3.3 generic version in IBM-AIX OS 6.1.
Regards,
S.Vinoth babu.
June 15th, 2014 on 10:30 pm
hi jaysen sharma,
I want a script to monitor,high cpu utilization,diskspace,process,out of memory issues,please do the needful.
thanks
khan