Hi,
Thread Dumps are one of the very important JVM reports which we can use to analyze Server/JVM Hang Situations and the Root Cause of it… Here are few very common techniques which we can use to take Thread Dumps…
Note: Taking only one or two Thread Dumps is not very useful. Always make sure that you collect atleast 5-6 Thread Dumps taken in the interval of 10-12 Seconds….Ehich will be really helpful to investigate the Hang Situation.
————————————————–
Option 1). In Windows Machines: Just Press (Ctrl + Break) Buttons together in the CommandWindow ehere your Server in running.
In Solaris/Linux …etc Unix Based Boxes you need to findout the Servers Process ID…by running the following command from the Same box where the Server is running:
A). ps -ef | grep java
qProcessbox 20650 1 0 Mar 26 ? 104:51 /opt/app/qProcessbox/java/jdk1.5.0_22/bin/java
B). Now use kill -3 20650
Syntax: kill -3 <ProcessID>
The OutPut of the Thread Dump will be generated in the Server STDOUT.
————————————————–
Option 2). We can Collect Thread Dump Using “weblogic.Admin” which is deprecated …but still available …and may be available in near future as well…As i think because it is one of the best debugging utility for Admins.
java weblogic.Admin -url t3://AdminHostName:7001 -username weblogic -password weblogic THREAD_DUMP java weblogic.Admin -url t3://MS1HostName:7003 -username weblogic -password weblogic THREAD_DUMP java weblogic.Admin -url t3://MS2HostName:7005 -username weblogic -password weblogic THREAD_DUMP
The THREAD_DUMPS will be generated in the Servers STDOUT file. If you want to See the Thread Dump In your Same Command Window Where you are running the “weblogic.Admin” utility…then Enable the “redirect-stdout-to-serverlog” Or by adding the JAVA_OPTION in serverStartScript (-Dweblogic.log.RedirectStdoutToServerLogEnabled=true
) flag from Admin Console…sothat the STDOUT entries also will be redirected to the Server Logs…then you can read The Server Logs using the following command remotely/locally.
java weblogic.Admin -url t3://MS2HostName:7005 -username weblogic -password weblogic SERVERLOG
————————————————–
Option 3). Through WLST you need to write the WLST script like “AdminThreadDump.py” In this case you will see the Thread Dumps on the Same Shell Prompt…. not need to look into the Server STDOUT
connect('weblogic','weblogic','t3://localhost:7001') cd ('Servers') ls() cd ('AdminServer') ls() threadDump() cd('..') cd('ManagedServerOne') threadDump()
Now open a command Shell and then run the “setWLSEnv.sh”
then run the WLST script like:
java weblogic.WLST AdminThreadDump.py
or
java weblogic.WLST /opt/bea/MyScripts/AdminThreadDump.py
=====================================
Option 4). Simplest option to take Thread Dump is :
Login to AdminConsole—>Server —> Monitoring —> Threads
Option 5). You can even collect THREAD_DUMPS using JMX code like mentioned in the below link …using “JVMRuntimeMBean” taking-threaddump-along-with-jrockitruntime-details/
Option 6). Taking ThreadDump Using T3ServicesDef
Option 7). Collecting Thread Dumps Using Jstack utility
Option 8). If you are using WebLogic Server as a Windows Service then you can collect the Thread dump as Described by Former GSS India Oracle Mentor SriDevi :http://middlewaremagic.com/weblogic/?p=823#comment-1011
Option 9). Taking the JVM Details (Heap size and Jdk Details including OS details) including Thread Dump:
java weblogic.Admin -url t3://ServerHostName:7001 -username weblogic -password weblogic GET -pretty -type JVMRuntime
.
.
Thanks
Jay SenSharma
April 1st, 2010 on 1:05 pm
One more way – if you’re using jrockit on windows (not sure about other os’s), you can use the “JRCMD” utility and do a “JRCMD print_threads”
eg:
jrcmd 1284 print_threads will print thread info for process 1284.
jrcmd 0 print_threads will print thread info for all java processes on the machine.
The output from this can be re-directed to an output file if needed using normal windows syntax and this method is useful because weblogic.Admin needs at least one thread to execute and if you have the situation where there are no threads available jrcmd can help.
Pete
April 1st, 2010 on 1:52 pm
Hi Pete,
Thanks a Lot… I have just now tried it and it worked for me. That’s a very good option.
Keep Posting 🙂
Thanks
Jay SenSharma
April 22nd, 2010 on 11:05 pm
Hi Jay,
threadDump() is not working with weblogic version 8.1 SP5.
I had downloaded the wlst.jar and jython.jar from the below.
http://www.4shared.com/file/116721496/8c3533ea/WLST_WLS_81.html
Thanks
Surya.
April 27th, 2010 on 10:08 pm
Hi Surya,
Sorry for Late response…. WLS81 SP5 doesnt support all the Online Commands. Even WLs81SP6 as well doesnt support all the Online commands. Full support for all WLST Online commands are available from WLS9 onwards.
.
.
Thanks
Jay SenSharma
April 27th, 2010 on 11:16 pm
Thanks Jay for the clarification.
July 13th, 2010 on 10:29 pm
If we are using Weblogic server as a windows service, then we can use as follows:
E:bea921weblogic92serverbinbeasvc.exe -dump -svcname:”WLS 921 domainname servername service”
Thanks
SriDevi ASNL
July 13th, 2010 on 10:40 pm
Thank you very Much Sreedevi…For making us aware of this option.
I added it as Option-8) for Taking Thread Dumps, in the Above Post.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
July 14th, 2010 on 4:08 am
Hi Jay,
In older versions of Linux, kill -3 does not work. In such cases kill -QUIT would be the best option to take thread dumps.
In fact, WLS 8.1 has an issue with kill -3 and the best part is kill -QUIT is compatible with any version of Linux
Regards,
Sadha
July 14th, 2010 on 9:05 am
Hi Sadha,
Thanks Sadha. U shared a best information with us. Really Great. You were always a Best Oracle WebLogic Support Engineer, Thanks for Sharing.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
July 14th, 2010 on 4:11 am
If we are using the option 4, we should be aware that one of the Execute threads would be used for collecting thread dumps.
Regards,
Sadha
August 2nd, 2010 on 6:58 pm
Hi Jay,
We can generate thread dump in windows os using SendSignal.exe I tried to use the same, I am not sure where it is generating thread dumps.
Syntax: SendSignal
=========================================
C:HeapDumps>SendSignal.exe 5976
Sending signal to process 5976…
C:HeapDumps>
=========================================
Please do analyse on this and let me know. Here are some useful links.
1)http://www.latenighthacking.com/projects/2003/sendSignal/
2) http://www-01.ibm.com/support/docview.wss?rs=2359&context=SS6JMN&dc=DB520&uid=swg21259615&loc=en_US&cs=UTF-8&lang=en&rss=ct2359websphere
Note: We can also monitor thread dumps, heap using jconsole. Please review this and share here.
Thanks in Advance
Rajendra
August 2nd, 2010 on 7:02 pm
Hi Jay,
As you mentioned in the above option2, can you please provide some details, like where it is exaclty generating threads. Please explaing elaborately. Thanks.
Rajendra
August 2nd, 2010 on 10:58 pm
Hi Rajendra,
When we issue the following command :
ava weblogic.Admin -url t3://AdminHostName:7001 -username weblogic -password weblogic THREAD_DUMP
Then the Thread dump gets generated in the Server Stdout. Stdout may be your command prompt or nohup file(in Unix based os). If you want to collect the Thread dumps in a separate file then u can use the following JAVA_OPTION in your WebLogic Server start script to redirect all the stdout data in a file:
-Dweblogic.Stdout=”stdout-filename”
-Dweblogic.Stderr=”stderr-filename”
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
October 14th, 2010 on 1:26 pm
Hi Jay,
Can you please a post on Core Dump.
How to produce a Core dump and how to analyze a core dump.
Thanks
Sathya Prakash
October 14th, 2010 on 2:31 pm
Hi SatyaPrakash,
Thank you for visiting Wonders. We are planning to upload a content on Core Dump analysis…but it may take some more time and more documentation as reproducing a Crash is little complex and tricky. Even different Operating Systems have different crash formats in the core dump and the analysis tools is also different for those Operating Systems…So still finding a way to make a generalized post on it.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 22nd, 2010 on 10:24 pm
Hi,
I am using weblogic.Admin utility to take thread dumps but I see “Thread Dump has been printed to the servers standard out (by default, the shell in which the server is running).” message being printed. I am not able to figure out where the dumps are getting printed. Where is this server standard out file? I tried redirecting to some txt file but the same message is also printed in the txt file.
Environment: BEA WebLogic Server 8.1 SP4
Please reply.
Thanks in advance.
Regards,
Sanjay
November 22nd, 2010 on 10:45 pm
Hi Sanjay,
If you want to collect the ThreadDump in the same command/Shell prompt where you are running the weblogic.Admin Utility then please use the following command:
java weblogic.Admin -url t3://ServerHostName:7001 -username weblogic -password weblogic GET -pretty -type JVMRuntime
For more details please refer to the Option9) in the following Post: http://middlewaremagic.com/weblogic/?p=823
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 23rd, 2010 on 4:18 pm
Thanks Jay
After running the following command, I am still not getting Thread Stack Dump describing various WebLogic Execute Queues and other important information.
See below output.
/opt/bea/wls/8.1sp4/weblogic81/server/bin>java -cp /opt/bea/wls/8.1sp4/weblogic81/server/lib/weblogic.jar weblogic.Admin
-url t3://serverhostname:7001 -username weblogic -password weblogic GET -pretty -type JVMRuntime
—————————
MBeanName: “domain1:Location=admin_domain1,Name=admin_domain1,ServerRuntime=admin_domain1,Type=JVMRuntime”
CachingDisabled: true
HeapFreeCurrent: 265869208
HeapSizeCurrent: 531628032
JavaVendor: Sun Microsystems Inc.
JavaVersion: 1.4.2_05
Name: admin_domain1
OSName: SunOS
OSVersion: 5.9
ObjectName: admin_domain1
Parent: admin_domain1
Registered: false
Type: JVMRuntime
Please help.
Thanks.
November 23rd, 2010 on 5:24 pm
Hi Sanjay,
The “weblogic.Admin” utility in changed a bit from WLS9.x onwards so in WLS8.1 u may not be getting the THREAD DUMP along with the JVM Details. As i can see that you are using WLS8.1 so it will be good if you will run the previous command…
java weblogic.Admin -url t3://ServerHostName:7001 -username weblogic -password weblogic THREAD_DUMP
Now as soon as you will run the above command ..the Utility will instruct the JVM in which the WLS is running….to Generate the THREAD DUMP in the Server’s STDOUT…(Standard Output…by default Standardx output is the command prompt or the Shell window where you are running the WebLogic Server).
If you thing that the default STDOUT is changed for your environment then in that case you can apply the following JVM OPTION in the server start script… -Dweblogic.log.RedirectStdoutToServerLogEnabled=true or you can even enable it from AdminConsole logging page for the respective Server. Sothat the STDOUT entries also will be redirected to the Server Logs…then you can read The Server Logs using the following command remotely/locally.
java weblogic.Admin -url t3:// t3://ServerHostName:7001 -username weblogic -password weblogic SERVERLOG
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
java weblogic.Admin -url t3://MS2HostName:7005 -username weblogic -password weblogic SERVERLOG
November 23rd, 2010 on 7:08 pm
Many Thanks Jay,
It worked out well as you said. The problem was my servers were running on Solaris as background process using “nohup startWeblogic.sh 1>/dev/null 2>/dev/null &”
So, I had to stop my server and start the server again without background so the server log was printing on the shell where the server was running. then I opened antoher telnet window and fired the Thread Dump command and Thread dump was displayed along with the server log. This was done for Admin Server.
1. My question is what if I have to take Thread Dumps of the Running servers provided the fact that the servers are in background mode as I mentioned above.
2. Also, we use nodemanager to start/stops Managed Servers so how to take Thread dumps for such runnign managed servers which are controlled by Node Manager.
Please guide.
Thanks,
Sanjay
November 23rd, 2010 on 10:14 pm
Hi Sanjay,
Thread dump generation has nothing to do with how we are starting the WebLogic Server (With NodeManager or Without NodeManager) And Only the JVM is responsible for generating the Thread Dumps. As soon as it gets a Operating System Signal to generate the Thread dump it generates it.
There are many options available for generating Thread dumps …I would request you to please go through the following link once again….: Taking Thread Dumps …I am very sure that out of 9 different ways mentioned in the above link…some of them will be suitable for you.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 25th, 2010 on 11:58 am
Hi Sanjay,
are u able to collect the THREAD DUMP of your NodeManager based Managed Server? Are u facing any issue in collecting the Thread Dumps?
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
December 16th, 2010 on 10:37 pm
Hi Jay,
Can we something for analysing Thread Dumps;
Please let me know if the Post already exist..
Thanks
Sathya
December 17th, 2010 on 12:15 am
Hi Sathya,
The “Debuggers” page of Middleware Magic has some stuff which related to Thread Dump analysis. Analyzing Thread Dumps is a very huge area…and All the Support is based on that only… But soon we are going to write some articles with Hands On demos on thread dumps so that Magic Users can even reproduce those things to analyze the thread dumps. Soon you will see more on Threads.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
December 27th, 2010 on 7:24 pm
Hello jay!
hope u good !
can u be more instructive about thread dumps topic,viz following
1.when should we take thred dumps,what is the the best approache for it,how to analyse it if u have any eaampls that u can share wouod be great ,whic tool we use for it eect…
December 27th, 2010 on 8:45 pm
Hi Ritu,
Thread Dumps tells us the State of all the Threads of a Running JVM process. It helps us to see what all threads have been doing. We must collect the Thread dumps whenever we see a very slow response from the Server or When we see a High CPU Utilization by the JVM process in which our Server or any Java Process is running.
Please refer to the following Post to see what actually we look inside the Thread dump in case of High CPU:
1). Thread Dump Analysis on Linux/Solaris High CPU: http://middlewaremagic.com/weblogic/?p=4884
2). Thread Dump Analysis on AIX for High CPU: http://middlewaremagic.com/weblogic/?p=4348
Thread dump analysis is a very vast topic …so i developed some part of it in the above posts, it may give you some idea.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
March 4th, 2011 on 3:46 pm
Hi Jay,
WebLogic Server uses software modules called muxers to read incoming requests on the server and incoming responses on the client.
These muxers are of two classified into Two types.
1.Java muxer
2.Native muxer.
Native provide superior scalability because they implement a non-blocking thread model.
When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests.
BEA recommends using the default setting of selected for the Enable Native IO parameter which allows the server automatically selects the appropriate muxer for the server to use.
Most application servers use thread pools to manage execution tasks of a certain type. A thread pool is merely a collection of threads set aside for a specific task. In the example thread above, I’ve shown you a thread from the WebLogic thread pool (or queue) named “weblogic.socket.Muxer”. A pool of these “Muxer threads” is set aside by WebLogic to manage reading and writing data for network connections coming into WebLogic.
Thread dump log:-
“ExecuteThread: ‘1’ for queue: ‘weblogic.socket.Muxer'” id=28 idx=0x68 tid=24640 prio=5 alive, in native, blocked, daemon — Blocked trying to get lock: java/lang/String@0x15a00898[fat lock] at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method) at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1679)[optimized] at jrockit/vm/Locks.lockFat(Locks.java:1780)[optimized] at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized] at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized] at jrockit/vm/Locks.monitorEnter(Locks.java:2466)[optimized] at weblogic/socket/EPollSocketMuxer.processSockets(EPollSocketMuxer.java:153) at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145) at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117) at jrockit/vm/RNI.c2java(IIIII)V(Native Method
In Production Environment We must enable Native I/O. The default value is 33.I am also using the Same value but frequently i get “”ExecuteThread: ‘1’ for queue: ‘weblogic.socket.Muxer'” id=28 idx=0x68 tid=24640 prio=5 alive, in native, blocked, daemon — Blocked trying to get lock: ” .
In oracle document the default value is based on your enviroment. What calculation we have to change the Socket Readers Default value.
Regards,
S.Vinoth Babu
March 26th, 2011 on 9:48 pm
Hi VinothBabu,
Most of the time the tuning data we get after performance testing and then we refine the runing again after performance tuning. There are no standard formula to calculate these things. Thats the only reason we don’t find any Formulas for performance tuning on Google or internet … even the Product Vendors like Oracle / IBM/ redHat also do not provide any formula to calculate the tuning parameter values.
The only thing you can do as an Administrator to arrange more and more performance testing.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
April 23rd, 2011 on 8:30 am
I am using weblogic 10.3.4 and when I have the console window open in windows or linux I am able to see the thread dump but if I want to redirect it to AdminServer.log file I put in the server start tab of AdminServer the following parameter and restart the server
Dweblogic.log.RedirectStdoutToServerLogEnabled=true
Then when I take the thread dump using weblogic.Admin I do no see it my AdminServer log Am I looking at the wrong file the only other log file is the domain log file and nodemanager log file.
April 23rd, 2011 on 9:36 am
Hi testab,
The JVM argument changes through the Admin Console is applicable only if you are starting your Respective Servers with the help of NodeManager. Like following:
http://middlewaremagic.com/weblogic/?p=780
-Dweblogic.log.RedirectStdoutToServerLogEnabled=true JAVA_OPTIONS is responsible to redirect the STDOUT informations of the JVM to the respective server log. For example:
1). Add the -Dweblogic.log.RedirectStdoutToServerLogEnabled=true in your server JAVA_OPTIONS then Collect the Thread Dump like “kill -9 $PID”
2). Check the Respective server log whose PID you have passed above. You should be able to see the Thread Dump….
But the more better option to collect the Thread dumps are available which will allow you to see the Thread dump in the same Command Window / Shell prompt where you want it to be collected:
http://middlewaremagic.com/weblogic/?p=823
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
May 5th, 2011 on 6:50 pm
Hi all, I hope you can help me on this,
I’d like to execute some WLST commands and scripts on a server (linux) without any Weblogic installed (our Weblogic platform are 100% windows).
Looking on the web, I create a wlfullclient.jar following the following URL (http://download.oracle.com/docs/cd/E11035_01/wls100/client/jarbuilder.html)
But when I execute java weblogic.WLST, I only have the message below:
Initializing WebLogic Scripting Tool (WLST) …
Problem invoking WLST – java.lang.RuntimeException: error in finding weblogic.Home
looking on the net, some people say ‘weblogic.home’ is hardcoded in the jar file.
Have you ever been able to run WLST from a machine not having Weblogic?
Thanks for your answer.
May 5th, 2011 on 8:25 pm
Hi fipnova,
Yes, It looks like a BUG with “wlfullclient.jar” Please report it to Oracle WebLogic Server Support.
I can see that the same JAR “wlfullclient.jar” works fine for most of the clients applications like any WebService/JMS/EJB…etc clients but it fails if we try to use it with the WLST.
WORKAROUND:
You can use the following CLASSPATH setting to AVOID installing the WebLogic in a Clients machine just to connect to WebLogic using WLST….
Just copy and paste the following 3 JARs inside the Clients Machine and then set them in the CLASSPATH …
1). C:bea103wlserver_10.3serverlibweblogic.jar
2). C:bea103modulescom.bea.cie.config_5.2.0.0.jar;
3). C:bea103modulesfeaturesweblogic.server.modules_10.3.0.0.jar
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
May 5th, 2011 on 8:52 pm
Thanks but I have another issue
prism@prismuat01 ~/uat/tools $ ls -l *jar
-rw-r–r– 1 prism gprime 3750330 Sep 20 2010 com.bea.cie.config_6.1.0.0.jar
-rw-r–r– 1 prism gprime 35657906 Feb 22 13:57 weblogic.jar
-rw-r–r– 1 prism gprime 2519 Feb 22 13:57 weblogic.server.modules_10.3.4.0.jar
prism@prismuat01 ~/uat/tools $ export CLASSPATH=/home/prism/uat/tools/weblogic.jar:/home/prism/uat/tools/com.bea.cie.config_6.1.0.0.jar:/home/prism/uat/tools/weblogic.server.modules_10.3.4.0.jar
prism@prismuat01 ~/uat/tools $ java -version
java version “1.6.0_12”
Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)
prism@prismuat01 ~/uat/tools $ java weblogic.WLST
Exception in thread “main” java.lang.NoClassDefFoundError: weblogic/descriptor/DescriptorClassLoader
at weblogic.WLST.main(WLST.java:19)
Caused by: java.lang.ClassNotFoundException: weblogic.descriptor.DescriptorClassLoader
I’ll work on my Windows server meanwhile…
May 5th, 2011 on 9:10 pm
Hi fipnova51,
Please refer to the following link:
http://middlewaremagic.com/weblogic/?page_id=241#comment-3621
Above link will help you to findout in which WebLogic Jar you have the missing class “weblogic.descriptor.DescriptorClassLoader” Once you find the Jar add it in the CLASSPATH as well.
In WebLogic 10.3.0 You could find it inside
“Library Path: C:bea103modulescom.bea.core.descriptor_1.4.0.0.jar”
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
June 24th, 2011 on 3:48 am
Hi Team,
I tried to take thread dump using the above methods suggested.
1) Took from console, clicked managed server>monitoring> thread ..dump thread stack.It worked.
But sometimes it hangs the console.
2) Tried kill -3 PID ,didnt work
3)Tried kill – QUIT PID ,didnt work.
3)Tried java weblogic.Admin command ,didnt work.
Says:
exception in therad ‘main’ java.lang.NoClassDefFoundError: weblogic/Admin
Can you please help,why kill -3 PID is not working.
Details:
Weblogic server 9.2
SUN os 5.10
Managed server has “Redirect stout logging enabled”
NO node manager. Every managed server has one log file only.
August 1st, 2011 on 7:43 am
Hello Middleware Admins,
Can you please guide me how and what is threading concept and how it works in Weblogic. Also back end scenario as to how weblogic works.. Like what exactly does it call behind the scenes and what are modules or pieces touched behind..Please guide me in this..
Regards
Tom.
August 1st, 2011 on 2:34 pm
Hi Tom,
WebLogic Threading Model is a very vast topic and it is very difficult to explain the same in Comment, But there is a very good link i have found which explains the types and functionalities of different WebLogic Threads :
WebLogic Threading Model https://sites.google.com/site/sureshsvnsite/weblogic-intro
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
August 2nd, 2011 on 7:15 am
Thank you for sharing the link Jay. Will go through it for now and get back to you.
Regards
Tom
September 15th, 2011 on 11:16 am
Hi MiddleWare Admins,
I just tried 2 ways to collect thread-dumps –
**First**
$ kill -3
And you said that the OutPut of the Thread Dump will be generated in the Server STDOUT , where’s this location ?
I’m not able to understand / or get this location.
**Second**
$ kill -3 > thread.txt
(here i want my output to get into a seperate text file so i’m redirecting to thread.txt
I can see thread.txt file , but empty file , no data.
Can someone help me here..
Thanks
September 15th, 2011 on 11:37 am
Hi Kiran,
When we use kill -3 $PID then the thread dump is getting generated in the same SHELL / Command prompt in which you are running the “startWebLogic.sh”/”startWebLogic.cmd” command.
If you want to see the Generated Thread dump inside the Server log then please add the following JAVA_OPTION in your server start script “-Dweblogic.log.RedirectStdoutToServerLogEnabled=true” so that the Generated thread dumps will start appearing in the server log from next time on wards.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
September 15th, 2011 on 1:47 pm
Thanks Jay.
Option-1 working fine! ( kill -3 $pid )
Can you show me how to write the output to some file ?
I tried like –
kill -3 $pid > testfile.txt
And this testfile.txt gets created under same path , but no data. so output was not moving to this file.
Thanks.
September 15th, 2011 on 4:07 pm
HI Kiran,
“Kill -3 $PID” simply sends a signal to the JVM process to generate the Thread dump ….so after getting the signal the JVM generates the Thread dump in it’s own STDOUT (where the JVM process is running)….
“Kill -3” will not generate any kind of output or thread dump in the shell prompt where it is running so redirecting the output of “Kill -3” to a file is Useless.
.
.
Thanks
Jay SenSharma
September 15th, 2011 on 4:59 pm
That’s good information for me.
Now i understood what exactly happening over here related to thread-dump generation.
Thanks Jay , everything makes clear now!
Thanks
Kiran
September 15th, 2011 on 8:21 pm
Hi Jay
i am new to this blog…. your blog is good and it is providing good knowledge for WL admins.
i have doubt regarding stuck thread..
how we will find process id of particular stuck thread
did process id also logged in log file.
we need to know process id to take thread dump to anlyse stuck thread..
pleade provide me stuck thread detection and resolution
Thanks,
Dino
September 16th, 2011 on 11:44 pm
Hi Dino,
Whenever a thread is marked as STUCK thread it’s clearly visible in the Thread dump. Like following:
“[STUCK] ExecuteThread: ’25’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=87495 idx=0x274 tid=15308 prio=1 alive, in native, blocked, daemon
So to find out which thread is stuck and what that stuck thread was doing you simply need to collect the thread dumps. If you want to findout which thread is consuming High CPU then please refer to the following article:
http://middlewaremagic.com/weblogic/?p=4884
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
September 16th, 2011 on 3:55 pm
Hi Jay,
I’m using “JRockit as JVM in my project”.
Is that possible to get jrockit-dump generated under log directory? I mean something where we can setup entries in JAVA_OPTIONS.
Any possibility? , Reason for asking.. something app fails due to JVM crash…, so i just want dump to get generated.
thanks
September 16th, 2011 on 6:37 pm
Hi Kiran,
JRockit does not generate Heap Dumps link Sun JDK, so you will need to refer to the following link to know how you can use Mission Control for JRockit Monitoring:
http://middlewaremagic.com/weblogic/?p=6930
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 24th, 2011 on 8:56 am
Hi,
when I run the below Query in command line for getting ThreadDump in webloig(java weblogic.Admin -url(servername) -username (XYZ) -password(XYZ) THREAD_DUMP) I got the output.
Thread Dump has been printed to the servers standard out.
where can we find STDOUT file?
how can we enable the STDOUT file in CommandLine?
November 24th, 2011 on 12:40 pm
Hi Dhilip,
Please refer to the following link: http://middlewaremagic.com/weblogic/?p=422#comment-692
The STDOUT is the shell prompt output where you run the startWebLogic.sh command. If you are running your WebLogic in nohup mode then you will find the thread dumps generated inside the nohup file.
If you want to see the generated thread dumps inside your server log then you need to add the following JAVA_OPTIONS in your servers startup script: -Dweblogic.log.RedirectStdoutToServerLogEnabled=true
This flag will instruct JVM to redirect its STDOUT inputs to the Server Logs….Now you can see the Thread Dumps in your Server Logs.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 24th, 2011 on 6:16 pm
Hi Jay,
Thanks for reply.
I have run these query in commandline.
step1,java weblogic.Admin -url(servername) -username (XYZ) -password(XYZ) THREAD_DUMP
step2,Dweblogic.log.RedirectStdoutToServerLogEnabled=true
step3,java weblogic.Admin -url(servername) -username (XYZ) -password(XYZ) SERVER_LOG
But i am getting Weblogic logs only in SERVERLOG.
Regards,
Dhilip
November 24th, 2011 on 8:02 pm
Hi Dhilip,
Can you try the following “weblogic.Admin” command to generate the Thread Dump?
java weblogic.Admin -url t3://localhost:7001 -username weblogic -password weblogic123 GET -pretty -type JVMRuntime
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 30th, 2011 on 7:44 am
Hi Jay,
yes using the above command i am capturing ThreadDump.
how can we capture threadDump for Secured port(https)?
Regards,
Dhilip
November 30th, 2011 on 10:39 am
Hi Dhilip,
webLogic.Admin utility also provides a “t3s” Based secured access. For more information on this please refer to the following link:
http://docs.oracle.com/cd/E13222_01/wls/docs81/admin_ref/cli.html#1336082
Also if you don’t want to use “t3s” protocol and just want to use “https” then you will need to enable the HTTP Tunneling on your WebLogic Server like following : http://docs.oracle.com/cd/E13222_01/wls/docs81/ConsoleHelp/servers.html#Configuring_HTTP_Protocol
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 30th, 2011 on 11:29 am
Hi Jay,
are you saying with the help of the below code we can get ThreadDump for Secured port?
java weblogic.Admin -url t3://server:5000 -username weblogic -password weblogic123 GET -pretty -type JVMRuntime
afer running this i am capturing ThreadDump for non secured port.
but if i run the same query for secured port i am getting below error.
Failed to connect to t3://server:5101: Destination unreachable; nested excption is:java.io.IOException: Empty server reply; No available router to destination.
Regards,
Dhilipkumar
November 30th, 2011 on 12:53 pm
Hi Jay,
Is there any SSL settings required for capturing ThreadDump(secured port) in CommandLine.
Regards,
Dhilipkumar
January 13th, 2012 on 5:14 pm
Hi Team ,
Can we have a post on how to analyze thread dumps .
I know we have tools like samurai .
I would like to know the best available tool and the way we need to analyze . Yes its a vast post but some basic inputs will be more helpful .
-Kiran
April 9th, 2012 on 4:30 pm
Hi Jay, thanks for sharing.
As per option 1, if I execute kill -3
whether that process will be killed or will it just take thread dump without killing the process ?
I need to execute this in prod env. so just want to check before executing.
Also I want to take heap dump – so is it better to do it after thread dump or before thread dump ?
*************************
A). ps -ef | grep java
qProcessbox 20650 1 0 Mar 26 ? 104:51 /opt/app/qProcessbox/java/jdk1.5.0_22/bin/java
B). Now use kill -3 20650
Syntax: kill -3
The OutPut of the Thread Dump will be generated in the Server STDOUT.
*************************
June 13th, 2012 on 12:01 pm
Hi,
i am getting thread dump but i am not understand the thread dump, how to understand that please tell me?
Thanks,
Ram
July 27th, 2012 on 12:23 am
Hi,
i am forget the weblogic Admin server password, how to get the the weblogic Admin server password?
Ram
August 5th, 2012 on 4:18 pm
Thanks René van Wijk
September 13th, 2012 on 11:54 am
HI Kiran,
I faced the same issue as below
——————————————————-
Can you show me how to write the output to some file ?
I tried like –
kill -3 $pid > testfile.txt
And this testfile.txt gets created under same path , but no data. so output was not moving to this file.
Thanks.
——————————————————-
you can try this:- this is just a method to collect the thread dumps from your STDOUT to a separate file, while doing kill -3 PID.
$ tail -f STDOUT_file >> threadDump.log & PID_last=”$!”
what it does:-
1. tail -f STDOUT_file (whatever will come next to this file, Eg. thread dump on executing kill -3 PID)
2. >> threadDump.log (should go in threadDump.log file)
3. & (run this process in background)
4. PID_last=”$!” (PID_last is a variable which has value of last backgroung process run. ie, tail command).
All steps below:-
1.$ tail -f STDOUT_file >> threadDump.log & PID_last=”$!”
2.$ kill -3 PID
3.$ kill -9 $PID_last (killing the tail process from step 1.)
4. Now you won’t find your threadDump.log EMPTY 🙂
September 21st, 2012 on 4:41 pm
Hi Jay,
Need some info about stuck threads. I notice that details of where the thread is stuck can be known by thread dumps( analyzing the dumps in log file/admin console etc.). I have a requirement that there be a flag variable which detects if a thread is stuck and calls a predetermined method. Can we know it in any way?
Thanks.
November 27th, 2012 on 12:29 pm
Hi,
I am new to this blog.
I have a problem about the STUCK THREAD. Few different ways to generate STUCK THREAD.
1. Generate by program
2. JRCMD
3. DUMP THREAD STACKS on AdminConsole
I have an application (actually by oracle) that will cause high CPU usage on the managed server, after a few minutes, STUCK THREAD will automatically generate in SERVER LOG, at this moment, everything is fine.
However, when STUCK THREAD happen on SERVER LOG, then i used JRCMD and DUMP THREAD STACKS immediately to capture the STUCK THREAD. There you are, something funny on the stack trace when compare SERVER LOG with JRCMD and DUMP THREAD STACKS.
I extracted a sample stack trace which cause by high CPU usage application and as below.
SERVER LOG – staplerz.servlets.HighCPUUsage.service(HighCPUUsage.java:27)
JRCMD – at staplerz/servlets/HighCPUUsage.service(HighCPUUsage.java:38)[optimized]
DUMP THREAD STACKS – at staplerz/servlets/HighCPUUsage.service(HighCPUUsage.java:38)
You can see the java line number which is not identical in SERVER LOG when compare with others.
For java line 38, which show the exact application execution line. For java line 27, which show the beginning of a METHOD. For us and my team, it is hard to trace the code when incident happen as the code is not provided by us.
Does anyone know that issue? Any solution?
p.s.: Service Request is opened in Oracle and withing for their reply.
Sorry for my poor English.
Thanks for the kindly help.
Best regrads,
Kelvin
November 27th, 2012 on 1:42 pm
Dear René van Wijk ,
Of course.
1.) Server Log
]”, which is more than the configured time (StuckThreadMaxTime) of “180” seconds. Stack trace:
Thread-24 “[STUCK] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” {
java.lang.StrictMath.tan(StrictMath.java:???)
java.lang.Math.tan(Math.java:136)
staplerz.common.Benchmark.trig(Benchmark.java:98)
staplerz.common.Benchmark.doBenchmark(Benchmark.java:20)
staplerz.servlets.HighCPUUsage.service(HighCPUUsage.java:27)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:183)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:168)
}
2.) JRCMD
“[STUCK] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=24 idx=0xd4 tid=25452 prio=1 alive, native_blocked, daemon
at staplerz/common/Benchmark.doBenchmark(Benchmark.java:29)[optimized]
at staplerz/servlets/HighCPUUsage.service(HighCPUUsage.java:38)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:183)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3717)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
— end of trace
3.) Dump Thread Stacks
“[STUCK] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=24 idx=0xd4 tid=25452 prio=1 alive, native_blocked, daemon
at staplerz/common/Benchmark.doBenchmark(Benchmark.java:28)[optimized]
at staplerz/servlets/HighCPUUsage.service(HighCPUUsage.java:38)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:183)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3717)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
— end of trace
Best regards,
Kelvin
November 27th, 2012 on 2:50 pm
Note that in the last two cases, the code has been optimized by JRockit. JRockit compiles the code to quick and dirty assembly when it first needs to run the code.
After a while (when the method is called a lot, i.e., it is hot) is recompiled by JRockit and the code is optimized. Just a little example to show this
When we run the class, we can use something such as: java -Xverbose:opt,gc model.test.Circle, which leads to the following output
– Note that the garbage collections stop after the optimizations have completed
– This is because the optimizer was able to determine that the Circle objects created in the getAreaFromRadius method
are not escaping from the scope of the method (they are only used for an area calculation)
– Once the call to c.getArea is inlined, it becomes clear that the entire lifecycle of the Circle objects is spent in the getAreaFromRadius method
– A Circle object just contains a radius field, a single double, and is thus easily represented just as that double if we know it has a limited lifespan
– An allocation that caused significant garbage collection overhead was removed by intelligent optimization
The above just gives an example of what happens to code when JRockit optimizates, over simply stated it does the following:
do {
1) get rid of calls, exposing more control flow through aggressive inlining
2) apply optimizations on enlarged code mass, try to shrink it
} while (“enough time left” && “code not growing too fast”);
You can influence the code optimization (not recommended) by using:
– logging: -Xverbose:codegen and -Xverbose:opt
– turning off optimizations: -XnoOpt (note that programs compile faster but run slower)
Also note that the threads are not really stuck. This is warning mechanism that WebLogic uses and which can be configured at will, for example
by using the admin console and set the max stuck thread time in the configuration, overload tab of a particular server. You can also configure
work managers and map this an application (examples of which are presented here http://middlewaremagic.com/weblogic/?p=5665 and
http://docs.oracle.com/cd/E17904_01/web.1111/e13701/self_tuned.htm#g1084820).
For the web application that does the bench mark, you can also ignore the stuck threads. To tell WebLogic to ignore them as well you can
define a work manager, for example in weblogic.xml,
Is your server environment becoming unstable when you have the stuck threads? (note that weblogic gives a health warning, but are you experiencing
some performance degradations because of this)
November 27th, 2012 on 3:35 pm
Dear René van Wijk,
Thanks for your quickly response.
I think i might not point out my problem correctly.
The problem is about to showing JAVA LINE NUMBER is different in the STUCK THREAD when application has stuck/error happen.
Here is a part of the sample code:
package staplerz.servlets;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletConfig;
import javax.servlet.ServletContext;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import staplerz.common.Benchmark;
public class HighCPUUsage extends HttpServlet
{
private static ServletContext sc;
public void init(ServletConfig paramServletConfig)
throws ServletException
{
super.init(paramServletConfig);
sc = getServletContext();
}
protected void service(HttpServletRequest paramHttpServletRequest, HttpServletResponse paramHttpServletResponse)
throws ServletException, IOException
{
PrintWriter localPrintWriter = paramHttpServletResponse.getWriter();
Object localObject = null;
paramHttpServletResponse.setContentType(“text/html”);
ServletContext localServletContext = getServletContext();
localPrintWriter.println(“High CPU Usage”);
localPrintWriter.println(“High CPU Usage“);
if (paramHttpServletRequest.getParameter(“numberLoops”) != null)
{
int i = new Integer(paramHttpServletRequest.getParameter(“numberLoops”)).intValue();
Benchmark localBenchmark = new Benchmark();
for (int j = 1; j <= i; j++)
Benchmark.doBenchmark();
}
localPrintWriter.println("“);
Server Log
Thread-24 “[STUCK] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” {
java.lang.StrictMath.tan(StrictMath.java:???)
java.lang.Math.tan(Math.java:136)
staplerz.common.Benchmark.trig(Benchmark.java:98)
staplerz.common.Benchmark.doBenchmark(Benchmark.java:20)
staplerz.servlets.HighCPUUsage.service(HighCPUUsage.java:27)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:183)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:168)
}
JRCMD
“[STUCK] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=24 idx=0xd4 tid=25452 prio=1 alive, native_blocked, daemon
at staplerz/common/Benchmark.doBenchmark(Benchmark.java:29)[optimized]
at staplerz/servlets/HighCPUUsage.service(HighCPUUsage.java:38)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:183)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3717)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
— end of trace
DUMP THREAD STACKS
“[STUCK] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” id=24 idx=0xd4 tid=25452 prio=1 alive, native_blocked, daemon
at staplerz/common/Benchmark.doBenchmark(Benchmark.java:28)[optimized]
at staplerz/servlets/HighCPUUsage.service(HighCPUUsage.java:38)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:183)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3717)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
— end of trace
You can ignore the “optimized” and try to compare the JAVA LINE NUMBER of each THREAD DUMP which took in different way. The LINE NUMBER in SERVER LOG is not SAME as JRCMD and DUMP THREAD STACKS. This is the main problem of my question.
Many thanks with your kindly help. ^^
Best regards,
Kelvin
November 28th, 2012 on 2:08 pm
Actually, you cannot ignore the optimized indication, as JRockit changes your code (usually it inlines different methods into one
which are then becoming larger – this is probably the reason why you are seeing a different line number).
JRockit also does some bookkeeping, when for example a choice it made in optimizing the code tends to be wrong (such as an exception
is thrown while it assumes it will not) – thus it has to change the code again.
The bookkeeping holds on to an original version of the code (which you see in the weblogic stack trace) and the JRockit trace shows the
optimized one (which is not the same as the original) – that is why I gave you the example concerning code optimizations, so you
have some idea in what is going on.
December 13th, 2012 on 5:37 am
I am able to take thread dump of the admin server fine using java weblogic.ADMIN utility
How can I take a thread dump of a managed server
java weblogic.Admin -url t3://vdaxta01.server.com:11101 -username weblogic -password weblogic THREAD_DUMP
throws
Failed to connect to [Login failed for an unknown reason: HTTP/1.1 404 Not Found]
July 19th, 2013 on 6:41 pm
Excel·lent post
I want to do a thread dump just before stop a server.
With stopserver.sh, any problem, but if I stop a managed from console admin with nodemanager, how can I take thread dump?
Thanks!
June 12th, 2014 on 12:21 pm
Hi Friends,
i am new to this blog.i registerd today.I hope this blog will improve my technical skill @ weblogic.
ThreadDump:
i have takem thread dump using kill -3 Pid In Linux os after that i check the log file domainhome/servers/mng1/logs but there is no STDOUT log file is not generate.Please help me where can i check thread dump file.
Thanks&Regards,
LakshmiNarayana.K
March 7th, 2016 on 3:34 pm
When I am using Step4, to get thread dump, I am getting following:
“server must be running before thread stacks can be displayed”
I am using WebLogic Server Version: 10.3.6.0.
Any idea why I am unable to get the thread dump.
March 7th, 2016 on 3:36 pm
I am using WebLogic Server Version: 10.3.6.0
When I use step4, to get thread dump, I get following:
“server must be running before thread stacks can be displayed”
March 7th, 2016 on 6:29 pm
Hello Swaraj,
It’s a known issue with WLS 10.3.6.0 Specifically on JDK 1.7
I remember that there is a patch also available for the same issue for WLS 10.3.6 release. Please contact Oracle Support to get that patch. This issue is fixed in later version of WLS like WLS 12.1.1
You should also see the following kind of log message in your server log that indicates this issue:
***** This VM does not support thread dumps*****
Regards
Jay SenSharma
April 24th, 2016 on 10:23 pm
When I use option 3 given at the top of the page I am able to get ThreadDump only from AdminServer, but not from the Managed Server.
Am I missing something here? Are we supposed to deploy anything extra on the managed server for the thread dump o work on it?