Once applications are deployed, the problems start: Multiple users connect to the application and complain about the speed. Where do you start to locate the problem? Is it the operating system? What about garbage collection? Is the architecture of the application a problem? In this post we give a starting point in how to analyze a problem and act accordingly.

We outline best practices for solving production problems in three parts:

• System performance
• Performance in general
• Solve performance problems

Before we start with tuning the performance, it is important to have an overview of the internal architecture of the WebLogic Server. The processing of requests is based on the following components: listen threads, the socket muxer and the execute queue. When the server boots, the server mounts a listen thread to each configured port. The socket muxer detects an incoming request and places it in the execute queue. A free execute thread picks up the request from the queue and executes it. The execute thread processes the request in whole, in other words, the call to a servlet, the servlet call to an EJB and the EJB call to JDBC to query a database, are handled by one execute thread.

An important observation is that if application code blocks an execute thread for a long time, the server can not use this execute thread to process other requests. If the application is in a state where each thread is blocked for an indefinite period, the server stops responding or additional threads are created. Creation of additional threads that eventually will block does not help the situation. Threads that are not processed in the foreseeable future, does not benefit the performance.

System performance

A good understanding of the operating system, network, JVM and server resources (such as connection pools, JMS servers) and the associated tuning options help enormously in applying best practices. In general it is not enough to know what to do, we should also know why it works.

When we design an application, we must first understand the application itself and how users interact with it. We should investigate all system components to identify and understand the interaction between these components. By determining the workload over all layers, we get an understanding which components are most affected by the activity of users. A good understanding of the system allows us to choose the right system architecture. Once the system architecture is set, we can begin to focus on the architecture of the application. Some very wise man once said: “Architecture matters, and in systems of scale and systems that require availability, architecture matters absolutely! Failure to achieve a solid architecture will doom in advance any hope of significant scalability, and will leave the effects of failure within the system to pure chance.”

If after well-considered decisions, there are still performance problems in production and we want the maximum from the server, we need to tune. The tuning is done at different layers within the production environment. In the following discussion we start at the bottom and then work our way steadily upwards.

Tuning the operating system

In general, Java EE applications have some kind of Web interface. Usually, this type of applications have a few thousand concurrent users. As a result that a high number of connections between the browser and the server are opened and closed. These connections are nothing more than TCP sockets on operating system level. Most operating systems handle sockets as a form of file access and use file descriptors to keep track of which sockets are open. To contain the resources per process, the operating system restricts the number of file descriptors per process.

A TCP connection that is properly closed by an application is in the TIME_WAIT state before it is returned to the operating system. While the connection is in TIME_WAIT, all the used resources (including the file descriptor) stay allocated to the process. The result is that the file descriptor table can fill up. This means that we have to tune the operating system so that a scalable application does not run against an operating system restriction. Tuning basically means following recommendations from the hardware vendor.

A number of useful tools which can be used are netstat (to determine the number of sockets in TIME_WAIT state, for example by using, netstat –a | grep TIME_WAIT | wc –l) or iostat (to determine disk I/O on the operating system, for example, iostat 5 5). Note that iostat is part of the sysstat package, which can be installed by using: yum -y install sysstat.

Tuning the Java Virtual Machine (JVM)

The JVM used to run the WebLogic Server and the deployed applications is of great importance in the final performance of the WebLogic Server. Keep in mind that performance is nothing if we do not have stability – fast applications do not do the users any good if they are not running. So when choosing a JVM look for stability first, then the performance.

The garbage collection is the most important factor in tuning a JVM. Poorly tuned garbage collectors or applications that create an unnecessary numbers of objects, can have dramatic effects on the performance of the application. A proper tuning of the garbage collector, greatly reduces the processing time resulting in a significant improvement in application performance. More information on JVM tuning is presented in this post.

Tuning the server

During performance tests it is important to enlarge the listen queue. WebLogic Server uses the Accept Backlog parameter to specify the size of the queue. If client requests are rejected it may be that the size of the queue is too small. The login timeout parameter is used to specify the maximum time to set up a connection. By default, this value is 5 seconds (25 seconds for SSL), which may be too small for systems with a heavy load.

Thread management optimization is handled by the WebLogic Server itself. By collecting performance information, the number of execute threads are adapted to the workload of the application. By default every deployed application is assigned to the default WorkManager. This gives each application the same priority and prevents applications to use more than their fair share of server resources. In general, the default WorkManager suffices. A number point are of interrest in this respect:

• Database Connection Pool – If the application depends on database connections, it is important to note that no more concurrent requests can be processed than there are connections in the pool. In this case we can override the default WorkManager by imposing a maximum thread constraint. This constraint is set to be equal to the number of connections in the connection pool.
• Server Deadlock – Some applications call resources in a different server instance, which in turn calls resources in the calling server. In this case, we must take extra care so that a server deadlock does not occur. A deadlock occurs when all threads are waiting for requests from the remote application and no thread is available to handle the callback process. In this case we can configure a minimum thread constraint so that a number of execute threads are available at all times to the process the callback.
• Server Overload – If we tune the server to achieve a certain response time, it is important to note that at some point the server must no longer accept new requests. If a request is in the queue, it takes some time before it can be processed (the request queued is waiting for an execute thread). To ensure that the queue is not cluttered with long waiting requests, we setup a capacity constraint. If the capacity is exceeded, a 503 response is send back to the user.

WebLogic Server uses resource pooling to optimize performance. For example, stateless session beans and message-driven beans are pooled. If all resources are used, the server will increase the pool to suit the requests of the application’s requirements. Increasing the pool adds overhead to process a request. This overhead depends on the type of resource. When the pool reaches its maximum, the server can not increase the pool. If this situation occurs, requests must wait until a resource is released before the request can be processed. An obvious rule of thumb is to ensure that the pool is large enough to process the number of simultaneous requests.

Once optimum settings are found, there usually can be achieved some improvements in garbage collection.

Before an application is put into production, it is important to conduct long-term load tests. These tests generally reveal problems, that only surface in production after a long time, such as memory leaks.

Performance in general

An appropriate architecture (the right patterns) can improve performance. Understanding where the application runs, such as in a Web and/or EJB container, and what these containers have to offer for is very important.

Design

Good application performace starts with a good design. Too complex an application will always perform badly, and tuning will not help. Proper use of design patterns could lead to significant advantages, such as standardizing default design problems. A number of design patterns also offer a performance enhancement, examples of these are the session facade and the command pattern:

• Session facades improve performance by offering high level business operations. Especially, when calling EJB by using remote interfaces (also when calling EJBs locally, because the EJB container provides services such as security, lifecycle management and transaction management).
• The command pattern uses one command object to process requests. By using this pattern with the session facade the number of remote calls can be reduced, and as such we are also reducing remote network calls.

Web container

Many applications complement the HTTP protocol by storing objects in the HTTP session. In the same browser session, the session data is available to subsequent requests. In a high-available application, the session data must be available even when the original server fails. This means that the session data must be replicated to other servers within a cluster. Keep in mind that there is a cost attached to replicating session data. If a web application changes the HTTP session object, the server must store these changes at the end of each request. The amount of data that the server is to store depends on the data structure of the HTTP session object. If the web application changes a small amount of data in a large object structure, the server replicates this large object structure not just the changes, so it is important to divide large structures into small pieces. An HTTP session object also uses resources, so it is important to clean it up when you are done.

EJB container

EJB components can have a dramatic effect on performance, because they generally do more work than a web component. We can optimize calls to EJBs. If the caller and the EJB are in the same JVM and loaded by the same classloader, we can ensure that a pass-by-reference optimization occurs. This optimization must be turned on by using the deployment override weblogic-ejb-jar.xml, i.e., set the enable-call-by-reference element of a particular EJB component to true.

Database access

Efficient database access is vital to obtain scalability and a high throughput. If database access is slow all other tuning, such as JVM, EJB container etcetare, will be futile. First, we need an effective database design. A classic in this area is “An Introduction to Database Systems” written by CJ Tate. Many database designers have a preference for normalizing their design. With the result that multi-way joins are needed to retrieve data for a business object. A design that looks good on paper, is generally a disaster in production. If performance is the issue at hand, it is time to sit down with the database designer and denormalize critical tables.

The next step is that the physical database must meet the performance requirements. A DBA must use all the optimization options in order to achieve the best possible performance. Here it is important that the DBA has a complete list of all queries within the application so that the appropriate indexes can be made.

WebLogic Server supports both local and global transactions. Global operations include multiple resources. Global transactions have additional logging and extra network I/O is needed, which makes it slower than local transactions. If possible, use local transactions.

Solve performance problems

Now that the application and the environment are tuned to perfection, users are happy and the system shamelessly handles hundreds of requests per second, or not? As mentioned above a requirement for successful troubleshooting we need a good understanding of the system and its components. Each system is different and every performance problem is probably different. There are, however, a number of best practices that can help pinpoint problems.

Preparation

Solving problems is difficult and time-consuming. If users are unhappy, we need the right infrastructure, processes and people to solve the problem.

First, the application must be thoroughly tested. It is important to find out how the application behaved in the test, in order to know if the problem we have to find a solution for is normal during peak loads. Test results also gives us insight in the resource usage. Testing is half the work for solving production problems.

Additionally, all performance monitoring mechanisms must be in place in order to provide information on system performance and activity. Unfortunately, performance problems are not there on request, so we must have some form of logging in order to reconstruct the resource usage and activity during a given period.

Last be not least; it is not a bad idea to have a multi-disciplinary team available before the problems arise.

Identifying and correcting bottlenecks

A bottleneck is a resource within a system, which reduces the throughput of a system or affects the response time significantly. To fix bottlenecks in a distributed system is not one of the easiest tasks in IT. In general, there are multi-disciplinary teams required. The advent of performance monitoring tools makes fixing problems somewhat easier. Bottlenecks can occur in the web server, application code, application server, database, network, hardware or operating systems.

Examples

Web Server
Problem description: WebLogic proxy plug-in closes keep-alive connections to clients randomly.

Configuration

KeepAliveTimeout 90
MaxKeepAliveRequests 0
KeepAliveSecs 55

Why does the WebLogic proxy plug-in ignore the KeepAlive directives and decides to the close connection to the client by itself?

• Is this related to the WebLogic proxy plug-in?
• What about the thread configuration in the Web Server?

From the Web Server documentation: In addition to the set of active child processes, there may be additional child processes which are terminating but where at least one server thread is still handling an existing client connection. Up to MaxClients terminating processes may be present, though the actual number can be expected to be much smaller. This behavior can be avoided by disabling the termination of individual child processes, which is achieved by the following:

• set the value of MaxRequestsPerChild to zero.
• set the value of MaxSpareThreads to the same value as MaxClients.

By applying these guidelines in the Web Server worker configuration the problem got resolved

<IfModule mpm_worker_module>
	StartServers 2
	MinSpareThreads 25
	MaxSpareThreads 75 --> 2000
	ThreadLimit 200
	ThreadsPerChild 200
	MaxClients 2000
	MaxRequestsPerChild 0
</IfModule>

JVM and Code.
Problem description: On Linux 64-bit machine and WebLogic 10.3.2 we are getting an out-of-memory very frequently (java.lang.OutOfMemoryError: class allocation...). Configured parameters of the JRockit JVM: -Xms2048m -Xmx2048m -Xns:1536m -XXsetGC:genparcon.

Symptoms indicate a native memory leak due to dynamically creating classes and not letting them be garbage collected. We need to have information on the class loading. To check if the number of loaded classes constantly increase, we can use -Xverbose:class.

Obtained information: Many classes are created as follows [INFO ][class ] created: #23722 com/beans/FactuurResponse$JobInfo$JaxbAccessorF_subjobInfo.

Now what; we need Java knowledge:

• The classes are used to marshal and unmarshal XML payload from a web service request/response.
• It looks like JAXB is creating the subjobInfo and many similar classes on the fly.
• Invoking JAXBContext.newInstance() causes a reload of all classes as per specification, it is not designed as a singleton.
• Solution is to create only one instance (singleton) of a JAXBContext and reuse it.

Application Server I
Problem description: Illegal memory access [54].

This you have to know

• JRockit has issues with Apache Axis and it crashes whenever a SOAP request is submitted to a web service.
• The reason for the crash is that there are two versions of javax.xml.soap.SOAPPart, and the wrong one is loaded before the right one.
• So this is actually not a JVM related issue, but an issue with the configured classpath on the application server.

Steps to resolve the issue

• Add the JVM option -Xverify:all to check to correctness of the loaded bytecode.
• Edit the classpath so that the right version of the class is loaded.
• Add the axis.jar to the WEB-INF/lib directory.
• Edit the deployment override weblogic.xml to tell WebLogic to prefer the application classes before the system classes, i.e.

<container-descriptor>
	<prefer-web-inf-classes>true</prefer-web-inf-classes>
</container-descriptor>

Another option to edit the classpath is to add the jars to the APP-INF/lib directory of the EAR file and use a filtering classloader.

Application Server II
Problem description: Running SOA suite 11g in a single server environment. Everything was running fine until the application adapters for SAP were deployed – after a restart of the server the enterprise manager does not start anymore.

Scanning the logging for errors: java.lang.NoSuchMethodError: org.apache.xerces.util.DOMEntityResolverWrapper.setEntityResolver(...). NoSuchMethodError is thrown when the definition of a class has incompatible changed. Again a very weird issue concerning classloading.

The application adapter iwafjca.rar contains the malefactor, i.e., the jar file xercesimpl.jar which contains the wrong version of the class.

Some Final Words

As you can tell from the wide-ranging discussion, there is a lot to know and consider when tuning or troubleshooting a complex Java EE system. Nothing beats experience, go home, make your users angry and do some performance tuning.

References

[1] Tate, “Bitter Java”, Manning, 2002. This book explores the topic of antipatterns with specific examples. There is a lot of code, which is refactored many times to improve performance and readability until the final examples are decidedly sweeter.
[2] System Tuning Info for Linux Servers.
[3] Linux Performance and Tuning Guidelines.
[4] Tuning Garbage Collection with the Java Virtual Machine.
[5] Performance White Paper.
[6] Top Tuning Recommendations for WebLogic Server.

Hi,

Jay SenSharma

DISCLAIMER:

In this article we may see an abnormal behaviors of weblogic. Which may not be necessarily a BUG but it is always to be aware of such behavior while using Weblogic. The idea behind making this page is just to make awareness among the WebLogic Admins to be alert specially when the some of these behaviors are related to WebLogic Security.

Some of the behaviors of WebLogic which may be due to inappropriate Security implementation in the Security system of WebLogic, Even if in some cases it is work as designed, Still it suggests to keep an eye on it and try to make those features more enhanced. Some of them are now fixed by the Application Server Vendor but still some need to be fixed or enhanced. The intentions here are not to point to the weak points of any Application Server but solely to make people aware about such strange or uncommon behaviors.

==========================================================================
Any WebServer or Application Server which runs on below mentioned JVM are not safe due to the security breach. For example if you just want to hang A server then just sent the following request to the Server using any HttpClient like JMeter or any other Utility which allows you to send the Http Header of your Choice.

Once you are able to send the following Http Request Header successfully to the Java based Application/Web Server …the Server will try to parse the Http Request Header and it will Hang  while processing this request.

“GET”,”/”,headers={“Accept-Language”: “en-us;q=2.2250738585072012e-308″}

Just for an example try to run the following simple Java program which just tries to parse a Double value 2.2250738585072012e-308. As soon as you will run this program you will see that your JVM will Hang….and the CPU Utilization will be around 100%  

class HangJVM
{
  public static void main(String[] args)
    {
      System.out.println("Test:");
      double d = Double.parseDouble("2.2250738585072012e-308");
      System.out.println("Value: " + d);
     }
}

The JVMs which are affected are as following:
Java SE
JDK and JRE 6 Update 23 and earlier for Windows, Solaris, and Linux
JDK 5.0 Update 27 and earlier for Solaris 9
SDK 1.4.2_29 and earlier for Solaris 8

Java for Business
JDK and JRE 6 Update 23 and earlier for Windows, Solaris and Linux
JDK and JRE 5.0 Update 27 and earlier for Windows, Solaris and Linux
SDK and JRE 1.4.2_29 and earlier for Windows, Solaris and Linux

Save your JVM and your Application Server From Attack or Contact your Support

Or Please get a Fix from Support  Which Updates the “rt.jar” of the JVM The Fix Details are available in the following Link  http://middlewaremagic.com/weblogic/?p=5393#comment-2821

And

Regarding the “Oracle Security Alert for CVE-2010-4476″You can get the Temp Security Patches related to this issue from the following link:http://www.oracle.com/technetwork/topics/security/alert-cve-2010-4476-305811.html

.
Thanks
Jay SenSharma

Jay SenSharma

Java Process Architecture Diagram

What Are these Different Parts?

Eden Space:

Survivor Spaces:

Young Generation: (-XX:MaxNewSize)

Virtual Space-1: (MaxNewSize – NewSize)

Java Heap Area: (-Xmx and -Xms)

Java Heap is a Memory area inside the Java Process which holds the java objects.  Java Heap is a combination of Young Generation Heap and Old Generation Heap. We can set the Initial Java Heap Size using -Xms JVM parameter similarly if we want to set the Maximum Heap Size then we can use -Xmx JVM parameter to define it.

Virtual Space-2: (MaxHeapSize – InitialHeapSize)

PermGen Space: (-XX:MaxPermSize)

Native Area:

Hi,

Jay SenSharma

Many times as a Weblogic Admin We find difficulty to identify the Process Id of our WebLogic Server …using (ps   -ef   |   grep   java) command Or the Step described in the Step2). of the following Link:  weblogic-heap-dump-genration-using-jmap/ …When There are more than one WebLogic Servers are running on the same Physical Box. So in this case we can use the “jps.exe” utility to easily get the process ID of each WebLogic Server.This utility can be seen inside ANY “<JAVA_HOME>\bin\jps.exe”.

NOTE: jps.exe utility is available from JDK1.5 Onwards By Default.

Step1). Run Two or More than 2 WebLogic Server Instances in the same Physical Box.

Step2). Now open a command prompt and move inside the “<JAVA_HOME>\bin” directory …And then run the command:

jps.exe     -v

Uniquely Finding Different WLS instances Process ID

In the above outout u can see that  when we run the “jps.exe    -v”  then it shows all the JAVA_OPTIONS of the Server including the WebLogic Property…  -Dweblogic.Name which represents the configured name of the Running WebLogic  Server.  This helps us to easily distinguish the correct process ID for our running Servers.

Not only the Process ID, Rather using the above tool we can see all the System property which has been taken by the WLS server as well as the Heap Settings…etc

Some other Options which can be used with jps.exe utility:

-m
Output the arguments passed to the main method. The output may be null for embedded JVMs.
-l
Output the full package name for the application’s main class or the full path name to the application’s JAR file.
-v
Output the arguments passed to the JVM.
-V
Output the arguments passed to the JVM through the flags file

.
.
Thanks
Jay SenSharma

Jay SenSharma

Many times we want to Generate the Heap Dump to see what all objects are created inside the JVMs Heap space. The Heap Dump tells us that what all objects are created inside the JVM and what is the size of those Objects. Based on this analysis we can findout that what all objects (Application Objects/Platform Objects) are consuming more memory inside the heap.

@REM
@REM Call setDomainEnv here.
set DOMAIN_HOME=C:\bea103\user_projects\domains\WS_Security_Domain
for %%i in ("%DOMAIN_HOME%") do set DOMAIN_HOME=%%~fsi
call "%DOMAIN_HOME%\bin\setDomainEnv.cmd" %*
set SAVE_JAVA_OPTIONS=%JAVA_OPTIONS% -Xrunhprof:format=b,file=MyHeapDump.hprof
set SAVE_CLASSPATH=%CLASSPATH%

#
# Call setDomainEnv here.
DOMAIN_HOME="C:/bea103/user_projects/domains/WLST_Domain"
. ${DOMAIN_HOME}/bin/setDomainEnv.sh $*
SAVE_JAVA_OPTIONS="${JAVA_OPTIONS}"   -Xrunhprof:format=b,file=MyHeapDump.hprof
SAVE_CLASSPATH="${CLASSPATH}"

Heap Dump Location

Wait For Some time 4-5 Minutes to get the complete heap Dump

Opening Jhat Console

jhat Histogram Analysis

Jhat Object Query Language Editor

Jhat Non Product Instance Counter

Jhat Platform/Non-Platform Instance Counter

(OQL) Object Query Language Querie

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

(OQL) Object Query Language Querries

java.io.IOException: Unrecognized heap dump sub-record type:  0

at com.sun.tools.hat.internal.parser.HprofReader.readHeapDump(HprofReader.java:489)

at com.sun.tools.hat.internal.parser.HprofReader.read(HprofReader.java:226)

at com.sun.tools.hat.internal.parser.Reader.readFile(Reader.java:79)

at com.sun.tools.hat.Main.main(Main.java:143)

HPROF ERROR: Missing jclass when fields needed [hprof_class.c:647]

HPROF ERROR: Trouble with unprepared classes [hprof_reference.c:384]

HPROF ERROR: Missing jclass when fields needed [hprof_class.c:647]

HPROF ERROR: Trouble with unprepared classes [hprof_reference.c:384]

allocation sites … done.

<Jul 11, 2010 8:11:06 PM IST> <Error> <> <BEA-000000> <THREAD DUMP from JVM taken at ‘Sun Jul 11 20:07:47 2010′>

C:\bea103\jdk160_05\bin>jmap -dump:format=b,file=heapJMap.bin 5552
Dumping heap to C:\bea103\jdk160_05\bin\heapJMap.bin …
Heap dump file created

C:\bea103\jdk160_05\bin>jhat -J-Xmx1024m heapJMap.bin
Reading from heapJMap.bin…
Dump file created Mon Jul 12 10:05:58 IST 2010
Snapshot read, resolving…

Chasing references, expect 239 dots………….
…………………………………………
Eliminating duplicate references…………….
…………………………………………
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.

Hi,

Jay SenSharma

Thread Dumps are one of the very important JVM reports which we can use to analyze Server/JVM Hang Situations and the Root Cause of it… Here are few very common techniques which we can use to take Thread Dumps…

Note: Taking only one or two Thread Dumps is not very useful. Always make sure that you collect atleast 5-6 Thread Dumps taken in the interval of 10-12 Seconds….Ehich will be really helpful to investigate the Hang Situation.

————————————————–

Option 1). In Windows Machines:   Just Press (Ctrl + Break) Buttons together in the CommandWindow ehere your Server in running.

In Solaris/Linux …etc Unix Based Boxes you need to findout the Servers Process ID…by running the following command from the Same box where the Server is running:

A). ps -ef | grep java

qProcessbox 20650 1   0   Mar 26 ?         104:51 /opt/app/qProcessbox/java/jdk1.5.0_22/bin/java

B). Now use     kill -3 20650

Syntax:  kill -3 <ProcessID>

The OutPut of the Thread Dump will be generated in the Server STDOUT.

————————————————–

Option 2). We can Collect Thread Dump Using “weblogic.Admin” which is deprecated …but still available …and may be available in near future as well…As i think because it is one of the best debugging utility for Admins.

java weblogic.Admin -url t3://AdminHostName:7001 -username weblogic -password weblogic THREAD_DUMP

java weblogic.Admin -url t3://MS1HostName:7003 -username weblogic -password weblogic THREAD_DUMP

java weblogic.Admin -url t3://MS2HostName:7005 -username weblogic -password weblogic THREAD_DUMP

The THREAD_DUMPS will be generated in the Servers STDOUT file.  If you want to See the Thread Dump In your Same Command Window Where you are running the “weblogic.Admin” utility…then Enable the “redirect-stdout-to-serverlog” Or by adding the JAVA_OPTION in serverStartScript (-Dweblogic.log.RedirectStdoutToServerLogEnabled=true) flag from Admin Console…sothat the STDOUT entries also will be redirected to the Server Logs…then you can read The Server Logs using the following command remotely/locally.

java weblogic.Admin -url t3://MS2HostName:7005 -username weblogic -password weblogic SERVERLOG
————————————————–

Option 3). Through WLST you need to write the WLST script like “AdminThreadDump.py” In this case you will see the Thread Dumps on the Same Shell Prompt…. not need to look into the Server STDOUT

connect('weblogic','weblogic','t3://localhost:7001')
cd ('Servers')
ls()
cd ('AdminServer')
ls()
threadDump()
cd('..')
cd('ManagedServerOne')
threadDump()

Now open a command Shell and then run the “setWLSEnv.sh”

then run the WLST script like:

java weblogic.WLST AdminThreadDump.py
or
java weblogic.WLST /opt/bea/MyScripts/AdminThreadDump.py
=====================================

Option 4). Simplest option to take Thread  Dump is :

Login to AdminConsole—>Server —> Monitoring —> Threads

Option 5). You can even collect THREAD_DUMPS using JMX code like mentioned in the below link …using “JVMRuntimeMBean”     taking-threaddump-along-with-jrockitruntime-details/

Option 6). Taking ThreadDump Using T3ServicesDef

Option 7). Collecting Thread Dumps Using Jstack utility

Option 8). If you are using WebLogic Server as a Windows Service then you can collect the Thread dump as Described by Former GSS India Oracle Mentor SriDevi :http://middlewaremagic.com/weblogic/?p=823#comment-1011

Option 9). Taking the JVM Details (Heap size and Jdk Details including OS details) including Thread Dump:

java weblogic.Admin -url t3://ServerHostName:7001 -username weblogic -password weblogic GET -pretty -type JVMRuntime

Collecting Thread Dump Using JVMRuntime

.

.

Thanks

Jay SenSharma

Hi,

Jay SenSharma

*******JVM Crash Investigation*********

JVM Core Dump is the most important File to investigate the JVM Crash. By default the Core Dump will be genarated. But Just in case if JVM is not able to generate the Core Dump then there may be the following reasons:

If there is not enough disk space or quota to write the file in your File System.
If JVM is not having to create or write a file in the directory.
if another file exists in the same directory with that is read-only or write-protected.

Unix/Linux-specific: Use the limit or ulimit commands to determine if core dumps are disabled.
Example, on Linux, the command “ulimit -c unlimited” enables core dumps to be written, no matter what their size. Core dump sizes can be restricted if disk space limitations are a concern.

It may be possible to get a thread dump before the process exits.  HotSpot supports the Java_Option -XX:+ShowMessageBoxOnError; the corresponding JRockit option is -Djrockit.waitonerror.  When the JVM is crashing, it may prompt the user :::Do you want to debug the problem?::: This pauses the process, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.  However, this does not work in all cases (for eg., in case of stack overflow).

=========================

*******Crash Because of OutOfMemory:*******

Please apply the Following Flag in your JAVA_OPTIONS in the start Script of your Server: -XX:+HeapDumpOnOutOfMemoryError

Details:
1) -XX:+HeapDumpOnOutOfMemoryError option available in 1.5.0_07 and 1.4.2_12, producing an hprof binary format heap dump (By default the DUMP will be generated in your Systems TEMP directory….In Window machine we can easily findout the TEMP directory by running the command “echo %TEMP%”)

2) Analyse hprof heap dumps using HAT, or jhat or YourKit (has an hprof import option)
hprof heap dumps are platform independent and so you don’t need to analyze the dump on the same system that produced it

3) running with -XX:+HeapDumpOnOutOfMemoryError does not impact performance – it is simply a flag to indicate that a heap dump should be generated when the first thread throws OutOfMemoryError.

4) NOTE: -XX:+HeapDumpOnOutOfMemoryError does not work with -XX:+UseConcMarkSweepGC in 1.5.0_07

=========================
TestCase:

public class TestXss {
public static void main(final String[] args) throws Exception {
// start the given number of threads
for (int i = 1; i <= Integer.parseInt(args[0]); i++) {
System.out.println("Starting Thread " + i);
final Thread t = new Thread("T[" + i + "]") {
public void run() {
try {
while (true) {
Thread.sleep(1000);
}
} catch (Exception e) {
e.printStackTrace();
}
}
};
t.setDaemon(true);
t.start();
Thread.sleep(5);
}
// wait
Thread.sleep(1000000);
}
}

———————————OUTPUT————————————
NOTE:   java -Xmx1496m -Xss1m TestXss 5000
Starting Thread 411
Starting Thread 412
Starting Thread 413
Starting Thread 414
Starting Thread 415
Starting Thread 416
Starting Thread 417
Exception in thread “main” java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at TestXss.main(TestXss.java:18)

————-

Note: if you see OOM in the Native Space  ( unable to create new native thread) Then Please first of all Try to decrease the -Xss size. It’s default value is -Xss512K   ———>Reduce it to ——–>  -Xss256K

Like this you will see that JVm is able to create more Native Threads…But beware that if u will decrease it more than a certain limit …u may start getting “java.lang.StackOverflowError”.

———————————————-

Thread Related Definitions

• CompletedRequestCount. Number of completed requests in the priority queue.
• ExecuteThreadIdleCount. Number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives.
• ExecuteThreadTotalCount. Total number of threads in the pool.
• HoggingThreadCount. Number of threads that are being hogged by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
• MinThreadsConstraintsCompleted. Number of requests with min threads constraint picked up out of order for execution immediately since their min threads requirement was not met. This does not include the case where threads are idle during schedule.
• MinThreadsConstraintsPending. Number of requests that should be executed now to satisfy the min threads requirement.
• PendingUserRequestCount. Number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
• QueueLength. Number of pending requests in the priority queue. This is the total of internal system requests and user requests.
• StandbyThreadCount. Number of threads in the standby pool. Surplus threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed.
• Throughput. Mean number of requests completed per second

How to rotate .out ( stdout) log file in weblogic 9.2 (solaris/linux)

By default Weblogic wont have options to rotate the *.out file, If you are running in unix or Linux boxes, you can add the following snippet

under /etc/logrotate.conf file append a function to handle *.out files

<Location of logs directory>/*.out

{
copytruncate
rotate
size=5Mb
}

————————————————————-

Thanks

Jay SenSharma