Hi All,

Jay SenSharma
.
In most of the environments we usually see some kind of Memory related issues like OutOfMemory/Server Slowless kind of things. most of the times it is related to the in accurate tuning of the JVM and some times it happens due to the Application Frameworks Bug/ In accurate tuning configuration or it may happen due to the Application Code as well ..like Object Leakin in the Application code.
.
.
Here we are going to see that what causes the OutOfMemory issues and Why it happens along with some basic First Aid Steps to debug this kind of issues.
What is OutOfMemory?
An OutOfMemory is a condition in which there is not enough space left for allocating required space for the new objects or libraries or native codes. OutOfMemory can be divided in tow main categories:
1). OutOfMemory in Java Heap:
This happens when the JVM is not able to allocate the required memory space for a Java Object. There may be many reasons behind this…like
Point-1). Very Less Heap Size allocation. Means setting the MaxHeapSize (-Xmx) parameter to a very less value.
.
Point-2). The Leaking of Objects. Either the Application is not unreferencing the unused Objects or the Third part frameworks (Hibernate/Spring/Seam…etc) might not be releasing the references of the objects due to some inaccurate configurations.
.
Point-3). In Many cases it may be the reason that Application codes are getting the JDBC connections objects from the DataSource are not being released back to the Connection Pool.
.
Point-4). Garbage Collection strategy may be in correct according to the environmental/application requirements.
.
Point-5). In-accurate setting of Application/Frameworks Cache.
Example:
Exception in thread "Thread-10" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStr ingBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java: 119)
at java.lang.Throwable.toString(Throwable.java:344)
2). Native OutOfMemory:
Native OutOfMemory is a scenario when the JVM is not able to allocate the required Native Libraries and JNI Codes in the memory.
Native Memory is an area which is usually used by the JVM for it’s internal operations and to execute the JNI codes. The JVM Uses Native Memory for Code Optimization and for loading the classes and libraries along with the intermediate code generation.
The Size of the Native Memory depends on the Architecture of the Operating System and the amount of memory which is already commited to the Java Heap. Native memory is an Process Area where the JNI codes gets loaded or JVM Libraries gets loaded or the native Performance packs and the Proxy Modules gets loaded…
Native OutOfMemory can happen due to the following main reasons:
.
Point-1). Setting very small StackSize (-Xss). StackSize is a memory area which is allocated to individual threads where they can place their thread local objects/variables.
.
Point-2). Usually it may be seen because of Tuxedos incorrect setting. WebLogic Tuxedo Connectors allows the interoperability between the Java Applications deployed on WebLogic Server and the Native Services deployed on Tuxedo Servers. Because Tuxedos uses JNI code intensively.
.
Point-3). Less RAM or Swap Space.
.
Point-4). Usually it may occur is our Application is using a very large number of JSPs in our application. The JSPs need to be converted into the Java Code and then need to be compiled. Which reqires DTD and Custom Tag Library resolution as well. Which usually consumes more native memory.
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at TestXss.main(TestXss.java:18)
3). OutOfMemory in PermGen Space:
Permanent Generation is a Non-Heap Memory Area inside the JVM Space. Manytimes we see OutOfMemory in this Area. PermGen Area is NOT present in JRockit JVMs. For more details on this area please refer to:
http://middlewaremagic.com/weblogic/?p=4456.
.
The PermGen Area is measured independently from the other generations because this is the place where the JVM allocates Classes, Class Structures, Methods and Reflection Objects. PermGen is a Non-Heap Area.It means we DO NOT count the PermGen Area as part of Java Heap.
The OutOfMemory in PermGen Area can be seen because of the following main reasons:
Point-1). Deploying and Redeploying a very Large Application which has many Classes inside it.
.
Point-2). If an Application is getting deployed/Updated/redeployed repeatedly using the Auto Deployment feature of the Containers. In that case the Classes belonging to the application stays un cleaned and remains in the PermGen Area without Class Garbage Collection.
.
Point-3). If “-noclassgc” Java Option is added while starting the Server. In that case the Classes instances which are not required will not be Garbage collected.
.
Point-4). Very Less Space for allocated the “=XX:MaxPermGen”
Example: you can see following kind of Trace in the Server/Stdout Logs:
<Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias DemoIdentity from the jks keystore file D:ORACLEMIDDLE~1WLSERV~1.3serverlibDemoIdentity.jks.>
Exception in thread "[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
What to do in case of OutOfMemory In JavaHeap?
Whenever we see an OutOfMemory in the server log or in the stdout of the server. We must try to do the following things as first aid steps:
.
Point-1). If possible enable the following JAVA_OPTIONS in the server start Scripts to get the informations of the Garbage Collection status.
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/opt/app/GCLogsDirectory/gc.log
.
Point-2). It is always needed to see what all objects were present when the OutOfMemory error occured to identify whether those objects belongs to the Application Code/ Application Framework Codes/ The Application Server APIs. Sothat we can isolate the issue. In order to get the details of the Heap Objects collect “HeapDump” either using JHat (not a better tool) or JMap (Much Better compared to the Jhat tool). Plese refer to the post to know how to do it :
http://middlewaremagic.com/weblogic/?p=2241
.
Point-3). Once we collected the Heap Dump we can easily monitor the Heap Details using best GUI toold like “Jhat Web Browser” or using “Eclipse Memory Analyzer”.
OutOfMemoryError: GC overhead limit exceeded?
The “GC overhead limit exceeded “ indicates that, more than 98% of the total time is spent doing garbage collection and less than 2% of the heap is recovered.
The “GC overhead limit exceeded” in general represent the following cause:
Point-1). When the heap is too small or the current size might not be suitable for your application. Try increasing the -Xmx value while starting your process.
Point-2). There might be a memory leak which means a particular kind of object might be getting created again and again but might not be getting garbage collected due to a leak in the code (application code/ third party application code, Application Server code leak, Or it may be a JVM memory leak).
Point-3). The old generation size of the heap might be very small compared to the new generation. So that the object might be getting passed to the Old Generation prematurely. And we know that GC happens less frequently in Old Generation compared to the Young Generation.
Point-4). If increasing the Heap size (-Xmx) OR tuning the Old Generation size does not help then it might be a memory leak in the application code/container code.
Better to take a heap dump and see what kind of objects are getting filled up inside the Heap, That will indicate which might be leaking or if the heap size is sufficient or not.
What to do in case of Native OutOfMemory?
Point-1). Usually Native OutOfMemory causes Server/JVM Crash. So it is always recommended to apply the following JAVA_OPTIONS flags in the Server Start Script to instruct the JVM to generate the HeapDump “-XX:+HeapDumpOnOutOfMemoryError“
By default the heap dump is created in a file called java_pidpid.hprof in the working directory of the VM, as in the example above. You can specify an alternative file name or directory with the “-XX:HeapDumpPath=C:/someLocation/“
.
Note: Above Flags are also suitable to collect HeapDump in case of JavaHeap OutOfMemory as well. But these flags never gurantees that the JVM will always generate the Heap Dump in case of any OutOfMemory Situation.
.
Point-2). Usually in case of Native OutOfMemory a “hs_err_pid.log” file is created in case of Sun JDK and “xxxx.dump” file is created in case of JRockit JDK. These log files are usually Text Files and tells about the Libraries which caused the Crash. These files need to be collected and analyzed to find out the root cause.
.
Point-3). Make Sure that the
-XX:MaxHeapSize is not set to a Very Large Space…because it will cause a very less Native Space allocation. Because as soon as we increase the HeapSize, the Native Area decreases. Please see the Post:
http://middlewaremagic.com/weblogic/?p=4456
.
Point-4). Keep Monitoring the process’s memory using the Unix utility ‘ps’ like following:
ps -p <PID> -o vsz
Here you need to pass the WebLogic Server’s PID (Process ID) to get it’s Threading Details with respect to the Virtual Memory Space.
.
Point-5). If the Heap Usages is less Or if you see that Your Application usages less Heap Memory then it is always better to reduls the MaxHeapSize so that the Native Area will automatically gets increased.
.
Point-6). Sometimes the JVMs code optimization causes Native OutOfMemory or the Crash…So in this case we can disable the Code Optimization feature of JVM.
(Note: disabling the Code Optimization of JVM will decrease the Performance of JVM)
For JRockit JVM Code Optimization can be disabled using JAVA_OPTION –Xnoopt
For Sun JDK Code Optimization can be disabled using JAVA_OPTION -Xint
What to do in case of OutOfMemory In PermGen?
Point-1). Make Sure that the PermGen Area is not set to a very less value.
.
Point-2). Usually if an Application has Many JSP Pages in that case every JSP will be converted to a *.class file before JSP Request Process. So a large number of JSPs causes generation of a Large number of *.class files all these classes gets loaded in the PermGen area.
.
Point-3). There is no standard formula to say which value of MaxPermSize will suit your requirement. This is because it completely depends on the kind of framework,APIs, number of JSPs…etc you are using in your application. The number of class which has to be loaded will vary based on that. but if you want to really tune the MaxPermSize then you should first start with some base value like 512M or 256M and then If you still get the OutOfMemory then please follow below instruction to troubleshoot it.
Point-4). If you are repeatedly getting the OutOfMemory in PermGen space then it could be a Classloader leak….
May be some of the classes are not being unloaded from the permgen area of JVM . So please try to increase the -XX:MaxPermSize=512M or little more and see if it goes away.
If not then add the following JAVA_OPTIONS to trace the classloading and unloading to find out the root cause :
-XX:+TraceClassloading and -XX:+TraceClassUnloading
Point-5).
If users want to investigate which kind of classes are consuming more PermGen space then we can use the “$JAVA_HOME/bin/jmap” utility as following:
$JAVA_HOME/bin/jmap -permstat $PID >& permstat.out
Above utility will dump the list of classes loaded in that JVM process Process (we are passing the processID to this command as $PID). This helps us in understanding if there is any classloader leak or if a particular class is consuming more memory in PermGen…etc Collecting HeapDump also gives a good idea on this.
.
.
Thanks
Jay SenSharma
November 21st, 2010 on 12:12 am
Thanks a lot for the post Jay , it helped a lot in clarifying basics on memory issues.
i will definitely bring more doubts soon 🙂
Thanks
Sanjib’
November 21st, 2010 on 11:57 am
Jay,
If I have a 30GB RAM machine.
Current heap size specified is Xmx=1024.
Now I want to deploy new application, Accordingly I have to increase the heap Size.
To What value should I increase the Heap Size?
How should i decide the value?
November 21st, 2010 on 12:23 pm
Hi Sathya,
The Theoretical limitation of Maximum Heap size for a 32 bit JVM is upto 4GB. Because of the Memory Fragmentation, Kernel Space Addressing, Swap memory usages and the Virtual Machine Overheads are some factors which doesnot allow us the allocate whole 4GB memory for Heap in a 32 bit JVM. So usually on 32-bit Windows Operating Systems the Maximum can be from 1.4 GB to 1.6 GB.
If we want a Larger memory allocation according to our application requirement then we must choose the 64-bit operating systems with 64 bit JVM. 64-bit JVM provides us a larger address space. So we can have much larger Java Heap with the increased number of Threads allocation area. Based on the Nature of your Operating system in a 64 bit JVM you can even set the Maximum Heap size upto 32GB.
Example: -Xms32g -Xmx32g -Xmn4g
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 21st, 2010 on 5:57 pm
Jay,
Is there any formula kind for calculating, Heap Size?
November 21st, 2010 on 7:48 pm
Hi Sathya,
There is no formula available for calculating the Heap Size. It totally depends on the Nature of Application, Operating System, OS Architecture, Number of Users Requests…etc. The Actual Heap Value is always environment specific which may vary with the change of the environment. Thats why there is no Standard/Formula available for Deciding what should be the HeapSize.
The Performance Testing Team and the ApplicationServer Administrators need to find the exact value for their environment for the heap size by looking at the Performance Testing Data.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
November 21st, 2010 on 11:22 pm
Again, Thanks Jay..
December 15th, 2010 on 2:54 pm
Jay,
This is regarding JSP pages may cause out of memory. Can we use the precompile option in weblogic.xml so that at least jsp pages does not get cause exhaustion of premGen and native memory at RUN time.
Will this option be helpful ? What could be the side effect on using this option.
Thanks
Sumit
December 15th, 2010 on 4:06 pm
Hi Sumit,
As long as we have many class to be loaded in the Permanent Generation there is a Chance of OutOfMemory in PermGen, because if the Number of classes are very large and the Permgen Size is less comparatively then we will get OutOfMemory in that memory area.
Even if we precompile the JSPs ultimately these JSPs will be converted into classes so if the Number of these classes is very high then there are high chances of Permgen OutOfMemoryError if we have set the Permanent Generatent Generation Size very low.
So the ase line is ….If we have many classes (either precompiled or Post compiled) to be loaded then we must have the have Sufficient Permanent Generation Size.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
December 15th, 2010 on 4:25 pm
Jay,
What is my point here is .
1) At the time of server start itself all our JSP’s gets pre-complied with this option [ Assuming sufficient memory for per-compilation ].
2) Then at RUN time these JSP will not get complied . So if they will not be required to compile at RUN time , then these classes will not load into PremGen.[ Correct me if i am wrong on this point].
Thanks
Sumit
December 15th, 2010 on 4:31 pm
Hi Sumit,
Classes will be always classes whether you precompile then using jspc or appc or any other utility …These classes will be ultimately loaded into the Permanent generation Only. So it really doesnt matter much whether we precompile the JSPs or not. Pre-compiling JSPs will save servers extra effort to compile the JSPs as soon as they receive the FIRST request for that JSP page. Once the JSPs are compiled the compiled version of JSP classes will go inside the Permgen only.
So the Point here is “If you have many classes to be loaded then increase/tune the Permanent generation size accordingly”
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
December 15th, 2010 on 4:54 pm
Thanks Jay,
So the JSP’s alone may not be the problem alone.Its the total number of class files which will be there in the PermGen at a particular time.
So as soon as there is no reference to these classes, the GC will clean them.This type of GC is minor or major?
Thanks
Sumit
December 15th, 2010 on 8:18 pm
Hi Sumit,
Not Sure whether it is Minor or major GC….I just know it is GC.
December 29th, 2010 on 1:08 pm
Hi Jay ,
I have a qucik question on JVM .
– How the physical RAM will be used by JVM in a weblogic domain .
Fror Eg : I have a weblogci domian with one admin server and 4 manage servers . I have setup memory args with min max as 1G . The physical server has 8GB RAM .
– Now how the distribution of the RAM happens to the total 5 JVMs in this domain .
Just want to know the process .
thanks,
Kiran
December 30th, 2010 on 7:23 am
Hi Jay ,
Can you please advice me on the above comment regarding JVM memory usage and allocation .
thanks,
Kiran
December 30th, 2010 on 10:17 am
Hi Kiran,
Prerequisit: http://middlewaremagic.com/weblogic/?p=4456
It depends on the Operating System and it’s architecture like 32 bit or 64 bit. Java Process Size will be around 2 GB for a 32 bit Operating system as default. So if you allocate 1 GB of Heap Memory then rest 1 GB of that Java process will be allocated for Native Memory of that Java Process.
So if for 8 GB RAM if you are planning to run 5 JVMs with 1 GB Heap size (1 GB Native) each then Operating system may start using the Paging (Swapping of Memory) to manage your servers run. How the Operating System will swap and how efficiently it will swap the memory it depends on the Operating System Configuration and tuning.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
December 30th, 2010 on 3:38 pm
Hi jay ,
Here is the config :
OS : Sun solaris 10 64 bit version
RAM:8GB
Domains : 1 domain with 5 JVMS( Admin + 4 manage servers )
if i set the memory args to 1GB , How much memory will be utilized .
thanks,
Kiran
December 30th, 2010 on 5:52 pm
There is no such Formula ….to calculate how much memory will be used…
Because Memory Allocation happens in the Runtime….
.
.
Thanks
Jay SenSharma
January 3rd, 2011 on 9:45 am
Hi Jay .
Thats fine . Is there any tool to find out how much memory is being used in a server ( Solaris ) especially for weblogic domains .
thanks,
Kiran
January 3rd, 2011 on 11:02 pm
Hi Kiran,
At Operating System level if you will try to findout the tool then you won’t get much in deapth details about the memory consumed by different part of JVM. Because Operating System Utilities only tells about the total memory utilization by a Process (for example a Java Process)…But it won’t tell anything about the Memory utilized by different Areas or parts of the Java Process…Like Heap Utilization, Native Memory Utilization, Permanent Generation Memory Utilization…
So if you want to analyze them in that case you will need to install some Memory Profilers like JProbe, Jconsole (JConsole comes with JDK you can find it insode ‘bin’ directory of the JDK), JProfiler, YourKit …etc.
But as these Profilers consumes a lots of Server resources and thus degrades the performance of the over all server…So these tools are not recommended for the Production Environments. Because they do a lots of profiling which doesnot even required.
So if you want to just anayze the memory utilization part of your Server (JVM Process) then JMX is the best way which allows us to selectively analyze (or to collect runtime statistics) whatever actually we want.
http://middlewaremagic.com/weblogic/?p=906
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
January 6th, 2011 on 2:08 am
hi Jay,
I want to how m/y is allocated to JVM?(i mean process) plz explain by example let say by Kiran’s question i.e. I have a weblogci domian with one admin server and 4 manage servers . I have setup memory args with min max as 1G . The physical server has 8GB RAM .
– Now how the distribution of the RAM happens to the total 5 JVMs in this domain .
i cudnt get ur comment compeletely : “if for 8 GB RAM if you are planning to run 5 JVMs with 1 GB Heap size (1 GB Native) each then Operating system may start using the Paging (Swapping of Memory) to manage your servers run”. plz explain…thanx in advance
January 6th, 2011 on 3:24 am
Hi Jay,
how to do analysis of core dump as its also in bin format? is their any tool for that or we shud use the same procedure which we use for heap dump.
thanx in advance…
January 6th, 2011 on 9:28 am
Hi Abhishek,
Core Dump analysis is a very vast topic. As the core dump is created by the Operating System whenever any process suddenly crashes. There is no tutorial or article available on internet which tells exactly how to analyze the Core Dumps, The Main reason behind this
1). Core dump is generated By Operating System which contains operating system related Loaded Libraries information which is different for every operating System.
2). Different JDKs which are installed in different Operating Systems contains different Native Library sets.
3). The addressing is different for 32 bit and 64 bit JDKs and Operating Systems.
4). Most of the time the core dump analysis is done only in the Physical box where it got generated because sometimes OS level incorrect patches also causes the Crash.
Due to all these reasons all the OS vendors suggests that whenever a Core Dump is produced it should be investigated by the Operating System Vendor Support Team. So please do not think of investigating the Core Dump files. Operating System has different tools like gdb, dbx…etc which is usually used to investigate the Core dump files but these tools varies from Operating System to Operating System. Even middleware Support Engineers do not analyze the Core Dumps because it has to be investigated the Operating System vendor Support Team.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
January 31st, 2011 on 6:49 am
Hi Jay,
I have a 8Gb Ram and one of the application is set to heap of -xms 1024m and -xmx 1024m, Now we are planning to increase the Heap to -Xmx2048… But there is considerations that other process on the machine should not get affected.. Please suggest… From where to start?
January 31st, 2011 on 10:57 pm
Guys,
I am using WebLogic Server Version: 10.3.0.0.This is a dev environment with 2 clusters.
This is my situation:
Frequently getting Out-Of-Memory errors in managed server console logs and even the managed servers are restarting frequently?? Anyone faced this issue before??
I checked the state of the managed servers and this is what I have now:
Managed_server1–> Force shutting down
Managed_server2—>Failed_not_restarttable
How do I tackle this issue?
Thanks,
Prasad
January 31st, 2011 on 11:05 pm
Hi Pradeep,
OutOfMemory has various types…like Native OutOfMemory, Java Heap OutOfMemory…PermGenOutOfMemory…etc So we need to check what kind of OOM are you facing. Also please let us know the MEM_ARGS settings of your Server.
It will be best if you can provide us the “gc.log” and the Server logs of the servers wherever you are faing OutOfMemory that will help more to analyze this issue. You can mail us your Logs in “contact@middlewaremagic.com”
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
January 31st, 2011 on 11:55 pm
Hi Jay,
This is my setting:
/opt/bea/jrockit_160_11/bin/java -jrockit -Xms2560m -Xmx2560m -Xverbose:memory
I have also forwarded the logs to contact@middlewaremagic.com.
Appreciate your help! Meanwhile even I am going through oracle forums as well as other postings in middlewaremagic just to check if someone else has faced this issue before!
Thanks,
Pradeep
February 1st, 2011 on 3:15 am
This is what I see with tail -f of console logs:
Exception in thread “RMI RenewClean-[10.101.110.28:40825]” java.lang.OutOfMemoryError: class allocation, 172056720 loaded, 159121408 footprint JVM@check_alloc (src/jvm/model/classload/classalloc.c:118). 356 bytes r
at sun.rmi.transport.DGCClient$EndpointEntry.processPhantomRefs(DGCClient.java:602)
at sun.rmi.transport.DGCClient$EndpointEntry.access$1400(DGCClient.java:153)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:537)
at java.lang.Thread.run(Thread.java:619)
[INFO ][memory ] 237243.936-237244.696: GC 612352K->345432K (2621440K), 760.371 ms
[INFO ][memory ] 240845.264-240847.608: GC 573561K->344925K (2621440K), 2343.363 ms
[INFO ][memory ] 244448.172-244448.876: GC 572821K->343419K (2621440K), 703.887 ms
[INFO ][memory ] 248049.431-248050.962: GC 571580K->343344K (2621440K), 1531.349 ms
[INFO ][memory ] 251651.519-251652.082: GC 571580K->343015K (2621440K), 562.874 ms
February 1st, 2011 on 10:59 pm
Hi Cappradeep,
For data integrity purpose i have changed the IP Address of your Server mentioned in above stackTrace.
The Error “java.lang.OutOfMemoryError: class allocation” which sometimes happens due to dynamic Class Loading and Creation. It’s a kind of Class Loader Leak. Please refer ot the following link for more details: http://middlewaremagic.com/weblogic/?p=2477#comment-2701
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
February 2nd, 2011 on 2:16 am
So are you saying that there is a possibility of memory leak in this case? Should I follow the same debug steps that you suggested adrian?
Thanks,
Prasad
February 2nd, 2011 on 11:17 am
Hi Cappradeep,
Yes i am suspecting some kind og classloading leak (class loading happens in Native Area in JRockit)….or else you need to decrease the Max heap size so that the Native Area will get more space to expand.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
February 2nd, 2011 on 10:00 pm
Thanks Jay. I will follow these steps and let you know the outcome!
Lot of things on my plate now! This is a dev server so I have couple more days to fix it. But I want to solve this soon.
I will keep you updated!
Thanks again!
February 8th, 2011 on 1:09 am
Update: Developer is checking the code again and causes for memory leak. I will keep you posted.
Thanks,
Prasad
May 23rd, 2011 on 9:32 pm
very nicely written…keep up the good work…
May 27th, 2011 on 9:33 pm
May be its more dumb Question but i am confused lot because of reading diff diff documentation, may be. When my domain has 1 admin sever and 1 managed server how many JVMs are running. The statements like ‘Oracle Service Bus 11gR1 (11.1.1.3) supports running in same JVM as SOA’ made me feel that all the servers in single domain uses the same JVM.
Please clarify on this JVM stuff..
Thanks
Siva
May 28th, 2011 on 6:26 pm
Hi Siva,
When you are running an AdminServer and 1 Managed server on the same machine then there are two JVM’s running, one been used by AdminServer and Second would be used by the Managed server having there own memory arguments, JVM flags.
Thanks,
Ravish Mody
May 30th, 2011 on 6:08 pm
Hi Middleware Magic Team,
I have a small doubt,i just started using jstat tool
the output is as follows
bash-3.00$ jstat -gcutil 22475 1 1
S0 S1 E O P YGC YGCT FGC FGCT GCT
100.00 0.00 42.56 82.87 67.23 3182 100.742 45 12.884 113.626
My YGCT is much more then that of FGCT..which i guess is not an idle case,Even my FGC is taking lot of time.
I have set Arguments -Xms1024m -Xmx1024m -XX:MaxPermSize=256m,is it necessary to set -Xmn as well..if yes,is there any default percentage for -Xmn ie whether it should be 50% or 60% of that of Java heap size.
Please help,Thanks in advance
Regards
Fabian
May 30th, 2011 on 8:23 pm
Hi Fabian,
By default NewRation is (-XX:NewRatio=2) Ratio of new/old generation sizes. Explicitly you can define the NewSize using (-XX:NewSize=) parameter and -XX:MaxNewSize Parameters. The Minor Garbage Collection keep happens on the Young Generation. So we need not to worry about that usually. If on Regular basis/Continuously Full Garbage Collection takes more than 5-6 Seconds or above for 2 GB heap Size or If Full Garbage Collection takes more than 9-10 Seconds for 4 GB heap space then we should check the Garbage Collection Log and the Heap Dump Or we should use some Profilers like JConsole.
So please collect the Full Garbage Collection Logs and then Analyze it using Garbage Cat tool (in Few Seconds you can get the complete GC report) Please refer to the following article:
http://middlewaremagic.com/weblogic/?p=5131
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
August 4th, 2011 on 11:58 am
Hi Jay/Ravish,
We have generated the Heap info using JMAP utility and below is the complete stack trace for the heap on peck time.
We have total of 5GB heap and out of this we set min as Max as 5GB. Also we conf’ed permgen space as 512MB & newsize (Xmn) as 2GB & new raatio =3.
As per below statistics, New generation value is almost 100%. If we reaches this value, did we get any memory issues?
Could you please kindly explain me what will happen in this scenario. How we can taken care this?
Also do we need to increase the heap on this case?
Please assist on this.
<>
With regards,
Gopal
August 4th, 2011 on 12:05 pm
[ sourcecode language=”java” wraplines=”false” ]
Heap info missed out in above post. Here is the deatils.
using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 5368709120 (5120.0MB)
NewSize = 2147483648 (2048.0MB)
MaxNewSize = 2147483648 (2048.0MB)
OldSize = 5439488 (5.1875MB)
NewRatio = 3
SurvivorRatio = 8
PermSize = 536870912 (512.0MB)
MaxPermSize = 536870912 (512.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 1932787712 (1843.25MB)
used = 1932387280 (1842.8681182861328MB)
free = 400432 (0.3818817138671875MB)
99.9792821530521% used
Eden Space:
capacity = 1718091776 (1638.5MB)
used = 1718091776 (1638.5MB)
free = 0 (0.0MB)
100.0% used
From Space:
capacity = 214695936 (204.75MB)
used = 214295504 (204.3681182861328MB)
free = 400432 (0.3818817138671875MB)
99.81348878443605% used
To Space:
capacity = 214695936 (204.75MB)
used = 0 (0.0MB)
free = 214695936 (204.75MB)
0.0% used
concurrent mark-sweep generation:
capacity = 3221225472 (3072.0MB)
used = 1755007824 (1673.7058868408203MB)
free = 1466217648 (1398.2941131591797MB)
54.48261350393295% used
Perm Generation:
capacity = 536870912 (512.0MB)
used = 104103336 (99.28067779541016MB)
free = 432767576 (412.71932220458984MB)
19.390757381916046% used
[ /sourcecode ]
August 9th, 2011 on 2:46 pm
could you help me with above my post?
Also I wanted to know.. how much memeory i can use any any of the SOlaris 10 box.
example:: I have 22MB RAM and have 5 app instances. how mcuh memeory I can use for Heap out of 22MB?
With regards,
Gopal
August 9th, 2011 on 4:22 pm
Hi Gopal,
There is no standard formula which can be used to provide the exact values for the JVM tuning parameters because it totally depends on various factor like Operating System, Load on the Server, Nature of the application like which kind of framework we are using (Spring, Swing, Clustered/Non Clustered, EJBs based, Struts etc), Which kind of objects are created by the application like Long living objects or Short Living Objects and their ration, Whether application uses Native implementations or not like JNI, How many concurrent users can access the application concurrently and how much Average Min and Max duration users interacts with the application…. etc.
So as these various factors vary environment to environment basis, the only solution to find out the exact tuning values will be to perform a load testing on the server. If you face any issue like OutOfMemory or any kind of error/exception then it can be analysed. But without load testing report or issues it is not possible to suggest the tuning parameter values.
Please refer to the following articles to get more insight on tuning the JVM :
http://middlewaremagic.com/weblogic/?page_id=2261
And
http://middlewaremagic.com/weblogic/?p=4456
And
http://middlewaremagic.com/weblogic/?p=4500
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
October 4th, 2011 on 8:53 pm
Hi
i have a problem in a application based in ejb2 run in weblogic 8.1. In 3 weeks or maybe a month the weblogic falls down with problem of OutOfMemory. It runs with JDk1.4 Sun. We passed an analizer memory as JProbe and Memory Analizer, and detect a big occupation in memory of the clases java.lang.ref.Finalizer, weblogic.ejb20.manager.DBManager but i dont detect the problem. Please help me. Thanks.
October 4th, 2011 on 9:08 pm
Hi Aggn007,
The very first thing is “This is not recommended to run any Server instance continuously for 3 weeks”. Because the most well written application servers can also fall into OutOfMemory issues if they are not restarted in a feasible interval. So it would be better if you have a clustered environment then you can shutdown and restart your WLS when the load on the server is not very high or can have a scheduled maintenance period in a weeks time.
Regarding the OutOfMemory on hava.lang.ref.Finalizer on “”weblogic.ejb20.manager.DBManager” is being shown by your monitoring tools because DBManager class has a method “flushModified()” to clean the flushed data. When it is invoked then the java.lang.ref.Finalizer get activated by the JVM to perform the cleanup operation. But again i really don;t think that it is good to run a server for more than 21 days continuously.
This is where we actually need a clustered setup so that once in a while we can restart any of the Cluster Node without interrupting clients requests and Can perform a Rolling Bounce on the Servers (means bringing one server down & restarted at a time while others will be running)
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
April 18th, 2012 on 4:23 pm
Hi Jay
We have an out of memory issue in our dev server, please find the environment details below :
JAVA Memory arguments: -Xms512m -Xmx2048m -Xss256k -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled
java.specification.vendor = Sun Microsystems Inc.
java.specification.version = 1.6
java.vendor = Sun Microsystems Inc.
java.vendor.url = http://java.sun.com/
java.vendor.url.bug = http://java.sun.com/cgi-bin/bugreport.cgi
java.version = 1.6.0_20
java.vm.info = mixed mode
java.vm.name = OpenJDK 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Sun Microsystems Inc.
java.vm.specification.version = 1.0
java.vm.vendor = Sun Microsystems Inc.
java.vm.version = 19.0-b09
Server Log stdout :
******************(Exception started from this line of log)
Exception in thread “[ACTIVE] ExecuteThread: ‘4’ for queue: ‘weblogic.kernel.Default (self-tuning)'” java.lang.OutOfMemoryError: PermGen space
Can you please suggest what should be the root cause for this and how we can handle it.
It is an urgent , Thanks in advance.
Thanks & Regards
Ramkumar R
July 26th, 2012 on 12:52 pm
hi jay,
how to identified out the OOM exception & how to solve this OOM exception error? Please explain this one
Thanks,
Ram