Hi,
Sometime we find Session Replication related Issues while using the Clustering Features of Weblogic. like: <BEA-000126> <All session objects should be serializable to replicate. Check the objects in your session. Failed to replicate non-serializable object.>
USEFUL Debuging techinques:
Enable the Debug Flags to track Session Replication Failures:
You could enable the flags DebugCluster, DebugClusterAnnouncements, DebugFailOver, DebugReplication, DebugReplicationDetails.
To Enable: You can use the weblogic.Admin command line utility to dynamically turn the debug options on and off.
For example, to turn on DebugCluster on all administration instances of ServerDebug Mbean (i.e., Admin Server or al Managed Server):
java weblogic.Admin -url t3://localhost:7001 -username system -password weblogic SET -type ServerDebug -property DebugCluster true
***Symptoms***:
1). After Logging into the Application which is deployed on WebLogic Cluster… Users suddenly finds themself to be redirected to Login page again even if they haven’t clicked on Logout Button.
2). In these cases There may be many reasons behind this…There may be Multicast Issues(Network Issues) in the Network where the WebLogic Clusters are running. Means Managed Servers which is part of the Cluster are not able to send Heartbeat messages to eachother. In these kind of scenario first of all we need to Apply the Multicast Test Utility provided as part of WebLogic. For more Informations you can refer to “http://download-llnw.oracle.com/docs/cd/E13222_01/wls/docs81/admin_ref/utils25.html” Multicast Util Informations.
3). Getting java.io.NotSerializableException” in Server Logs
4). May be the Default Session Cookie Name is different in “weblogic.xml” and It’s not same as the Web Proxy Default Cookie Name.
Parameter | Default | Description | Applicable to |
---|
CookieName |
JSESSIONID |
If you change the name of the WebLogic Server session cookie in the WebLogic Server Web application, you need to change the CookieName parameter in the plug-in to the same value. The name of the WebLogic session cookie is set in the WebLogic-specific deployment descriptor, in the <session-descriptor>element.Note: The CookieName parameter has been renamed as WLCookieName . A warning message is issued if you continue to use the old parameter name. |
NSAPI, ISAPI, and Apache plug-in, HttpClusterServlet, and HttpProxyServlet |
In most of the Cases i have observed that Users get Logged off from our applications which is targeted to a Cluster .is because of the Application. If an application is storing some NonSerializable Data Objects inside the HttpSession as an attribute then the WebLogic Cluster is not able to replicate these Non-Serializable data to the replicated HttpSession on another Secondary Managed Servers. For a Large scale applications it is not an easy task for developers to check the complete code again…sothat they can identify that all the HttpSession Attributes are implementing the java.io.Serializable interface or not.
Here is a simple JSP page which is able to display all the HttpSession attributes in a tabular format along with the report that Which Session Attribute is Not Serializable. We need to Just bundle this JSP inside our existing application which is causing the issue. Then we need to login to our application we need to access this additional page “SerializationTest.jsp” sothat if any HttpSession attribute is already stored inside the HttpSession then it can display it’s complete information.
***NOTE:
Here i am providing a complete TestCase for this JSP…to demonstrate How actually it works..But you need to add only the “SerializationTest.jsp” page inside your application which is causing the issue
“SerializationTest.jsp”
—————————-
<%@ page import=”java.util.*” %>
<table border=10%>
<TR><TH>Session Attribute Name</TH><TH>Session Attribute Value</TH><TH>STATUS</TH></TR>
<%
Enumeration en=session.getAttributeNames();
while(en.hasMoreElements())
{
String attrName=(String)en.nextElement();
if(session.getAttribute(attrName) instanceof java.io.Serializable)
out.println(“<TR><TD>”+attrName+”</TD><TD>”+session.getAttribute(attrName)+”</TD><TD><font color=green> SERIALIZABLE</font></TD></TR>”);
else
out.println(“<TR><TD>”+attrName+”</TD><TD>”+session.getAttribute(attrName)+”</TD><TD><font color=red><b>NOT SERIALIZABLE</b></font></TD></TR>”);
}
%>
</table>
—————————-
*********Complete TestCase Below:**********
Just to know how it actually works Just develop a Sample Web Application with the following details
Step1). Develop a classes “Emp.java” here Emp class should NOT implement the “java.io.Serializable” interface.
package pack;
public class Emp
{
String name=”Default Name”;
public String getName()
{
return this.name;
}
public void setName(String name)
{
this.name=name;
}
}
Step2). Develop a classes “Salesman.java” here Emp class should implement the “java.io.Serializable” interface.
package pack;
public class Salesman implements java.io.Serializable
{
String name=”Default Salesman Name”;
public String getName()
{
return this.name;
}
public void setName(String name)
{
this.name=name;
}
}
Step3). Compile the above two classes and place them inside your WebApplications “WEB-INF/classes” directory
D:DELETESessionTestWEB-INFclasses> javac -d . Emp.java
D:DELETESessionTestWEB-INFclasses> javac -d . Salesman.java
Step4). Now provide the “index.jsp” Page as below:
“index.jsp”
————————
<%
session.setAttribute(“EmpAttribute”,new pack.Emp());
session.setAttribute(“SalesmanAttribute”,new pack.Salesman());
%>
<center>
<h2> Session Object Created with the Following Attributes: </h2><BR>
</center>
<h4>
<BR>
session.setAttribute(“EmpAttribute”,new pack.Emp());
<BR>
session.setAttribute(“SalesmanAttribute”,new pack.Salesman());
<BR><BR><BR>
<center>
<a href=”SerializationTest.jsp”> Click Here To Check Which one is Serializable</a>
</center>
————————
Step5). Now provide the Sample “web.xml” we don’t need to enter much information in this file …as it is just a TestCase.
<web-app xmlns=”http://java.sun.com/xml/ns/j2ee“>
</web-app>
Step6). Now If you want to enable Session Replication for your WebApplication then provide the “weblogic.xml” file like below…(In our TestCase at present this is not required…as we are just going to test The HttpSession attribute is Serializable or Not. it can be tested in a NonClustered Managed Server as well)
<?xml version=”1.0″ encoding=”UTF-8″?>
<weblogic-web-app xmlns=”http://www.bea.com/ns/weblogic/90“>
<session-descriptor>
<persistent-store-type>replicated</persistent-store-type>
</session-descriptor>
</weblogic-web-app>
Step7). Now Deploy your Application in WebLogic Server…
Step 8). Access the “index.jsp” page of your application then you will find the following output
NOTE:
The HttpSession replication happens Only when WebLogic creates a New HttpSession or Container encounters “session.setAttribute(obj,obj)” method…It means the Data is replicated to the Secondary JVM HttpSession only when application calls the setAttribute method of HttpSession. And in this case only the modified part of data is replicated NOT the complete HttpSession data.
So i think u can try adding a Simple temporary attribute in the HttpSession like
session.setAttribute(“dummyData”,””+new java.util.Date());
Now make sure that the above code is placed in each and every page of your application…sothat whenever a client will request the because of setAttribute() the Session modified data (dummyData) will be replicated…and if there are more setAttribute() on that page then all the those also will be replicated…
Example:
Map map = new HashMap();
session.setAttribute(“map”, map); ——-> Here Session data Will be replicated to Node-2 (bcoz of setAttribute())
map.put(“one”, “aaa”); No Replication (Means No Session data Refresh…bcoz setAttribute is not called…) then from page1.jsp I go to page2.jsp and I do this:
Map map = (Map)session.getAttribute(“map”);
map.put(“two”, “bbb”); No Replication (Means No Session data Refresh…bcoz setAttribute is not called…)
and then from page2.jsp I go to page3.jsp and I do this:
Map map = (Map)session.getAttribute(“map”);
map.put(“three”, “ccc”); No Replication (Means No Session data Refresh…bcoz setAttribute is not called…)
.
.
Thanks
Jay SenSharma
June 7th, 2010 on 4:50 pm
Hi Jay,
I am not able to get the session replication done.
When I check the web logic server logs then I can see the errors like: No secondary server… JsessionId none!
I have a 2 node clustered environment.
In my weblogic.xml I have placed an entry for
how to proceed with this.
Thanks in Advance!!
Vicky
June 8th, 2010 on 12:16 pm
any pointers in this, will be of real help.
Thanks
Vicky
June 8th, 2010 on 1:31 pm
Hi Vicky,
Please provide the informations like:
1). Your “weblogic.xml”
2). How Many Servers are there in Cluster.
3). Are u using any Proxy Server or any Load balancer in front of the cluster?
4). What is the entry in the Server Logs and the Proxy Logs at the same time stamp …better if u provide the complete logs. What was the activity happening (Server State) on Other cluster members Server Logs. Do u see any kind of n/w Disconnect or Multicast related messages in your server logs?
Your previous post is not showing the XML content which u tried to paste in your query….So please provide the details in “weblogicwonders@yahoo.in”
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
August 26th, 2010 on 10:08 pm
Hi Jay,
We are getting but we don’t see a stack trace in WL log (not in server not in domain logs)
But if I start WL from command line – I can see error and stacktrace in Windows Console. So how do I get the stacktrace in any of the logs?
Thanks
August 26th, 2010 on 10:23 pm
Hi Lach_v,
You are able to see the eror in the Windows Command Prompt (console) it means the errors are Standard Output Errors…so if you want to capture those errors in a Separate file then please do the following:
Open the “startWebLogic.cmd” (may be your custom server start script as well) and then add the following JAVA_OPTION
-Dweblogic.Stdout=”stdout-filename” and -Dweblogic.Stderr=”stderr-filename”
above JAVA_OPTIONS will redirect all the Stdout and stderror in a separate file.
Example: following lines in your “startWebLogic.cmd”
set SAVE_JAVA_OPTIONS=%JAVA_OPTIONS% -Dweblogic.Stdout=MyStdout.txt -Dweblogic.Stderr=MyStdErr.txt
once you will start your server now then u will see the above files created in <DOMAIN_HOME> location
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
October 8th, 2010 on 5:22 am
Hi Jay,
I have 2 Managed Server but some time we are getting the session replication issue , we have a load balancer infront of the 2 managed server.
this is the error we get , we were migration from OAS to WLS Server, In OAS session replication is there by default.
can you please suggest ?
<Error looking up session with id:vQLgM1yMmZLJrJsTYQg93JfyRx3pbCZzkxMwVhvl8Tdm7hQ12V2v!1187848188!634421960
java.rmi.MarshalException: error marshalling return; nested exception is:
java.io.NotSerializableException: com.agile.ui.pcm.common.treegrid.RowNode
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
at weblogic.cluster.replication.ReplicationManager_1030_WLStub.fetch(Unknown Source)
at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
Truncated. see log file for complete stacktrace
java.io.NotSerializableException: com.agile.ui.pcm.common.treegrid.RowNode
at java.lang.Throwable.(Throwable.java:67)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1089)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:332)
at java.util.HashMap.writeObject(HashMap.java:848)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
Truncated. see log file for complete stacktrace
>
March 6th, 2011 on 9:11 pm
hi sarath,
first of all full stack trace is required! any way by looking at the given trace I understand the problem is with serialization. Ensure that the code you are migrating follow SUN STANDARDS. Particularly com.agile.ui.pcm.common.treegrid.RowNode must implement serializable interface. I look this as a coding problem.
regards,
sri…
March 6th, 2011 on 11:06 pm
Hi Sri,
As you have helped Sarath with his issue and gave him a pointer we are glad to give you additional 20 Magic Points .
Keep posting 🙂
April 19th, 2011 on 3:08 pm
Hi Jay/Ravish,
What type of data you can replicate in session replication. Is session replication is possible using multicast? Is automatic migration of singleton services possible using multicast?
Thanks for your help and time.
Thanks & Regards,
Sohel
April 19th, 2011 on 4:48 pm
Hi Sohel,
Multicast is only a utility and messaging type which is required for Cluster Members to send and receive their small amount of Heart Beat byte messages among the other nodes of the same cluster. Sending and receiving heart beat messages ensures that the Cluster members are alive.
Session replication happens only between Primary and Secondary servers …Suppose if there are 10 Member nodes are present in the Cluster and a Clients request comes to Node 3 then the Primary session will be created in Node-3 but the Secondary session may be created in any of the other cluster member (Secondary Node). For every unique clients request the Primary and Secondary Server combination will be different.
Only The Serializable Data Object which we place inside the HttpSession by calling session.setAttribute(“key”,object) gets replicated to the Secondary Server. Non-Serializable data objects cannot be replicated. Also refer to : http://middlewaremagic.com/weblogic/?p=290
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
April 20th, 2011 on 8:52 am
Jay,
“For every unique clients request the Primary and Secondary Server combination will be different”..
Means,
Two different clients will never have same primary and secondary servers same is it?
Thanks
Sathya
April 20th, 2011 on 9:43 am
Hi Sathya,
For two clients the Primary and Secondary Server configuration may or may not be same.
.
.
Thanks 🙂
Jay SenSharma
October 24th, 2011 on 11:39 pm
Which cluster flags of Weblogic 10.3 print the classes of non-serializable objects in HttpSession in case of failure of replication ?
DebugReplication / DebugReplicationDetails / ?
Thanks
Umesh
October 25th, 2011 on 10:16 am
Hi Umesh,
The “DebugReplication” provides little less details about the replicated objects compared to “DebugReplicationDetails” debug flag. But usually both the debug options are required in order to get the Replication debug data. the DebugReplication triggers the Debugging informations related to Replication and the next flag displays the details of it.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
January 2nd, 2012 on 11:24 am
Hi
this is swami, i am new to weblogic administration.i have some doubts.Plz suugest me to improve my skills.
January 2nd, 2012 on 11:30 am
Hi Friends
i have doubts ,please give me reply for this
1)can one domain have two nodemanagers
2)what is the difference berween unicast and multicast messaging in clusters with example.
3)what is the command to know the free memorry of jvm
4)give me some common exceptions occurs in weblogic production support, and how we solved it.
5)on what situations we will get out of memmorry exceptions
6)
January 2nd, 2012 on 12:58 pm
Hi Swami,
1) Can one domain have two nodemanagers
Answer: A Node Manager process is NOT associated with a specific WebLogic domain but with a machine. Node Manager process to control server instances in any WebLogic Server domain until the server instances reside on the same machine as the Node Manager process. Please refer to the following article for more insight:
http://middlewaremagic.com/weblogic/?p=1914
2) What is the difference berween unicast and multicast messaging in clusters with example.
Answer: Uncasting or Multicasting are two modes of cluster communication (means how the cluster members will communicate with each other). Unicast mode of communication is made as a Default option from WebLogic 10 onwards. Earlier to WLS10 the Multicasting was the only option. Unicast communication is considered as a reliable and poerformance wise better communication. Which is explained better in the following link:
http://docs.oracle.com/cd/E11035_01/wls100/cluster/features.html#wp1007001
3) What is the command to know the free memorry of jvm?
Answer: At Operating system level if you want to know what is the free memory status then you can use some Unix based OS utilities like “vmstat” and for windows you can use “TaskManager” in general.
However as JVM has different partitions like Native Space/ HeapSpace/ permSpace then it would be better if you cna use the VisualVM/JConsole kind of memory profilers in order to find the exact memory usages by various parts of the JVM. But before that we should know what is the architecture of the JVM.
So to know the archicteture of the JVM first you can refer to the following link:
http://middlewaremagic.com/weblogic/?p=4456
For JRockit you can use the JRockit JVM specific tools like JRMC as mentioned in the following link: http://middlewaremagic.com/weblogic/?page_id=7132
4) Give me some common exceptions occurs in weblogic production support, and how we solved it.
Answer: Some very common Production issues & troubleshooting steps you will see in the following link: http://middlewaremagic.com/weblogic/?page_id=2261
5) On what situations we will get out of memmorry exceptions:
Answer: Please refer to the following link: http://middlewaremagic.com/weblogic/?p=4464
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
March 29th, 2012 on 3:22 am
Hello Jay,
Is there anyway in Weblogic 10.3 to execute a particular URL say abc.do in a separate JVM ? Our Report generation URL is taking huge amount of memory. It is not throwing OOM. But memory spikes are observed when it is executed
Thanks,
Umesh
August 2nd, 2013 on 10:14 am
Hi JS,
I have a j2ee product which does NOT generate session cookie JSESSIONID. As it is a product, I don’t have the luxury of touching the code. Our proxying/load-balancing are very simple using an IIS webserver and the iisproxy.dll plug-in supplied by weblogic.
We are not in a position to buy any commercial load balancing solution just for this product alone.
But we must need session stickiness between LB and weblogic instance from the first request onwards.
Ours is a simple 2 node weblogic cluster
can you please suggest some other alternative how to achieve this ?
We are absolutely OK even if the solution is kind of Active-Passive setup. Our main intention is a successful Failover from Primary to Secondary weblogic instance. Load balancing is not of priority as this product’s functions are of very light processing.
Thanks & Rgds,
Ram
May 8th, 2018 on 9:42 pm
Hi,
We have upgraded weblogic version form 10.3.5 to 10.3.6 and jdk version from 1.6.64 to 1.6.181.
The issue is, When we restart the server, for sometime we will be able to login to the application. But after sometimes it gives Exception – session replication issue/ Serialization issue. We are working in clustered environment under 2 servers.
Kindly help to solve to solve this issue.