This is a post where I would like to share some of my experience working with Apache web server, the reason behind writing this post is because during some test we and some of customers of WLS faced this similar issue and during solving this issue got a good set of information which would be helpful for others to understand how Apache, WLS plugin and WLS works to-gather and whose fault was it to show 404 error.
Let me give you some background information about the issue, by default dynamic server list is ON hence if any of the servers goes down or if that server is only allowed to server admin requests then the list should get updated and sent back by removing that particular server, so that client would not get 404 error. This was not happing and client were getting 404 error again and again.
The reason behind this with the a solution is given below which would give you an idea how things works and hope this would help you however this information is applied to any platform and version of Oracle Weblogic Server
Issue:
When any managed server is put in “Suspended” state the Apache server continues to deliver requests to the suspended server instance which leads to a 404 http error on the client side.
Reason behind this issue?
Apache is a multi-processed and multi-threaded, which means that every process maintains it’s own server list.
So, if a response comes back from WLS with a new server list which excludes the non working servers is sent back to the process which had sent the request will update it’s own server list ONLY and will NOT share the new server list with the other processes, until a request is served by them explicitly and each of them gets there own new server list in the response.
This is how Apache is been design, hence its an Apache issue and weblogic plugin plays no part in it.
How to Solve this?
The solution is to use “worker mpm” instead of “prefork mpm” and set both StartServers and ServerLimit to 1, and then tune the number of threads within the process which might help in resolving this issue.
More details can be found in the link below:
http://httpd.apache.org/docs/2.0/mod/worker.html
Do share your experience’s with us also so that others can get benefits out of it and help them learn new stuff 🙂
Regards,
Ravish Mody
January 26th, 2011 on 10:38 pm
I conclude bu your above application that every process maintains there own list of active servers my ques is
Is dere any way to see the active server list in the processes
also could u please explain wht is prefork mpm?
January 27th, 2011 on 11:33 am
Hi Ritu,
Is dere any way to see the active server list in the processes
You can set the Debug as “ALL” and then in the proxy log you can see the server list for particular process which would be something like < 123456789 > and number of servers ip addresses
also could u please explain wht is prefork mpm?
You can go through the below link for more info about prefork mpm
http://httpd.apache.org/docs/2.0/mod/prefork.html
January 27th, 2011 on 3:58 pm
ravish can u please share the stack trace or be specific which apache logs can i look in ,i looked in access logs
image logs and apache errro logs ,were else i can look for it ,also in the satacktrace can u highlight an instance and explain ,
thanks
RItu
January 27th, 2011 on 11:07 pm
Hi Ritu,
I am talking about Apache log file, in your httpd.conf you have to specify it like these parameters to get the correct logs
Now from the below example proxy.log sample you can see that 1382912804066002 process gets its list in the line-10 same way other process might get different list if any one of the servers go down
January 27th, 2011 on 4:18 pm
Also,
is there any way to see the heartbeat produced by the
dervers in the cluster in which way they put there attendance as i am alive…
January 27th, 2011 on 11:12 pm
Hi Ritu,
This happens internal and I think it can been seen using TCP dump, however it uses internal APIs and we do not have access to it. So would request you to open a service ticket with the WLS support team to get more information on it.
February 4th, 2011 on 3:21 am
Hi Ravish,
i think the server list (on apache side) we are talking here is about the cluster members in weblogic cluster.
if any member in a cluster is dead then cluster will take care of routing the next requests to alive cluster member (am I correct ?), then why does the server list on apache side not getting updated (if any cluster member is dead) will infact affect us?
Please correct if my understanding is wrong
Thanks
Sumanth
February 4th, 2011 on 8:15 pm
Hi Sumanth,
Yes, Server List is the members in the clusters.
In the above scenario MS-1 was send a response 404 error and updates the server list to that process which had send the request (i.e. you are not a admin, hence cant access the app) thus removes the MS-1 for the list. And the same request was not sent to the MS-2 when it got 404 because when a request comes then Apache tells on which server it has to go in the header.
February 7th, 2011 on 12:47 pm
Thanks Ravish.
Not sure where to start a new thread, so posting the question here
What is the difference between JMS Vs IBM MQ Series, in the sense that why MQ Series when WebLogic already providing JMS?
One answer i can think of is JMS works if 2 application systems are on same version (say 2 systems are on Weblogic 10 MP1), if 2 systems are on different version or use different platforms (say 1 weblogic, another websphere) them JMS does not work we have to use MQSeries. Am I Correct?
can you/someone throw some light on this please?
February 7th, 2011 on 11:38 pm
Hi Sumanth,
JMS
JMS is a java based Message Oriented Middleware (MOM) API for sending/receiving messages between two or more clients.
IBM MQ Series
Is a JMS provider which has implemented JMS API’s and interfaces for a Message Oriented Middleware (MOM) same way weblogic is also a JMS provider.
Weblogic works fine with the different versions of WLS using JMS as well as if anyone is using WLS and websphere then you can use Foreign JMS to work between WLS and foreign JMS provider like websphere.
February 8th, 2011 on 12:35 am
thanks Ravish,
so there is not much a point to use MQSeries, but why do companies still use MQSeries when JMS is already provided by WebLogic?
what are the advantages of using MQSeries over WebLogic provided JMS?
February 8th, 2011 on 1:00 am
Hi Sumanth,
IBM MQ is a Message-oriented middleware (MOM). In other words a platform that provides the messaging features via MQ queues. It supports both synchronous and asynchronous messsaging and is generally used in conjunction to a specific EAI tool. This is the kind of transport you would like to use for your communication across the different applications/systems that you would like to integrate..
It’s been a long time that MQ had captured Messaging Market…But now Applications Servers like JBoss and WebLogic itself contains a very strong Messaging Module which is not less than MQ series. Whatever MQ series can do the same things can be easily achieved with these applications servers now.
MQ has some extra features like it can work independently as well means standalone, It has it’s own Message Browser to easily see and delete messages…etc.
But JMS modules are integrated in Application Servers. So if you just want to use Messaging (nothing else) then MQ is very light weight/cheap compared to Application Servers…But if you want to use Application Server functionality as well to make your Messaging more applicable then you can go with WebLogic/JBoss Messaging as well.
The Advantage of using MQ Series or a Separate Messaging is if the Application Server goes down/Crashes still you have MQ running to store your Messages… 🙂 It will be running separately and will consuming Critical Messages. It means Separation of Tasks.
.
.
Keep Posting 🙂
Thanks
Jay SenSharma
February 8th, 2011 on 1:33 am
Hi Jay,
thanks a lot
so based on your explanation the big advantage of usig MQ is, MQ is useful if we need only messaging service without any app server functionality
one more thing JMS is asynchronous messaging service compared to MQ which is both synchronous & asynchronous
so it just means MQ has captured messaging market.
regarding the storage of messages when app server crashes , we can use persistence so that messages can come online when server is back UP again..but yes..MQ allows receiver not to wait until app server comes UP
March 16th, 2011 on 10:10 am
Hi Ravish/Jay, we have a setup with couple of domains of weblogic server.
The web servers are I-plant servers(Sun one servers), we are also getting 404 error often to our users, not sure what would be the reason, do you think the same kind of issue what you mentioned about (for Apache) may cause to Sun one also ??
Thanks a lot
Viswa
March 16th, 2011 on 10:26 am
Hi Viswa,
The above post issue was happening due to the Apache side as the its process was not sharing the dynamic list with other processes of Apache, so this issue was more specific to Apache and right now cant be said that the same issue is happing for your case.
Regards,
Ravish Mody
March 21st, 2011 on 4:56 pm
Hi Ravish,
I am new to weblogic, can u pls help me how to clean up logs..
Thanks in Advance
Nag
March 21st, 2011 on 10:24 pm
Hi Nag,
Oracle has given an option to rotate a log file by Size or by Time, so you can use them as per your requirement. However the default the log file rotation is done by size 500 MB. Hence would suggest you to go with this feature and take the best out of it 😉
More information can be found in the below link
Topic: Rotate log files
Also if you need to force rotation of the log then you can go through the below post.
Topic: Force ServerLog Rotation
Regards,
Ravish Mody