404 error with Apache and Suspended Weblogic managed server

Nov/10

404 error with Apache and Suspended Weblogic managed server

Ravish Mody

This is a post where I would like to share some of my experience working with Apache web server, the reason behind writing this post is because during some test we and some of customers of WLS faced this similar issue and during solving this issue got a good set of information which would be helpful for others to understand how Apache, WLS plugin and WLS works to-gather and whose fault was it to show 404 error.

Let me give you some background information about the issue, by default dynamic server list is ON hence if any of the servers goes down or if that server is only allowed to server admin requests then the list should get updated and sent back by removing that particular server, so that client would not get 404 error. This was not happing and client were getting 404 error again and again.

The reason behind this with the a solution is given below which would give you an idea how things works and hope this would help you however this information is applied to any platform and version of Oracle Weblogic Server

Issue:

When any managed server is put in “Suspended” state the Apache server continues to deliver requests to the suspended server instance which leads to a 404 http error on the client side.

Reason behind this issue?

Apache is a multi-processed and multi-threaded, which means that every process maintains it’s own server list.

So, if a response comes back from WLS with a new server list which excludes the non working servers is sent back to the process which had sent the request will update it’s own server list ONLY and will NOT share the new server list with the other processes, until a request is served by them explicitly and each of them gets there own new server list in the response.

This is how Apache is been design, hence its an Apache issue and weblogic plugin plays no part in it.

How to Solve this?

The solution is to use “worker mpm” instead of “prefork mpm” and set both StartServers and ServerLimit to 1, and then tune the number of threads within the process which might help in resolving this issue.

More details can be found in the link below:
http://httpd.apache.org/docs/2.0/mod/worker.html

Do share your experience’s with us also so that others can get benefits out of it and help them learn new stuff 🙂

Regards,

Ravish Mody

Buffer

If you enjoyed this post, please considerleaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.

404 error, Apache, Proxy

17 Comments for this entry

ritu_21
January 26th, 2011 on 10:38 pm
I conclude bu your above application that every process maintains there own list of active servers my ques is
Is dere any way to see the active server list in the processes
also could u please explain wht is prefork mpm?

Log in to Reply
- Ravish Mody
  January 27th, 2011 on 11:33 am
  Hi Ritu,
  
  Is dere any way to see the active server list in the processes
  You can set the Debug as “ALL” and then in the proxy log you can see the server list for particular process which would be something like < 123456789 > and number of servers ip addresses
  
  also could u please explain wht is prefork mpm?
  You can go through the below link for more info about prefork mpm
  http://httpd.apache.org/docs/2.0/mod/prefork.html
  
  Log in to Reply

ritu_21
January 27th, 2011 on 3:58 pm

ravish can u please share the stack trace or be specific which apache logs can i look in ,i looked in access logs
image logs and apache errro logs ,were else i can look for it ,also in the satacktrace can u highlight an instance and explain ,

thanks
RItu

Ravish Mody
January 27th, 2011 on 11:07 pm

Hi Ritu,

I am talking about Apache log file, in your httpd.conf you have to specify it like these parameters to get the correct logs

<IfModule mod_weblogic.c>
            .
            .
             WebLogicCluster xxx.xxx.xxx.xxx:7102,xxx.xxx.xxx.xxx:7101
             Debug All
             WLLogFile logs/proxy.log
             .
             .
</IfModule>

Now from the below example proxy.log sample you can see that 1382912804066002 process gets its list in the line-10 same way other process might get different list if any one of the servers go down

================New Request: [GET /myApp/path HTTP/1.1] =================
.
.
Thu Jul 01 14:30:00 2010 <1382912804066002> INFO: SSL is not configured
Thu Jul 01 14:30:00 2010 <1382912804066002> Using Uri /myApp/path
Thu Jul 01 14:30:00 2010 <1382912804066002> After trimming path: '/myApp/path'
Thu Jul 01 14:30:00 2010 <1382912804066002> The final request string is '/myApp/path'
Thu Jul 01 14:30:00 2010 <1382912804066002> SEARCHING       id=[xxx.xxx.xxx.xxx:7102,xxx.xxx.xxx.xxx:7101] from current ID=[xxx.xxx.xxx.xxx:7102,xxx.xxx.xxx.xxx:7101]
Thu Jul 01 14:30:00 2010 <1382912804066002> The two ids matched
Thu Jul 01 14:30:00 2010 <1382912804066002> @@@FOUND...id=[xxx.xxx.xxx.xxx:7102,xxx.xxx.xxx.xxx:7101], server_name=[xxx.xxx.xxx.xxx], server_port=[80]
Thu Jul 01 14:30:00 2010 <1382912804066002> attempt #0 out of a max of 5
.
.

ritu_21
January 27th, 2011 on 4:18 pm
Also,

is there any way to see the heartbeat produced by the
dervers in the cluster in which way they put there attendance as i am alive…

Log in to Reply
- Ravish Mody
  January 27th, 2011 on 11:12 pm
  Hi Ritu,
  
  This happens internal and I think it can been seen using TCP dump, however it uses internal APIs and we do not have access to it. So would request you to open a service ticket with the WLS support team to get more information on it.
  
  Log in to Reply
sumanth716
February 4th, 2011 on 3:21 am
Hi Ravish,
i think the server list (on apache side) we are talking here is about the cluster members in weblogic cluster.
if any member in a cluster is dead then cluster will take care of routing the next requests to alive cluster member (am I correct ?), then why does the server list on apache side not getting updated (if any cluster member is dead) will infact affect us?
Please correct if my understanding is wrong

Thanks
Sumanth

Log in to Reply
- Ravish Mody
  February 4th, 2011 on 8:15 pm
  Hi Sumanth,
  
  Yes, Server List is the members in the clusters.
  
  In the above scenario MS-1 was send a response 404 error and updates the server list to that process which had send the request (i.e. you are not a admin, hence cant access the app) thus removes the MS-1 for the list. And the same request was not sent to the MS-2 when it got 404 because when a request comes then Apache tells on which server it has to go in the header.
  
  Log in to Reply
sumanth716
February 7th, 2011 on 12:47 pm
Thanks Ravish.

Not sure where to start a new thread, so posting the question here

What is the difference between JMS Vs IBM MQ Series, in the sense that why MQ Series when WebLogic already providing JMS?
One answer i can think of is JMS works if 2 application systems are on same version (say 2 systems are on Weblogic 10 MP1), if 2 systems are on different version or use different platforms (say 1 weblogic, another websphere) them JMS does not work we have to use MQSeries. Am I Correct?

can you/someone throw some light on this please?

Log in to Reply
- Ravish Mody
  February 7th, 2011 on 11:38 pm
  Hi Sumanth,
  
  JMS
  JMS is a java based Message Oriented Middleware (MOM) API for sending/receiving messages between two or more clients.
  
  IBM MQ Series
  Is a JMS provider which has implemented JMS API’s and interfaces for a Message Oriented Middleware (MOM) same way weblogic is also a JMS provider.
  
  Weblogic works fine with the different versions of WLS using JMS as well as if anyone is using WLS and websphere then you can use Foreign JMS to work between WLS and foreign JMS provider like websphere.
  
  Log in to Reply
sumanth716
February 8th, 2011 on 12:35 am
thanks Ravish,

so there is not much a point to use MQSeries, but why do companies still use MQSeries when JMS is already provided by WebLogic?

what are the advantages of using MQSeries over WebLogic provided JMS?

Log in to Reply
- JaySenSharma
  February 8th, 2011 on 1:00 am
  Hi Sumanth,
  IBM MQ is a Message-oriented middleware (MOM). In other words a platform that provides the messaging features via MQ queues. It supports both synchronous and asynchronous messsaging and is generally used in conjunction to a specific EAI tool. This is the kind of transport you would like to use for your communication across the different applications/systems that you would like to integrate..
  
  It’s been a long time that MQ had captured Messaging Market…But now Applications Servers like JBoss and WebLogic itself contains a very strong Messaging Module which is not less than MQ series. Whatever MQ series can do the same things can be easily achieved with these applications servers now.
  
  MQ has some extra features like it can work independently as well means standalone, It has it’s own Message Browser to easily see and delete messages…etc.
  But JMS modules are integrated in Application Servers. So if you just want to use Messaging (nothing else) then MQ is very light weight/cheap compared to Application Servers…But if you want to use Application Server functionality as well to make your Messaging more applicable then you can go with WebLogic/JBoss Messaging as well.
  
  The Advantage of using MQ Series or a Separate Messaging is if the Application Server goes down/Crashes still you have MQ running to store your Messages… 🙂 It will be running separately and will consuming Critical Messages. It means Separation of Tasks.
  .
  .
  Keep Posting 🙂
  Thanks
  Jay SenSharma
  
  Log in to Reply
sumanth716
February 8th, 2011 on 1:33 am
Hi Jay,
thanks a lot

so based on your explanation the big advantage of usig MQ is, MQ is useful if we need only messaging service without any app server functionality

one more thing JMS is asynchronous messaging service compared to MQ which is both synchronous & asynchronous

so it just means MQ has captured messaging market.

regarding the storage of messages when app server crashes , we can use persistence so that messages can come online when server is back UP again..but yes..MQ allows receiver not to wait until app server comes UP

Log in to Reply
viswa
March 16th, 2011 on 10:10 am
Hi Ravish/Jay, we have a setup with couple of domains of weblogic server.

The web servers are I-plant servers(Sun one servers), we are also getting 404 error often to our users, not sure what would be the reason, do you think the same kind of issue what you mentioned about (for Apache) may cause to Sun one also ??

Thanks a lot
Viswa

Log in to Reply
- Ravish Mody
  March 16th, 2011 on 10:26 am
  Hi Viswa,
  
  The above post issue was happening due to the Apache side as the its process was not sharing the dynamic list with other processes of Apache, so this issue was more specific to Apache and right now cant be said that the same issue is happing for your case.
  
  Regards,
  Ravish Mody
  
  Log in to Reply
Nag
March 21st, 2011 on 4:56 pm
Hi Ravish,
I am new to weblogic, can u pls help me how to clean up logs..

Thanks in Advance
Nag

Log in to Reply
- Ravish Mody
  March 21st, 2011 on 10:24 pm
  Hi Nag,
  
  Oracle has given an option to rotate a log file by Size or by Time, so you can use them as per your requirement. However the default the log file rotation is done by size 500 MB. Hence would suggest you to go with this feature and take the best out of it 😉
  
  More information can be found in the below link
  
  Topic: Rotate log files
  
  Also if you need to force rotation of the log then you can go through the below post.
  
  Topic: Force ServerLog Rotation
  
  Regards,
  Ravish Mody
  
  Log in to Reply