52 Comments for this entry

• murali
  May 13th, 2010 on 1:45 pm

  How to separate the Static content and dynamic content Apace and weblogic

  Log in to Reply
  • Administrator
    May 13th, 2010 on 5:18 pm

    Hi Murali,

    I developed a Fresh sample to demonstrate ..how we can add and access static contents in Apache proxy level… Please refer to http://middlewaremagic.com/weblogic/2010/05/13/apache-proxy-as-static-content-repository/
    Let us know if you have any doubts.

    .
    .
    Keep Posting
    Thanks
    Jay SenSharma

    Log in to Reply
• rajni_kant_lal
  June 16th, 2010 on 1:05 pm

  We are facing strange issue with weblogic proxy plug-in.

  Here is description of our setup:

  1. We have two different kind of servers. Say server 1 and server 2
  2. We have two solaris box where different instance of server 1 is running and forming a server-1 cluster
  3. We have two solaris box where different instance of server 2 is running and forming a server-2 cluster
  4. We have configured proxy plugin at both instance of server 1 so that each instance of server 1 can talk to both instance of server 2. E.g

  WebLogicCluster
  server2_1:8001|server2_2:8001
  5. By doing above we are making sure that whenever a request comes to server1_1 then it redirects this request to either server2_1 or server2_2 depends on availability.
  6. We have a kind of service which takes around 30 minutes to complete
  7. To make sure that proxy-plugin doesn’t redirect this request to server2_2 within 30 minutes( if server2_1 is already processing) we have configured WLIOTimeoutSecs to 40 minutes e.g.

  WLIOTimeoutSecs^M
  2400 ^M

  8. However, we are observing that proxy-plugin in redirecting this request to server2_2 after 300 second(default value for WLIOTimeoutSecs).

  Could you please suggest us what mistake we are doing?

  Log in to Reply
• sandeep
  August 14th, 2010 on 2:50 am

  we have 12 instances in the cluster, with 4 physical machines, admin server on one machine and 4 server instances on each physical machine. When we do a load test with unicast configuration the load is not evenly balanced, but when we use multicast the load is evenly balanced in the cluster. We track the load balancing on each server with the no of sessions on each server instance. We are using Apache as proxy .

  Log in to Reply
• vicky
  September 16th, 2010 on 7:02 pm

  Hi,

  Would like to know that in case we are trying to upload a large file(about 13 MB) then we are getting session time out.

  Can you suggest us some parameters that can be tuned?
  We tried setting post time out to max value 120 and also tried max message size.

  We are having a clustered environment, NIC’s are configured and requests route to the servers through them.

  Please let us know how can we proceed on this

  Regards,
  Vicky

  Log in to Reply
• abrahamissac
  October 14th, 2010 on 6:10 pm

  Hi Jay / team,

  I am trying to find a solution for the following problem:

  – I have Sune One Web Server with weblogic proxy plug-in.
  – The traffic is load balanced to 4 weblogic managed servers

  – Say, when application is not running on one of the managed servers, is there any way the plug-in can understand that its unavailable and should not route the load to that particular managed server.

  – As far I understand there is no option in the webserver plug-in to prevent routing to weblogic managed servers when its running and the application deployed on it is not available.

  If you came across similar issues, let me know.

  Thanks!
  Abraham

  Log in to Reply
  • JaySenSharma
    October 14th, 2010 on 7:56 pm

    Hi Abraham,
    If your 4 Managed Servers are available as part of a Cluster ….then Proxy will detect if any of the Server has been shut down or the application is not present in that. Because WebLogic Cluster Maintains “Dynamic Server List” in a frequent interval …so as soon as any of the Server which is part of Cluster is down or has Application Unavailability then the Proxy will get to know that the “Dynamic Server List” is not having that Server Name in it. So Proxy will not send the request to that Server next time until the Dynamic Server list is Updated by the WebLogic Server with that Server Name.
    But If your ManagedServers are not part of any Cluster then in that Case…the proxy will not be able to detect the ManagedServer Failure and it will keep on sending the requests to all those 4 Managed Servers in a RoundRobin Fashion.
    In this case it will be preferred that u use the DNS level failover….it means the DNS name (example: abc.com) will represent all the 4 Managed Server IP Addresses. So as soon as you find a Managed Server is Down or causing failure you can immediately ask your N/W team to remove the Effected ManagedServer’s IP Address entry from the DNS Mapping.
    .
    .
    Keep Posting
    Thanks
    Jay SenSharma

    Log in to Reply
• abrahamissac
  October 15th, 2010 on 3:01 pm

  Thanks Jay for responding.

  All 4 managed servers are part of the cluster. Let me make my question more clear

  – We have a jsp deployed part of a war file and it is running on all managed servers.
  – If for some reason the application deployed on managed server 4 is down. But managed server is still up and running.
  – In the above case, plugin finds server is running and forwards the traffic to access jsp on managed server4.

  – But you will get a page not found error.

  – In short there is no intelligence in webserver proxy plugin to identify whether the application deployed on managed server 4 is still accessible.

  My question was , what can we do to avoid this situation.

  For example, CSS load balancer will probe on some jsp page to make sure that the application on that node is reachable to forward the traffic.

  In my case i can check only at managed server port level, but dont have the capability to check whether the application is reachable on the managed server.

  Hope i tried to explain my issue better this time

  Thanks!
  Abraham

  Log in to Reply
  • JaySenSharma
    October 15th, 2010 on 5:47 pm

    Hi Abraham,
    This is correct that Until any of the Managed Server (Which is part of Cluster) will not be down physically (means closes it’s listen port) …the Proxy/LoadBalancer will not be able to detect this application failure. In your case Only the Application is not present//down on one of the Server (Means only the Application went down but still the Server is up and running) … so the Load Balancer will not be able to detect that and will keep on sending the request to that Managed Server as well.
    From WebLogic side no such failure detection feature implemented yet. So we need to look for some out of the box solution. I will try to reproduce the issue and will try to implement some workarounds but right now i dont have any ideato avoid this kind of scenario.
    .
    .
    Keep Posting
    Thanks
    Jay SenSharma

    Log in to Reply
• abrahamissac
  October 15th, 2010 on 7:10 pm

  Thanks! Jay.

  We are also thinking of an approach to mark managed server as failed or kill the JVM, when deployed application’s health is not OK.

  Not sure whether this would be a good approach.

  -Abraham

  Log in to Reply
  • JaySenSharma
    October 15th, 2010 on 7:47 pm

    Hi Abraham,
    Thats a good idea to kill the Server which has a Failed (Undeployed) Application in it. May be u can use the following JMX Code to continuously monitor the State of a particular application is ACTIVE or not. http://middlewaremagic.com/weblogic/2010/01/23/application-status-checker-jmx-for-wls9-x-onwards/
    If the Application is not Active then you can use the following JMX Code to Kill/Shutdown that particular Managed Server Instance: http://middlewaremagic.com/weblogic/2009/08/17/force-shutingdown-wls-using-jmx/
    .
    .
    Keep Posting
    Thanks
    Jay SenSharma

    Log in to Reply
• fabianromerorcc
  October 25th, 2010 on 10:25 pm

  Hi Jay,

  First of all I want to thank you your amazing work! I am learning a lot.

  I got a question about weblogic plugin: we are using apache http server with weblogic plugin, but I am wondering if I need to enable the option “WebLogic Plug-In Enabled” in environment -> clusters -> configuration -> general tab. I see that it’s working, but, Could it work better?

  Thanks!

  Fabián.

  Log in to Reply
  • JaySenSharma
    October 25th, 2010 on 11:01 pm

    Hi Fabrian,
    Actually the actual use of “WebLogic-Plugin-In Enabled” is when we want to get the Actual client’s IP Address inside the Application or logs. When this is set to true if the cluster will receive requests from a proxy plug-in or HttpClusterServlet. When WeblogicPluginEnabled is true, a call to httpServletRequest.getRemoteAddr will return the address of the browser client from the proprietary WL-Proxy-Client-IP header, instead of the web server.
    Example:
    Client (Browser)——>Apache Web Server —-> WebLogicServer
    In Above scenario Apache forwards the request to the WebLogic Server so in that case (If the WebLogic-Plugin-InEnabled is FALSE) then when we will try to Track the end client’s IP Address using httpServletRequest.getRemoteAddr…it will return Apache WebServers IP Address because Apache is sending the request to the WebLogic…..
    So to get the Actual End Clients IP Address or Host Name we must set “WebLogic-Plugin-In Enabled=true”…so that we can get the actual request Originators IP Address to Track them.
    .
    .
    Keep Posting
    Thanks
    Jay SenSharma

    Log in to Reply
• fabianromerorcc
  October 25th, 2010 on 11:25 pm

  Very clear!, thanks again.

  Fabián.

  Log in to Reply
• sumit
  October 26th, 2010 on 7:16 pm

  Jay,

  I have question over “DynamicServerList” parameter.

  I am setting this parameter ON and the MS instance define in “WebLogicCluster” is shutdown. The other instance in the cluster is up and running which is not defined in “WebLogicCluster” .

  What i am seeing is the request is getting process by running instance even for the first time. How this is possible?

  What i was thinking is that , when the proxy configuration is done for the first then proxy plugin gets the Dynamic server list from the server define in “WebLogicCluster” . But this instance is down so ideally the proxy should not get info abt other instance for the first time.

  Is this is expected behavior. If yes then any idea how the proxy gets the list of all the servers.

  This link define “DynamicServerList” http://download.oracle.com/docs/cd/E12840_01/wls/docs103/plugins/apache.html

  says that proxy gets the list ‘After the first request is routed to one of these servers”

  Can you please let me know what is the exact behavior of this parameter.

  Thanks
  Sumit

  Log in to Reply
  • JaySenSharma
    October 26th, 2010 on 10:46 pm

    Hi Sumit,
    WebLogic cluster is actually responsible to continuously monitor the Heartbeats of each and every member Managed Server of the Cluster and to Maintain the “DynamicClusterList”. When we enable the DynamicServerList=ON at Apache proxy level then the Proxy starts receiving the Fresh List from the Cluster as soon as there is any Update made by the Cluster in that List.
    So Basically The Plugin work as a Listener to the “DynamicProxyList” which is sent by the Cluster to it. And Accordingly it sends the request. Means actual DynamicServerList is maintained by the ClusterMBean and not by the Proxy….Proxy just use that List to take decision.

    .
    .
    Keep Posting
    Thanks
    Jay SenSharma

    Log in to Reply
• joeljos
  October 27th, 2010 on 12:43 am

  Jay,

  We have 3 servers in the cluster and two of them kept down for resource issues. we have all three nodes mentioned in the apache conf file.

  The apache keeps trying to connect to the nodes those are down for months and throwing “connection refused” exception in the logs. And our monitors are getting triggered when they see error in the log file.

  Can we avoid that? I don’t want my apache to keep looking for the nodes thaose are down.

  JJ

  Log in to Reply
  • JaySenSharma
    October 27th, 2010 on 9:24 am

    Ji Joeljos,
    Please make sure that u set the following 3 Parameters in your “httpd.conf” file…
    DynamicServerList=ON (ON is the default value)
    When set to OFF, the plug-in ignores the dynamic cluster list used for load balancing requests proxied from the plug-in and only uses the static list specified with the WebLogicCluster parameter. Normally this parameter should remain set to ON.
    There are some implications for setting this parameter to OFF:
    If one or more servers in the static list fails, the plug-in could waste time trying to connect to a dead server, resulting in decreased performance.
    If you add a new server to the cluster, the plug-in cannot proxy requests to the new server unless you redefine this parameter. WebLogic Server automatically adds new servers to the dynamic server list when they become part of the cluster.
    Idempotent=ON (ON is the default value)
    When set to ON and if the servers do not respond within WLIOTimeoutSecs (new name for HungServerRecoverSecs), the plug-ins fail over.
    If set to “OFF” the plug-ins do not fail over. If you are using the Netscape Enterprise Server Plug-In, or Apache HTTP Server you can set this parameter differently for different URLs or MIME types.
    WLIOTimeoutSecs=15
    Defines the amount of time the plug-in waits for a response to a request from WebLogic Server. The plug-in waits for HungServerRecoverSecs for the server to respond and then declares that server dead, and fails over to the next server. The value should be set to a very large value. If the value is less than the time the servlets take to process, then you may see unexpected results.
    Minimum value: 10
    Maximum value: Unlimited
    .
    .
    Keep Posting
    Thanks
    Jay SenSharma

    Log in to Reply
• sumit
  November 20th, 2010 on 2:14 pm

  Jay,

  We need understand the need to keep dynamic server list on .
  What i understand is that this dynamic server list is sent the weblogic server instance to the client as part of response ??

  Secondly each server instance sent the list of only servers in its cluster for which it is able to communicate through multicast IP??

  I would like to know how exactly dynamic server list is constructed by each server instance and when it sents this list to proxy on the client side.??

  Let me explain my question with an example

  We have an application “A” and application “B”

  .Application “A” runs on 4 solaris boxes with each box has 2 instances and thus total 8 instances of application A forms a cluster.

  Same way application “B” runs on 4 solaris boxes with each box having 2 instances and thus total 8 instances of application B forms a cluster.

  We have deployed a proxy on application “A” . The cluster list configuration have IP:PORT of application B

  Same way we have deployed proxy on application “B”. The cluster list configuration have IP:PORT of application A.

  Thus these 2 applications “A” and “B” communicate through proxy.

  CASE1: Initially communication starts b/w these 2 application and proxy deployed on each application knows about the application instances on other application. Say after some time 2 instances of application “A” goes down.

  Now as soon as next communication happens b/w these 2 application,the proxy on application “B” will update the list of servers by removing those 2 down instacnes. Right???.

  Now i want to know,As the communication happened b/w once specific instance of application “A” and “B” and this dynamic server list is returned ( as part of response ) by one of the instance of application “A” to only one of instance of application “B” . This is correct???

  Then how will the other proxy instance deployed on each instances of application “B” will update its server list???

  CASE 2: Let say the application instance A11,A12 (on box 1) does not gets the heart beats of instance A31,A32(on box 3). Thus A11 and A12 removes A31,A32 from there cluster list.
  However the communication b/w application “A” and application “B” is very fine with all instance. So how the dynamic server list will look like on proxy B.

  This because when the communication happens b/w A11 or A12 and any instance of B , the A11, A12 will return server list that will not have A31,A32.
  While other instances of A will send complete of severs.

  Jay, Let me know if i am unclear any where.??

  Thanks
  Sumit

  Log in to Reply
  • Ravish Mody
    November 21st, 2010 on 2:33 pm

    Hi Sumit,

    Welcome to Middleware Magic.

    Below are your questions answers

    What i understand is that this dynamic server list is sent the weblogic server instance to the client as part of response ??
    – WLS sends the dynamic server list as a part of a response to the proxy (example Apache) NOT to the client.

    Secondly each server instance sent the list of only servers in its cluster for which it is able to communicate through multicast IP??
    – Only the cluster master would send the list after collecting the details which servers are up and read to servers client request, NOT each servers sends the list.

    I would like to know how exactly dynamic server list is constructed by each server instance and when it sents this list to proxy on the client side.??
    – Above answer-2 is applied here.

    Now as soon as next communication happens b/w these 2 application,the proxy on application “B” will update the list of servers by removing those 2 down instacnes. Right???.
    – Correct.

    Now i want to know,As the communication happened b/w once specific instance of application “A” and “B” and this dynamic server list is returned ( as part of response ) by one of the instance of application “A” to only one of instance of application “B” . This is correct???
    
    – No, as the list is sent by the cluster master described as above answer-2. However I dint get your architecture.

    Box-1 = (MS-1, MS-2) => C1
    Box-2 = (MS-3, MS-4) => C1
    Box-3 = (MS-5, MS-6) => C1
    Box-4 = (MS-7, MS-8) => C1

    So are both the application (App-A & App-B) are deployed on cluster C1?

    OR

    Box-5 = (MS-9, MS-10) => C2
    Box-6 = (MS-11, MS-12) => C2
    Box-7 = (MS-13, MS-14) => C2
    Box-8 = (MS-15, MS-16) => C2

    App-A is deployed on cluster C1 and App-B is deployed on cluster C2 ?
    Are you using two proxies (Proxy-A and Proxy-B) for each application (App-A and App-B)?
    Which proxies are you using?
    Also is Proxy-A has IP & Port from (MS-9 to MS-16) and Proxy-B has IP & Port from (MS-1 to MS-8)?

    Regards,
    Ravish Mody

    Log in to Reply
• sumit
  November 21st, 2010 on 3:43 pm

  Hello Ravish,

  1) What is cluster master? what all criteria it takes to construct this server list?

  2) So does it means that server instances does not play any role in returing this dynamic server list to proxy on client side.

  3) Architecture is like that application A and B both can behave as client and server and thus proxy is there on the application.

  The application “A” with MS from MS1 to MS8 forms one cluster say C1. The application “B” with MS from MS9 to MS16 forms cluster say C2.
  There is a proxy deployed on cluster C1 which has IP &Port of Application B ( MS9 to MS16 ) and there is a proxy deployed on C2 which has IP & PORT of Application A ( MS1 to MS8)

  Thus my concerns here is MS1,MS2 which are on box 1 does not get heart beat from MS3 ,MS4 which are on box2. Thus MS1 and MS2 will remove MS1 and MS4 from its cluster list , correct ??

  At the same time there is a communication b/w application B and application A with say MS1 is communicating with MS9 and MS5 is communicating with MS 10 . In this case MS1 will send the dynamic server list which will not have MS3 and MS4.
  However MS5 will send the list which will have all the 8 instances ( MS1 to MS8 ) correct??

  How the proxy on Application B will behave ??? I mean will it sends the request to MS3 and MS4.

  4) What mechanism is WL using to prepare this dynamic server list when it sends to proxy on client side ?? Does it consider cases like server slowness which cannot respond to request etc??

  Thanks
  Sumit

  Log in to Reply
  • Ravish Mody
    November 21st, 2010 on 10:11 pm

    Hi Sumit,

    1) What is cluster master? what all criteria it takes to construct this server list?

    – Cluster master/lead means from n-number of managed servers in a cluster one of them would become the master/lead which will have the data which all servers are running in the cluster and are ready to server the requests. This is been done by ever servers in the cluster by replying the heartbeats sent by the master/lead server.

    – There few criterias on which the master declars that if any servers should be in the list or not. Like if the server is in shutdown state, if for some reason any server is not able to reply the heartbeats which can be a network issue also, if the server is in Admin state which means only the adminstors request should be served etc… then these servers are not kept in the list as these servers cannot server the request of the clients.

    2) So does it means that server instances does not play any role in returing this dynamic server list to proxy on client side.

    – The cluster master is main source to send the list to the proxy and other server has to reply the hearbeats, hence they only play that role to be in the server list.

    3) Architecture is like that application A and B both can behave as client and server and thus proxy is there on the application.

    The application “A” with MS from MS1 to MS8 forms one cluster say C1. The application “B” with MS from MS9 to MS16 forms cluster say C2.
    There is a proxy deployed on cluster C1 which has IP &Port of Application B ( MS9 to MS16 ) and there is a proxy deployed on C2 which has IP & PORT of Application A ( MS1 to MS8)

    Thus my concerns here is MS1,MS2 which are on box 1 does not get heart beat from MS3 ,MS4 which are on box2. Thus MS1 and MS2 will remove MS1 and MS4 from its cluster list , correct ??
    

    – Correct, but if we consider MS1 is the cluster master.

    At the same time there is a communication b/w application B and application A with say MS1 is communicating with MS9 and MS5 is communicating with MS 10 . In this case MS1 will send the dynamic server list which will not have MS3 and MS4.

    However MS5 will send the list which will have all the 8 instances ( MS1 to MS8 ) correct??

    – Now MS5 would not send the list, MS1 would send the list as its the cluster master and in above case the list would not have MS3 & MS4.

    How the proxy on Application B will behave ??? I mean will it sends the request to MS3 and MS4.

    – Now if proxy on Application B gets the above list then it would not send the requests to MS3 and MS4

    4) What mechanism is WL using to prepare this dynamic server list when it sends to proxy on client side ?? Does it consider cases like server slowness which cannot respond to request etc??

    – The answer-1 (part-2) would answer this question.

    Hope you got all the information.

    Regards,
    Ravish Mody

    Log in to Reply
• sumit
  November 21st, 2010 on 11:22 pm

  Ravish,

  1) Which among the MS becomes a cluster master? How this get decide which server which will be master??

  Can you please point me to any WL document which have reference of cluster master . I found one(http://download.oracle.com/docs/cd/E13222_01/wls/docs90/cluster/failover.html) but that does not say anything that we are discussing here.

  2) It may be possible that request from client side may not come to cluster master ?

  And one more thing, what i have observed that when ever there is any n/w issue in the cluster and which causes one machine to go out of n/w, then each member says that it is removing the particular member ( from the logs) . And this what WL document (The WebLogic Server “Heartbeat” at http://download.oracle.com/docs/cd/E13222_01/wls/docs90/cluster/failover.html) also says.

  Then what role cluster master is playing here??

  3) Will it possible in any case that server is not able to process the request but still able to send the heart beats to its cluster member?Then what lists will the cluster master will send??

  Thanks
  Sumit

  Log in to Reply
  • Ravish Mody
    November 22nd, 2010 on 12:35 pm

    Hi Sumit,

    – The cluster master/leader is chosen based on the managed server that has been running the longest within a cluster.

    – The cluster master/leader will share the list with all the servers in the list, so that if the request is severed by any server then they know which server are working properly. However in this all done internally hence it looks that a particular server is sending the list.

    – If the list has been sent to the proxy saying MS-3 is up, but after the list is sent the MS-3 goes down then when the request comes to MS-3 it would not be able to send any response and if the “Idempotent” is ON with “WLIOTimeoutSecs” set then the request would be failed over on the other server and declaring MS-3 as dead. And from the new server the new list will come.

    However all this is more of a conceptual and internal informations, As Cuuster master is a webLogic Internal Implementation …many documents are not available on it’s internal functionality…. so if you need more details or documents on it we would request you to open a ticket with WLS support team and they would provided you the docs if they have with them.

    Also do share the information with us once you contact support, so that others can take benefits out of it.

    Hope this would help you.

    Regards,
    Ravish Mody

    Log in to Reply
• sumit
  November 22nd, 2010 on 2:44 pm

  Ravish,

  Thanks for all the explanation. I will ping you on this if need any more info

  And regarding oracle support, it looks they also do not have any document. They always point document where dynamic server list definition is there.

  Thanks
  Sumit

  Log in to Reply
• sumit
  November 22nd, 2010 on 2:58 pm

  Ravish,

  One more thing , say if the cluster master goes down then who will become cluster master . Any idea what protocol they use for becoming cluster master.

  Thanks
  Sumit

  Log in to Reply
  • Ravish Mody
    November 22nd, 2010 on 3:02 pm

    Hi Sumit,

    The oldest member in the cluster would become the master once the present cluster master goes down.

    Regards,
    Ravish Mody

    Log in to Reply
• sumit
  November 22nd, 2010 on 4:08 pm

  Ravish,

  In my architecture. We have proxy on both application .

  Application “B” send the request to application “A” . To send this it uses proxy deployed on it and send the request to any 1 instance of application A.

  Now that instance of application “A” uses prxoy to sent the request to any instance of “B”. Moreover this instance of “A” will sent the dynamic server list.

  According to your post this list will get updated on proxy plugin deployed on application “B” . Correct?
  How this proxy-plugin gets list.

  Does it means that the response send by application “A” will first go to proxy on application “B” and then to a specific instance??
  This is something unclear to me now.

  Thanks
  Sumit

  Log in to Reply
  • Ravish Mody
    November 22nd, 2010 on 5:14 pm

    Hi sumit,

    See the following architecture image which we have created and see the explanation below

    

    New Request from App-B
    -Now from App-B say MS-5 sends a request to App-A using Proxy-A, which can go to any one of the managed servers, lets say it goes to MS-1.

    – Then the response + the DSL (Dynamic Server List) sent by MS-1 will go to the Proxy-A [as it is a response not a new request which would use Proxy-B], thus this way the Proxy-A would get the new DSL.

    Same way the App-A with the Proxy-B would behave.

    Hope this would help you.

    Regards,
    Ravish Mody

    Log in to Reply
• sumit
  November 22nd, 2010 on 5:50 pm

  Ravish,

  Got it. Thanks for all help. This now for more information . WL provides “__WebLogicBridgeConfig” using which we can get to know what all servers are there in proxy list.

  The o/p of this URL gives like

  General Server List:
  10.197.1.7:8011:8012
  10.197.1.7:8001:8002
  .
  .
  .
  Here 8001 is clear port and 8012 is SSL port.

  However sometime the serverlist looks like
  General Server List:
  10.197.1.7:8011:443
  10.197.1.7:8001:443.

  Why this 443 comes.??

  Thanks
  Sumit

  Log in to Reply
• sumit
  December 1st, 2010 on 3:53 pm

  Hi Ravish,

  I am testing this in continuation with the above test bed

  1) All servers are up and running

  2) I have create scenario where all the threads in one server are starting getting stuck and thus server will not respond to any HTTP request at some point of time later.Correct?

  3) Now the cluster master should remove this server instance from the list when all the thread comes to stuck. Moreover the server which i am making stuck/hang is lastly started .So it will not be cluster master.Correct

  Thus expected result of this test should be that DL should not have hang/stuck instance .Correct??

  Thanks
  Sumit

  Log in to Reply
• sumit
  December 1st, 2010 on 6:40 pm

  Ravish,

  The managed instance is not able to process the request and thus request is going to other member of cluster. However dynamic server list is not getting updated.

  Thus every time the request is going to hang server first and then timing out and going to other instances.

  I am not able to assimilate this behavior.

  It looks like Managed instance is able to send the multicast packet but cannot process the request.

  can you please give some comments on this??

  Thanks
  Sumit

  Log in to Reply
  • JaySenSharma
    December 1st, 2010 on 11:23 pm

    Hi Sumit,
    We had a very long discussion on Plugin Scenario which u tried to explain….However it looks like your issue is little different and requires explanation from the WebLogic Vendor because Oracle Support has access to all the Source codes and internal documents on weblogic so they may provide you some proof for this behavior of WebLogic. I will request you to please open a support ticket with oracle if you see that the behavior of apache plugin or WebLogic is not correct.
    .
    .
    Keep Posting
    Thanks
    Jay SenSharma

    Log in to Reply
• sumit
  December 1st, 2010 on 11:57 pm

  Hi Jay/Ravish,

  Thanks buddy for all the discussion . Actually the above discussion helps us a lot to analyze on how actually the cluster behaves.

  Thanks
  Sumit

  Log in to Reply
• shiva
  March 30th, 2011 on 12:36 am

  I have a apache webserver with Siteminder authentication. Once authenticated it redirects to the weblogic cluster. The Ifmodule has a cluster configuraion with only one server (DynamicServerList=OFF), So request is directed to one weblogic server. We have proxied for all the contents in the page like .js, .jsp and so on. Once the request hits the weblogic an image needs to generated and displayed on page.

  We face an issue when the page is displayed, the image and the .js are not loaded. These gets loaded only on a page reload. The requests are directed to only one weblogic instance as per the apache webserver logs. The page seems to load quickly. I am planning to set WLIOTimeoutSecs to higher value. But would this help.

  It would be great if you could provide some help.

  Regards!
  Shiva

  Log in to Reply
• muffin_top
  April 11th, 2011 on 7:46 pm

  Hi,

  This may be a stupid question but if we do not have the parameter ‘DynamicServerList=ON’ explicitly set in the Apache configuration file, is it set by default? OR, should we explicitly set this?

  Thanks,

  Log in to Reply
  • Ravish Mody
    April 11th, 2011 on 8:47 pm

    Hi Muffin,

    You do not have to explicitly set it, as by default its set and it ON for DynamicServerList. To get all the details about other parameters check out the below link

    Topic: Entering Parameters in Web Server Plug-In Configuration Files

    Regards,
    Ravish Mody

    Log in to Reply
• t8mas062184
  January 11th, 2012 on 11:01 pm

  Hello there!

  I’m trying to figure out what’s happening with our current setup. We have 2 weblogic clustered and two web servers. However, I’m starting to see some errors as below in our web servers:

  Manager Server 1
  ‘CONNECTION_REFUSED [os error=0, line 1734 of URL.cpp]: Error connecting to host’
  ‘PROTOCOL_ERROR [line 843 of URL.cpp]: Backend Server not responding’

  Manager Server 2
  ‘FAILOVER_REQUIRED [line 1790 of proxy.cpp]: Service Unavailable’
  ‘CONNECTION_REFUSED [os error=0, line 1734 of URL.cpp]: Error connecting to host’

  Here is the part of the SJSWS obj.conf:

  Service fn=”wl_proxy” WebLogicCluster=”IP1:7002,IP2:7002″ DynamicServerList=”OFF”

  We are using:
  Sun Java System Web Server 7.0
  WebLogic Server 10.3

  I got into this site while looking for some answers. Can you also help me out here or at least guide me where can I start? Thanks in advance.

  Log in to Reply
• dhiraj_dixit
  March 15th, 2012 on 9:31 pm

  we have ISAPI plugin & unicast cluster in weblogic. But load is not getting distributed properly to all manage servers. But if i change cluster messaging type from unicast to multicast, the load gets distributed to all manage servers. Has anybody faced similar issue with unicast ? what is reason behind this ?

  Log in to Reply
• zia
  March 25th, 2012 on 2:35 pm

  Hello,
  I have an apache server, which redirects to the weblogic cluster. The cluster has an proxyservlet web app which it redirects to another host m/c. The prb is ofr the first time when the user clicks, the webapp redirects to host m/c . After the first request, all the requests the host is very slow. Below is the conf in apache:
  httpd.conf

  WebLogicCluster 10.0.0.30:8001,10.0.0.18:8003
  Debug ON
  WLProxySSL OFF
  DynamicServerList ON
  WLLogFile /tmp/weblogic.log
  WLTempDir /tmp
  DebugConfigInfo ON

  SetHandler weblogic-handler

  SSL.conf:

  WebLogicCluster 10.0.0.30:8001,10.0.0.18:8003
  Debug ON
  WLProxySSL OFF
  DynamicServerList ON
  WLLogFile /tmp/weblogic_ssl.log
  WLTempDir /tmp
  DebugConfigInfo ON

  SetHandler weblogic-handler

  SSLProxyEngine On
  SSLEngine On

  webapp1 is a proxyservlet which will redirect to another host.

  Kindly help me on this issue.

  Log in to Reply
• ritu_21
  August 8th, 2012 on 11:15 am

  Hi

  We have an encountered an issue were the session between the client and the app server expired before the database transaction completed
  and throws unexpected errors,the application team wants us to increase the time out

  not sure how and were can i do the same.they say the code time outs while the database transaction is in progress
  they say when it runs on ba infrastructure it fails due to lost connection between the client code and the app server.
  can any body suggest were should i increase the time out..

  My understanding is

  1.at apache layer
  2.At apache weblogic-lugin by using WLtimeout Parameter

  any more suggestions please..?

  Log in to Reply
• ritu_21
  August 8th, 2012 on 6:02 pm

  at present my WL plugin has got the below entry

  WebLogicCluster bquat100-02.baplc.com:5035,bquat100-01.baplc.com:5035
  ConnectTimeoutSecs 20
  ConnectRetrySecs 4
  ErrorPage /temp_outage.html
  MatchExpression *
  WLIOTimeoutSecs 7200
  KeepAliveEnabled ON
  KeepAliveSecs 7200
  WLExcludePathOrMimeType /temp_outage.html
  WLTempDir /bq/logs/apache/bquat102

  Can u suggest if it looks alright,what is the max value that i can set for the managed server->configuration->Protocol->HTTP-> Duration ?can this be set to same as KeepAliveSecs in plugin

  Also it is getting a bit harder because the code is a third party code and they run the code into ba infrastructures env and get the connection reset error.but when the application is run from within BA’s network ,no such error is found ,below is the stack trace provided by them

  java.net.SocketException: Connection reset
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
  at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1202)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1196)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:885)
  at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:367)
  at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:318)
  at com.smgtec.framework.remote.HttpConnector.queryServer(HttpConnector.java:252)
  at com.smgtec.framework.remote.HttpConnector.execute(HttpConnector.java:138)
  at com.smgtec.framework.remote.HttpProxy.proxyExecute(HttpProxy.java:318)
  at com.smgtec.framework.remote.HttpProxy.access$5(HttpProxy.java:311)
  at com.smgtec.framework.remote.HttpProxy$3.perform(HttpProxy.java:227)
  at com.smgtec.framework.remote.NonblockingWorker$ExecutionThread.run(NonblockingWorker.java:38)
  at com.smgtec.framework.remote.NonblockingWorker.execute(NonblockingWorker.java:58)
  at com.smgtec.framework.remote.HttpProxy.execute(HttpProxy.java:231)
  at com.smgtec.framework.remote.HttpProxy.access$2(HttpProxy.java:212)
  at com.smgtec.framework.remote.HttpProxy$2.invoke(HttpProxy.java:177)
  at $Proxy11.provideRunSecurityAutomationOnRuleIfRequired(Unknown Source)
  at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:585)
  at com.borland.dx.ejb.SessionBeanConnection.provideDataSets(Unknown Source)
  at com.borland.dx.ejb.SessionBeanProvider.provideData(Unknown Source)
  at com.borland.dx.dataset.StorageDataSet.refresh(Unknown Source)
  at com.smgtec.framework.dx.dataset.RemoteMethod.invokeMethod(RemoteMethod.java:68)
  at com.smgtec.wp.client.dm.WorkPackageClientDM.runSecurityAutomationOnRuleIfRequired(WorkPackageClientDM.java:717)
  at com.smgtec.wp.client.ui.query.WorkPackageOpenerTask.run(WorkPackageOpenerTask.java:149)
  Caused by: java.net.SocketException: Connection reset
  at java.net.SocketInputStream.read(SocketInputStream.java:168)
  at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:284)
  at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:319)
  at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:720)
  at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:677)
  at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
  at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
  at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
  at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:606)
  at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:554)
  at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:571)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:936)
  at sun.net.www.protocol.http.HttpURLConnection.getHeaderFields(HttpURLConnection.java:1888)
  at sun.net.www.protocol.https.HttpsURLConnectionImpl.getHeaderFields(HttpsURLConnectionImpl.java:263)
  at com.smgtec.framework.remote.HttpConnector.queryServer(HttpConnector.java:248)
  … 19 more

  Log in to Reply
• ritu_21
  August 14th, 2012 on 9:25 pm

  WLIOTimeoutSecs (default 300 seconds)
  Also known as HungServerRecoverSecs (old name)
  Defines the amount of time the plug-in waits for a response to a request from WebLogic Server
  The plug-in waits for WLIOTimeoutSecs for the server to respond and then declares that server dead, and fails over to the next server
  The value should be set to a very large value. If the value is less than the time the servlets take to process, then you may see unexpected results

  Hi jay,

  With reference to the above can I ask which servlet are you referring to here.

  Regards
  Ritu

  Log in to Reply
• René van Wijk
  August 15th, 2012 on 5:54 pm

  Servlet, you probably mean server. With server it is meant the WebLogic Servers configured in the plug-in, either through the WebLogicHost and WebLogicPort or WebLogicCluster directives.

  Log in to Reply
• ankur.bhargava
  October 23rd, 2012 on 5:24 pm

  Hi Jay,

  I am getting issue with session in weblogic 10. The issue is when I click on logout in my jsp by using I am forwarding to main controller.java/jpf in my app in begin methods I check the param “islogout=true” if I find it then I am invalidating the session and before invalidating the session I extract some value for e.g. hostname from session in temporary variable. then I am reset the hostname value again in session (newly created one). Then I am forwarding my control to “index.jsp” this JSP is responsible for forwarding to my LoginPortal. I am using . In frame’src I am attaching JSESSION id of session (newly created session in my main controller) but in LoginController I am getting the new
  session (second newly created session) and I am losing the value which I set in (first session) which I got created after session invalidate in my main controllers begin. Please see the “index.jsp” code below.

  Index.jsp code :

  <frame id = 'cmsTop' src="/CMSLogin.portal;JSESSIONID=”+”?hostname=” name=”redirect_main” application=”yes” HEIGHT=”100%” WIDTH=”90%” scrolling=”yes” frameborder=”0″>

  Please help me to get to know why I am getting the new session after redirection. I tried to enable true in my weblogic-application.xml file under It seems my problem got solved but it starts generating the exception on logs. I searched the exception in oracle knowledge base but they suggested to disable it again and its now allow to enable with portal project.

  Please help.

  Thanks,
  Ankur

  Hi Jay, I posted wrong code of jsp here is the code which I am using in index.jsp

  Index.jsp code :

  <frame id = 'cmsTop' src="/CMSLogin.portal;JSESSIONID=” name=”redirect_main” application=”yes” HEIGHT=”100%” WIDTH=”90%” scrolling=”yes” frameborder=”0″>
  

  thanks,
  Ankur

  Index.jsp code :

  <frame id = 'cmsTop' src="/CMSLogin.portal;JSESSIONID= ” name=”redirect_main” application=”yes” HEIGHT=”100%” WIDTH=”90%” scrolling=”yes” frameborder=”0″>
  

  * JSESSIONID = session.getId()

  I dont know why my scriptlet code is disappearing.

  Thanks,
  Ankur

  Log in to Reply
• Avneesh
  December 13th, 2012 on 5:09 am

  Hi,

  I have two node A and B in cluster and both are reverse proxy by an ohs 11g.

  Is there anyway i can configure ohs as it will continue to forward all incoming requests to only one node like node A and when node A is down it will start forwarding requests to node B.

  Thanks,
  Avneesh

  Log in to Reply
• Ramprasad
  May 14th, 2013 on 8:38 pm

  Hi Rene,

  I have a question about the parameter – “Idempotent”. Is this set to ON by default, even though we don’t see it in the httpd.conf file? Or do we need to include it in the httpd.conf file and set to ON if we need it?

  Can you or someone please clarify?

  Thanks,
  -Ramprasad

  Log in to Reply
• prashi89
  July 31st, 2013 on 9:51 am

  run time statistics can be exactly viewed only by this URL http://hostname:portNo/contextPath/servletPath/pathInfo_WebLogicBridgeConfig

  can’t we View this by main LB URL(VIP URL) , instead of hostname and port ..

  because when i access main url like http://google.com/contextPath/servletPath/pathInfo_WebLogicBridgeConfig
  it returns only one weblogic server in list, but if i access individual apache server URL it returns exact list..

  Log in to Reply
• debojitk1
  March 31st, 2014 on 11:38 pm

  Hello,
  I am posting a very typical load related problem that I am not able to solve.
  Let me describe the problem a little bit.
  We have two apaches load balancing a weblogic cluster of 4 nodes. There is a spring based web service hosted on the wl cluster. The weblogic delegates the requests to tuxedo server via WTC connector that does all the heavy lifting. The peak load volume is 20-25 concurrent requests. Now the tuxedo average processing time is 200-500 ms, and we logged the timings in weblogis access log and found the same timings there also. That means weblogic processing does not have much overload in itself. Now when we see the apache access log we see an evidence of 3-4 secs of trip time sometimes during peak load and that timings some times goes beyond 16-20 seconds, and even worse, however the weblogic access log never shows these types of timings.
  My question is when tuxedo is doing the all processing and still returning within an acceptable time limit, then why apache timings and weblogic timings are way apart (sometimes).
  Our investigation and progress so far.
  1. To identify every request at apache and weblogic level we have added apache unique id in the request header that we log in weblogic access log along with timings. To get the unique id in access log we have overridden the weblogic extended log format.
  2. On weblogic we have 4 gb ram and using deterministic garbage collection with 30 ms pause time.
  3. We have a dedicated work manager with fairshare value as 1000 specific for this particular service.

  During load time we see there is more that 20 idle thread in the work manager, however in the application workload section we see 3-4 requests in the pending column.

  Sample logs
  weblogic access log:
  2014-03-31 17:56:23.330 2014-03-31 17:56:23.721 391 7i1Uk6OmXTEAAFPQJ7cAAAAE /webservices/getupgradeavailability POST 200
  apache log:
  [31/Mar/2014:17:56:18 +0100] “POST /webservices/getupgradeavailability HTTP/1.1” 200 918 “-” “7i1Uk6OmXTEAAFPQJ7cAAAAE” 443 |TRIP:4746339| X
  u see that in the weblogic log the reuest took 391 msec, where as at apache level it took 4746339 us sec i.e., 4.74 sec. I just can’t understand this.
  Please help me resolving this.

  apache wl_module config:
  LoadModule weblogic_module /opt/wls_plugin/x86_64/mod_wl_22.so
  #
  WebLogicHost ymuat-01.baplc.com
  WebLogicPort 41059
  #

  WebLogicCluster ymuat100-01.baplc.com:5039,ymuat100-02.baplc.com:5039
  ConnectTimeoutSecs 20
  ConnectRetrySecs 4
  ErrorPage /temp_outage.html
  MatchExpression *
  WLExcludePathOrMimeType /temp_outage.html,/outage_page.html,/cgi-bin/,/admin-cgi/,/ebiz-test-pages
  WLTempDir /ym/logs/apache/ymuat101
  MaxPostSize 1048576

  In the _wl_proxy folder I could see some accumulation that was being removed very quickly though.
  Thanks in advance.
  Debojit

  Log in to Reply
• René van Wijk
  August 8th, 2012 on 12:47 pm

  With session you mean the HttpSession object created for particular clients?

  Note that the session time-out is set to 1800 seconds by default in WebLogic.

  Or are you referring to something else?

  When you want to set parameters on the WebLogic plug-in side, this document can help: http://docs.oracle.com/cd/E21764_01/web.1111/e14395/plugin_params.htm#CIHJHEHH
  – KeepAliveSecs: The length of time after which an inactive connection between the plug-in and WebLogic Server is closed. You must set KeepAliveEnabled to true (ON when using the Apache plug-in) for this parameter to be effective. The value of this parameter must be less than or equal to the value of the Duration field set in the Administration Console on the Server/HTTP tab, or the value set on the server Mbean with the KeepAliveSecs attribute.

  Log in to Reply
• René van Wijk
  August 8th, 2012 on 6:14 pm

  Note that the admin server online help shows min and max values for managed server->configuration->Protocol->HTTP-> Duration

  When you look here (http://docs.oracle.com/javase/7/docs/api/java/net/SocketException.html) the reason of why
  the exception is thrown is shown “Thrown to indicate that there is an error creating or accessing a Socket.”

  Could be the client is sending requests to a connection that has already been closed.

  Log in to Reply
• René van Wijk
  December 13th, 2012 on 1:40 pm

  This behavior is what you get out-of-the-box.

  Log in to Reply