To configure a Uniform Distributed Queue you can follow the steps provided in the post Steps to Create Uniform Distributed Queue (UDQ) on Weblogic Server.
In this post I would be using a stand-alone java code for producing messages as a Producer/Clint and for consuming the messages as a Consumer/Listener. By which we can make sure that if something goes wrong then we can narrow down the issue if its an issue with WLS or the code, as many times their an issue with the Producer or a Consumers.
.
Most Common Problems with UDQ’s
- Un-even load balancing
- No load balancing
The most of the time people come across the issue that the UDQ’s not able to load balance the messages evenly when they are using uniform distributed queues, or that the messages are only going to one server and not to the other servers.
Tips to solve these issues
- Make sure the Server Affinity Enabled parameter should be un-checked (disable) which under the [ Connection factory –> Configuration (tab) –> Load Balance (sub-tab)]
- Disable the Server Affinity Enabled parameter for the connection factory whish is been used by your UDQ.
- All the managed servers are in the same cluster.
- If the managed serves are in different boxes make sure the listing address is given correctly which is under [ Machine –> Configuration (tab) –> Node Manager (sub-tab) ]
- Test if you are able to PING the servers on different boxes and make sure that there is no network issues and you are able to communicate with the servers.
Steps of test the UDQ
- Create a Directory somewhere in your file system like: “D:OracleJMSQueue” to write the QueueSend.java and QueueReceive.java programs.
- In QueueSend.java copy the below command
import java.io.*; import java.util.Hashtable; import javax.jms.JMSException; import javax.jms.Queue; import javax.jms.QueueConnection; import javax.jms.QueueConnectionFactory; import javax.jms.QueueSender; import javax.jms.QueueSession; import javax.jms.Session; import javax.jms.TextMessage; import javax.naming.Context; import javax.naming.InitialContext; import javax.naming.NamingException; public class QueueSend { public final static String JNDI_FACTORY="weblogic.jndi.WLInitialContextFactory"; // ******* Give the CONNECTION FACTORY JNDI name which is UDQ_CF ******** public final static String JMS_FACTORY="UDQ_CF"; // ******* Give the UNIFORM DISTRIBUTED QUEUE JNDI name which is UDQ ******** public final static String QUEUE="UDQ"; private QueueConnectionFactory qconFactory; private QueueConnection qcon; private QueueSession qsession; private QueueSender qsender; private Queue queue; private TextMessage msg; public void init(Context ctx, String queueName) throws NamingException, JMSException { qconFactory = (QueueConnectionFactory) ctx.lookup(JMS_FACTORY); qcon = qconFactory.createQueueConnection(); qsession = qcon.createQueueSession(false, Session.AUTO_ACKNOWLEDGE); queue = (Queue) ctx.lookup(queueName); qsender = qsession.createSender(queue); msg = qsession.createTextMessage(); qcon.start(); } public void send(String message) throws JMSException { msg.setText(message); qsender.send(msg); } public void close() throws JMSException { qsender.close(); qsession.close(); qcon.close(); } public static void main(String[] args) throws Exception { if (args.length != 1) { System.out.println("Usage: java QueueSend WebLogicURL"); return; } InitialContext ic = getInitialContext(args[0]); QueueSend qs = new QueueSend(); qs.init(ic, QUEUE); readAndSend(qs); qs.close(); } private static void readAndSend(QueueSend qs) throws IOException, JMSException { String line="Test Message Body with counter = "; BufferedReader br=new BufferedReader(new InputStreamReader(System.in)); boolean readFlag=true; System.out.println("ntStart Sending Messages (Enter QUIT to Stop):n"); while(readFlag) { System.out.print("<Msg_Sender> "); String msg=br.readLine(); if(msg.equals("QUIT") || msg.equals("quit")) { qs.send(msg); System.exit(0); } qs.send(msg); System.out.println(); } br.close(); } private static InitialContext getInitialContext(String url) throws NamingException { Hashtable env = new Hashtable(); env.put(Context.INITIAL_CONTEXT_FACTORY, JNDI_FACTORY); env.put(Context.PROVIDER_URL, url); return new InitialContext(env); } }
- In QueueReceive.java copy the below command
import java.util.Hashtable; import javax.jms.*; import javax.naming.Context; import javax.naming.InitialContext; import javax.naming.NamingException; public class QueueReceive implements MessageListener { public final static String JNDI_FACTORY="weblogic.jndi.WLInitialContextFactory"; // ******* Give the CONNECTION FACTORY JNDI name which is UDQ_CF ******** public final static String JMS_FACTORY="UDQ_CF"; // ******* Give the UNIFORM DISTRIBUTED QUEUE JNDI name which is UDQ ******** public final static String QUEUE="UDQ"; private QueueConnectionFactory qconFactory; private QueueConnection qcon; private QueueSession qsession; private QueueReceiver qreceiver; private Queue queue; private boolean quit = false; public void onMessage(Message msg) { try { String msgText; if (msg instanceof TextMessage) { msgText = ((TextMessage)msg).getText(); } else { msgText = msg.toString(); } System.out.println("nt<Msg_Receiver> "+ msgText ); if (msgText.equalsIgnoreCase("quit")) { synchronized(this) { quit = true; this.notifyAll(); // Notify main thread to quit } } } catch (JMSException jmse) { jmse.printStackTrace(); } } public void init(Context ctx, String queueName) throws NamingException, JMSException { qconFactory = (QueueConnectionFactory) ctx.lookup(JMS_FACTORY); qcon = qconFactory.createQueueConnection(); qsession = qcon.createQueueSession(false, Session.AUTO_ACKNOWLEDGE); queue = (Queue) ctx.lookup(queueName); qreceiver = qsession.createReceiver(queue); qreceiver.setMessageListener(this); qcon.start(); } public void close()throws JMSException { qreceiver.close(); qsession.close(); qcon.close(); } public static void main(String[] args) throws Exception { if (args.length != 1) { System.out.println("Usage: java QueueReceive WebLogicURL"); return; } InitialContext ic = getInitialContext(args[0]); QueueReceive qr = new QueueReceive(); qr.init(ic, QUEUE); System.out.println("JMS Ready To Receive Messages (To quit, send a "quit" message from QueueSender.class)."); // Wait until a "quit" message has been received. synchronized(qr) { while (! qr.quit) { try { qr.wait(); } catch (InterruptedException ie) {} } } qr.close(); } private static InitialContext getInitialContext(String url) throws NamingException { Hashtable env = new Hashtable(); env.put(Context.INITIAL_CONTEXT_FACTORY, JNDI_FACTORY); env.put(Context.PROVIDER_URL, url); return new InitialContext(env); } }
- Now open 3 – command promt Windows and run “setWLSEn.cmd” in all the three Command Windows to set the Environment (PATH & CLASSPATH).
- Then compile the QueueSend.java and QueueReceive.java programs.
- Now run the code with the following command
- Once this is been done try to send 4 messages in the Prompt one which is sending to the port 7003, however you will notice that the message would be received on Prompt-3 also which is listening on port 7004 as shown in the below snap-shot.
Prompt-1
java QueueSend t3://localhost:7003
Prompt-2
java QueueReceive t3://localhost:7003
Prompt-3
java QueueReceive t3://localhost:7004
Note:
With the “java QueueSend” and “java QueueReceive” command you have to give the<Protocol>://<IP-address>:<Port number>of the managed servers in different prompts.
.
Thus this way we can see that even when we send messages to a particular server through UDQ the messages gets distributed evenly on all the servers in the cluster.
.
Thanks,
Ravish Mody
December 2nd, 2010 on 6:50 pm
Hi Ravish,
What is the purpose of using migratable target instead of normal server instance.
Thanks
Sumit
December 8th, 2010 on 5:12 pm
Hi Sumit,
You can target JMS server to normal server instance too, however if we target the JMS server on a migratable server then we get the advantage of the Migration framework implementation of WebLogic, Which includes both scheduled migrations as well as (Auto Migration) migrations in response to a WebLogic Server failure.
Regards,
Ravish Mody
July 17th, 2011 on 10:12 pm
Hi Ravish,
Thanks for this post. But here are two things I have observed:
1. Sometimes my messages are not getting distributed at all. (Only Server-B is receiving)
2. When I am passing ‘quit’ (from server-A) only B prompt is responding to it (coming back to the prompt with QUIT)
I have configured the same setting you have mentioned here. Please let me know the reason..
Thanks in advance.
Vidhan/-
July 17th, 2011 on 10:21 pm
Hi Vidhan,
I would suggest you to try out the tips given in the above post in the section “Tips to solve these issues” and see if that solves your issue.
Regards,
Ravish Mody
August 25th, 2011 on 3:16 pm
Hi Ravish,
can we use distributed queue as we dont have any connection factory for our MDB’s?
August 25th, 2011 on 6:37 pm
Hi RK,
Yes, you can use distributed queue with your MDB’s. However this MDB has to be deployed on the same server on which the queue resides so that it can pull the messages from it.
Regards,
Ravish Mody
August 26th, 2011 on 12:48 pm
Thanks Ravish,
We are using migratable targets in my environment.Can you please tell me below things.
1.If we have 4 managed servers do we need to create 4 jms servers?
2.You said some issues with distibuted queue we need to disable the server affinity.But we dont have any connection factory.In this case what we need to do?
Thanks
RK
August 29th, 2011 on 5:25 pm
Hi RK,
1.If we have 4 managed servers do we need to create 4 jms servers?
Yes, its always been recommended to do that and it would be even better if you target the JMS servers to a migratable managed server when in cluster which would help you in not to have a single point of failure using JMS automatic migration check out the below article
Steps to Configure JMS Automatic Migration Using Consensus Leasing
2.You said some issues with distibuted queue we need to disable the server affinity.But we dont have any connection factory.In this case what we need to do?
If you are taking about from your MDB point of view then you dont have to worry about the load-balancing as MDB’s are the listeners which picks up the messages from queue/topics. However that point was given for the publisher/producer which sends messages to the UDQ, hence to send messages to a particular queue you would need a connection from the connection factory to connect with the queue.
Regards,
Ravish Mody
October 13th, 2011 on 6:22 am
Hey Ravish,
Thanks for your post – it really helped me understand a lot of things about UDQs however I have few questions.
This is how my setup looks like:
Physical Servers: 2 (call SrvrA, SrvrB)
Admin Server (Server SrvrB)
MS1 (SrvrA) MachineA
MS2 (SrvrB) MachineB
Cluster AB (MS1 and MS2)
I setup as you indicated:
MS1_JMS on MS1 using JDBC store – table prefix MS1_JMS_
MS2_JMS on MS2 using JDBC store – table prefix MS2_JMS_
JMS Module – xyzJmsModule – Cluster – all servers in cluster
Cnx Factory – xyzUDQ_CF – Cluster – all servers in cluster
SubDeployments – xyzSubUDQ – MS1_JMS and MS2_JMS
Distributed Q – xyzUDQ – Allocate Members Uniformly, RRobin – xyzSubUDQ
I am producing the messages using SpringFramework and consuming using MDB. Both the producer and consumers are the same application. The application is deployed on both MS1 and MS2. We are using WebLogic Integration 9.2.2 MP2.
So here are my questions:
1) How can I check the messages are being sent to both MS1 and MS2?
2) If MS1 goes down, will MS2 process messages that were stored in the MS1’s JDBC store (MS1_JMS_WLstore)? Are the messages being written to both MS1_JMS and MS2_JMS JDBC store when received? This also brings me to the question – do the tables need to be different for each JMS server?
3) Given that the JMS servers are on the same servers as the application, I don’t set the URL for performance. How can I check which server processed the message?
4) If you were wondering why we are using the JMS to send data to the same application then here is my explanation. We want to asynchronously process certain logic for which the original creator does not need to wait on. This improves the original creator’s process response time as the extra work – not critical to flow – is not done synchronously. Also if the backend to which the extra work dealt with was done, the original process is not impacted. I hope that makes sense. Any comments to this idea?
I guess that’s enough questions for now. I hope you or anyone else can provide input on this.
Thanks.
October 13th, 2011 on 11:37 pm
Hi Stupify,
1) How can I check the messages are being sent to both MS1 and MS2?
With your application which is a producer and a consumer itself would be difficult to know on which server has the messages is been sent, if you can stop the consumer somehow then you can make out with the messages getting stored in the queue which can be seen from the monitoring tab in the “message current” which shows the current number of messages in the destination. Or else you can use a simple java code as shown in the above post to just send messages without running the receiving code, the same way your code would also work.
2) If MS1 goes down, will MS2 process messages that were stored in the MS1′s JDBC store (MS1_JMS_WLstore)? Are the messages being written to both MS1_JMS and MS2_JMS JDBC store when received? This also brings me to the question – do the tables need to be different for each JMS server?
If the MS1 itself goes down then the messages would not be processed by MS2, however to over come this issue as shown in the configuration of UDQ we have used migratable target which would take care of this issue. More information on migratable target can be found in the below article
Topic:Steps to Configure JMS Automatic Migration Using Consensus Leasing
http://middlewaremagic.com/weblogic/?p=4153
However if the consumer of MS1 goes down then by default messages would not be processed by consumer of MS2 because the value for “Forward Delay” found under Configuration tab is -1, however if you set some value which is in seconds then if the messages are not been picked up by the consumer then it would get forwarded to the queue on MS2 in the given number of seconds.
3) Given that the JMS servers are on the same servers as the application, I don’t set the URL for performance. How can I check which server processed the message?
Frankly I dint get this question, however I believe you are talking about the who has consumed the messages from the UDQ. The server is just a base on top of which UDQ is been placed and server does not processes the messages the consumer pick up the messages from the queue and then process the messages. In your scenario as explained in the question 1 would be difficult.
4) If you were wondering why we are using the JMS to send data to the same application then here is my explanation. We want to asynchronously process certain logic for which the original creator does not need to wait on. This improves the original creator’s process response time as the extra work – not critical to flow – is not done synchronously. Also if the backend to which the extra work dealt with was done, the original process is not impacted. I hope that makes sense. Any comments to this idea?
It does make little sense to me, however would suggest you to keep in mind to know the time consumer takes to process the messages because here you are only thinking one sided from the producer side which should not be affected, but if the consumer is not able to process messages as compared to the producer processing the messages then you might get lot of pending messages in the queue and that my cause a bottle neck for your whole process.
Regards,
Ravish Mody
October 14th, 2011 on 4:05 am
For #1 I had figured it out – started up both MS and looked at their output logs to see which one was processing the message. However I have to do some digging to better check if MS1 is processing the messages on its “local queue” only or can pull from the MS2’s queue too. Not too critical for us. So i will get back to this later.
#2 – I will check it out.
#3 – The question is given that I have an application deployed to both MS1 and MS2 that contains the producer and consumer (MDB), I want to make sure that if MS1 produces the message then it goes to MS1_JMS and that MDB on the MS1 processes the message. Similarly for MS2. This is preferrable so that it limits the network traffic going to the other MS to put messages or consume messages. Is there a way to ensure that does take place while also making it set up for clustered processing and failover?
#4 – The amount of messages being produced should be relatively small – 10000/day – where the system easily processes over 1 Million transactions /day. And the consumer’s task is not too demanding either. So I believe the consumers should be fine.
October 18th, 2011 on 10:51 am
Hi Ravish,
I have 2 Uniform Distributed Queues in my weblogic environment and 8 managed instances (8 JMS servers).
UDQ1 – Targetted to SubDeployment –> 8 JMSServers separately
UDQ2 – Default Targetting –> Cluster
I could able to navigate through the UDQ1 and manage the messages inside the queue. If i click on UDQ1, i can see 8 copies of the UDQ1.
Where as i can not even able to do the above with UDQ2.
Please help me how can i solve this.
Thanks,
Pavan
October 18th, 2011 on 11:41 am
Hi Pavan,
This issue you are seeing because you have not used a SubDeployment for UDQ2 and used default targetting, just use a SubDeployment and you would be able to see the queues and messages inside them also.
Regards,
Ravish Mody
November 3rd, 2011 on 12:32 pm
Hi Ravish,
Had a query on JMS & MDB implementation in weblogic 9.2.3.
I have a J2EE application targeted on a cluster of 4 managed servers and app having 3 MDB’s. These MDB’s listens to a JMS UDQ which is also targeted on same cluster.
The JMS Queue receives some events from another system which has a constraint of configuring only 1 VIP of the server. so basically when the other component push events on to JMS queue it does on a single JMS server – say MS2.
My query is when my application MDB’s will listen to UDQ will the MDB of MS2 only read those messages and pass it to my application to process which will mean event being processed by only one server from cluster or it will get distributed to all?
Thanks,
Ranjan
November 10th, 2011 on 8:57 pm
@Stupify on Oct, 14:
To answer #3: To limit network traffic in a manner you described, you can use ConnectionFactory’s Server Affinity property. That will override DQ’s load-balancing and always try to connect to a member on a local server. Try and check it.
November 10th, 2011 on 9:14 pm
Ravish,
This is really good stuff!! Looking at the clarity of your thoughts around complex subjects like messaging, I am sure you would have tried and tested this yourself! Appreciate your efforts.
I have a few queries around JMS which I hope you would answer soon! I am preparing common guidelines for Weblogic JMS setup and use. I need to know the following:
1) Under what circumstances one should prefer a JDBC Persistence store to a File based store?
1.1) Assume we have a weblogic cluster configuration and each member instance is in a separate physical machine. Should each server instance have a local file store OR file store should be shared across these server instances? Which configuration is better and why?
2) We can use message selectors so that a single queue (or topic) can suffice needs of many consumers. When one should consider creating a new destination (Queue or topic)?
3) Assume one has many different organization-wide projects (managed by different teams) that are going to use JMS on the same Weblogic server platform. How one should consider partitioning the JMS resources for these projects? E.g. how many JMS Servers, File stores, Modules, Destinations etc?
Thanks in advance for your answers!! 🙂
Ashish.
November 11th, 2011 on 7:15 pm
Hi Ashish
1) Under what circumstances one should prefer a JDBC Persistence store to a File based store?
You can get this question answer after reading the below link
Topic: Comparing File Stores and JDBC Stores
http://download.oracle.com/docs/cd/E15523_01/web.1111/e13701/store.htm#CNFGD226
1.1) Assume we have a weblogic cluster configuration and each member instance is in a separate physical machine. Should each server instance have a local file store OR file store should be shared across these server instances? Which configuration is better and why?
For high performance you can go with shared store but you have to make sure that they are targeted to the same server instance or migratable target, more information can be found in below link.
Topic: High-Performance Throughput and Transactional Support
http://download.oracle.com/docs/cd/E15523_01/web.1111/e13701/store.htm#i1126514
2) We can use message selectors so that a single queue (or topic) can suffice needs of many consumers. When one should consider creating a new destination (Queue or topic)?
This totally depends on the load of the incoming messages, if the load is high and seeing lot of pending messages either you can set a quota or else go with the UDQ which a proper load testing so see which one is better choice for your environment and requirement.
3) Assume one has many different organization-wide projects (managed by different teams) that are going to use JMS on the same Weblogic server platform. How one should consider partitioning the JMS resources for these projects? E.g. how many JMS Servers, File stores, Modules, Destinations etc?
I would suggest you to create different Managed servers depending on the organisation departments on top of which the its own JMS servers and file stores would created and different modules for each departments which would help you to overcome the conflicts like having a suffix for each department which is been used in all the components. Regarding the number that totally depends on the requirement, if the one department does not have much load then less number of servers and if the second department has load then more servers for them.
Regards,
Ravish Mody
November 16th, 2011 on 1:37 pm
Hi Ravish,
Thanks for taking time off to answer my queries. Appreciate your efforts.
I know these are difficult questions to answer as there are no thumb-rules and it all depends on use-case at hand.
With regards-
Ashish
September 27th, 2012 on 2:43 am
Ravish
I don’t know if you are the person to ask this, but here goes:
We’re running into an issue with synchronization of code across multiple JVMs. Here is our configuration:
WLS 10.3.6 installed onto a shared file system (/global/beahome)
2 Machines: web1 and web2, with NodeManager running on both and the domains registered with the AdminServer
1 cluster
4 managed servers – (A thru D, with A and B on web1, and C and D on web2), all in cluster
We have a JSP page to test the session persistence, with the following pseudo-Java-code:
synchronized(session)
{
Integer value = (Integer) session.getAttribute(“Value”);
if (value==null)
{
value = new Integer(0);
}
value = new Integer(value.intValue() + 1); // Increment
Thread.sleep(5000);
session.setAttribute(“Value”,value);
}
// Produce message to a JMS Queue, with “value” as TextMessage.
We also have simple Java applications/daemons that consume the JMS messages, and print them out.
What we are finding is that when the requests are all served by the same server (Server A), the code is synchronized correctly, and the session value is incremented correctly. Even with concurrent access for the same Session, the synchronized() call forces the JSP page to wait until all other requests to the same Java block have completed.
However, when the requests are spread across A, B, C, and D (separate JVMs), the synchronized() call does not synchronize access to the Session object, which causes situations like:
1) Browser makes request to server A
2) Server A reads “Value” from Session, and gets “12”
3) Server A increments “Value” to “13”
4) Browser makes request to server B
5) Server B reads “Value” from Session, and gets “12”
6) Server B increments “Value” to “13”
7) Server A stores “13” into “Value” in the Session
8) Server B stores “13” into “Value” in the Session
9) Server A produces a Message on the JMS queue, with text “13”.
10)Server B produces a Message on the JMS queue, with text “13”.
How can this scenario be avoided?
If you cannot answer, can you please direct us to someone who can?
Thanks
September 28th, 2012 on 7:43 pm
Thanks for the reply Rene.
Got another one.
Oracle docs say “You use session persistence to permanently store data from an HTTP session object to enable failover and load balancing across a cluster of WebLogic Servers. When your applications stores data in an HTTP session object, the data must be serializable.”
Is there any way to do this if the data is not serialized?
The reason I’m asking is our apps have a lot of time and effort in their creation (beginning the start of their testing) and it was never believed they would be in a clustered environment, so objects weren’t serialized.
Thanks
November 1st, 2012 on 4:49 am
Ravish
Can i use the same JMS configuration with UDQ for a clustered env with out node managers?
December 12th, 2012 on 4:07 am
Ravish,
In your example you have 2 consumers, right?
Is possible configure the following scenario, where I have a unique consumer to UDQ ?
java QueueSend t3://localhost:7003
java QueueReceive t3://localhost:7003, localhost:7004
January 7th, 2013 on 1:38 pm
Hi Ravish,
I have a similar problem using path service to load balance the UUO.
I would like to control the distribution of the messages while using path service with distributed destination. For example : Supposing we have 2 big UUOs ( that will have much more messages) and 2 small UUOS, distributed on 2 JMS Servers.
Do you have any trick(s) that could distribute 1big UUO on each JMS Server in order to avoid big difference in utilization of both machine?
Thank you so much for your help.
January 8th, 2013 on 2:34 pm
Thx René for your reply.
UOO messages sent to a distributed queue reside on only one physical member of the distributed queue so I am wondering how we can control the loading by forcing messages of UOO1 sent to JMSserver1 and messages of U002 sent to JMSserver2.
January 9th, 2013 on 6:13 am
Thanks ,
Even using Unit-of-Order,The queue still could load balance the messages ?
Cause only one distributed destination member consumes messages for a particular Unit-of-Order.
April 18th, 2013 on 1:36 am
Ravish Mody
Do you have any example with request and reply within the same code and creating TemporaryQueue ?
November 12th, 2013 on 6:53 am
Hi,
Can some one help me to resolve Load balancing issue.
I have double checked all the configuration as mention above.It’s Looks All Good.
Scenario are as below
A(Mainframes System) -> B (OSB) ->C(wls10.3.0.0).
The B interface application is posting the message to C interface UDQ. But the message are flowwing in single instance only. It’s not load balancing the message.
Thanks In advance!!!
January 13th, 2014 on 5:57 pm
I tried to configure fail over of JMS clustering in weblogic (Persistent Store).
But I get the below error,
Exception in thread “main” javax.naming.NameNotFoundException: Unable to resolve ‘DistributedSterlingQCF’. Resolved ” [Root exception is javax.naming.NameNotFoundException: Unable to resolve ‘DistributedSterlingQCF’. Resolved ”]; remaining name ‘DistributedSterlingQCF’
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
Eventhough I have a distributed queue configured. Any help or suggestions?
I would provide further information if required.
January 22nd, 2014 on 12:25 pm
Hi Rene,
I have a distributed queue targeted for two JMS Servers under cluster say SterlingJMSServer, FailoverJMSServer.
Configuration:
Name – SterlingDistributedQueue, Type – Uniform Distributed Queue, JNDI Name – SDQ, Subdeployment – SterlingSubDep, Targets- FailoverJMSServer, SterlingJMSServer.
Is there any thing I need to work on the JNDI naming part?
Please reply.