JMS Message Delivered To Busy Instance of JMS Consumer

Shashidhar Mruthyunjaya

Shashidhar Mruthyunjaya

DISCLAIMER:

In this article we may see an abnormal behaviors of weblogic. Which may not be necessarily a BUG but it is always to be aware of such behavior while using Weblogic. The idea behind making this page is just to make awareness among the WebLogic Admins to be alert specially when the some of these behaviors are related to WebLogic Security/JMS.

Some of the behaviors of WebLogic which may be due to inappropriate Security/JMS implementation in the Security/JMS system of WebLogic, Even if in some cases it is work as designed, Still it suggests to keep an eye on it and try to make those features more enhanced. Some of them are now fixed by the Application Server Vendor but still some need to be fixed or enhanced. The intentions here are not to point to the weak points of any Application Server but solely to make people aware about such strange or uncommon behaviors.

==========================================================================

We specially thank to “Mr. Shashidhar Mruthyunjaya” (MiddlewareMagic Subscriber) for making us aware about an abnormal behavior of WebLogic JMS subsystem as part of his comment in http://middlewaremagic.com/weblogic/?page_id=6231#comment-5504

The below code is shared by Shashidhar, we have just  made few changes in sending the messages just to make it more understanding for others and we have tested this code as well as were able to replicated this issue in WLS 10.3.0.

Again thanking Mr. Shashidhar for taking time for us and sharing the complete details about the issue and the steps to reproduce it. Shashidhar also proactively reported this issue to Oracle WebLogic Support team and it resulted in

Bug 13436706 – JMS MESSAGE DELIVERED TO BUSY INSTANCE OF CONSUMER

The Abnormal behavior is happening on weblogic 10.3.2, but I think this bug is lurking in all weblogic servers starting from weblogic 6.x onwards I think. Send 17 messages quickly in succession, 1st one as Long running and other 16 short running messages. You will notice that all the 16 messages run, but the 17th message gets stuck behind the 1st message which is a long running one (and is still running). This is because we have 16 sessions by default and due to weblogic bug, it just cycles around these 16 sessions without checking for free sessions (which there are plenty as short running messages complete quickly and free their JMS sessions)

Basic buggy behavior with weblogic jms code is that it is not looking for free JMS session (or consumer), but going around in a cycle allocating new JMS message arrived to the next session – NOT the next free session as expected. So if the session is already running a long-running message(transaction), the new message will get stuck behind this and will wait until the long-running message finishes. This was causing us frustration as we have some transaction running for hours together!!

==========================
Instructions to simulate
==========================
JMS Configuration: Create JMS step-up by using the WLST script given in the article Setup JMS with Unit-Of-Order using WLST , Script , however you can put “unit.of.order.value = 0″ in the domain.properties file.

Step1) You need to create an EJB project and include ResourceAdapterMDB.java in the same, deploy this

Step2) Create a new Java Project and include this TestMsgSender.java file within that project src.

Step3) Once all setup is done and checking that the EJB is active and listening to the Queue, run the TestMsgSender which has a main method,
which will send 17 messages, 1st one Long running (10 min) and other 16 short running messages. You will notice that the 17th message gets stuck behind the 1st message which is a long running one.

NOTE: Change the queue name and connection factory name and other values to match yours in both these files if required.

ResourceAdapterMDB.java:

import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.ejb.TransactionAttribute;
import javax.ejb.TransactionAttributeType;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;

import weblogic.javaee.TransactionTimeoutSeconds;

/**
* Message driven bean for the Resource Adapter This bean is configured to
* listen to the ResAdptQueue queue
*
* @author Shashidhar Mruthyunjaya
*
*/
@MessageDriven(activationConfig = {
	@ActivationConfigProperty(propertyName = "messageListenerInterface", propertyValue = "javax.jms.MessageListener"),
	@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
	@ActivationConfigProperty(propertyName = "MaxDeliveryCnt", propertyValue = "1"),
	@ActivationConfigProperty(propertyName = "ConnectionFactoryName", propertyValue="My_CF")
}
, mappedName="My_Q")
/*
* @TransactionTimeoutSeconds – Resource Adapter Transaction is set to 24 hours
* 24 x 60 x60 = 86400 seconds.
*/
@TransactionTimeoutSeconds(86400)
public class ResourceAdapterMDB implements MessageListener {

@TransactionAttribute(TransactionAttributeType.REQUIRED)
public void onMessage(Message message) {

TextMessage textMessage = (TextMessage) message;

long millisToSleep = 1000 * 2 ;//2 sec

try {
if("msg= LONG_RUNNING".equals(textMessage.getText())) {
 millisToSleep = 1000 * 60 * 1 ; // Sleep for 1 min
}
} catch (JMSException e) {
e.printStackTrace();
}

try {
Thread.sleep(millisToSleep);
System.out.println("t==> MyMDB Received: "+ textMessage.getText() +", time= "+ millisToSleep/1000 +" sec.");
}
catch (InterruptedException e) {
e.printStackTrace();
}
catch (JMSException e) {
            e.printStackTrace();
        }
}
}

Step4) Following is the code for the JMS Message Sender “TestMsgSender.java” :

import java.util.Date;
import javax.jms.JMSException;
import javax.jms.Queue;
import javax.jms.QueueConnection;
import javax.jms.QueueConnectionFactory;
import javax.jms.QueueSender;
import javax.jms.QueueSession;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.naming.Context;
import javax.naming.NamingException;
import weblogic.jndi.Environment;

/**
* Provides the capability to send asynchronous messages to the defined target
* queues
*
* @author Shashidhar Mruthyunjaya
*
*/
public class TestMsgSender {
private static final String QUEUE = "My_Q";
private static final String CONN_FACTORY = "My_CF";
private static final String WEBLOGIC_USER = "weblogic";
private static final String WEBLOGIC_PASSWORD = "weblogic";
private static final String WEBLOGIC_URL = "t3://localhost:7001";

public void sendMsg(String textMessage) {
QueueConnection connection = null;
QueueSender sender = null;
QueueSession session = null;

try {
QueueConnectionFactory qcf = (QueueConnectionFactory) jndiLookup(CONN_FACTORY);
Queue targetQ = (Queue) jndiLookup(QUEUE);
connection = qcf.createQueueConnection();
session = connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
connection.start();
sender = session.createSender(targetQ);
TextMessage textMsg = session.createTextMessage();
textMsg.setText(textMessage);
sender.send(textMsg);
System.out.println(textMsg+": Sent at "+(new Date()));
} catch (Exception e) {
System.out.println("Exception occurred while sending message: " + e.getMessage());
throw new RuntimeException(e);
} finally {
if (connection != null) {
try {
connection.close();
} catch (JMSException e) {
System.out.println("Exception occurred while closing jms connection " + e.getMessage());
throw new RuntimeException(e);
} finally {
connection = null;
}
}
}
}

/**
* Helper method to perform JNDI lookup
*
* @param name
* @return
* @throws NamingException
*/
private Object jndiLookup(String name) throws NamingException {
Environment env = new Environment();
env.setProviderUrl(WEBLOGIC_URL);
env.setSecurityPrincipal(WEBLOGIC_USER);
env.setSecurityCredentials(WEBLOGIC_PASSWORD);
Context ctx = env.getInitialContext();

// Context ctx = new InitialContext();
Object ret = ctx.lookup(name);
ctx.close();

return ret;
}

public static void main(String[] args) throws InterruptedException {
TestMsgSender genericMsgSender = new TestMsgSender();

String textMessage;

// Send a long running message here
textMessage = "msg= LONG_RUNNING";
genericMsgSender.sendMsg(textMessage);

// Send 16 short running message here
for (int i = 1; i < 17; i++) {
genericMsgSender.sendMsg("msg= "+i);
Thread.sleep(1000 * 1); // wait for 1 sec
}
System.out.println("++++++++++++++++++ DONE Sending all messages ++++++++++++++++++");
}
}

Out put of the above code can be seen in the below snap-short which would make things much more easier to understand

Now from the above snap-shot you can see that the first messages “msg= LONG_RUNNING” was sent which takes 60 seconds to get consumed and just after that 16 more messages are been sent which should get consumed within 2 seconds, that makes total of 17 messages are been sent. You would notice that the messages from msg=1 to msg=15 has been consumed by the MDB within 2 seconds for each, but as the first message is still not been consumed due to which the last messages msg=16 (i.e. 17th number message) is been stuck behind and only gets consumed when the first message get consumed.

Workaround/Solution for this Issue:

================================================
Updated on : 16-Dec-2011 Based on Comment: http://middlewaremagic.com/weblogic/?p=7653#comment-5662

We need to define connection-factory-jndi-name” in “weblogic-ejb-jar.xml” along with setting MaximumMessages to 1 . This somehow makes this issue go away. After these changes the messages get allocated to a free consumer!!
.

.
.
Thanks


A Successful Year of MiddlewareMagic

Hi All,

Middleware Magic has completed it’s 1-year today (7-Nov_2011). It was a great learning for us and your contribution and interest for this Community of Middleware guys are really appreciated. A year back we wanted to start a platform/community where Middleware guys can discuss their issues and can learn new things and technologies involved in Middleware World.

.
René van Wijk

René van Wijk

Jay - Ravish

Jay And Ravish

The Middleware Magic team René van Wijk (Oracle ACE), Ravish Mody and Jay SenSharma has put a lots of effort in building Middleware Magic community by replying to various issues with a proper solution and by writing Production ready and various useful articles which can be used be Middleware Admins and troubleshooters.  Magic team is keep focusing on innovative technologies available on Middleware World to make it more simpler for everyone.
.

In this one year Middleware Magic has grown a lot and has a great database of knowledge where you may find some of the resolutions of your issues. Thank you again for your visits and appreciating our efforts, which can be seen in the following charts still counting…
As you can see in the above chart the Unique hits and page views for Middleware Magic is keeps on growing from the very first day. Also we have seen that many Middleware Magic subscribers also started replying each others queries and discussing their issues with others. These discussions on technical issues are very positive and helping many others to not to fall on those issues again.
.
We specially thank to some of our Middleware Magic Subscribers who keep visiting and sharing their experiences with us and the queries asked by them are helping others.
[TABLE=5]
The Middleware Magic subscribers are exponentially growing all over the world and chosing the MiddlewareMagic to share their experiences/issues with us, as you can see in the following cluster map.

So What’s New in Middleware Magic ?

As the Middleware Magic is keep growing world wide and many of our Magic Subscribers are asking us to add few contents on usages of JBoss Application Server and it’s features, specially when the New Generation JBoss AS7 is released with a lots of amazing and great features of it.  As the demand is keep on growing to see the contents and articles related to JBoss AS7 so we have started writing on JBoss Application Server, which can be visited in the following link:

http://middlewaremagic.com/jboss

JBoss AS7 – The Next Generation
.
.
.
.
Thank You All
.
.
.

Setup JMS with Unit-Of-Order using WLST

Ravish Mody

Yesterday one of our subscribers Miguel had asked us using comment how to configure the Unit-of-Order property using WLST script, as he was already able to create a JMSServer, JMSModule and JMS Connection Factory using a wlst script but was having an issue setting the  Unit-of-Order property, hence it seemed to be a great topic to create an article on it. With below single script you can create and target following components.

  1. JMSServer
  2. JMSModule
  3. Queue
  4. ConnectionFactory
  5. Unit-Of-Order

As usual we have used a properties file which has all the required details in it like the JMSServer, JMSModule, Queue, ConnectionFactory names, targets name etc. This way you would not have to modify anything in your actual WLST script.

Setup JMS with Unit-Of-Order using WLST

Step1). Create a Directory somewhere in your file system like : “C:WLSTJMS

Step2). Write a Properties filedomain.properties inside “C:WLSTJMS” like following:

# 1 - Connecting details
server.url = t3://localhost:7001
username = weblogic
password = weblogic

# 2 - JMSServer details
jms.server.name = My_JMSServer
store.name = MyJDBCStore
tragated.jms.server.name = AdminServer

# 3 - SystemModule Details
system.module.name = My_SystemModule
tragated.system.module.name = AdminServer

# 4 - ConnectionFactory Details
connection.factory.name = My_ConnectionFactory
connection.factory.jndi.name = My_CF

# 5 - Unit Of Order Details
unit.of.order.value = 1

# 6 - SubDeployment & Queue Details
queue.sub.deployment.name = Sub_My_Queue
queue.name = My_Queue
queue.jndi.name = My_Q

Step2). Now in the same directory write the following WLST Script “JMS_Setup.py” like following:

#############################################################################
#
# @author Copyright (c) 2010 - 2011 by Middleware Magic, All Rights Reserved.
#
#############################################################################

from java.io import FileInputStream
import java.lang
import os
import string

propInputStream = FileInputStream("domain.properties")
configProps = Properties()
configProps.load(propInputStream)

# 1 - Connecting details
serverUrl = configProps.get("server.url")
Username = configProps.get("username")
Password = configProps.get("password")

# 2 - JMSServer details
jmsServerName = configProps.get("jms.server.name")
storeName = configProps.get("store.name")
tragatedJMSServerName = configProps.get("tragated.jms.server.name")

# 3 - SystemModule Details
systemModuleName = configProps.get("system.module.name")
tragatedSystemModuleName = configProps.get("tragated.system.module.name")

# 4 - ConnectionFactory Details
connectionFactoryName = configProps.get("connection.factory.name")
ConnectionFactoryJNDIName = configProps.get("connection.factory.jndi.name")

# 5 - Unit Of Order Details
unitOfOrderValue = configProps.get("unit.of.order.value")

# 6 - SubDeployment & Queue Details
queueSubDeploymentName = configProps.get("queue.sub.deployment.name")
queueName = configProps.get("queue.name")
queueJNDIName = configProps.get("queue.jndi.name")

redirect('wlst.log','false')

# 1 - Connecting to the Destination 
connect(Username,Password,serverUrl)

edit()

# 2 - JMSServer details
print "================== JMSSever ==================="
startEdit()
cmo.createJMSServer(jmsServerName)
print "Created a JMSServer !!"
cd('/Deployments/'+jmsServerName)
cmo.setPersistentStore(getMBean('/JDBCStores/'+storeName))
print "PersistentStore has been set for the JMSServer !!"
set('Targets',jarray.array([ObjectName('com.bea:Name='+tragatedJMSServerName+',Type=Server')], ObjectName))
print "Targeted the JMSServer !!"
activate()
print ""

# 3 - SystemModule Details
print "================== SystemModule ==================="
startEdit()
cd('/')
cmo.createJMSSystemResource(systemModuleName)
print "Created a SystemModule !!"
cd('/SystemResources/'+systemModuleName)
set('Targets',jarray.array([ObjectName('com.bea:Name='+tragatedSystemModuleName+',Type=Server')], ObjectName))
print "Targeted the SystemModule !!"
activate()
print ""

# 4 - ConnectionFactory Details
print "================== ConnectionFactory ==================="
startEdit()
cd('/JMSSystemResources/'+systemModuleName+'/JMSResource/'+systemModuleName)
cmo.createConnectionFactory(connectionFactoryName)
cd('/JMSSystemResources/'+systemModuleName+'/JMSResource/'+systemModuleName+'/ConnectionFactories/'+connectionFactoryName)
cmo.setJNDIName(ConnectionFactoryJNDIName)
print "Created a ConnectionFactory !!"
cd('/JMSSystemResources/'+systemModuleName+'/JMSResource/'+systemModuleName+'/ConnectionFactories/'+connectionFactoryName+'/SecurityParams/'+connectionFactoryName)
cmo.setAttachJMSXUserId(false)
cd('/JMSSystemResources/'+systemModuleName+'/JMSResource/'+systemModuleName+'/ConnectionFactories/'+connectionFactoryName)
cmo.setDefaultTargetingEnabled(true)
print "Targeted the ConnectionFactory !!"
activate()
print ""

# 5 - Unit Of Order Details
print "================== Unit Of Order ==================="
startEdit()
cd('/JMSSystemResources/'+systemModuleName+'/JMSResource/'+systemModuleName+'/ConnectionFactories/'+connectionFactoryName+'/DefaultDeliveryParams/'+connectionFactoryName)
cmo.setDefaultUnitOfOrder(unitOfOrderValue)
print "Changed Unit Of Order !!"
activate()
print ""

# 6 - SubDeployment & Queue Details
print "================== SubDeployment & Queue ==================="
startEdit()
cd('/SystemResources/'+systemModuleName)
cmo.createSubDeployment(queueSubDeploymentName)
print "Created a SubDeployment for Queue !!"
cd('/JMSSystemResources/'+systemModuleName+'/JMSResource/'+systemModuleName)
cmo.createQueue(queueName)
print "Created a Queue !!"
cd('/JMSSystemResources/'+systemModuleName+'/JMSResource/'+systemModuleName+'/Queues/'+queueName)
cmo.setJNDIName(queueJNDIName)
cmo.setSubDeploymentName(queueSubDeploymentName)
cd('/SystemResources/'+systemModuleName+'/SubDeployments/'+queueSubDeploymentName)
set('Targets',jarray.array([ObjectName('com.bea:Name='+jmsServerName+',Type=JMSServer')], ObjectName))
print "Targeted the Queue to the created subdeployment !!"
activate()
print ""

cmd = "rm -f wlst.log"
os.system(cmd)

Step3). Now Open a Command/Shell Prompt and then run the “setWLSEnv.sh” script to set the CLASSPATH and PATH environment variables. Run the “. ./setWLSEnv.sh” by adding two DOTs separated by a single space …..before the actual script like following : (use ‘cd’ command to move inside the <BEA_HOME>/wlserver_10.3/server/bin) then run the following command….
. ./setWLSEnv.sh

Note: Here The first DOT represents that set the Environment in the current Shell, AND the second ./ represents execute the script from the current directory.

Step4). Run the Above WLST Script like following:

java weblogic.WLST JMS_Setup.py

Following would be the Output

java weblogic.WLST JMS_Setup.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands


================== JMSSever ===================
Created a JMSServer !!
PersistentStore has been set for the JMSServer !!
Targeted the JMSServer !!

================== SystemModule ===================
Created a SystemModule !!
Targeted the SystemModule !!

================== ConnectionFactory ===================
Created a ConnectionFactory !!
Targeted the ConnectionFactory !!

================== Unit Of Order ===================
Changed Unit Of Order !!

================== SubDeployment & Queue ===================
Created a SubDeployment for Queue !!
Created a Queue !!
Targeted the Queue to the created subdeployment !!

Regards,

Ravish Mody


Copyright © 2010-2012 Middleware Magic. All rights reserved. |