Middleware Magic

This page is having a slideshow that uses Javascript. Your browser either doesn't support Javascript or you have it turned off. To see this page as it is meant to appear please use a Javascript enabled browser.

WebLogic Command-Line Monitoring

by René van Wijk under JMX, Monitoring, WLST

René van Wijk

Recently, I was working on a project that involved JBoss AS7. JBoss AS7 has a very handy command-line interface that enables browsing management beans by using Linux commands, such as, cd, ls etcetera. It also has the ability to use the tab button for completion. Afterwards, you start to wonder if this is also possible when browsing management beans in a WebLogic environment. Of course, there is the WebLogic Scripting Tool (WLST), with which we can build (Jython) scripts in order to access the WebLogic runtime environment. Sometimes we want to access some runtime information on the fly, and a tool such as the JBoss AS7 command-line interface lets you do that, i.e., connect to a server and start browsing the management beans of interest (of course, one can also use a tool such as jconsole). After some browsing I stumbled upon this site, which introduced (to me) tools that enable the server command-line monitoring we are looking for. The site discusses jmx4perl (that includes the sought after command-line interface) and the agent jolokia.

To set-up jmx4perl (on Linux) we can follow these steps. First download a jmx4perl distribution. Next, execute the following commands:

tar xvfz jmx4perl-1.05.tar.gz
cd jmx4perl-1.05
perl Build.PL

Install 'jmx4perl' ? (y/n) [y ]y
Install 'check_jmx4perl' ? (y/n) [y ]n (can be used with a nagios agent)
Install 'cacti_jmx4perl' ? (y/n) [y ]n (can be used with a cacti agent)
Install 'j4psh' ? (y/n) [y ]y (this is the command-line interface)
Use Term::ReadLine::Gnu ? (y/n) [n ]n
Install 'jolokia' ? (y/n) [y ]n (this we will deploy separately to WebLogic as a web application)

It could be that the perl Build.PL command complains about necessary (extra) dependencies. The before mentioned site has packaged these dependencies in an easy to use tar:

wget http://download.op5.com/tmp/jmx4perl_dependencies.tar.gz
tar xvfz jmx4perl_dependencies.tar.gz
rpm -Uvh perl-ExtUtils-ParseXS-2.20.02-1.el5.rf.noarch.rpm
rpm -Uvh perl-ExtUtils-CBuilder-0.2603.01-1.el5.rf.noarch.rpm
rpm -Uvh perl-Module-Build-0.3607-1.el5.rf.noarch.rpm

During the creation of the build script a message might appear that there are still some dependencies missing, which can be installed by running

./Build installdeps

Next, we can build, test and install the packages

./Build
./Build test
./Build install

The following step we need to perform is deploying the Jolokia agent to the WebLogic Server. A distribution of Jolokia can be downloaded here. Download the .war file and deploy it to the WebLogic Server we want to monitor. When the agent is deployed successfully, it can be accessed by using: http://192.168.1.50:7001/jolokia-war-1.0.3, which will show something like:

{"timestamp":1336387435,"status":200,"request":{"type":"version"},"value":{"protocol":"6.1","agent":"1.0.3","info":{"product":"weblogic","vendor":"Oracle","version":"12.1.1.0"}}}

It is also possible to access management beans by entering the right URL, for example by using: http://192.168.1.50:7001/jolokia-war-1.0.3/read/com.bea:Name=jolokia-war-1.0.3,ServerRuntime=AdminServer,Type=ApplicationRuntime, which will lead to the following output:

{"timestamp":1336387635,"status":200,"request":{"mbean":"com.bea:Name=jolokia-war-1.0.3,ServerRuntime=AdminServer,Type=ApplicationRuntime","type":"read"},"value":{"WseeRuntimes":[],"ApplicationName":"jolokia-war-1.0.3","Parent":{"objectName":"com.bea:Name=AdminServer,Type=ServerRuntime"},"EAR":false,"CoherenceClusterRuntime":null,"ApplicationVersion":null,"Type":"ApplicationRuntime","WseeV2Runtimes":[],"LibraryRuntimes":null,"ClassRedefinitionRuntime":null,"WorkManagerRuntimes":[{"objectName":"com.bea:ApplicationRuntime=jolokia-war-1.0.3,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime"}],"MinThreadsConstraintRuntimes":[],"HealthState":{"critical":false,"subsystemName":null,"state":0,"reasonCode":[],"mBeanType":null,"mBeanName":null},"Name":"jolokia-war-1.0.3","ActiveVersionState":2,"KodoPersistenceUnitRuntimes":[],"OptionalPackageRuntimes":[],"PersistenceUnitRuntimes":[],"RequestClassRuntimes":[],"ComponentRuntimes":[{"objectName":"com.bea:ApplicationRuntime=jolokia-war-1.0.3,Name=AdminServer_\/jolokia-war-1.0.3,ServerRuntime=AdminServer,Type=WebAppComponentRuntime"}],"QueryCacheRuntimes":[],"MaxThreadsConstraintRuntimes":[]}}

To use the jmx4perl command-line interface we can use

[oracle@axis-into-ict ~]$ j4psh 

[j4psh] : connect http://192.168.1.50:7001/jolokia-war-1.0.3
Connected to 192.168.1.50:7001 (http://192.168.1.50:7001/jolokia-war-1.0.3).

[192.168.1.50:7001] :
quit    history cd      ls      error   help    servers connect
[192.168.1.50:7001] :

By hitting the tab button we can see the available commands. To get information about the Jolokia deployment we can use:

[192.168.1.50:7001 com.bea] : cd Name=jolokia-war-1.0.3,Type=App
Name=jolokia-war-1.0.3,Type=AppDeployment           Name=jolokia-war-1.0.3,Type=Application
[192.168.1.50:7001 com.bea] : cd Name=jolokia-war-1.0.3,Type=AppDeployment
[192.168.1.50:7001 com.bea:Name=jolokia-war-1.0.3, ...] :

A good explanation of how the WebLogic Server MBeans are organized is provided here. WebLogic Server follows a convention in which object names for child MBeans contain part of its parent MBean object name. WebLogic Sever naming conventions encode its MBean object names as follows:

com.bea:Name=name,Type=type[,TypeOfParentMBean=NameOfParentMBean][,TypeOfParentMBean1=NameOfParentMBean1]...

In the preceding MBean object name convention:

com.bea: – is the JMX domain name.
Name – is the unique string provided when the resource was created that the MBean represents. In the example above the name is created during the deployment of the application, which is this case equals jolokia-war-1.0.3.
Type – is, for configuration MBeans and run-time MBeans, the short name of the MBean’s type. The short name is the unqualified type name without the MBean suffix. For example, for an MBean that is an instance of the AppDeploymentMBean, use AppDeployment.
TypeOfParentMBean – To create a hierarchical namespace, WebLogic Server MBeans use one or more instances of this attribute in their object names. The levels of the hierarchy are used to indicate scope. For example, the ApplicationRuntimeMBean that is a child of the AdminServer ServerRuntimeMBean uses ServerRuntime=AdminServer in its object name: com.bea:ApplicationRuntime=jolokia-war-1.0.3,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime.

When we have reached the MBean of interest use cat (and tab) to see the available attributes, for example,

[192.168.1.50:7001 com.bea:Name=jolokia-war-1.0.3, ...] : cat
InstallDir                             LocalInstallDir                        OnDemandContextPaths                   AutoDeployedApp
SubDeployments                         PlanDir                                Parent                                 AbsolutePlanDir
Type                                   VersionIdentifier                      LocalSourcePath                        RootStagingDir
AbsolutePlanPath                       DeploymentOrder                        LocalPlanDir                           InternalApp
BackgroundDeployment                   SecurityDDModel                        ApplicationName                        AppMBean
DeploymentPrincipalName                PlanPath                               CompatibilityName                      Name
DeploymentPlan                         DeploymentPlanExternalDescriptors      LocalPlanPath                          OnDemandDisplayRefresh
SourcePath                             StagingMode                            Notes
AbsoluteSourcePath                     ModuleType                             AbsoluteInstallDir
ValidateDDSecurityData                 ApplicationIdentifier                  Targets

[192.168.1.50:7001 com.bea:Name=jolokia-war-1.0.3, ...] : cat Targets
    [
      {
        objectName => 'com.bea:Name=AdminServer,Type=Server'
      }
    ]

To obtain runtime information about created resources such as, for example, a save-and-forward agent, we can use

[192.168.1.50:7001 com.bea] : cd Name=SAFAgent@AdminServer,ServerRuntime=AdminServer,Type=JMSServerRuntime
[192.168.1.50:7001 com.bea:Name=SAFAgent@AdminServer, ...] : cat
BytesCurrentCount                      MessagesCurrentCount                   DestinationsHighCount                  PagingAllocatedWindowBufferBytes
MessagesHighCount                      MessagesPagedInTotalCount              MessagesThresholdTime                  BytesPendingCount
PendingTransactions                    ConsumptionPaused                      MessagesPendingCount                   InsertionPausedState
ProductionPausedState                  ConsumptionPausedState                 BytesHighCount                         BytesPagedInTotalCount
BytesPagedOutTotalCount                InsertionPaused                        MessagesPageableCurrentCount           DestinationsTotalCount
SessionPoolsCurrentCount               BytesReceivedCount                     DestinationsCurrentCount               HealthState
Transactions                           SessionPoolsHighCount                  Parent                                 Name
Type                                   PagingAllocatedIoBufferBytes           BytesPageableCurrentCount              PagingPhysicalWriteCount
SessionPoolRuntimes                    BytesThresholdTime                     Destinations                           ProductionPaused
MessagesPagedOutTotalCount             SessionPoolsTotalCount                 MessagesReceivedCount

[192.168.1.50:7001 com.bea:Name=SAFAgent@AdminServer, ...] : cat *Curr*
 BytesCurrentCount               0
 SessionPoolsCurrentCount        0
 MessagesCurrentCount            0
 MessagesPageableCurrentCount    0
 DestinationsCurrentCount        1
 BytesPageableCurrentCount       0

Note the use of the wild cards in the previous command. To obtain some information about the Java Virtual Machine we can use

[192.168.1.50:7001 com.bea] : cd Name=AdminServer,ServerRuntime=AdminServer,Type=JRockitRuntime
[192.168.1.50:7001 com.bea:Name=AdminServer, ...] : cat
Uptime                         HeapFreeCurrent                FreeHeap                       JvmProcessorLoad               TotalNumberOfThreads
OSName                         HeapSizeCurrent                JavaVendor                     LastGCEnd                      Concurrent
NumberOfDaemonThreads          UsedPhysicalMemory             HeapFreePercent                TotalHeap                      UsedHeap
ThreadStackDump                FreePhysicalMemory             JVMDescription                 PauseTimeTarget                JavaVMVendor
Type                           Version                        Vendor                         AllProcessorsAverageLoad       Name
TotalNurserySize               JavaVersion                    GcAlgorithm                    TotalPhysicalMemory            Parallel
Incremental                    OSVersion                      LastGCStart                    Generational                   GCHandlesCompaction
NumberOfProcessors             HeapSizeMax                    Parent                         TotalGarbageCollectionCount    TotalGarbageCollectionTime

[192.168.1.50:7001 com.bea:Name=AdminServer, ...] : cat *Total*
 TotalNurserySize                103269328
 TotalHeap                       536870912
 TotalPhysicalMemory             3158638592
 TotalGarbageCollectionCount     321
 TotalNumberOfThreads            42
 TotalGarbageCollectionTime      15324

[192.168.1.50:7001 com.bea:Name=weblogic.socket.M...] : pwd
com.bea:Name=weblogic.socket.Muxer,ServerRuntime=AdminServer,Type=ExecuteQueueRuntime

[192.168.1.50:7001 com.bea:Name=weblogic.socket.M...] : cat *Count
 ExecuteThreadCurrentIdleCount   0
 PendingRequestCurrentCount      0
 ServicedRequestTotalCount       3
 ExecuteThreadTotalCount         3

The last two entries show information about the execute queue and the use of the pwd command.

References

[1] Jolokia Download.
[2] JMX4Perl Download.
[3] Open Source Monitoring.
[4] Developing Custom Management Utilities With JMX for WebLogic Server.

JMX, Monitoring, weblogic, WLST 2 Comments more...

Apr/12

WebLogic Messaging Bridge and Store-and-Forward Service

by René van Wijk under Cluster, JMS

René van Wijk

In this post we first set-up a WebLogic environment that uses the WebLogic Messaging Bridge to forward messages. Next, we use the store-and-forward service to do the same. We will use Spring to test the set-ups. We end the post by looking at some performance considerations.

Messaging bridge

The WebLogic messaging bridge is a forwarding mechanism that provides interoperability between WebLogic JMS implementations, and between JMS and other messaging products. Use the Messaging Bridge to integrate messaging applications between:

Any two implementations of WebLogic JMS, including those from separate releases of WebLogic Server
WebLogic JMS implementations that reside in separate WebLogic domains
WebLogic JMS and a third-party JMS product

A messaging bridge instance forwards messages between a pair of bridge source and target destinations. These destinations are mapped to a pair of bridge source and target destinations. The messaging bridge reads messages from the source bridge destination and forwards those messages to the target bridge destination. For WebLogic JMS and third-party JMS products, a messaging bridge communicates with source and target destinations using the Java EE Connector Architecture (JCA) resource adapters provided with WebLogic Server. Two resource adapters are provided, i.e.,

eis.jms.WLSConnectionFactoryJNDIXA – Provides transaction semantics. Used when the required quality of service (QOS) is exactly-once (the message will be delivered from the sending destination to the receiving destination using XA transactions so that the receiver gets exactly one copy of each message). This envelopes a received message and sends it within a user transaction (XA/JTA). The following requirements apply to use of this resource adapter:
- Any WebLogic Server implementation being bridged must be release 7.0 or later
- The source and target JMS connection factories must be configured to use the XAConnectionFactory
eis.jms.WLSConnectionFactoryJNDINoTX – Provides no transaction semantics. Used when the required QOS is atmost-once (Makes sure that the receiving destination receives only a single copy of the message or does not receive it at all) or duplicate-okay (the bridge acknowledges receiving the message from the source destination only after writing it to the target destination, because this is done outside the scope of a transaction, failures after writing the message to the target and before acknowledging the source can result in duplicate messages being delivered but should never result in a message being lost, this type of delivery is better known as at-least-once delivery). If the requested QOS is atmost-once, the resource adapter uses AUTO_ACKNOWLEDGE mode. If the requested QOS is duplicate-okay, CLIENT_ACKNOWLEDGE is used.

An instance of the messaging bridge maps each source destination with a target destination. Each destination is configured using one of the adapters. Each bridge instance is targeted to a specific WebLogic Server instance. If the source is a distributed destination, the JMS consumer load balancing rules will associate the bridge with a single destination. In this case, it is best to connect a separate bridge instance to each member of the source destination. This leads to a proliferation of bridge instances that must be reconfigured if the cluster membership changes. The messaging bridge must be used when storing and forwarding messages between JMS destinations where one or both destinations are either hosted by foreign JMS providers or running on older versions of WebLogic Server (prior 9.0) that do not support the store-and-forward service.

Configuration

To set-up a messaging bridge, we need to first set-up the JMS resources involved in the bridge, i.e., the connection factories and destinations. In the example below we use two WebLogic environments. On one WebLogic environment we create the following:

Persistent store targeted to the admin server
JMS Server targeted to the admin server
JMS Module targeted to the admin server with the following resources:
- Connection Factory default targeting enabled, with JNDI jms/BridgeConnectionFactory (and is XA enabled)
- Destination (Queue) targeted through a sub deployment to the JMS Server, with JNDI jms/BridgeCompanyQueue

On the other environment we create the following:

A cluster consisting of two managed servers
Two JMS servers each targeted to a migratable target belonging to a managed server in the cluster (note that a migratable target is automatically created when a managed server is clustered
Two persistent stores each targeted to a managed server in the cluster
JMS module targeted to the cluster with the following resources:
- Connection Factory default targeting enabled, with JNDI jms/ConnectionFactory (and is XA enabled)
- Uniform distributed destination (queue) targeted through a subdeployment to the JMS Servers, with JNDI jms/CompanyQueue

Next, we configure two JMS bridge destinations. One source destination from which the messaging bridge instance reads messages and one target destination where the messaging bridge instance sends the messages it receives from the source destination. In the admin console

Open the tree services, messaging, bridges
Click JMS bridge destinations, click new and enter the following parameters:
- Name: SourceDestination
- Adapter JNDI Name: eis.jms.ConnectionFactoryJNDINoTx
- Adapter Classpath: When connecting to a third-party JMS product, the bridge destination must supply the product’s CLASSPATH in the WebLogic Server CLASSPATH.
- Connection URL: t3://192.168.1.50:7001
- Connection Factory JNDI Name: jms/BridgeConnectionFactory
- Destination JNDI Name: jms/BridgeCompanyQueue
Click OK

The target destination is configured as

Open the tree services, messaging, bridges
Click JMS bridge destinations, click new and enter the following parameters:
- Name: TargetDestination
- Adapter JNDI Name: eis.jms.ConnectionFactoryJNDINoTx
- Adapter Classpath: When connecting to a third-party JMS product, the bridge destination must supply the product’s CLASSPATH in the WebLogic Server CLASSPATH.
- Connection URL: t3://192.168.1.50:9001,192.168.1.50:9002
- Connection Factory JNDI Name: jms/ConnectionFactory
- Destination JNDI Name: jms/CompanyQueue
Click OK

The actual messaging bridge can be created as follows:

Open the tree services, messaging, bridges
Click new and enter the following parameters:
- Name: WebLogicToWebLogicBridge
- Selector: can be left empty
- Quality Of Service: atmost-once
- Started: enabled
Click next and select the source destination in our case this is SourceDestination
Click next and select the messaging provider in our case this is WebLogic Server 7.0 or higher
Click next and select the target destination in our case this is TargetDestination
Click next and select the messaging provider for the target in our case this is WebLogic Server 7.0 or higher
Click next and target the messaging bridge to the server that holds to the source destination, which is our case is the adminserver
Click next and click finish

Note that after the creation the messaging bridge can be fine tuned.

WLST

The following script shows an example how the messaging bridge can be set-up using WLST

print 'CONNECT TO ADMIN SERVER';
connect('weblogic', 'magic12c', 't3://192.168.1.50:7001');

print 'START EDIT MODE';
edit();
startEdit();

print 'CREATE SOURCE JMS BRIDGE DESTINATION';
cmo.createJMSBridgeDestination('SourceDestination');
sourcedestination = cmo.lookupJMSBridgeDestination('SourceDestination');
sourcedestination.setClasspath('');
sourcedestination.setConnectionURL('t3://192.168.1.50:7001');
sourcedestination.setAdapterJNDIName('eis.jms.WLSConnectionFactoryJNDINoTX');
sourcedestination.setConnectionFactoryJNDIName('jms/BridgeConnectionFactory');
sourcedestination.setDestinationJNDIName('jms/BridgeCompanyQueue');

print 'CREATE TARGET JMS BRIDGE DESTINATION';
cmo.createJMSBridgeDestination('TargetDestination');
targetdestination = cmo.lookupJMSBridgeDestination('TargetDestination');
targetdestination.setClasspath('');
targetdestination.setConnectionURL('t3://192.168.1.50:9001,192.168.1.50:9002');
targetdestination.setAdapterJNDIName('eis.jms.WLSConnectionFactoryJNDINoTX');
targetdestination.setConnectionFactoryJNDIName('jms/ConnectionFactory');
targetdestination.setDestinationJNDIName('jms/CompanyQueue');

print 'CREATE MESSAGING BRIDGE';
cmo.createMessagingBridge('Bridge');
bridge = cmo.lookupMessagingBridge('Bridge');
adminserver = cmo.lookupServer('AdminServer');
targets = bridge.getTargets();
targets.append(adminserver);
bridge.setTargets(targets);
bridge.setSourceDestination(sourcedestination);
bridge.setTargetDestination(targetdestination);
bridge.setStarted(true);
bridge.setSelector('');
bridge.setQualityOfService('Atmost-once');

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

Test

To test if the set-up works we will use a Spring client. Make sure the wlclient and wljmsclient jar files are on the classpath. To set-up a JMS producer using Spring we can use the following configuration:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="sourceJndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <property name="environment">
            <props>
                <prop key="java.naming.factory.initial">weblogic.jndi.WLInitialContextFactory</prop>
                <prop key="java.naming.provider.url">t3://192.168.1.50:7001</prop>
            </props>
        </property>
    </bean>
    <bean id="sourceConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="sourceJndiTemplate"/>
        <property name="jndiName" value="jms/BridgeConnectionFactory"/>
    </bean>
    <bean id="sourceDestination" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="sourceJndiTemplate"/>
        <property name="jndiName" value="jms/BridgeCompanyQueue"/>
    </bean>
    <bean id="sourceJmsTemplate" class="org.springframework.jms.core.JmsTemplate">
        <property name="connectionFactory" ref="sourceConnectionFactory"/>
        <property name="defaultDestination" ref="sourceDestination"/>
    </bean>
    <bean id="sender" class="model.logic.JMSSender">
        <property name="jmsTemplate" ref="sourceJmsTemplate"/>
    </bean>
</beans>

To send a message by using the configured JMSTemplate we can use the following class:

package model.logic;

import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.core.MessageCreator;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.Session;
import javax.jms.TextMessage;
import java.util.Random;

public class JMSSender {

    private Random generator = new Random();
    private JmsTemplate jmsTemplate;

    public JMSSender() {
    }

    public JmsTemplate getJmsTemplate() {
        return jmsTemplate;
    }

    public void setJmsTemplate(JmsTemplate jmsTemplate) {
        this.jmsTemplate = jmsTemplate;
    }

    public void sendMessage() {
        getJmsTemplate().send(new MessageCreator(){
            public Message createMessage(Session session) throws JMSException {
                TextMessage message = session.createTextMessage();
                message.setText(Long.toString(Math.abs(generator.nextLong()), 36));
                return message;
            }
        });
    }
}

To run the class we can use:

package model.test;

import model.logic.JMSSender;
import model.utils.SpringUtilities;

public class JMSTest {

    public static void main(String[] args) {
        JMSSender jmsSender = SpringUtilities.getJMSSender();
        jmsSender.sendMessage();
    }
}

To consume a message we create another client using Spring’s SimpleMessageListenerContainer. We have the following configuration:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="targetJndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <property name="environment">
            <props>
                <prop key="java.naming.factory.initial">weblogic.jndi.WLInitialContextFactory</prop>
                <prop key="java.naming.provider.url">t3://192.168.1.50:9001,192.168.1.50:9002</prop>
            </props>
        </property>
    </bean>
    <bean id="targetConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="targetJndiTemplate"/>
        <property name="jndiName" value="jms/ConnectionFactory"/>
    </bean>
    <bean id="targetDestination" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="targetJndiTemplate"/>
        <property name="jndiName" value="jms/CompanyQueue"/>
    </bean>
    <bean class="org.springframework.jms.listener.SimpleMessageListenerContainer">
        <property name="connectionFactory" ref="targetConnectionFactory"/>
        <property name="destination" ref="targetDestination"/>
        <property name="messageListener" ref="receiver"/>
    </bean>
    <bean id="receiver" class="model.logic.JMSReceiver"/>
</beans>

To receive a message asynchronously we use:

package model.logic;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;

public class JMSReceiver implements MessageListener {
    public JMSReceiver() {
    }

    public void onMessage(Message message) {
        TextMessage text = (TextMessage)message;
        try {
            message.acknowledge();
            System.out.println("received the following message: " + text.getText());
        } catch (JMSException e) {
            e.printStackTrace();
        }
    }
}

Store-and-forward service

The Store-and-Forward (SAF) service enables WebLogic Server to deliver messages reliably between applications that are distributed across WebLogic Server instances. JMS modules utilize the SAF service to enable local JMS message producers to reliably send messages to remote queues or topics without worrying about the availability of the remote environment. If the destination is not available at the moment the messages are sent, either because of network problems or system failures, then the messages are saved on a local server instance, and are forwarded to the remote destination once it becomes available. The SAF Service should be used when forwarding JMS messages between WebLogic Server 9.x or later domains. The SAF service can deliver messages:

Between two stand-alone server instances
Between server instances in a cluster
Across two clusters in a domain
Across separate domains

When not to use the SAF service:

Forwarding messages to prior releases of WebLogic Server
Interoperating with third-party JMS products. For these tasks, we should use the WebLogic Messaging Bridge
When using temporary destinations with the JMSReplyTo field to return a response to a request

Additionally, when using JMS SAF, an application can only receive messages directly from a remote server, and only when the remote server is available. Both the store-and-forward service and the messaging bridge provide JMS applications a message forwarding agent-based technology. A thing to note is that the store-and-forward service uses an internal duplicate elimination algorithm that does not require XA transactions and thus will perform better for the exactly-once quality of service. The store-and-forward service uses agents to store and forward the messages between the sending and receiving sides. We must configure store-and-forward agents to support sending, receiving, or both depending on the set-up being used:

JMS sending – only a sending agent
JMS receiving – no agent needed
Web Services Reliable Messaging sending – sending and receiving agent
Web Services Reliable Messaging receiving – receiving agent

SAF agents are similar to JMS servers in that they have persistent stores, paging directories, and destinations, as well as quotas, thresholds, and other similar configuration parameters. The primary difference is that SAF agents only support imported destinations, a local representation of remote destinations to which messages are stored locally and then forwarded. SAF agents also support targeting to migratable targets to enable SAF agent service migration. Reliability in SAF is time-based in that the time-to-live attribute determines how long the agent will attempt to forward the message before expiring it. When setting a time-to-live on one of the SAF objects, a value of -1 means that the value is not set, 0 means that the message never expires, and a positive value defines the number of milliseconds after the message was created that the message will expire. If a message expires, SAF error handling provides four expiration policies from which to choose: Discard, Log, Redirect, and Always-forward. Always-forward ignores the time-to-live setting on the imported destinations and any message expiration time and forwards the message even after it has expired. Typically, this option would be used if the application had expiration policies set up on the remote destinations and we want the expired messages to be handled using these policies.

Configuration

To set-up a store-and-forward service we can use the following steps. First, we create a SAF agent:

In the admin console click services, messaging, store and forward agent
Click new and enter the following parameters:
- name: SAFAgent
- persistent store: AdminFileStore (or create a new one)
- agent type: sending-only
Click next and enter the following parameters:
- target: AdminServer (the same as the target of the persistent store)

Next, create a JMS module or use an existing one and create a subdeployment for the SAF agent

Click subdeployments, click new and enter the following parameters:
- subdeployment name: SAFSubDeployment
Click next and select as target the created SAF agent
Click finish

Subsequently, we create a remote SAF context

Click the configuration tab of the JMS module
Click new, select remote SAF context, click next and enter the following parameters:
- name: RemoteSAFContext
- URL: t3://192.168.1.50:9001,192.168.1.50:9002
- username: weblogic (admin username)
- password: magic12c (admin password)
- confirm password: magic12c
Click OK

Next, create SAF imported destinations

Click the configuration tab of the JMS module
Click new, select SAF imported destinations, click next and enter the following parameters:
- name: SAFImportedDestinations
- JNDI prefix: jms/ (this can be used for the local destination counterparts of remote destinations to which the SAF agent is connecting)
- remote SAF context: RemoteSAFContext
Click next and click advanced targeting
Select SAFSubDeployment and click finish

In the last step we will add remote queues to the created SAF imported destination

Click the create SAF imported destination
Click the queues, configuration tab, click new and enter the following parameters:
- name: SAFQueue
- remote JNDI name: jms/CompanyQueue
Click OK
Click SAFQueue and set the local JNDI name to SAFCompanyQueue (note that we can now look up the local queue by using jms/SAFCompanyQueue, in which jms/ is the JNDI prefix we set earlier)
Save the configuration

To check if the configuration is correct check the server logging, something like the following should be present

####<Apr 16, 2012 10:43:49 AM CEST> <Info> <JMS> <axis-into-ict.nl> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1334565829666> <BEA-040506> <The JMS store-and-forward (SAF) forwarder has successfully connected to the remote destination "t3://192.168.1.50:9001,192.168.1.50:9002/jms/CompanyQueue".>

When the servers to which the SAF agent is connecting are not present the following exception is observed

####<Apr 16, 2012 10:43:07 AM CEST> <Info> <JMS> <axis-into-ict.nl> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1334565787556> <BEA-040507> <The JMS store-and-forward (SAF) forwarder failed to connect to the remote destination "t3://192.168.1.50:9001,192.168.1.50:9002/jms/CompanyQueue", because of javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://192.168.1.50:9001,192.168.1.50:9002: Destination unreachable; nested exception is:
	java.net.ConnectException: Connection refused; No available router to destination]
	at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
	at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:767)
	at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:366)
	at weblogic.jndi.Environment.getContext(Environment.java:315)
	at weblogic.jndi.Environment.getContext(Environment.java:285)
	at weblogic.jndi.Environment.createInitialContext(Environment.java:208)
	at weblogic.jndi.Environment.getInitialContext(Environment.java:192)
	at weblogic.jndi.Environment.getInitialContext(Environment.java:170)
	at weblogic.jndi.Environment.getContext(Environment.java:215)
	at weblogic.jms.forwarder.Forwarder.getInitialContext(Forwarder.java:428)
	at weblogic.jms.forwarder.Forwarder.connectTarget(Forwarder.java:447)
	at weblogic.jms.forwarder.Forwarder.reconnect(Forwarder.java:270)
	at weblogic.jms.forwarder.Forwarder.timerExpired(Forwarder.java:335)
	at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:293)
	at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.net.ConnectException: t3://192.168.1.50:9001,192.168.1.50:9002: Destination unreachable; nested exception is:
	java.net.ConnectException: Connection refused; No available router to destination
	at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
	at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
	at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:165)
	at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:345)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
	at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:340)
	... 14 more
Caused by: java.rmi.ConnectException: Destination unreachable; nested exception is:
	java.net.ConnectException: Connection refused; No available router to destination
	at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:470)
	at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:321)
	at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:260)
	at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:197)
	at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
	at weblogic.rjvm.RJVMFinder.findOrCreateRemoteCluster(RJVMFinder.java:316)
	at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:205)
	... 20 more
.>

As can be seen from the previous log line (which in time is happening later), when the servers become available the SAF agent connects to the servers.

WLST

The following script shows an example how the save-and-forward service can be set-up using WLST

print 'CONNECT TO ADMIN SERVER';
connect('weblogic', 'magic12c', 't3://192.168.1.50:7001');

print 'START EDIT MODE';
edit();
startEdit();

print 'CREATE FILE STORE';
cmo.createFileStore('SAFFileStore');
saffilestore = cmo.lookupFileStore('SAFFileStore');
saffilestore.setDirectory('/home/oracle/weblogic12.1.1/configuration/applications/base_domain');
targets = saffilestore.getTargets();
adminserver = cmo.lookupServer('AdminServer');
targets.append(adminserver);
saffilestore.setTargets(targets);

print 'SAVE, ACTIVATE CHANGES AND START EDIT';
save();
activate(block='true');
startEdit();

print 'CREATE SAF Agent';
cmo.createSAFAgent('SAFAgent');
safagent = cmo.lookupSAFAgent('SAFAgent');
safagent.setStore(saffilestore);
safagent.setTargets(targets);
safagent.setServiceType('Sending-only');

print 'SAVE, ACTIVATE CHANGES AND START EDIT';
save();
activate(block='true');
startEdit();

print 'CREATE JMS MODULE';
cmo.createJMSSystemResource('jms-saf-module');
safmodule = cmo.lookupJMSSystemResource('jms-saf-module');
safmodule.setTargets(targets);
safmodule.createSubDeployment('SAFSubDeployment');
cd('/JMSSystemResources/jms-saf-module/SubDeployments/SAFSubDeployment');
set('Targets',jarray.array([ObjectName('com.bea:Name=SAFAgent,Type=SAFAgent')], ObjectName));
cd('/');

print 'SAVE, ACTIVATE CHANGES AND START EDIT';
save();
activate(block='true');
startEdit();

print 'OBTAIN JMS RESOURCE';
resource = safmodule.getJMSResource();

print 'CREATE CONNECTION FACTORY';
resource.createConnectionFactory('SAFConnectionFactory');
connectionfactory = resource.lookupConnectionFactory('SAFConnectionFactory');
connectionfactory.setJNDIName('jms/SAFConnectionFactory');
connectionfactory.setDefaultTargetingEnabled(true);
connectionfactory.getTransactionParams().setTransactionTimeout(3600);
connectionfactory.getTransactionParams().setXAConnectionFactoryEnabled(true);
connectionfactory.getSecurityParams().setAttachJMSXUserId(false);
connectionfactory.getClientParams().setClientIdPolicy('Restricted');
connectionfactory.getClientParams().setSubscriptionSharingPolicy('Exclusive');
connectionfactory.getClientParams().setMessagesMaximum(10);

print 'SAVE, ACTIVATE CHANGES AND START EDIT';
save();
activate(block='true');
startEdit();

print 'CREATE SAF REMOTE CONTEXT';
resource.createSAFRemoteContext('RemoteSAFContext');
safcontext = resource.lookupSAFRemoteContext('RemoteSAFContext');
safcontext.getSAFLoginContext().setLoginURL('t3://192.168.1.50:9001,192.168.1.50:9002');
safcontext.getSAFLoginContext().setUsername('weblogic');
safcontext.getSAFLoginContext().setPassword('magic12c');

print 'SAVE, ACTIVATE CHANGES AND START EDIT';
save();
activate(block='true');
startEdit();

print 'CREATE SAF IMPORTED DESTINATIONS';
resource.createSAFImportedDestinations('SAFImportedDestinations');
importeddestinations = resource.lookupSAFImportedDestinations('SAFImportedDestinations');
importeddestinations.setJNDIPrefix('jms/');
importeddestinations.setSAFRemoteContext(safcontext);
importeddestinations.setSAFErrorHandling(None);
importeddestinations.setTimeToLiveDefault(0);
importeddestinations.setUseSAFTimeToLiveDefault(false);
importeddestinations.setSubDeploymentName('SAFSubDeployment');

print 'SAVE, ACTIVATE CHANGES AND START EDIT';
save();
activate(block='true');
startEdit();

print 'ADD QUEUE TO THE SAF IMPORTED DESTINATIONS';
importeddestinations.createSAFQueue('SAFQueue');
safqueue = importeddestinations.lookupSAFQueue('SAFQueue');
safqueue.setRemoteJNDIName('jms/CompanyQueue');
safqueue.setLocalJNDIName('SAFCompanyQueue');

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

Test

To test the set-up we will again use Spring. To set-up the sending end, we will use the following configuration

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="sourceJndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <property name="environment">
            <props>
                <prop key="java.naming.factory.initial">weblogic.jndi.WLInitialContextFactory</prop>
                <prop key="java.naming.provider.url">t3://192.168.1.50:7001</prop>
            </props>
        </property>
    </bean>
    <bean id="sourceConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="sourceJndiTemplate"/>
        <property name="jndiName" value="jms/SAFConnectionFactory"/>
    </bean>
    <bean id="sourceDestination" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="sourceJndiTemplate"/>
        <property name="jndiName" value="jms/SAFCompanyQueue"/>
    </bean>
    <bean id="sourceJmsTemplate" class="org.springframework.jms.core.JmsTemplate">
        <property name="connectionFactory" ref="sourceConnectionFactory"/>
        <property name="defaultDestination" ref="sourceDestination"/>
    </bean>
    <bean id="timerListener" class="org.springframework.scheduling.commonj.ScheduledTimerListener">
        <property name="delay" value="10000"/>
        <property name="period" value="30000"/>
        <property name="runnable" ref="sender"/>
    </bean>
    <bean id="timerFactory" class="org.springframework.scheduling.commonj.TimerManagerFactoryBean">
        <property name="timerManagerName" value="java:comp/env/tm/default"/>
        <property name="resourceRef" value="true"/>
        <property name="scheduledTimerListeners">
            <list>
                <ref bean="timerListener"/>
            </list>
        </property>
    </bean>
    <bean id="sender" class="model.logic.JMSSender">
        <property name="jmsTemplate" ref="sourceJmsTemplate"/>
    </bean>
</beans>

in which the sender bean looks as follows

package model.logic;

import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.core.MessageCreator;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.Session;
import javax.jms.TextMessage;
import java.util.Random;

public class JMSSender implements Runnable {

    private Random generator = new Random();
    private JmsTemplate jmsTemplate;

    public JMSSender() {
    }

    public JmsTemplate getJmsTemplate() {
        return jmsTemplate;
    }

    public void setJmsTemplate(JmsTemplate jmsTemplate) {
        this.jmsTemplate = jmsTemplate;
    }

    public void sendMessage() {
        getJmsTemplate().send(new MessageCreator(){
            public Message createMessage(Session session) throws JMSException {
                TextMessage message = session.createTextMessage();
                message.setText(Long.toString(Math.abs(generator.nextLong()), 36));
                return message;
            }
        });
    }

    public void run() {
        sendMessage();
    }
}

To set-up the receiving end, we will use the following configuration

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="targetJndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <property name="environment">
            <props>
                <prop key="java.naming.factory.initial">weblogic.jndi.WLInitialContextFactory</prop>
                <prop key="java.naming.provider.url">t3://192.168.1.50:9001,192.168.1.50:9002</prop>
            </props>
        </property>
    </bean>
    <bean id="targetConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="targetJndiTemplate"/>
        <property name="jndiName" value="jms/ConnectionFactory"/>
    </bean>
    <bean id="targetDestination" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="targetJndiTemplate"/>
        <property name="jndiName" value="jms/CompanyQueue"/>
    </bean>
    <bean id="taskExecutor" class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor">
        <property name="workManagerName" value="java:comp/env/default"/>
        <property name="resourceRef" value="true"/>
    </bean>
    <bean class="org.springframework.jms.listener.SimpleMessageListenerContainer">
        <property name="connectionFactory" ref="targetConnectionFactory"/>
        <property name="destination" ref="targetDestination"/>
        <property name="messageListener" ref="receiver"/>
        <property name="taskExecutor" ref="taskExecutor"/>
    </bean>
    <bean id="receiver" class="model.logic.JMSReceiver"/>
</beans>

in which the receiver bean looks as follows

package model.logic;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;

public class JMSReceiver implements MessageListener {
    public JMSReceiver() {
    }

    public void onMessage(Message message) {
        TextMessage text = (TextMessage)message;
        try {
            message.acknowledge();
            System.out.println("received the following message: " + text.getText());
        } catch (JMSException e) {
            e.printStackTrace();
        }
    }
}

We deploy the sender to the AdminServer and the receiver to the cluster (the server set-up can be found here). To deploy the application we add a web.xml with the following contents

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
           version="3.0">
    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath:spring-config.xml</param-value>
    </context-param>
    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>
    <!-- default weblogic work manager -->
    <resource-ref>
        <res-ref-name>default</res-ref-name>
        <res-type>commonj.work.WorkManager</res-type>
        <res-auth>Container</res-auth>
        <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>
    <!-- default weblogic timer manager -->
    <resource-ref>
        <res-ref-name>tm/default</res-ref-name>
        <res-type>commonj.timers.TimerManager</res-type>
        <res-auth>Container</res-auth>
        <res-sharing-scope>Shareable</res-sharing-scope>
    </resource-ref>
    <servlet>
        <servlet-name>TestServlet</servlet-name>
        <servlet-class>test.TestServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>TestServlet</servlet-name>
        <url-pattern>/testservlet</url-pattern>
    </servlet-mapping>
</web-app>

What happens when the applications run is that a JMS producer (sender) is connected to the SAF environment on the AdminServer. The JMS producer sends messages every 30 seconds, which is configured by using a commonj timer manager. The message send to the SAF queue is being forwarded to the jms/CompanyQueue (a distributed queue) to which a JMS consumer (receiver) is listening by using a SimpleMessageListenerContainer that is coupled to a commonj work manager in order to make sure the thread spawned by the listener is managed by the WebLogic Server. In the logging the following is observed

Logging server1 in the cluster
Apr 16, 2012 11:41:02 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization started
Apr 16, 2012 11:41:02 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.XmlWebApplicationContext@1e3c6703: display name [Root WebApplicationContext]; startup date [Mon Apr 16 11:41:02 CEST 2012]; root of context hierarchy
Apr 16, 2012 11:41:03 AM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [spring-config.xml]
Apr 16, 2012 11:41:03 AM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.XmlWebApplicationContext@1e3c6703]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1e4efd06
Apr 16, 2012 11:41:03 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1e4efd06: defining beans [sourceJndiTemplate,targetJndiTemplate,sourceConnectionFactory,sourceDestination,sourceJmsTemplate,targetConnectionFactory,targetDestination,taskExecutor,timerListener,timerFactory,org.springframework.jms.listener.SimpleMessageListenerContainer#0,sender,receiver]; root of factory hierarchy
Apr 16, 2012 11:41:03 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization completed in 842 ms
received the following message: d8nv7v6vgm5j
received the following message: 1be4omcwwiy61
received the following message: 19y6wh44hora6
received the following message: ha5hp6gaw6ql
received the following message: xercuxsl5qwf
received the following message: 1vvx5lptldi8r
received the following message: 1cgsdl10dkokz
received the following message: qfmd6v14tbb6
received the following message: 53hsnn8dfmor
received the following message: 8uaecn6919mi
received the following message: 8i93wuglm4lj
received the following message: 11d5bkqc5vdnq
received the following message: wc9hpcqmzjj9
received the following message: 17zj64mnvj7d9
received the following message: 1dt5g68vv9d50
received the following message: 13ch0ee5ofrzb
received the following message: 121t0btwasrqs
received the following message: 1lr8qb6il7nac
received the following message: 1l6lm4lge9tjx
received the following message: 15bgb2y3qzxl8
received the following message: vlryb3s3cn1o
received the following message: 189ahnkk5me1n
received the following message: 1wlhom58c3u01

Logging server2 in the cluster
Apr 16, 2012 11:41:02 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization started
Apr 16, 2012 11:41:02 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.XmlWebApplicationContext@1e9e9ebc: display name [Root WebApplicationContext]; startup date [Mon Apr 16 11:41:02 CEST 2012]; root of context hierarchy
Apr 16, 2012 11:41:03 AM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [spring-config.xml]
Apr 16, 2012 11:41:03 AM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.XmlWebApplicationContext@1e9e9ebc]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1ec29f17
Apr 16, 2012 11:41:03 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1ec29f17: defining beans [sourceJndiTemplate,targetJndiTemplate,sourceConnectionFactory,sourceDestination,sourceJmsTemplate,targetConnectionFactory,targetDestination,taskExecutor,timerListener,timerFactory,org.springframework.jms.listener.SimpleMessageListenerContainer#0,sender,receiver]; root of factory hierarchy
Apr 16, 2012 11:41:03 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization completed in 686 ms
received the following message: 1uwivfp2ngq1t
received the following message: bo4z7iakwqqv
received the following message: 1sdrelhucxxed
received the following message: 99xztxrc3f9f
received the following message: 1u8ej4ls0avqt
received the following message: hhkpei2bqyaw
received the following message: 2nid8k122rnd
received the following message: m2n72h6ajiao
received the following message: 1wpp89uiqttrw
received the following message: 18ecmmhgmmheu
received the following message: 1neja8ptey39a
received the following message: 1iu5em4cknpaa
received the following message: p89q16c2npbp
received the following message: 13gx5jqyzh57n
received the following message: 6lcjw5vjpttp
received the following message: 74wvvr4ey35s
received the following message: 1mcnjxzy2xp6f
received the following message: 12d6ybdpmi2wp

The extra messages in server1 are there because when running Spring as a client instead of being deployed to the WebLogic cluster, the receiver is only registered to one queue in the distributed queue. To test the set-up (without the timer manager and work manager configuration and) without deploying the application to WebLogic we can use

package model.test;

import model.logic.JMSSender;
import model.utils.SpringUtilities;

public class JMSTest {

    public static void main(String[] args) {
        JMSSender jmsSender = SpringUtilities.getJMSSender();

        while (true) {
            jmsSender.sendMessage();
            System.out.println("done sending message");
            try {
                Thread.sleep(10000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

Performance considerations

The following documents provide information on various methods to improve performance:

Another tuning parameter to consider is the chunk size. A chunk is a unit of memory that the WebLogic Server network layer, both on the client and server side, uses to read data from and write data to sockets. To reduce memory allocation costs, a server instance maintains a pool of these chunks. For applications that handle large amounts of data per request, increasing the value on both the client and server sides can boost performance. The default chunk size is about 4K. Use the following properties to tune the chunk size and the chunk pool size:

weblogic.Chunksize – sets the size of a chunk (in bytes). The primary situation in which this may need to be increased is if request sizes are large. It should be set to values that are multiples of the network’s maximum transfer unit (MTU), after subtracting from the value any Ethernet or TCP header sizes. Set this parameter to the same value on the client and server.
weblogic.utils.io.chunkpoolsize – sets the maximum size of the chunk pool. The default value is 2048. The value may need to be increased if the server starts to allocate and discard chunks in steady state. To determine if the value needs to be increased, monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.Chunk.
weblogic.PartitionSize – sets the number of pool partitions used (default is 4). The chunk pool can be a source of significant lock contention as each request to access to the pool must be synchronized. Partitioning the thread pool spreads the potential for contention over more than one partition.

For example, we can set the chunksize by using -Dweblogic.Chunksize=65536 on the command-line.

References

[1] Configuring and Managing JMS for WebLogic Server.
[2] Configuring and Managing the Messaging Bridge for WebLogic Server.
[3] Configuring and Managing Store-and-Forward for WebLogic Server.
[4] Performance and Tuning for WebLogic Server.

JMS, Messaging Bridge, Spring, Store and Forward, weblogic Leave a Comment more...

Apr/12

Spring and GlassFish JMS

by René van Wijk under JMS, Spring

René van Wijk

In this post we set-up a GlassFish Server environment that includes a JMS configuration that will be used by a Spring client. First, we look at some considerations when developing a client that uses JMS. Next, we look at the classloading, i.e., the classes needed in order for Spring to communicate with GlassFish and obtain JMS resources from JNDI. Finally, we show the configuration needed in Spring and test the set-up.

Some notes up front

The Java Messaging Specification was developed to abstract access to message-oriented middleware systems. A client that writes JMS code should be portable to any provider that implements this specification. If code portability is important, be sure to do the following in developing clients:

Make sure the code does not depend on extensions or features that are specific to Message Queue.
Look up, using JNDI, (rather than instantiate) administered objects for connection factories and destinations.

Another point to consider when developing a client is to use threads effectively, i.e., we need to balance performance, throughput, and resource needs. To do this, we need to understand JMS restrictions on thread usage, what threads Message Queue allocates for itself, and the architecture of the applications. The Java Messaging Specification mandates that a session not be operated on by more than one thread at a time. This leads to the following restrictions:

A session may not have an asynchronous consumer and a synchronous consumer.
A session that has an asynchronous consumer can only produce messages from within the onMessage() method (the message listener). The only call that we can make outside the message listener is to close the session.
A session may include any number of synchronous consumers, any number of producers, and any combination of the two. That is, the single-thread requirement cannot be violated by these combinations. However, performance may suffer.

The system does not enforce the requirement that a session be single threaded. If the client application violates this requirement, we will get a JMSIllegalState exception or unexpected results. When the Message Queue client runtime creates a connection, it creates two threads: one for consuming messages from the socket, and one to manage the flow of messages for the connection. In addition, the client runtime creates a thread for each client session. Thus, at a minimum, for a connection using one session, three threads are created. For a connection using three sessions, five threads are created, and so on. Managing threads in a JMS application often involves trade-offs between performance and throughput. Weigh the following considerations when dealing with threading issues. When we create several asynchronous message consumers in the same session, messages are delivered serially by the session thread to these consumers. Sharing a session among several message consumers might starve some consumers of messages while inundating other consumers. If the message rate across these consumers is high enough to cause an imbalance, we might want to separate the consumers into different sessions. To determine whether message flow is unbalanced, we can monitor destinations to see the rate of messages coming in. We can reduce the number of threads allocated to the client application by using fewer connections and fewer sessions. However, doing this might slow the application’s throughput. We might be able to use certain JVM runtime options to improve thread memory usage and performance.

A client application running in a JVM needs enough memory to accommodate messages that flow in from the network as well as messages the client creates. If the client gets OutOfMemoryError errors, chances are that not enough memory was provided to handle the size or the number of messages being consumed or produced. The client might need more than the default JVM heap space. Consider the following guidelines:

Evaluate the normal and peak system memory footprints when sizing heap space.
Adjust the heap size settings accordingly (the best size for the heap space depends on both the operating system and the JDK release).

In general, for better manageability, we can break large messages into smaller parts, and use sequencing to ensure that the partial messages sent are concatenated properly. We can also use a Message Queue JMS feature to compress the body of a message. Compression only affects the message body; the message header and properties are not compressed. Although message compression has been added to improve performance, such benefit is not guaranteed. Benefits vary with the size and format of messages, the number of consumers, network bandwidth, and CPU performance. For example, the cost of compression and decompression might be higher than the time saved in sending and receiving a compressed message. This is especially true when sending small messages in a high-speed network. On the other hand, applications that publish large messages to many consumers or who publish in a slow network environment, might improve system performance by compressing messages.

Persistent messages guarantee message delivery in case of broker failure. The broker stores these message in a persistent store until all intended consumers acknowledge that they have consumed the message. Broker processing of persistent messages is slower than for nonpersistent messages for the following reasons:

A broker must reliably store a persistent message so that it will not be lost should the broker fail.
The broker must confirm receipt of each persistent message it receives. Delivery to the broker is guaranteed once the method producing the message returns without an exception.
Depending on the client acknowledgment mode, the broker might need to confirm a consuming client’s acknowledgment of a persistent message.

A transaction guarantees that all messages produced in a transacted session and all messages consumed in a transacted session will be either processed or not processed (rolled back) as a unit. A message produced or acknowledged in a transacted session is slower than in a non-transacted session for the following reasons:

Additional information must be stored with each produced message.
In some situations, messages in a transaction are stored when normally they would not be. For example, a persistent message delivered to a topic destination with no subscriptions would normally be deleted, however, at the time the transaction is begun, information about subscriptions is not available.
Information on the consumption and acknowledgment of messages within a transaction must be stored and processed when the transaction is committed.

Other than using transactions, we can ensure reliable delivery by having the client acknowledge receiving a message. The messaging provider can sort messages according to criteria specified in the message selector associated with a consumer and deliver to that consumer only those messages whose property value matches the message selector. Creating consumers with selectors lowers performance (as compared to using multiple destinations) because additional processing is required to handle each message. When a selector is used, it must be parsed so that it can be matched against future messages. Additionally, the message properties of each message must be retrieved and compared against the selector as each message is routed. However, using selectors provides more flexibility in a messaging application and may lower resource requirements at the expense of speed.

Message size affects performance because more data must be passed from producing client to broker and from broker to consuming client, and because for persistent messages a larger message must be stored. However, by batching smaller messages into a single message, the routing and processing of individual messages can be minimized, providing an overall performance gain. In this case, information about the state of individual messages is lost. JMS supports five message body types, shown below roughly in the order of complexity:

Bytes: Contains a set of bytes in a format determined by the application.
Text: Is a java.lang.String.
Stream: Contains a stream of Java primitive values.
Map: Contains a set of name-and-value pairs.
Object: Contains a Java serialized object.

While, in general, the message type is dictated by the needs of an application, the more complicated types (map and object) carry a performance cost – the expense of serializing and deserializing the data. The performance cost depends on how simple or how complicated the data is.

Developing the JMS client with Spring

When using GlassFish JMS in a Spring client we need the right classes on the class path. The gf-client.jar contains all the necessary references, i.e., the MANIFEST.MF contained in the jar file looks as follows

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.6.5
Created-By: Apache Maven
Archiver-Version: Plexus Archiver
Built-By: java_re
Build-Jdk: 1.6.0_10
Package: org.glassfish.appclient.client.acc
Main-Class: org.glassfish.appclient.client.AppClientFacade
GlassFish-ServerExcluded: true
PreMain-Class: org.glassfish.appclient.client.acc.agent.AppClientContainerAgent
Class-Path: ../modules/jtype.jar ../modules/tools.jar ../modules/glassfish-corba-asm.jar ../modules/glassfish-corba-codegen.jar ../modules/glassfish-corba-csiv2-idl.jar ../modules/glassfish-corba-internal-api.jar ../modules/glassfish-corba-newtimer.jar ../modules/glassfish-corba-omgapi.jar ../modules/glassfish-corba-orb.jar ../modules/glassfish -corba-orbgeneric.jar ../modules/auto-depends.jar ../modules/config.jar ../modules/config-types.jar ../modules/hk2.jar ../modules/hk2-core.jar ../modules/osgi-adapter.jar ../modules/grizzly-comet.jar ../modules/grizzly-config.jar ../modules/grizzly-framework.jar ../modules/grizzly-http.jar ../modules/grizzly-http-servlet.jar ../modules/grizzly-lzma.jar ../modules/grizzly-portunif.jar ../modules/grizzly-rcm.jar ../modules/grizzly-utils.jar ../modules/grizzly-websockets.jar ../modules/javax.mail.jar ../modules/pkg-client.jar ../modules/jaxb-osgi.jar ../modules/activation.jar ../modules/el-api.jar ../modules/jaxr-api-osgi.jar ../modules/jaxrpc-api-osgi.jar ../modules/endorsed/jaxb-api-osgi.jar ../modules/stax-api.jar ../modules/junit.jar ../modules/stax2-api.jar ../modules/woodstox-core-asl.jar ../modules/javax.persistence.jar ../modules/org.eclipse.persistence.antlr.jar ../modules/org.eclipse.persistence.asm.jar ../modules/org.eclipse.persistence.core.jar ../modules/org.eclipse.persistence.jpa.jar ../modules/org.eclipse.persistence.jpa.modelgen.jar ../modules/org.eclipse.persistence.oracle.jar ../modules/endorsed/javax.annotation.jar ../modules/javax.ejb.jar ../modules/javax.enterprise.deploy.jar ../modules/javax.jms.jar ../modules/javax.management.j2ee.jar ../modules/javax.resource.jar ../modules/javax.security.auth.message.jar ../modules/javax.security.jacc.jar ../modules/javax.servlet.jar ../modules/javax.servlet.jsp.jar ../modules/javax.transaction.jar ../modules/simple-glassfish-api.jar ../modules/admin-core.jar ../modules/admin-util.jar ../modules/config-api.jar ../modules/monitoring-core.jar ../modules/acc-config.jar ../modules/gf-client-module.jar ../modules/gms-bootstrap.jar ../modules/amx-core.jar ../modules/amx-j2ee.jar ../modules/annotation-framework.jar ../modules/common-util.jar ../modules/container-common.jar ../modules/glassfish-api.jar ../modules/glassfish-ee-api.jar ../modules/glassfish-naming.jar ../modules/internal-api.jar ../modules/scattered-archive-api.jar ../modules/stats77.jar ../modules/connectors-inbound-runtime.jar ../modules/connectors-internal-api.jar ../modules/connectors-runtime.jar ../modules/work-management.jar ../modules/glassfish.jar ../modules/kernel.jar ../modules/logging.jar ../modules/deployment-common.jar ../modules/deployment-javaee-core.jar ../modules/dol.jar ../modules/ejb-container.jar ../modules/ejb-internal-api.jar ../modules/ldapbp-repackaged.jar ../modules/libpam4j-repackaged.jar ../modules/management-api.jar ../modules/flashlight-framework.jar ../modules/gmbal.jar ../modules/ha-api.jar ../modules/class-model.jar ../modules/asm-a ll-repackaged.jar ../modules/bean-validator.jar ../modules/jms-core.jar ../modules/endorsed/webservices-api-osgi.jar ../modules/webservices-extra-jdk-packages.jar ../modules/webservices-osgi.jar ../modules/orb-connector.jar ../modules/orb-iiop.jar ../modules/eclipselink-wrapper.pom ../modules/jpa-connector.jar ../modules/persistence-common.jar ../modules/cmp-internal-api.jar ../modules/appclient.security.jar ../modules/ejb.security.jar ../modules/jaspic.provider.framework.jar ../modules/security.jar ../modules/ssl-impl.jar ../modules/websecurity.jar ../modules/webservices.security.jar ../modules/jta.jar ../modules/jts.jar ../modules/transaction-internal-api.jar ../modules/el-impl.jar ../modules/jsp-impl.jar ../modules/war-util.jar ../modules/web-cli.jar ../modules/web-core.jar ../modules/web-embed-api.jar ../modules/web-glue.jar ../modules/web-gui-plugin-common.jar ../modules/web-naming.jar ../modules/jsr109-impl.jar ../modules/mimepull.jar ../modules/tiger-types.jar ../modules/shoal-gms-api.jar ../../mq/lib/imq.jar ../ ../mq/lib/imqadmin.jar ../../mq/lib/imqutil.jar ../../mq/lib/fscontext.jar ../lib/install/applications/jmsra/imqjmsra.jar ../lib/install/applications/__ds_jdbc_ra/__ds_jdbc_ra.jar ../lib/install/applications/__cp_jdbc_ra/__cp_jdbc_ra.jar ../lib/install/applications/__xa_jdbc_ra/__xa_jdbc_ra.jar ../lib/install/applications/__dm_jdbc_ra/__dm_jdbc_ra.jar ../../javadb/lib/derby.jar ../../javadb/lib/derbyclient.jar ../../javadb/lib/derbynet.jar ../../javadb/lib/derbytools.jar ../../javadb/lib/derbyrun.jar ../lib/install/applications/jaxr-ra/jaxr-ra.jar ../modules/aixporting-repackaged.jar
GlassFish-Conditional-Additions: ../modules/aixporting-repackaged.jar

which is of course quite a lot to have on the classpath when we just want to use JMS. Note that it also means we have to install GlassFish on the client. So naturally you start to look for alternative ways. To see if we can obtain an object from a GlassFish server by using JNDI, we can create a little test class, for example,

package model.test;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.util.Properties;

public class JMSTest {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.setProperty("java.naming.factory.initial", "com.sun.enterprise.naming.SerialInitContextFactory");
        props.setProperty("java.naming.factory.url.pkgs", "com.sun.enterprise.naming");
        props.setProperty("java.naming.factory.state", "com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl");
        props.setProperty("org.omg.CORBA.ORBInitialHost", "192.168.1.50");
        props.setProperty("org.omg.CORBA.ORBInitialPort", "3700");

        try {
            Context context = new InitialContext(props);
            System.out.println(context);
            System.out.println(context.lookup("jms/ConnectionFactory"));
        } catch (NamingException e) {
            e.printStackTrace();
        }
    }
}

When running this class we of course miss classes on the class path. To find which jar contains the missing class we can use

#!/bin/sh

PATH=${PATH}:/home/oracle/jdk1.6.0_31/bin
export PATH

LOOK_FOR="org/glassfish/api/admin/ProcessEnvironment$ProcessType"
export LOOK_FOR

for i in `find /home/oracle/glassfish3/glassfish/modules -name "*jar"`
do
	jar tvf $i | grep $LOOK_FOR > /dev/null
	if [ $? == 0 ]
	then
		echo "==> Found \"$LOOK_FOR\" in $i"
	fi
done

Here the LOOK_FOR variable is set to the value for the missing class and the first parameter in the find command is the path in which to look. After a while we get to a collection of these classes

auto-depends.jar
common-util.jar
glassfish-api.jar
glassfish-corba-internal-api.jar
glassfish-naming.jar
hk2-core.jar
internal-api.jar

and run into the following exception

Caused by: javax.naming.NamingException: Unable to acquire SerialContextProvider for SerialContext[myEnv={java.naming.provider.url=iiop://192.168.1.50:3700, java.naming.factory.initial=com.sun.enterprise.naming.SerialInitContextFactory, java.naming.factory.url.pkgs=com.sun.enterprise.naming, java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl} [Root exception is java.lang.NullPointerException]
	at com.sun.enterprise.naming.impl.SerialContext.getProvider(SerialContext.java:352)
	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:504)
	... 8 more
Caused by: java.lang.NullPointerException
	at com.sun.enterprise.naming.impl.SerialContext.getORB(SerialContext.java:365)
	at com.sun.enterprise.naming.impl.SerialContext.getProviderCacheKey(SerialContext.java:372)
	at com.sun.enterprise.naming.impl.SerialContext.getRemoteProvider(SerialContext.java:402)
	at com.sun.enterprise.naming.impl.SerialContext.getProvider(SerialContext.java:347)
	... 9 more

Now what? You go for the gf-client.jar and install GlassFish on the client. Not the nicest of solutions but one that works. The gf-client.jar file is located in the ${GLASS_HOME}/glassfish/lib directory. With gf-client.jar on the class path we get the following output when running the test

javax.naming.InitialContext@682bc3f5
Apr 12, 2012 4:59:02 PM org.hibernate.validator.util.Version <clinit>
INFO: Hibernate Validator 4.1.0.Final
Apr 12, 2012 4:59:02 PM org.hibernate.validator.engine.resolver.DefaultTraversableResolver detectJPA
INFO: Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.
Apr 12, 2012 4:59:02 PM com.sun.messaging.jms.ra.ResourceAdapter start
INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter: Version:  4.5.1  (Build 3-b) Compile:  Tue Jun 21 16:31:32 PDT 2011
Apr 12, 2012 4:59:02 PM com.sun.messaging.jms.ra.ResourceAdapter start
INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter starting: broker is REMOTE, connection mode is TCP
Apr 12, 2012 4:59:02 PM com.sun.messaging.jms.ra.ResourceAdapter start
INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter Started:REMOTE
com.sun.messaging.jms.ra.ConnectionFactoryAdapter@61672c01

Next we need to configure Spring

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <property name="environment">
            <props>
                <prop key="java.naming.factory.initial">com.sun.enterprise.naming.SerialInitContextFactory</prop>
                <prop key="java.naming.factory.url.pkgs">com.sun.enterprise.naming</prop>
                <prop key="java.naming.factory.state">com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl</prop>
                <prop key="org.omg.CORBA.ORBInitialHost">192.168.1.50</prop>
                <prop key="org.omg.CORBA.ORBInitialPort">3700</prop>
            </props>
        </property>
    </bean>
    <bean id="connectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="jndiTemplate"/>
        <property name="jndiName" value="jms/ConnectionFactory"/>
    </bean>
    <bean id="destination" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="jndiTemplate"/>
        <property name="jndiName" value="jms/CompanyQueue"/>
    </bean>
    <bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
        <property name="connectionFactory" ref="connectionFactory"/>
        <property name="defaultDestination" ref="destination"/>
    </bean>
    <bean class="org.springframework.jms.listener.SimpleMessageListenerContainer">
        <property name="connectionFactory" ref="connectionFactory"/>
        <property name="destination" ref="destination"/>
        <property name="messageListener" ref="receiver"/>
    </bean>
    <bean id="sender" class="model.logic.JMSSender">
        <property name="jmsTemplate" ref="jmsTemplate"/>
    </bean>
    <bean id="receiver" class="model.logic.JMSReceiver"/>
</beans>

The classes JMSSender and JMSReceiver can be found in the post WebLogic JMS Clustering and Spring. The post Fun with GlassFish shows the steps involved in how to create the JMS resources (connection factory and queue). To test the set-up we can use

package model.test;

import model.logic.JMSSender;
import model.utils.SpringUtilities;

public class JMSTest {

    public static void main(String[] args) {
        JMSSender jmsSender = SpringUtilities.getJMSSender();
        jmsSender.sendMessage();
        System.out.println("done sending message");
    }
}

which leads to the following output

Apr 12, 2012 5:30:45 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@78b5f53a: display name [org.springframework.context.support.ClassPathXmlApplicationContext@78b5f53a]; startup date [Thu Apr 12 17:30:45 CEST 2012]; root of context hierarchy
Apr 12, 2012 5:30:45 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [spring-config.xml]
Apr 12, 2012 5:30:45 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@78b5f53a]: org.springframework.beans.factory.support.DefaultListableBeanFactory@21b64e6a
Apr 12, 2012 5:30:45 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@21b64e6a: defining beans [jndiTemplate,connectionFactory,destination,jmsTemplate,org.springframework.jms.listener.SimpleMessageListenerContainer#0,sender,receiver]; root of factory hierarchy
Apr 12, 2012 5:30:49 PM org.hibernate.validator.util.Version <clinit>
INFO: Hibernate Validator 4.1.0.Final
Apr 12, 2012 5:30:49 PM org.hibernate.validator.engine.resolver.DefaultTraversableResolver detectJPA
INFO: Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.
Apr 12, 2012 5:30:49 PM com.sun.messaging.jms.ra.ResourceAdapter start
INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter: Version:  4.5.1  (Build 3-b) Compile:  Tue Jun 21 16:31:32 PDT 2011
Apr 12, 2012 5:30:49 PM com.sun.messaging.jms.ra.ResourceAdapter start
INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter starting: broker is REMOTE, connection mode is TCP
Apr 12, 2012 5:30:49 PM com.sun.messaging.jms.ra.ResourceAdapter start
INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter Started:REMOTE
received the following message: 8dxgbdo4u4ee
done sending message

References

[1] GlassFish Server Documentation.

GlassFish Server, JMS, Spring Leave a Comment more...

Apr/12

WebLogic JMS Clustering and Spring

by René van Wijk under Cluster, JMS, Spring

René van Wijk

In this post we set-up a WebLogic cluster that includes a JMS environment, which will be used by Spring. A distributed queue will be used in order to load balance the JMS message processing.

Create domain

To create our domain, we are going to use WLST. The following script, creates the domain in production mode, disables hostname verification, creates two managed server that are clustered and creates the artifacts needed to run the application.

beahome = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
adminusername = 'weblogic';
adminpassword = 'magic12c';
adminservername='AdminServer';
adminserverurl='t3://axis-into-ict.nl:7001';
domainname = 'base_domain';
domaindirectory = beahome + pathseparator + 'configuration' + pathseparator + 'domains' + pathseparator + domainname;
domaintemplate = beahome + pathseparator + 'wlserver_12.1' + pathseparator + 'common' + pathseparator + 'templates' + pathseparator + 'domains' + pathseparator + 'wls.jar';
jvmdirectory = '/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0';

print 'CREATE DOMAIN';
readTemplate(domaintemplate);
setOption('DomainName', domainname);
setOption('OverwriteDomain', 'true');
setOption('ServerStartMode', 'prod');
cd('/Security/' + domainname + '/User/weblogic');
cmo.setName(adminusername);
cmo.setUserPassword(adminpassword);
cd('/');
writeDomain(domaindirectory);

print 'START ADMIN SERVER';
startServer(adminservername, domainname, adminserverurl, adminusername, adminpassword, domaindirectory);

print 'CONNECT TO ADMIN SERVER';
connect(adminusername, adminpassword, adminserverurl);

print 'START EDIT MODE';
edit();
startEdit();

print 'DISABLE HOSTNAME VERIFICATION';
cd('/Servers/' + adminservername + '/SSL/' + adminservername);
cmo.setHostnameVerificationIgnored(true);
cmo.setHostnameVerifier(None);
cmo.setTwoWaySSLEnabled(false);
cmo.setClientCertificateEnforced(false);
cd('/');

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'SHUTDOWN THE ADMIN SERVER';
shutdown(block='true');

print 'START ADMIN SERVER';
startServer(adminservername, domainname, adminserverurl, adminusername, adminpassword, domaindirectory);

print 'CONNECT TO ADMIN SERVER';
connect(adminusername, adminpassword, adminserverurl);

print 'START EDIT MODE';
edit();
startEdit();

print 'CREATE MACHINE: machine1';
machine1 = cmo.createUnixMachine('machine1');
machine1.setPostBindUIDEnabled(true);
machine1.setPostBindUID('oracle');
machine1.getNodeManager().setListenAddress('axis-into-ict.nl');
machine1.getNodeManager().setNMType('ssl');

print 'CREATE CLUSTER: CLUSTER';
cluster = cmo.createCluster('cluster');
cluster.setClusterMessagingMode('unicast');

print 'CREATE MANAGED SERVER: server1';
server1 = cmo.createServer('server1');
server1.setListenPort(9001);
server1.setListenAddress('axis-into-ict.nl');
server1.setAutoRestart(true);
server1.setAutoKillIfFailed(true);
server1.setRestartMax(2);
server1.setRestartDelaySeconds(10);
server1.getServerStart().setJavaHome(jvmdirectory);
server1.getServerStart().setJavaVendor('Oracle');
server1.getServerStart().setArguments('-jrockit -Xms512m -Xmx512m -Xgc:throughput');

print 'CREATE MANAGED SERVER: server2';
server2 = cmo.createServer('server2');
server2.setListenPort(9002);
server2.setListenAddress('axis-into-ict.nl');
server2.setAutoRestart(true);
server2.setAutoKillIfFailed(true);
server2.setRestartMax(2);
server2.setRestartDelaySeconds(10);
server2.getServerStart().setJavaHome(jvmdirectory);
server2.getServerStart().setJavaVendor('Oracle');
server2.getServerStart().setArguments('-jrockit -Xms512m -Xmx512m -Xgc:throughput');

print 'ADD MANAGED SERVERS TO CLUSTER';
server1.setCluster(cluster);
server2.setCluster(cluster);

print 'ADD MANAGED SERVERS TO MACHINE';
server1.setMachine(machine1);
server2.setMachine(machine2);

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'START EDIT MODE';
startEdit();

print 'CREATE FILESTORE FOR SERVER1';
filestore1 = cmo.createFileStore('FileStore1');
filestore1.setDirectory(beahome + pathseparator + 'configuration' + pathseparator + 'applications' + pathseparator + domainname);
targets = filestore1.getTargets();
targets.append(server1);
filestore1.setTargets(targets);

print 'CREATE JMS SERVER FOR SERVER1';
jmsserver1 = cmo.createJMSServer('JMSServer1');
jmsserver1.setPersistentStore(filestore1);
jmsserver1.setTargets(targets);

targets.remove(server1);
targets.append(server2);

print 'CREATE FILESTORE FOR SERVER2';
filestore2 = cmo.createFileStore('FileStore2');
filestore2.setDirectory(beahome + pathseparator + 'configuration' + pathseparator + 'applications' + pathseparator + domainname);
filestore2.setTargets(targets);

print 'CREATE JMS SERVER FOR SERVER2';
jmsserver2 = cmo.createJMSServer('JMSServer2');
jmsserver2.setPersistentStore(filestore2);
jmsserver2.setTargets(targets);

targets.remove(server2);
targets.append(cluster);

print 'CREATE JMS SYSTEM MODULE';
module = cmo.createJMSSystemResource('SystemModule');
module.setTargets(targets);

print 'CREATE SUBDEPLOYMENT';
module.createSubDeployment('SubDeployment');
cd('/JMSSystemResources/SystemModule/SubDeployments/SubDeployment');
set('Targets',jarray.array([ObjectName('com.bea:Name=JMSServer1,Type=JMSServer'), ObjectName('com.bea:Name=JMSServer2,Type=JMSServer')], ObjectName));
cd('/');

resource = module.getJMSResource();

print 'CREATE CONNECTION FACTORY';
resource.createConnectionFactory('ConnectionFactory');
connectionfactory = resource.lookupConnectionFactory('ConnectionFactory');
connectionfactory.setJNDIName('jms/ConnectionFactory');
connectionfactory.setDefaultTargetingEnabled(true);
connectionfactory.getTransactionParams().setTransactionTimeout(3600);
connectionfactory.getTransactionParams().setXAConnectionFactoryEnabled(true);

print 'CREATE UNIFORM DISTRIBUTED QUEUE';
resource.createUniformDistributedQueue('DistributedQueue');
distributedqueue = resource.lookupUniformDistributedQueue('DistributedQueue');
distributedqueue.setJNDIName('jms/CompanyQueue');
distributedqueue.setLoadBalancingPolicy('Round-Robin');
distributedqueue.setSubDeploymentName('SubDeployment');

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'SHUTDOWN THE ADMIN SERVER';
shutdown(block='true');

Application

The Spring configuration file looks as follows

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <property name="environment">
            <props>
                <prop key="java.naming.factory.initial">weblogic.jndi.WLInitialContextFactory</prop>
                <prop key="java.naming.provider.url">t3://192.168.1.50:9001,192.168.1.50:9002</prop>
            </props>
        </property>
    </bean>
    <bean id="connectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="jndiTemplate"/>
        <property name="jndiName" value="jms/ConnectionFactory"/>
    </bean>
    <bean id="destination" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiTemplate" ref="jndiTemplate"/>
        <property name="jndiName" value="jms/CompanyQueue"/>
    </bean>
    <bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
        <property name="connectionFactory" ref="connectionFactory"/>
        <property name="defaultDestination" ref="destination"/>
    </bean>
    <bean id="taskExecutor" class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor">
        <property name="workManagerName" value="java:comp/env/default"/>
        <property name="resourceRef" value="true"/>
    </bean>
    <bean id="timerListener" class="org.springframework.scheduling.commonj.ScheduledTimerListener">
        <property name="delay" value="10000"/>
        <property name="period" value="30000"/>
        <property name="runnable" ref="sender"/>
    </bean>
    <bean id="timerFactory" class="org.springframework.scheduling.commonj.TimerManagerFactoryBean">
        <property name="timerManagerName" value="java:comp/env/tm/default"/>
        <property name="resourceRef" value="true"/>
        <property name="scheduledTimerListeners">
            <list>
                <ref bean="timerListener"/>
            </list>
        </property>
    </bean>
    <bean class="org.springframework.jms.listener.SimpleMessageListenerContainer">
        <property name="connectionFactory" ref="connectionFactory"/>
        <property name="destination" ref="destination"/>
        <property name="messageListener" ref="receiver"/>
        <property name="taskExecutor" ref="taskExecutor"/>
    </bean>
    <bean id="sender" class="model.logic.JMSSender">
        <property name="jmsTemplate" ref="jmsTemplate"/>
    </bean>
    <bean id="receiver" class="model.logic.JMSReceiver"/>
</beans>

Here, we use the WebLogic TimerManager to send messages every 30 seconds to the distributed queue. The messages are picked up by the SimpleMessageListenerContainer, for which the threads are managed by the default WebLogic WorkManager. The JMSSender looks as follows

package model.logic;

import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.core.MessageCreator;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.Session;
import javax.jms.TextMessage;
import java.util.Random;

public class JMSSender implements Runnable {

    private Random generator = new Random();
    private JmsTemplate jmsTemplate;

    public JMSSender() {
    }

    public JmsTemplate getJmsTemplate() {
        return jmsTemplate;
    }

    public void setJmsTemplate(JmsTemplate jmsTemplate) {
        this.jmsTemplate = jmsTemplate;
    }

    public void sendMessage() {
        getJmsTemplate().send(new MessageCreator(){
            public Message createMessage(Session session) throws JMSException {
                TextMessage message = session.createTextMessage();
                message.setText(Long.toString(Math.abs(generator.nextLong()), 36));
                return message;
            }
        });
    }

    public void run() {
        sendMessage();
    }
}

and JMSReceiver looks as follows

package model.logic;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;

public class JMSReceiver implements MessageListener {
    public JMSReceiver() {
    }

    public void onMessage(Message message) {
        TextMessage text = (TextMessage)message;
        try {
            message.acknowledge();
            System.out.println("received the following message: " + text.getText());
        } catch (JMSException e) {
            e.printStackTrace();
        }
    }
}

Deploy the application. Open the logging of the specific servers to which the application was deployed to check how the messages are balanced between the servers by the distributed queue:

Apr 11, 2012 11:23:15 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization started
Apr 11, 2012 11:23:15 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.XmlWebApplicationContext@1ea13672: display name [Root WebApplicationContext]; startup date [Wed Apr 11 11:23:15 CEST 2012]; root of context hierarchy
Apr 11, 2012 11:23:15 AM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [spring-config.xml]
Apr 11, 2012 11:23:15 AM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.XmlWebApplicationContext@1ea13672]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1eac97e6
Apr 11, 2012 11:23:15 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1eac97e6: defining beans [jndiTemplate,connectionFactory,destination,jmsTemplate,taskExecutor,timerListener,timerFactory,org.springframework.jms.listener.SimpleMessageListenerContainer#0,sender,receiver]; root of factory hierarchy
Apr 11, 2012 11:23:16 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization completed in 503 ms
received the following message: yqd0zvqllt58
received the following message: vnsw07fqhsmt
received the following message: 1secm492wxbhd
received the following message: 132hz54mtxvae
received the following message: 1nerzk2kcq41t
received the following message: orj9cmwyvvm4
received the following message: z7kz7vecuvoj
received the following message: 1elg1cszki0gz
received the following message: f283omdowrmm
received the following message: 4tfqfrek6tlm
received the following message: 1lka0mjnvjway
received the following message: n6uuuepdjqw1
received the following message: jz3rgzfp5myt
received the following message: iwvbwuvkotb2
received the following message: 1px4ymopgu0z4
received the following message: 1dvaf4do8nnb1
received the following message: vij9m5bilbtc
received the following message: 2vetjvvw9t1l

Apr 11, 2012 11:23:15 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization started
Apr 11, 2012 11:23:15 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.XmlWebApplicationContext@1e6a873c: display name [Root WebApplicationContext]; startup date [Wed Apr 11 11:23:15 CEST 2012]; root of context hierarchy
Apr 11, 2012 11:23:15 AM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [spring-config.xml]
Apr 11, 2012 11:23:15 AM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.XmlWebApplicationContext@1e6a873c]: org.springframework.beans.factory.support.DefaultListableBeanFactory@1e761a31
Apr 11, 2012 11:23:15 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1e761a31: defining beans [jndiTemplate,connectionFactory,destination,jmsTemplate,taskExecutor,timerListener,timerFactory,org.springframework.jms.listener.SimpleMessageListenerContainer#0,sender,receiver]; root of factory hierarchy
Apr 11, 2012 11:23:15 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization completed in 520 ms
received the following message: zvbtb316a5w0
received the following message: 1ur1serspbtls
received the following message: 1bcvfhi0f0vs9
received the following message: pfjts61vsryf
received the following message: 1nbp2sq86n1xm
received the following message: 1e6d9szm0sdx7
received the following message: 1juogzbawdyj9
received the following message: 148bbdq1t4mam
received the following message: 1mhaz3f8zzl1t
received the following message: 13hh354hcrf78
received the following message: 8y4n5g11ah5r
received the following message: c9zu7pcorr0r
received the following message: 4q3rgdshqjo9
received the following message: k3y8jygax551
received the following message: 108s63prt2rjl
received the following message: 1qgseev65213l
received the following message: 1utzdoiyw2ynt

By using a distributed queue the message processing load is balanced between the servers to which queue is targetted.

References

[1] WebLogic Server Documentation.

Cluster, JMS, Spring, weblogic 6 Comments more...

Mar/12

Building a Coherence Cluster with Multiple Application Servers

by René van Wijk under Cluster, Coherence, JVM Tuning, Monitoring

René van Wijk

A milestone, this is article number 50. Here, we go into the details of configuring the operating system for an application server and an in-memory data grid, such as Coherence; Present scripts to start and stop the environment; Deploy an application that uses JSF 1.2, JBoss RichFaces and Coherence to the domain we set-up in the Fun with JBoss post; Show how class loading in JBoss AS7 works; Add role-based security; Configure mod_cluster; and end with expanding the cluster to multiple machines.

Configure the operating system

When using an application server that runs an in-memory data grid such as Coherence some operating system tweaks are in order. Conifguration tweaks for the Coherence cluster can be found in the Set-up Coherence Clusters post.

Change the hostname and map it to a static ip address

To change the hostname we can follow the following steps:

Login as root.
Open the file /etc/sysconfig/network and enter a hostname, i.e., HOSTNAME=axis-into-ict.nl.
Retrieve the IP-address by using /sbin/ifconfig.
Open the file /etc/hosts and add, for example, 192.168.1.50 axis-into-ict.nl.
Use the hostname utility to change the hostname: hostname axis-into-ict.nl.
Run it again without any parameter to see if the hostname has been changed.
Restart the network to apply the changes: service network restart.
Logout and login again.

To set-up a static ip-address we can follow these steps. Edit /etc/sysconfig/network, the following gives an example

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=axis-into-ict.nl
GATEWAY=192.168.1.1

In the case of DHCP the eth0 configuration (/etc/sysconfig/network-scripts/ifcfg-eth0) usually looks like

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
HWADDR=00:0c:29:76:c8:2d

To have a static ip address we edit /etc/sysconfig/network-scripts/ifcfg-eth0 as follows:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
HWADDR=00:0c:29:76:c8:2d
IPADDR=192.168.1.50
NETMASK=255.255.255.0

When one or more DNS servers are available, edit /etc/resolv.conf and setup the DNS servers, for example,

nameserver 192.168.1.1
nameserver 192.168.1.1

Edit the /etc/hosts when necessary, for example,

127.0.0.1       localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
192.168.1.50    axis-into-ict.nl axis-into-ict

Restart the network service and check the configuration

[root@axis-into-ict ~]# service network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:                                [  OK  ]

[root@axis-into-ict ~]# /sbin/ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:0C:29:76:C8:2D
          inet addr:192.168.1.50  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:410 errors:0 dropped:0 overruns:0 frame:0
          TX packets:114 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:30288 (29.5 KiB)  TX bytes:12023 (11.7 KiB)

Verify the static ip configuration by using

[root@axis-into-ict ~]# ping -c 3 -M do -s 1468 192.168.1.60
PING 192.168.1.60 (192.168.1.60) 1468(1496) bytes of data.
1476 bytes from 192.168.1.60: icmp_seq=1 ttl=128 time=0.837 ms
1476 bytes from 192.168.1.60: icmp_seq=2 ttl=128 time=0.345 ms
1476 bytes from 192.168.1.60: icmp_seq=3 ttl=128 time=0.346 ms

--- 192.168.1.60 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.345/0.509/0.837/0.232 ms

Packet loss minimization

The operating system buffers must be large enough to handle incoming network traffic while the application is paused during garbage collection. Also when UDP (User Datagram Protocol) is used to transmit messages to server instances in a cluster we must limit the need to retransmit UDP messages. In this case the size of the operating system buffers must be set appropriately to avoid excessive UDP datagram loss. To configure Linux to allow for larger buffers we can follow these steps:

Edit the /etc/sysctl.conf and add the following lines:

		# increase TCP max buffer size
		net.core.rmem_max = 2096304
		net.core.wmem_max = 2096304
		net.core.rmem_default=2096304
		net.core.wmem_default=2096304

Run /sbin/sysctl -p to make the changes effective. By executing, for example, cat /proc/sys/net/core/rmem_max we can check the current values.

Maximum number of open file descriptors

Most operating systems handle sockets as a form of file access and use file descriptors to keep track of which sockets are open. To contain the resources per process, the operating system restricts the number of file descriptors per process. Linux limits the number of open file descriptors per process, by default this is equal to 1024. It could be that the 1024 limit does not offer optimal performance. To obtain the current open file limit we can use the ulimit command. The ulimit -aS command shows the current limit, the ulimit -aH command shows the hard limit. To increase the total number of file descriptors for all users we can add the following to the /etc/sysctl.conf file: fs.file-max = 344030. To check the current value we can use cat /proc/sys/fs/file-max. If we want to make sure that a process (running under the jboss user) gets a maximum of 8192 open file descriptors, we have to edit the /etc/security/limits.conf file:

jboss soft nofile 8192
jboss hard nofile 8192

Note that this number applies across all processes running under the jboss user. To make the adjustments active, execute the ulimit -u 8192 command and log out and log in again.

TCP/IP

On some systems the default value for the time wait interval is too high and needs to be adjusted. To determine the number of sockets in TIME_WAIT we can use netstat -a | grep TIME_WAIT | wc -l. When the number approaches the maximum number of file descriptors per process, the application’s throughput will degrade, i.e., new connections have to wait for a free space in the application’s file descriptor table. To check the current value we can use cat /proc/sys/net/ipv4/tcp_keepalive_time and cat /proc/sys/net/ipv4/tcp_keepalive_intvl. The default time wait interval on newer Linux systems is 60 seconds and as such there is no need to change this. If the need arises that TCP/IP settings must be adjusted follow these steps:

Edit the /etc/sysctl.conf and add the following lines:

		net.ipv4.tcp_rmem = 4096    87380   4194304
		net.ipv4.tcp_wmem = 4096    16384   4194304
		net.ipv4.tcp_sack = 1
		net.ipv4.tcp_timestamps = 1
		net.ipv4.tcp_window_scaling = 1
		net.ipv4.tcp_keepalive_time = 7200
		net.ipv4.tcp_keepalive_intvl = 75
		net.ipv4.tcp_fin_timeout = 60

Run /sbin/sysctl -p to make the changes effective. By executing, for example, cat /proc/sys/net/ipv4/tcp_keepalive_time we can check the current values.

Network interface card (NIC)

Configure the network card at it’s maximum link speed and at full duplex. To determine the configuration of the network card we can use:

[root@axis-into-ict jboss]# /sbin/ethtool eth0
Settings for eth0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
							100baseT/Half 100baseT/Full
							1000baseT/Full
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full
							100baseT/Half 100baseT/Full
							1000baseT/Full
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Supports Wake-on: d
	Wake-on: d
	Current message level: 0x00000007 (7)
	Link detected: yes

To change the maximum speed and duplex style we can use, for example, /sbin/ethtool -s eth0 speed 1000 duplex full.

Maximum transaction unit (MTU)

MTU is the maximum packet size, in bytes, that can be transmitted across a link. By default, an MTU of 1500 bytes is assumed and a default packet size of 1500 – 32 = 1468 bytes is used. To determine the MTU we can use the ping utility.

Swapping

Swapping, also known as paging, is the use of secondary storage to store and retrieve data for use in RAM. Swapping is automatically performed by the operating system and typically occurs when the available RAM memory is depleted. Swapping can have a significant impact on the performance of Coherence and should thus be avoided. One thing to avoid swapping is to have enough RAM available – a tool such as top can be used to monitor swap rates. The swappiness (swap out a process or some of its heap due to low resource usage) should also be avoided for Coherence. To obtain the current value for swappiness, we can use cat /proc/sys/vm/swappiness. For Coherence, a low value (0 if possible) should be set. To change the swappiness, open the /etc/sysctl.conf file and add the following:

# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

# increase TCP max buffer size
net.core.rmem_max = 2096304
net.core.wmem_max = 2096304
net.core.rmem_default=2096304
net.core.wmem_default=2096304

# set the swappiness to zero
vm.swappiness = 0

Large pages

Large pages are essentially blocks of contiguous physical memory addresses that are reserved for a process. Large pages improve performance of applications that access memory frequently (as is the case when using Coherence). When large pages are used the application uses the translation look-aside buffer (TLB) in the processor more effectively. The TLB is a cache of recently used virtual-to-physical address space translations stored in the processor memory. To obtain data from memory, the processor looks up the TLB to find out the physical addresses (RAM or hard disk) that hold the required data. In the case of large pages, a single entry in the TLB could represent a large contiguous address space and thereby potentially reducing the TLB look-up frequency and avoiding frequent look-ups in the hierarchical page table stored in-memory.

The kernel built with hugepage support (CONFIG_HUGETLBFS and CONFIG_HUGETLB_PAGE selected) should show the number of configured hugepages in the system by running cat /proc/meminfo:

[root@middleware-magic jboss]# cat /proc/meminfo
MemTotal:      4044532 kB
MemFree:       3411928 kB
Buffers:        167968 kB
Cached:         195860 kB
SwapCached:          0 kB
Active:         262596 kB
Inactive:       247148 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      4044532 kB
LowFree:       3411928 kB
SwapTotal:     6088624 kB
SwapFree:      6088536 kB
Dirty:             232 kB
Writeback:           0 kB
AnonPages:      146020 kB
Mapped:          52732 kB
Slab:            79676 kB
PageTables:      19360 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   8110888 kB
Committed_AS:   426808 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    265976 kB
VmallocChunk: 34359471959 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

where,

HugePages_Total is the size of the pool of hugepages.
HugePages_Free is the number of hugepages in the pool that are not yet allocated.
HugePages_Rsvd is short for reserved, and is the number of hugepages for which a commitment to allocate from the pool has been made, but no allocation has yet been made.
Hugepagesize is the configured hugepage size.

Also cat /proc/filesystems should show a filesystem of type hugetlbfs configured in the kernel:

[root@middleware-magic jboss]# cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   cpuset
nodev   binfmt_misc
nodev   debugfs
nodev   securityfs
nodev   sockfs
nodev   usbfs
nodev   pipefs
nodev   anon_inodefs
nodev   futexfs
nodev   tmpfs
nodev   inotifyfs
nodev   eventpollfs
nodev   devpts
        ext2
nodev   ramfs
nodev   hugetlbfs
        iso9660
nodev   mqueue
        ext3
nodev   rpc_pipefs
nodev   autofs

The following command can be used to dynamically allocate/deallocate hugepages: echo 1024 > /proc/sys/vm/nr_hugepages. This command will try to configure 1024 hugepages in the system. The success or failure of allocation depends on the amount of physically contiguous memory that is present in the system at this time. At times, the JVM might not be able to reserve large pages because the pages might not be contiguous. To ensure that the JVM can use large pages, these must be enabled in the operating system as soon as possible after the system starts; otherwise the memory can become too fragmented which leads to a reduction in the number of large pages. In order to accomplish this, we can put the command in the /etc/rc.d/rc.local script, for example,

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

# set the sharable memory size - in this case we set the memory size to 4GB:
# 1024*1024*1024*4 = 4294967296
echo 4294967296 > /proc/sys/kernel/shmmax

# set the number of huge pages based on the Hugepagesize, i.e., 2048kB
# when we want to reserve 2GB in huge pages we have to set the number of huge pages to:
# (1024*1024*1024*2)/(1024*1024*2) = 2147483648/2097152 = 1024
echo 1024 > /proc/sys/vm/nr_hugepages

# give permission to the group that runs the process to access the shared memory segment
# to this end open the /etc/group file and retrieve the group-id (jboss:x:500:)
echo 500 > /proc/sys/vm/hugetlb_shm_group

Next, we set the locked in-memory address space for the jboss user in the file /etc/security/limits.conf, i.e.,

# memlock - maximum locked in-memory address space (kB), we set this equal to:
# number_of_huge_pages * huge_page_size = 1024 * 2048 = 2097152
jboss  soft    memlock 2097152
jboss  hard    memlock 2097152

Large pages can be exploited by the Java Virtual Machine, for example,

-server -Xms1024m -Xmx1024m -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -XX:+UseTLAB -XX:LargePageSizeInBytes=2048k -XX:+UseLargePages

-server – select the JIT compiler.
-Xms – initial heap size.
-Xmx – maximum heap size.
-XX:NewRatio=N – sets the young generation to heap size / (1 + N).
-XX:-UseParallelGC – select the parallel collector for minor collections.
-XX:ParallelGCThreads – sets the number of garbage collector threads, i.e., the number of CPUs to be used.
-XX:MaxGCPauseMillis – sets the maximum pause time goal.
-XX:GCTimeRatio=N – sets the throughput goal that is measured in terms of the time spent doing garbage collection versus the time spent outside of garbage collection. In the case above with N=19, the garbage collection time to application time is set to 1 / (1+N) = 1 / 20, i.e., 5% of the total time is spent in garbage collection.
-XX:-UseParallelOldGC – selects the parallel collector for major collections.
-XX:+UseTLAB – enables thread-local object allocation.
-XX:LargePageSizeInBytes – sets the large page size used for the Java heap. We set this equal to the operating system parameter: Hugepagesize, which in our case is 2048kB
-XX:+UseLargePages – use large page memory.

One thing to note is that when large pages are not appropriately configured it can lead to an increase of swapping which should be monitored.

Start and stop scripts

The domain environment we presented in the Fun with JBoss post can be started by using domain.sh. By turning of the auto-start feature of the servers, the domain.sh script will start the process controller and the host controller. When these processes are started, we can use the command-line interface to start the individual servers. To disable auto-start we edit the host.xml file

<host name="axis-into-ict.nl" xmlns="urn:jboss:domain:1.1">
	...
    <jvms>
    	<jvm name="default">
            <heap size="512m" max-size="512m"/>
            <permgen size="256m" max-size="256m"/>
            <jvm-options>
                <option value="-server"/>
                <option value="-XX:NewRatio=2"/>
                <option value="-XX:+UseParallelGC"/>
                <option value="-XX:ParallelGCThreads=2"/>
                <option value="-XX:MaxGCPauseMillis=200"/>
                <option value="-XX:GCTimeRatio=19"/>
                <option value="-XX:+UseParallelOldGC"/>
				<option value="-Dtangosol.coherence.mode=prod"/>
				<option value="-Dtangosol.coherence.distributed.localstorage=false"/>
            </jvm-options>
        </jvm>
    </jvms>
    <servers>
        <server name="server1" group="ha-server-group" auto-start="false">
        </server>
        <server name="server2" group="ha-server-group" auto-start="false">
            <socket-bindings port-offset="1"/>
        </server>
    </servers>
</host>

In which we have also added two system parameters for Coherence. By setting the system property tangosol.coherence.distributed.localstorage to false, we disable storage for the Coherence nodes running on the application servers as we do not want to use the JVMs, the application servers are running in, for storage. To enable storage we are going to start separate stand alone JVMs that run Coherence cache servers. To start up the environment we can use the following script

#!/bin/bash

JBOSS_HOME="/home/jboss/jboss-as-7.1.0.Final"
export JBOSS_HOME

APACHE_HOME="/home/jboss/apache/"
export APACHE_HOME

start_apache() {
	echo "starting Apache HTTP Server"
	${APACHE_HOME}/bin/apachectl -k start
}

start_controllers() {
	echo "starting process controller and host controller"
	${JBOSS_HOME}/bin/domain.sh >/dev/null 2>/dev/null &
	echo "waiting a little for the host controller to finish starting"
	sleep 10
}

start_servers() {
	echo "starting servers"
	${JBOSS_HOME}/bin/jboss-cli.sh --file=${JBOSS_HOME}/scripts/start-servers.cli
}

start_controllers

start_servers

start_apache

In which the start-server.cli looks as follows

connect axis-into-ict.nl:9999
/host=axis-into-ict.nl/server-config=server1:start
/host=axis-into-ict.nl/server-config=server2:start

To stop the environment we can use

#!/bin/bash

JBOSS_HOME="/home/jboss/jboss-as-7.1.0.Final"
export JBOSS_HOME

APACHE_HOME="/home/jboss/apache/"
export APACHE_HOME

stop_apache() {
	echo "stopping Apache HTTP Server"
	${APACHE_HOME}/bin/apachectl -k stop
}

stop_servers() {
	echo "stopping JBoss processes"
	${JBOSS_HOME}/bin/jboss-cli.sh --file=${JBOSS_HOME}/scripts/stop-servers.cli
}

stop_apache

stop_servers

In which the stop-server.cli script looks as follows

connect axis-into-ict.nl:9999
/host=axis-into-ict.nl/server-config=server1:stop
/host=axis-into-ict.nl/server-config=server2:stop
/host=axis-into-ict.nl:shutdown

Class loading

The application presented here uses JSF 1.2, JBoss RichFaces 3.3.1 and Coherence 3.7.1. These classes are by default not present, and we need to do something. In the JBoss application server, instead of the more familiar hierarchical class loading environment, the class loading is based on modules that have to define explicit dependencies on other modules. Deployments are also modules, and do not have access to classes that are defined in jars in the application server unless an explicit dependency on those classes is defined. Even though modules are isolated by default, as part of the deployment process some dependencies on modules defined by the application server are set up automatically. For example, when deploying a Java EE application a dependency on the Java EE API’s will be added to the module automatically. Similarly if the module contains a beans.xml file a dependency on Weld will be added automatically, along with any supporting modules that Weld needs to operate. The same happens when a faces-config.xml is present, which leads to loading of the module javax.faces.api, i.e., the JSF2.1 API is automatically added. As we are using JSF 1.2, we have exclude the automatic JSF dependency by using the jboss-deployment-structure.xml file. A complete list of automatic dependencies can be found here.

Module dependencies are added in a specific order:

System Dependencies are added to the module automatically by the container, including the Java EE API’s.
User Dependencies are added through jboss-deployment-structure.xml or through the Dependencies in the manifest file.
Local Resources are class files packaged up inside the deployment itself, e.g. class files from WEB-INF/classes or WEB-INF/lib of a .war. A .war is considered to be a single module, so classes defined in WEB-INF/lib are treated the same as classes in WEB-INF/classes. All classes packaged in the .war will be loaded with the same class loader.
Inter deployment dependencies, i.e., dependencies on other deployments in an .ear deployment, which can include classes in an ear’s lib directory, or classes defined in other ejb jars. An .ear deployment is a multi-module deployment. This means that not all classes inside an .ear will necessarily have access to all other classes in the .ear, unless explicit dependencies have been defined. By default the ear/lib directory is a single module, and every Web or EJB deployment is also a separate module. A sub-deployment (.war and .jar) have a dependency on the parent module, which gives them access to classes in ear/lib, however they do not always have an automatic dependency on each other. This behaviour is controlled via the ear-subdeployments-isolated setting in the ee subsystem configuration. By default this is set to false, which allows a sub-deployment to see classes belonging to another sub-deployment within the .ear.

Our first step to deploy the application is define the appropriate modules. To this end, we create the following directory structure

${JBOSS_HOME}/modules
	/com/coherence
		/main
			module.xml
			coherence.jar
	/javax/faces/api
		/1.2.14
			module.xml
			commons-beanutils-1.7.0.jar
			commons-collections-3.2.jar
			commons-digester-1.8.jar
			commons-logging-1.0.4.jar
			jsf-api.jar
			jsf-facelets.jar
			jsf-impl.jar
			jstl-api-1.2.jar
			jstl-impl-1.2.jar
	/org/jboss/richfaces
		/main
			module.xml
			jhighlight-1.0.jar
			richfaces-api-3.3.1.GA.jar
			richfaces-impl-3.3.1.GA.jar
			richfaces-ui-3.3.1.GA.jar

The module.xml for the Coherence module looks as follows

<module xmlns="urn:jboss:module:1.1" name="com.coherence">
    <dependencies>
	<!-- This dependency is for javax.naming -->
        <module name="javax.api" export="true"/>
	<!-- This dependency is for sun.rmi.server -->
	<module name="sun.jdk" export="true"/>
    </dependencies>
    <resources>
        <resource-root path="coherence.jar"/>
    </resources>
</module>

Here an important note is in order. The module sun.jdk does by default not contain a path for sun.rmi.server and has to be added to the module

<module xmlns="urn:jboss:module:1.1" name="sun.jdk">
    <resources>...</resources>
    <dependencies>
        <system export="true">
            <paths>
				...
				<path name="sun/rmi/server"/>
				...
            </paths>
            <exports>...</exports>
        </system>
    </dependencies>
</module>

The module.xml for the JSF 1.2.14 module looks as follows

<module xmlns="urn:jboss:module:1.1" name="javax.faces.api" slot="1.2.14">
    <dependencies>
		<!-- This dependency is for javax.naming -->
        <module name="javax.api" export="true"/>
        <module name="javax.el.api" export="true"/>
        <module name="javax.servlet.api" export="true"/>
        <module name="javax.servlet.jsp.api" export="true"/>
        <module name="javax.servlet.jstl.api" export="true"/>
        <module name="javax.validation.api" export="true"/>
		<module name="org.apache.log4j" export="true"/>
    </dependencies>
    <resources>
		<resource-root path="commons-beanutils-1.7.0.jar"/>
		<resource-root path="commons-collections-3.2.jar"/>
		<resource-root path="commons-digester-1.8.jar"/>
		<resource-root path="commons-logging-1.0.4.jar"/>
        <resource-root path="jsf-api.jar"/>
		<resource-root path="jsf-facelets.jar"/>
		<resource-root path="jsf-impl.jar"/>
		<resource-root path="jstl-api-1.2.jar"/>
		<resource-root path="jstl-impl-1.2.jar"/>
    </resources>
</module>

Note that the commons-beanutils (org/apache/commons/beanutils), commons-collections (org/apache/commons/collections) and commons-logging (org/apache/commons/logging) are also present as modules and could have been used in the dependencies as well, instead of adding the .jar files. The slot definition will be used to specify the appropriate dependency in the jboss-deployment-structure.xml file. The module.xml for the JBoss RichFaces module looks as follows

<module xmlns="urn:jboss:module:1.1" name="org.jboss.richfaces">
    <dependencies>
		<module name="javax.faces.api" slot="1.2.14" export="true"/>
    </dependencies>
    <resources>
        <resource-root path="jhighlight-1.0.jar"/>
		<resource-root path="richfaces-api-3.3.1.GA.jar"/>
		<resource-root path="richfaces-impl-3.3.1.GA.jar"/>
		<resource-root path="richfaces-ui-3.3.1.GA.jar"/>
    </resources>
</module>

As we are dealing with a .war deployment, we have to add the jboss-deployment-structure.xml file to the WEB-INF directory. In the case of an .ear deployment it must be added in the root META-INF directory. The contents of jboss-deployment-structure.xml looks as follows

<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="javax.faces.api"/>
            <module name="com.sun.jsf-impl"/>
        </exclusions>
        <dependencies>
            <module name="com.coherence"/>
            <module name="javax.faces.api" slot="1.2.14"/>
            <module name="org.jboss.richfaces"/>
        </dependencies>
    </deployment>
</jboss-deployment-structure>

Here, we exclude the default JSF 2.1 API and load the appropriate modules by using the dependencies element. The .war file still needs to have all the impl jars in the WEB-INF/lib directory, i.e.,

Security.war
	/WEB-INF
		/classes
			/packages with compiled classes
		/lib
			jsf-facelets.jar
			jsf-impl.jar
			jstl-impl-1.2.jar
			richfaces-impl-3.3.1.GA.jar
			richfaces-ui-3.3.1.GA.jar
		faces-config.xml
		jboss-deployment-structure.xml
		web.xml

With the modules in place, we restart the servers by using

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.50:9999
[domain@192.168.1.50:9999 /] /host=axis-into-ict.nl/server-config=server2:restart
{
    "outcome" => "success",
    "result" => "STARTING"
}
[domain@192.168.1.50:9999 /] /host=axis-into-ict.nl/server-config=server1:restart
{
    "outcome" => "success",
    "result" => "STARTING"
}

Security

We add two different users (employee and manager) to the application realm. To this end run add-user.sh (${JBOSS_HOME}/bin) and follow the instructions given on the screen, for example,

[jboss@axis-into-ict bin]$ ./add-user.sh 

What type of user do you wish to add?
 a) Management User (mgmt-users.properties)
 b) Application User (application-users.properties)
(a): b

Enter the details of the new user to add.
Realm (ApplicationRealm) :
Username : employee
Password :
Re-enter Password :
What roles do you want this user to belong to? (Please enter a comma separated list, or leave blank for none) : EMPLOYEE
About to add user 'employee' for realm 'ApplicationRealm'
Is this correct yes/no? yes
Added user 'employee' to file '/home/jboss/jboss-as-7.1.0.Final/standalone/configuration/application-users.properties'
Added user 'employee' to file '/home/jboss/jboss-as-7.1.0.Final/domain/configuration/application-users.properties'
Added user 'employee' with roles EMPLOYEE to file '/home/jboss/jboss-as-7.1.0.Final/standalone/configuration/application-roles.properties'
Added user 'employee' with roles EMPLOYEE to file '/home/jboss/jboss-as-7.1.0.Final/domain/configuration/application-roles.properties'

When a user logs in with MANAGER privileges, the user is able to delete customers. When a user logs in with EMPLOYEE privileges, the delete option is not rendered.

Before we deploy the application, we start a Coherence cache server. The reason for this is that the application server nodes are configured as cache clients that do not store data and at least one node in a Coherence cluster must be present that can store data. To start a cache server we use the following script

#!/bin/sh

JMX_OPTIONS="-Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
COHERENCE_MANAGEMENT_OPTIONS="-Dtangosol.coherence.management=all -Dtangosol.coherence.management.remote=true"
COHERENCE_OPTIONS="-Dtangosol.coherence.cacheconfig=security-cache-config.xml -Dtangosol.coherence.mode=prod ${COHERENCE_MANAGEMENT_OPTIONS} ${JMX_OPTIONS}"
export COHERENCE_OPTIONS

JAVA_HOME="/home/jboss/jdk1.6.0_31"
export JAVA_HOME

MEM_ARGS="-server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing"
export MEM_ARGS

CLASSPATH="/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/coherence.jar:/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/test.jar"
export CLASSPATH

${JAVA_HOME}/bin/java ${MEM_ARGS} ${COHERENCE_OPTIONS} com.tangosol.net.DefaultCacheServer

To deploy the application we use the command-line utility

[domain@192.168.1.50:9999 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/Security.war --server-groups=ha-server-group
[domain@192.168.1.50:9999 /] undeploy Security.war --server-groups=ha-server-group

which also shows how to undeploy it. The Coherence logging shows how the cluster is created

[jboss@axis-into-ict scripts]$ ./default-cache-server.sh
2012-03-10 10:43:24.872/0.390 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/coherence.jar!/tangosol-coherence.xml"
2012-03-10 10:43:24.956/0.474 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/coherence.jar!/tangosol-coherence-override-prod.xml"
2012-03-10 10:43:25.089/0.607 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/test.jar!/tangosol-coherence-override.xml"
2012-03-10 10:43:25.093/0.611 Oracle Coherence 3.7.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified

Oracle Coherence Version 3.7.1.0 Build 27797
 Grid Edition: Production mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

2012-03-10 10:43:25.676/1.194 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Loaded cache configuration from "jar:file:/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/test.jar!/security-cache-config.xml"
2012-03-10 10:43:25.891/1.409 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Loaded Reporter configuration from "jar:file:/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/coherence.jar!/reports/report-group.xml"
2012-03-10 10:43:26.637/2.155 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, member=n/a): TCMP bound to /192.168.1.50:8088 using SystemSocketProvider
2012-03-10 10:43:57.408/32.926 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster, member=n/a): Created a new cluster "cluster:0xFCDB" with Member(Id=1, Timestamp=2012-03-10 10:43:26.685, Address=192.168.1.50:8088, MachineId=54112, Location=site:,machine:axis-into-ict,process:11068, Role=CoherenceServer, Edition=Grid Edition, Mode=Production, CpuCount=2, SocketCount=1) UID=0xC0A8013200000135FBFDC4DDD3601F98
2012-03-10 10:43:57.418/32.936 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Started cluster Name=cluster:0xFCDB

Group{Address=224.3.7.0, Port=37000, TTL=4}

MasterMemberSet(
  ThisMember=Member(Id=1, Timestamp=2012-03-10 10:43:26.685, Address=192.168.1.50:8088, MachineId=54112, Location=site:,machine:axis-into-ict,process:11068, Role=CoherenceServer)
  OldestMember=Member(Id=1, Timestamp=2012-03-10 10:43:26.685, Address=192.168.1.50:8088, MachineId=54112, Location=site:,machine:axis-into-ict,process:11068, Role=CoherenceServer)
  ActualMemberSet=MemberSet(Size=1
    Member(Id=1, Timestamp=2012-03-10 10:43:26.685, Address=192.168.1.50:8088, MachineId=54112, Location=site:,machine:axis-into-ict,process:11068, Role=CoherenceServer)
    )
  MemberId|ServiceVersion|ServiceJoined|MemberState
    1|3.7.1|2012-03-10 10:43:57.409|JOINED
  RecycleMillis=1200000
  RecycleSet=MemberSet(Size=0
    )
  )

TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}

2012-03-10 10:43:57.473/32.991 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2012-03-10 10:43:57.662/33.180 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache, member=1): Loaded POF configuration from "jar:file:/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/test.jar!/security-pof-config.xml"
2012-03-10 10:43:57.701/33.219 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache, member=1): Loaded included POF configuration from "jar:file:/home/jboss/jboss-as-7.1.0.Final/modules/com/coherence/main/coherence.jar!/coherence-pof-config.xml"
2012-03-10 10:43:57.763/33.281 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1
2012-03-10 10:43:57.788/33.306 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=1):
Services
  (
  ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1}
  InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
  PartitionedCache{Name=DistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
  )

Started DefaultCacheServer...

2012-03-10 10:46:35.557/191.075 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2012-03-10 10:46:35.396, Address=192.168.1.50:8090, MachineId=54112, Location=site:,machine:axis-into-ict,process:11537, Role=JbossModulesMain) joined Cluster with senior member 1
2012-03-10 10:46:35.666/191.184 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=3, Timestamp=2012-03-10 10:46:35.481, Address=192.168.1.50:8092, MachineId=54112, Location=site:,machine:axis-into-ict,process:11566, Role=JbossModulesMain) joined Cluster with senior member 1
2012-03-10 10:46:35.905/191.423 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 2 joined Service Management with senior member 1
2012-03-10 10:46:36.117/191.635 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 3 joined Service Management with senior member 1
2012-03-10 10:46:36.682/192.200 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 2 joined Service DistributedCache with senior member 1
2012-03-10 10:46:36.897/192.415 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 3 joined Service DistributedCache with senior member 1

Here, the last lines (after Started DefaultCacheServer) are the application server nodes that are joining the Coherence cluster

[jboss@axis-into-ict scripts]$ ./start-environment.sh
starting process controller and host controller
waiting a little for the host controller to finish starting
starting servers
{
    "outcome" => "success",
    "result" => "STARTING"
}
{
    "outcome" => "success",
    "result" => "STARTING"
}
starting Apache HTTP Server

When we stop the application servers

[jboss@axis-into-ict scripts]$ ./stop-environment.sh
stopping Apache HTTP Server
stopping JBoss processes
{
    "outcome" => "success",
    "result" => "STOPPING"
}
{
    "outcome" => "success",
    "result" => "STOPPING"
}
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}

the following is observed in the Coherence logging

2012-03-10 10:55:33.075/728.593 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): TcpRing disconnected from Member(Id=3, Timestamp=2012-03-10 10:46:35.481, Address=192.168.1.50:8092, MachineId=54112, Location=site:,machine:axis-into-ict,process:11566, Role=JbossModulesMain) due to a peer departure; removing the member.
2012-03-10 10:55:33.076/728.594 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 3 left service Management with senior member 1
2012-03-10 10:55:33.076/728.594 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 3 left service DistributedCache with senior member 1
2012-03-10 10:55:33.076/728.594 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=3, Timestamp=2012-03-10 10:55:33.076, Address=192.168.1.50:8092, MachineId=54112, Location=site:,machine:axis-into-ict,process:11566, Role=JbossModulesMain) left Cluster with senior member 1
2012-03-10 10:55:33.140/728.658 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): TcpRing disconnected from Member(Id=2, Timestamp=2012-03-10 10:46:35.396, Address=192.168.1.50:8090, MachineId=54112, Location=site:,machine:axis-into-ict,process:11537, Role=JbossModulesMain) due to a peer departure; removing the member.
2012-03-10 10:55:33.140/728.658 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 2 left service Management with senior member 1
2012-03-10 10:55:33.140/728.658 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 2 left service DistributedCache with senior member 1
2012-03-10 10:55:33.140/728.658 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2012-03-10 10:55:33.14, Address=192.168.1.50:8090, MachineId=54112, Location=site:,machine:axis-into-ict,process:11537, Role=JbossModulesMain) left Cluster with senior member 1

When the application servers are started again the following is observed in the Coherence logging

2012-03-10 10:58:43.009/918.527 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=4, Timestamp=2012-03-10 10:58:42.837, Address=192.168.1.50:8090, MachineId=54112, Location=site:,machine:axis-into-ict,process:12772, Role=JbossModulesMain) joined Cluster with senior member 1
2012-03-10 10:58:43.242/918.760 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 4 joined Service Management with senior member 1
2012-03-10 10:58:44.436/919.954 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 4 joined Service DistributedCache with senior member 1
2012-03-10 10:58:44.444/919.962 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=5, Timestamp=2012-03-10 10:58:44.264, Address=192.168.1.50:8092, MachineId=54112, Location=site:,machine:axis-into-ict,process:12801, Role=JbossModulesMain) joined Cluster with senior member 1
2012-03-10 10:58:44.777/920.295 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 5 joined Service Management with senior member 1
2012-03-10 10:58:45.259/920.777 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 5 joined Service DistributedCache with senior member 1

Configure mod_cluster

The advantage of using mod_cluster with respect to mod_jk (or mod_proxy) is a dynamic cluster configuration. As we saw in the Fun with JBoss post, to configure mod_jk we have to provide a static list of servers. When expanding the cluster, for example, to other machines (as will be done below) we have to add the servers to the mod_jk load balancer configuration. When using mod_cluster, servers in the cluster are discovered through an advertising mechanism (UDP messages on a multicast group). The set of proxies to which an application server will communicate is determined either by a static list (configure in the domain.xml file) or using dynamic discovery (by using the advertising mechanism). Load balance factors are calculated and provided by the application servers, rather than computing these in the proxy. Each server forwards any web application context lifecycle events (e.g. web-app deploy/undeploy) to the proxy informing it to start/stop routing requests for a given context to that server.

A mod_cluster distribution can be downloaded here. We will use mod_cluster 1.2.0.Final. Note that the binaries (for example, binaries linux2-x64) contain a full Apache distribution, while the dynamic libraries (for example, dynamic libraries linux2-x64) contain only the modules for mod_cluster (excluding mod_proxy.so, mod_proxy_ajp.so and mod_proxy_http.so). As we already have an Apache installed we will use the dynamic libraries distribution. To configure mod_cluster we first copy the libraries into ${APACHE_HOME}/modules, i.e.,

${APACHE_HOME}
	/conf
		httpd.conf
		jboss-workers.properties
		mod_jk.conf
		mod_cluster.conf
	/modules
		mod_advertise.so
		mod_jk.so
		mod_manager.so
		mod_proxy.so
		mod_proxy_ajp.so
		mod_proxy_http.so
		mod_proxy_cluster.so
		mod_slotmem.so

To configure mod_cluster, we create a mod_cluster configuration file, for example mod_cluster.conf. The configuration has the following contents

LoadModule proxy_module "/home/jboss/apache/modules/mod_proxy.so"
LoadModule proxy_ajp_module "/home/jboss/apache/modules/mod_proxy_ajp.so"
LoadModule proxy_http_module "/home/jboss/apache/modules/mod_proxy_http.so"
LoadModule slotmem_module "/home/jboss/apache/modules/mod_slotmem.so"
LoadModule manager_module "/home/jboss/apache/modules/mod_manager.so"
LoadModule proxy_cluster_module "/home/jboss/apache/modules/mod_proxy_cluster.so"
LoadModule advertise_module "/home/jboss/apache/modules/mod_advertise.so"

NameVirtualHost axis-into-ict.nl:8888

<VirtualHost axis-into-ict.nl:8888>
	EnableMCPMReceive
	ManagerBalancerName mycluster

	AllowDisplay On

	KeepAliveTimeout 60
	MaxKeepAliveRequests 0

	ServerAdvertise On
	AdvertiseGroup 224.0.1.105:23364
	AdvertiseFrequency 10

	<Directory />
    	Order deny,allow
		Deny from all
	    Allow from 192.168.1.
	</Directory>

	<Location /mod_cluster-manager>
		SetHandler mod_cluster-manager
		Order deny,allow
   		Deny from all
   		Allow from 192.168.1.
	</Location>
</VirtualHost>

The used directives are explained here. Next, add the following Include directive to httpd.conf

# put it near the end of the file where all the other includes are present
# mod_cluster configuration
Include conf/mod_cluster.conf

The protocol used by mod_cluster to communicate with the servers in based on the connectors configured in domain.xml. When an AJP connector is configured it will be used, otherwise an HTTP connector will be used. While AJP is generally faster, an HTTP connector can optionally be secured via SSL. In the case of an AJP connector make sure the scheme attribute is set to http, otherwise a request after a form-based log in is not redirected correctly

<subsystem xmlns="urn:jboss:domain:web:1.1" native="false" default-virtual-server="default-host">
	<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
	<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" enable-lookups="false" max-post-size="1048576" enabled="true" max-connections="150"/>
	<virtual-server name="default-host" enable-welcome-root="true">
		<alias name="localhost"/>
		<alias name="example.com"/>
	</virtual-server>
</subsystem>

Restart the Apache HTTP Server. By accessing the URL: http://hostname:8888/mod_cluster-manager, we get an overview of the configuration

mod_cluster/1.2.0.Final

start of "httpd.conf" configuration
mod_proxy_cluster.c: OK
mod_sharedmem.c: OK
Protocol supported: AJP
mod_advertise.c: OK
Server: axis-into-ict.nl
Server: axis-into-ict.nl VirtualHost: 192.168.1.50:8888 Advertising on Group 224.0.1.105 Port 23364 for http://192.168.1.50:8888 every 10 seconds
end of "httpd.conf" configuration

Auto Refresh show DUMP output show INFO output
Node f817d8c0-737d-3134-beb0-85dee2469701 (ajp://axis-into-ict.nl:9080):

Enable Contexts Disable Contexts
Balancer: mycluster,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Status: OK,Elected: 750,Read: 1310513,Transferred: 4869,Connected: 0,Load: 1
Virtual Host 1:

Contexts:

/, Status: ENABLED Request: 0 Disable
/Security, Status: ENABLED Request: 0 Disable
Aliases:

default-host
localhost
example.com
Node 236ade27-f8e4-30c5-aa80-ded57bfe7736 (ajp://axis-into-ict.nl:9081):

Enable Contexts Disable Contexts
Balancer: mycluster,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Status: OK,Elected: 383,Read: 850881,Transferred: 2112,Connected: 0,Load: 1
Virtual Host 1:

Contexts:

/, Status: ENABLED Request: 0 Disable
/Security, Status: ENABLED Request: 0 Disable
Aliases:

default-host
localhost
example.com

The mod_cluster configuration can also be viewed by using the command-line interface

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.50:9999
[domain@192.168.1.50:9999 /] /profile=full-ha/subsystem=modcluster/mod-cluster-config=configuration:read-resource
{
    "outcome" => "success",
    "result" => {
        "advertise" => true,
        "advertise-socket" => "modcluster",
        "auto-enable-contexts" => true,
        "balancer" => "mycluster",
        "excluded-contexts" => "ROOT,admin-console,invoker,jbossws,jmx-console,juddi,web-console",
        "flush-packets" => false,
        "flush-wait" => -1,
        "max-attemps" => 1,
        "node-timeout" => -1,
        "ping" => 10,
        "proxy-list" => "",
        "proxy-url" => "/",
        "socket-timeout" => 20,
        "ssl" => undefined,
        "sticky-session" => true,
        "sticky-session-force" => false,
        "sticky-session-remove" => false,
        "stop-context-timeout" => 10,
        "ttl" => 60,
        "worker-timeout" => -1
    }
}

When the advertising mechanism is not working (for example, multicast is disabled), we can set the ServerAdvertise directive to Off (also comment all the advertise related directives). When doing this we have to configure the modcluster subsystem within the domain and set the proxy list by hand, for example,

<domain xmlns="urn:jboss:domain:1.1">
    <extensions>
		...
        <extension module="org.jboss.as.modcluster"/>
		...
    </extensions>
    <system-properties>
        <property name="java.net.preferIPv4Stack" value="true"/>
    </system-properties>
    <profiles>
        <profile name="full-ha">
			...
            <subsystem xmlns="urn:jboss:domain:modcluster:1.0">
                <mod-cluster-config advertise-socket="modcluster proxy-list="192.168.1.50:8888"/>
            </subsystem>
			...
        </profile>
    </profiles>
    <interfaces>...</interfaces>
    <socket-binding-groups>
        <socket-binding-group name="ha-sockets" default-interface="public">
			...
            <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
			...
        </socket-binding-group>
    </socket-binding-groups>
    <deployments>
        <deployment name="Security.war" runtime-name="Security.war">
            <content sha1="5a864da538be690f7dcf282201f3574678edb139"/>
        </deployment>
    </deployments>
    <server-groups>
        <server-group name="ha-server-group" profile="full-ha">
            <jvm name="default"/>
            <socket-binding-group ref="ha-sockets"/>
            <deployments>
                <deployment name="Security.war" runtime-name="Security.war"/>
            </deployments>
        </server-group>
    </server-groups>
</domain>

Multiple machine set-up

Detailed steps can be found in the AS7 Cluster Howto. To get the environment on a different machine we just tar the whole ${JBOSS_HOME} directory and copy the tar to the other machine

[jboss@axis-into-ict temp]$ tar -cvpf /home/jboss/temp/jboss.tar /home/jboss/jboss-as-7.1.0.Final/

[jboss@axis-into-ict temp]$ scp jboss@192.168.1.50:/home/jboss/temp/jboss.tar jboss@192.168.1.51:/home/jboss/temp/
The authenticity of host '192.168.1.50 (192.168.1.50)' can't be established.
RSA key fingerprint is 20:59:8b:b7:0e:24:05:b8:e1:70:28:e4:15:05:fa:95.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.50' (RSA) to the list of known hosts.
jboss@192.168.1.50's password:
The authenticity of host '192.168.1.51 (192.168.1.51)' can't be established.
RSA key fingerprint is 20:59:8b:b7:0e:24:05:b8:e1:70:28:e4:15:05:fa:95.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.51' (RSA) to the list of known hosts.
jboss@192.168.1.51's password:
jboss.tar                                                                                       100%  220MB   2.9MB/s   01:17
Connection to 192.168.1.50 closed.

On the other machine we unpack the tar

[jboss@middleware-magic ~]$ cd /
[jboss@middleware-magic /]$ tar xvf /home/jboss/temp/jboss.jar

Next, we set-up the host controller, i.e., edit the host.xml (${JBOSS_HOME}/domain/configuration) file

<host name="jboss" xmlns="urn:jboss:domain:1.1">
    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
				<server-identities>
                     <secret value="bWFnaWMxMWc="/>
                </server-identities>
                <authentication>
                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
                </authentication>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <authentication>
                    <properties path="application-users.properties" relative-to="jboss.domain.config.dir" />
                </authentication>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket interface="management" port="9999"/>
            </native-interface>
            <http-interface security-realm="ManagementRealm">
                <socket interface="management" port="9990"/>
            </http-interface>
        </management-interfaces>
    </management>
    <domain-controller>
       <remote host="192.168.1.50" port="9999" security-realm="ManagementRealm"/>
    </domain-controller>
    <interfaces>
        <interface name="management">
           <nic name="eth0"/>
        </interface>
        <interface name="public">
           <nic name="eth0"/>
        </interface>
        <interface name="unsecure">
	   <nic name="eth0"/>
        </interface>
    </interfaces>
    <jvms>
    	<jvm name="default">
            <heap size="512m" max-size="512m"/>
            <permgen size="256m" max-size="256m"/>
            <jvm-options>
                <option value="-server"/>
                <option value="-XX:NewRatio=2"/>
                <option value="-XX:+UseParallelGC"/>
                <option value="-XX:ParallelGCThreads=2"/>
                <option value="-XX:MaxGCPauseMillis=200"/>
                <option value="-XX:GCTimeRatio=19"/>
                <option value="-XX:+UseParallelOldGC"/>
				<option value="-Dtangosol.coherence.mode=prod"/>
				<option value="-Dtangosol.coherence.distributed.localstorage=false"/>
            </jvm-options>
        </jvm>
    </jvms>
    <servers>
        <server name="server3" group="ha-server-group" auto-start="false">
        </server>
        <server name="server4" group="ha-server-group" auto-start="false">
            <socket-bindings port-offset="1"/>
        </server>
    </servers>
</host>

The choice of the name of the host is very important, as a corresponding user name must be present in the mgmt-users.properties file on the host where the domain controller runs. Recall, that we did this by using the add-user.sh script (see the Fun with JBoss post in which we added a user called jboss). Another important configuration is the server-identities element, which contains the base 64 code for the password of the user jboss. A handy base64-decoder-encoder is provided here. With the configuration in place we can start the process and host controller on the other machine by using domain.sh. When everything went alright the following should show up in the log of the domain controller

[Host Controller] 19:42:13,338 INFO  [org.jboss.as.domain] (management-handler-threads - 1) JBAS010918: Registered remote slave host "jboss", JBoss AS 7.1.0.Final "Thunder"

Now, we ready to start the other configured servers (server3 and server4). Before we do this, we first start a Coherence cache server on the other machine as well

2012-03-10 20:14:46.944/289.405 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=4, Timestamp=2012-03-10 20:14:46.554, Address=192.168.1.51:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:13750, Role=CoherenceServer) joined Cluster with senior member 1
2012-03-10 20:14:47.091/289.552 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 4 joined Service Management with senior member 1
2012-03-10 20:14:47.795/290.256 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 4 joined Service DistributedCache with senior member 1
2012-03-10 20:14:47.855/290.316 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache, member=1): 1> Transferring vulnerable PartitionSet{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127} to member 4 requesting 128
2012-03-10 20:14:48.084/290.545 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache, member=1): 1> Transferring 129 out of 129 partitions to a machine-safe backup 1 at member 4 (under 129)
2012-03-10 20:14:48.127/290.588 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache, member=1): Transferring 0KB of backup[1] for PartitionSet{128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256} to member 4

Note that the data gets partitioned among the Coherence cache servers. To start the servers we can use the command-line interface

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.50:9999
[domain@192.168.1.50:9999 /] /host=jboss/server-config=server3:start
{
    "outcome" => "success",
    "result" => "STARTING"
}
[domain@192.168.1.50:9999 /] /host=jboss/server-config=server4:start
{
    "outcome" => "success",
    "result" => "STARTING"
}

When the servers are started the nodes join the Coherence cluster

2012-03-10 20:19:23.858/566.319 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=5, Timestamp=2012-03-10 20:19:23.349, Address=192.168.1.51:8090, MachineId=65081, Location=site:,machine:middleware-magic,process:14024, Role=JbossModulesMain) joined Cluster with senior member 1
2012-03-10 20:19:24.605/567.066 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 5 joined Service Management with senior member 1
2012-03-10 20:19:26.447/568.908 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 5 joined Service DistributedCache with senior member 1
2012-03-10 20:21:49.001/711.462 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=6, Timestamp=2012-03-10 20:21:48.617, Address=192.168.1.51:8092, MachineId=65081, Location=site:,machine:middleware-magic,process:14202, Role=JbossModulesMain) joined Cluster with senior member 1
2012-03-10 20:21:49.708/712.169 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 6 joined Service Management with senior member 1
2012-03-10 20:21:50.849/713.310 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 6 joined Service DistributedCache with senior member 1

By using an MBean browser we can observe the Coherence cache statistics

How beautiful can software get? “Beautiful programs work better, cost less, match user needs, have fewer bugs, run faster, are easier to fix, and have a longer life span. Beautiful software is as small as it can be, by using existing computing resources where possible. Beautiful software is simple. Beautiful software is achieved by creating a ‘wonderful whole’…”. Coherence is beautiful software!

References

[1] Documentation JBoss AS7.1.
[2] Documentation mod_cluster.

Application Server, Cluster, Coherence Leave a Comment more...

Feb/12

Fun with JBoss

by René van Wijk under Cluster, DataSource, Deploy, EJB, JDBC, JVM Tuning, MDB

René van Wijk

In this post we show a step-by-step example of how to set-up a high available tuned Java EE environment. We start with installing a Java Virtual Machine (in this case HotSpot) and JBoss Application Server. We continue with configuring a standalone server, tune the operating system and the Java Virtual Machine. Subsequently, we show how to set-up data source and JMS resources that are used by an application (consisting of a servlet, persistence, a stateless enterprise bean and a message-driven bean). Have some words about thread-pooling and enterprise beans. Show how to use the command-line interface to obtain run-time statistics. Show the steps involved in setting up a cluster by using standalone servers and configure a load balancer. Finally, we take a look at how to configure a domain in order to manage multiple server instances from a single control point.

JBoss AS7 is designed around a new kernel, which is now based on:

JBoss Modules – handle class loading of resources in the container. Sort of a thin bootstrap wrapper for executing an application in a modular environment.
Modular Service Container – provides a way to install, uninstall, and manage services used by a container. It further enables resources injection into services and dependency management between services.

Preparing the environment

Some operating system tweaks are worth considering when we do not want to run against system restrictions:

Packet loss minimization – The operating system buffers must be large enough to handle incoming network traffic while the application is paused during garbage collection. Usually UDP (User Datagram Protocol) is used in order to transmit multicast messages to server instances in a cluster; to limit the need to retransmit UDP messages the size of the operating system buffers must be set appropriately to avoid excessive UDP datagram loss.
Maximum number of open file descriptors – Most operating systems handle sockets as a form of file access and use file descriptors to keep track of which sockets are open. To contain the resources per process, the operating system restricts the number of file descriptors per process. Linux limits the number of open file descriptors per process, by default this is equal to 1024. It could be that the 1024 limit does not offer optimal performance.
TCP/IP – On some systems the default value for the time wait interval is too high and needs to be adjusted. When the number approaches the maximum number of file descriptors per process, the application’s throughput will degrade, i.e., new connections have to wait for a free space in the application’s file descriptor table.
Timesources – Linux has several timesources to choose from; the fastest being TSC (time stamp counter) and is used by default. However, if during start-up inconsistencies are found Linux switches to a slower timesource. This can have a negative performance impact.
Network interface card (NIC) – Configure the network card at it’s maximum link speed and at full duplex.

In the post Tuning GlassFish for Performance all the tweaks are explained in detail and the steps involved in adjusting the operating system are explained as well. The post also contains suggestions in tuning the application server and Java Virtual Machine which are applicable to the JBoss application server as well.

Installation

Install a JDK (of which a distribution can be downloaded here).
Install JBoss (of which a distribution can be downloaded here).

We will use the versions: jdk-6u31-linux-x64.bin for the JDK and jboss-as-7.1.0.Final.zip for JBoss. The installation steps are straight forward, for the JDK we can just run the bin file. JBoss can be installed by unzipping the file. When we are done we have the following directory structure:

/home/jboss
	/jboss-as-7.1.0.Final (${JBOSS_HOME)
	/jdk1.6.0_31 (${JAVA_HOME})

After the set-up, we perform a startup test to validate that there are no major problems with the Java VM/operating system combination. To test the set-up navigate to ${JBOSS_HOME}/bin and run standalone.sh. Before running standalone.sh add the following lines:

#!/bin/sh

# These lines must be added
export JAVA_HOME=/home/jboss/jdk1.6.0_31
export PATH=${PATH}:${JAVA_HOME}/bin

To set JVM parameters we can edit standalone.conf, for example,

#
# Specify options to pass to the Java VM.
#
if [ "x$JAVA_OPTS" = "x" ]; then
   JAVA_OPTS="-server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
   JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
   JAVA_OPTS="$JAVA_OPTS -Djboss.server.default.config=standalone.xml"
else
   echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi

More information on tuning the JVM can be found here. Note that the default configuration is standalone.xml. This configuration does not by default include all subsystems, for example, messaging is not included. The configuration file standalone-full.xml has all the subsystems included. In the examples below we will work with standalone.xml and add specific subsystems when needed, hereby keeping the memory footprint of the application server to what is needed. When the standalone.sh is run the following is observed:

=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/jboss-as-7.1.0.Final

  JAVA: /home/jboss/jdk1.6.0_31/bin/java

  JAVA_OPTS:  -XX:+UseCompressedOops -XX:+TieredCompilation -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

=========================================================================

12:32:01,842 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
12:32:02,010 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
12:32:02,057 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
12:32:03,047 INFO  [org.xnio] XNIO Version 3.0.3.GA
12:32:03,063 INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
12:32:03,078 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
12:32:03,088 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.2.GA
12:32:03,145 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
12:32:03,151 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 26) JBAS016200: Activating ConfigAdmin Subsystem
12:32:03,212 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 44) JBAS013101: Activating Security Subsystem
12:32:03,216 INFO  [org.jboss.as.security] (MSC service thread 1-3) JBAS013100: Current PicketBox version=4.0.6.final
12:32:03,231 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
12:32:03,237 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 39) JBAS011940: Activating OSGi Subsystem
12:32:03,282 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 48) JBAS015537: Activating WebServices Extension
12:32:03,347 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 38) JBAS011800: Activating Naming Subsystem
12:32:03,399 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
12:32:03,406 INFO  [org.jboss.as.connector] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
12:32:03,478 INFO  [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: Starting Naming Service
12:32:03,605 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) JBAS015400: Bound mail session [java:jboss/mail/Default]
12:32:03,633 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-2) JBoss Web Services - Stack CXF Server 4.0.1.GA
12:32:03,716 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-4) Starting Coyote HTTP/1.1 on http--127.0.0.1-8080
12:32:03,976 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on /127.0.0.1:9999
12:32:03,984 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) JBAS015012: Started FileSystemDeploymentService for directory /home/jboss/jboss-as-7.1.0.Final/standalone/deployments
12:32:03,986 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on /127.0.0.1:4447
12:32:04,119 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
12:32:04,136 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 2726ms - Started 134 of 205 services (70 services are passive or on-demand)

To verify that the server is reachable, enter the following URL: http://localhost:8080.

To use the command line interface, we can start jboss-cli.sh (also add the JAVA_HOME and PATH variables to the script). To stop the started server, we can use

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect
[standalone@localhost:9999 /] :shutdown
{"outcome" => "success"}

We can also shuwdown the server by using:

[jboss@axis-into-ict bin]$ ./jboss-cli.sh --connect command=:shutdown
{"outcome" => "success"}

When browsing through the application server folders we see that its file system is divided into two parts: standalone servers (${JBOSS_HOME}/standalone) and domain servers (${JBOSS_HOME}/domain). An application server node which is not configured as part of a domain is qualified as a standalone server.

The ${JBOSS_HOME}/bin folder is where we start application server instances. As we already saw a standalone server can be started by using standalone.sh. When using a domain we will use domain.sh, which starts the application servers specified in the domain configuration file as we will see later on. The ${JBOSS_HOME}/modules folder contains the application server’s set of libraries, which are part of the server distribution. Each module is a pluggable unit. By using a static file system approach we can load modules, all we need to do is provide the location of the modules. When looking at the standalone.sh script (line 177 and on) we see an example of how to start the server and loading modules, for example, we could use

[jboss@axis-into-ict ~]$ cd jdk1.6.0_31/bin/
[jboss@axis-into-ict bin]$ ./java -jar /home/jboss/jboss-as-7.1.0.Final/jboss-modules.jar -mp /home/jboss/jboss-as-7.1.0.Final/modules/ org.jboss.as.standalone -Djboss.home.dir=/home/jboss/jboss-as-7.1.0.Final/

The module path (-mp) argument points to the root directory which will be searched by the module loader for module definitions. A module is defined by using an XML descriptor, for example,

<module xmlns="urn:jboss:module:1.1" name="javax.jms.api">
    <dependencies>
        <module name="javax.transaction.api" export="true"/>
    </dependencies>
    <resources>
        <resource-root path="jboss-jms-api_1.1_spec-1.0.0.Final.jar"/>
        <!-- Insert resources here -->
    </resources>
</module>

A module definition contains two main elements: the module dependencies and the resources defined in the module.

When we want to use the admin console, we first have to add a user. To this end run add-user.sh (${JBOSS_HOME}/bin) and follow the instructions given on the screen, for example,

[jboss@axis-into-ict bin]$ ./add-user.sh 

What type of user do you wish to add?
 a) Management User (mgmt-users.properties)
 b) Application User (application-users.properties)
(a): 

Enter the details of the new user to add.
Realm (ManagementRealm) :
Username : jboss
Password :
Re-enter Password :
About to add user 'jboss' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'jboss' to file '/home/jboss/jboss-as-7.1.0.Final/standalone/configuration/mgmt-users.properties'
Added user 'jboss' to file '/home/jboss/jboss-as-7.1.0.Final/domain/configuration/mgmt-users.properties'

Enter the following URL to access the admin console: http://hostname:9990/console and enter the log in credentials.

Configuration

The structure of the application server is maintained into a single file, which acts as a main reference for all server configurations. The default configuration files are named standalone.xml for standalone servers and domain.xml for an application server domain. An application server domain can be seen as a specialized server configuration, which also includes the domain and host controller set-up, which we will see later on. For now we will focus on the standalone configuration. The standalone.xml file is located in ${JBOSS_HOME}/standalone/configuration. An example looks as follows:

<server xmlns="urn:jboss:domain:1.1">
    <extensions>
        <extension module="org.jboss.as.clustering.infinispan"/>
        <extension module="org.jboss.as.configadmin"/>
		...
    </extensions>
    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <authentication>
                    <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <authentication>
                    <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket-binding native="management-native"/>
            </native-interface>
            <http-interface security-realm="ManagementRealm">
                <socket-binding http="management-http"/>
            </http-interface>
        </management-interfaces>
    </management>
    <profile>
        <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:configadmin:1.0"/>
        <subsystem xmlns="urn:jboss:domain:datasources:1.0">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:ee:1.0"/>
        <subsystem xmlns="urn:jboss:domain:ejb3:1.2">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:infinispan:1.1" default-cache-container="hibernate">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jca:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jmx:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:jpa:1.0">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:mail:1.0">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:naming:1.1"/>
        <subsystem xmlns="urn:jboss:domain:osgi:1.2" activation="lazy">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
        <subsystem xmlns="urn:jboss:domain:remoting:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:resource-adapters:1.0"/>
        <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
        <subsystem xmlns="urn:jboss:domain:security:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:threads:1.1"/>
        <subsystem xmlns="urn:jboss:domain:transactions:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:webservices:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
    </profile>
    <interfaces>
        <interface name="management">
            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
        </interface>
        <interface name="public">
            <inet-address value="${jboss.bind.address:127.0.0.1}"/>
        </interface>
    </interfaces>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="http" port="8080"/>
        <socket-binding name="https" port="8443"/>
        <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
        <socket-binding name="osgi-http" interface="management" port="8090"/>
        <socket-binding name="remoting" port="4447"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
    </socket-binding-group>
</server>

The application server contains a list of basic modules, called extensions, which are shared by all services. Extensions can be seen as a special type of module, which are used to extend the functionalities of the application server. They are stored in the ${JBOSS_HOME}/modules folder. Each extension is in turn picked up by the application server classloader at boot time, before any deployment. The application server has administration and management features that include a command-line interface (listens on port 9999) and an administration console (listens on port 9990). The profile can be seen as a collection of subsystems. Each subsystem contains a subset of functionalities used by the application server. For example, the web subsystem contains the definition of a set of connectors used by the container, the messaging subsystem defines the JMS configuration, and so on. The interfaces section contains the network interfaces and host names or IP addresses to which the application server is bound. Note that by default the server is bound to 127.0.0.1, which means it can only be reached locally. The IP address or host name can be changed at will, for example,

<interfaces>
	<interface name="management">
		<inet-address value="${jboss.bind.address.management:192.168.1.66}"/>
	</interface>
	<interface name="public">
		<inet-address value="${jboss.bind.address:192.168.1.66}"/>
	</interface>
</interfaces>

or even better gather the information from the network interface card, for example,

<interfaces>
	<interface name="management">
		<!--inet-address value="${jboss.bind.address.management:127.0.0.1}"/-->
		<nic name="eth0"/>
	</interface>
	<interface name="public">
		<!--inet-address value="${jboss.bind.address:127.0.0.1}"/-->
		<nic name="eth0"/>
	</interface>
</interfaces>

When we would now start the server the following is observed:

[jboss@axis-into-ict bin]$ ./standalone.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/jboss-as-7.1.0.Final

  JAVA: /home/jboss/jdk1.6.0_31/bin/java

  JAVA_OPTS:  -XX:+UseCompressedOops -XX:+TieredCompilation -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

=========================================================================

16:29:17,748 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
16:29:17,935 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
16:29:17,983 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
16:29:18,991 INFO  [org.xnio] XNIO Version 3.0.3.GA
16:29:18,998 INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
16:29:19,023 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
16:29:19,043 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.2.GA
16:29:19,070 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
16:29:19,086 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 26) JBAS016200: Activating ConfigAdmin Subsystem
16:29:19,167 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
16:29:19,210 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 48) JBAS015537: Activating WebServices Extension
16:29:19,214 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 44) JBAS013101: Activating Security Subsystem
16:29:19,224 INFO  [org.jboss.as.security] (MSC service thread 1-1) JBAS013100: Current PicketBox version=4.0.6.final
16:29:19,226 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 39) JBAS011940: Activating OSGi Subsystem
16:29:19,228 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 38) JBAS011800: Activating Naming Subsystem
16:29:19,290 INFO  [org.jboss.as.connector] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
16:29:19,351 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
16:29:19,367 INFO  [org.jboss.as.naming] (MSC service thread 1-2) JBAS011802: Starting Naming Service
16:29:19,371 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) JBAS015400: Bound mail session [java:jboss/mail/Default]
16:29:19,586 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-4) JBoss Web Services - Stack CXF Server 4.0.1.GA
16:29:19,694 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-3) Starting Coyote HTTP/1.1 on http--192.168.1.66-8080
16:29:19,915 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on /192.168.1.66:4447
16:29:19,916 INFO  [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100: Listening on /192.168.1.66:9999
16:29:19,949 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-3) JBAS015012: Started FileSystemDeploymentService for directory /home/jboss/jboss-as-7.1.0.Final/standalone/deployments
16:29:20,101 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
16:29:20,115 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 2615ms - Started 134 of 205 services (70 services are passive or on-demand)

The server is listening on another IP address than the 127.0.0.1, which is of course more useful. When using the command-line interface, we have to make sure we connect to the right host, for example,

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect
The controller is not available at localhost:9999
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /]

The command-line interface is a management tool for a domain or standalone server. It allows a user to connect to the domain controller or standalone server and execute management operations available through the de-typed management model. The native-management API is the entry point for management clients that rely on the application server’s native protocol to integrate with the management layer. It uses an open binary protocol and RPC style API based on a small number of Java types to describe and execute management operations against a domain or standalone server.

To get back at the configuration file. A socket binding makes up a named configuration of a socket. Here, we can configure the network ports, such as native management (command-line interface), which by default runs on port 9999.

Thread pools

Thread pools improve performance, and provide a means of resource management when a large number of tasks are executed asynchronously. By configuring the thread pool we can tune specific areas. The application server thread pool configuration can include:

A thread factory creates new threads when needed. The thread factory is not included by default in the server configuration as it relies on defaults, which in general do not need to be modified.
A bounded thread pool (or blocking bounded thread pool) is the most common kind of pool used by the application server, as it prevents resource exhaustion by defining a constraint on the thread pool size. This thread pool is also the most complex, i.e., it has to maintain a fixed-length queue and two pool sizes (core size and maximum size). When a new task is submitted, a new thread is created when the number of running threads is less than the core size. Otherwise, if there is room in the queue, the task is queued. When none of these options apply, the executor can create a new thread when the maximum size is not yet reached, otherwise the task can either be blocked (in the case of a blocking bounded thread pool) until there is room in the queue or it can be passed to a so-called hand-off executor (when one is specified, otherwise the task will be rejected).
A unbounded thread pool has a core size and a queue with no upper bound. When a task is submitted and the number of running threads is less than the core size, a new thread is created. Otherwise, the task is placed in a queue. If too many tasks are submitted, an out of memory error can occur.
A queueless thread pool (or blocking queueless thread pool) is a thread pool with no queue and has the same logic as the bounded thread pool (without the queue storage option).
A scheduled thread pool is used for activities on the server-side that require running periodically or with delays.

An example configuration looks as follows

<subsystem xmlns="urn:jboss:domain:threads:1.1">
	<blocking-bounded-queue-thread-pool name="http-thread-pool">
		<queue-length count="500" per-cpu="200"/>
		<core-threads count="5" per-cpu="2"/>
		<max-threads count="10" per-cpu="3"/>
		<keepalive-time time="30" unit="seconds"/>
	</blocking-bounded-queue-thread-pool>
</subsystem>

In the configuration above we did not define a thread-factory. By not defining a thread-factory an appropriate default thread factory will be used. To view the configuration we can use the following (note that by using read-resource-description we get a detailed description of all the elements the can be configured)

[standalone@192.168.1.66:9999 /] /subsystem=threads/blocking-bounded-queue-thread-pool=http-thread-pool:read-resource-description
{
    "outcome" => "success",
    "result" => {
        "description" => "A thread pool executor with a bounded queue where threads submittings tasks may block. Such a thread pool has a core and maximum size and a specified queue length.  When a task is submitted, if the number of running threads is less than the core size, a new thread is created.  Otherwise, if there is room in the queue, the task is enqueued. Otherwise, if the number of running threads is less than the maximum size, a new thread is created. Otherwise, the caller blocks until room becomes available in the queue.",
        "attributes" => {
            "core-threads" => {
                "type" => INT,
                "description" => "The core thread pool size which is smaller than the maximum pool size. If undefined, the core thread pool size is the same as the maximum thread pool size.",
                "expressions-allowed" => true,
                "nillable" => true,
                "min" => 0L,
                "max" => 2147483647L,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "no-services"
            },
            "current-thread-count" => {
                "type" => INT,
                "description" => "The current number of threads in the pool.",
                "expressions-allowed" => false,
                "nillable" => false,
                "access-type" => "metric",
                "storage" => "runtime"
            },
            "largest-thread-count" => {
                "type" => INT,
                "description" => "The largest number of threads that have ever simultaneously been in the pool.",
                "expressions-allowed" => false,
                "nillable" => false,
                "access-type" => "metric",
                "storage" => "runtime"
            },
            "keepalive-time" => {
                "type" => OBJECT,
                "description" => "Used to specify the amount of time that pool threads should be kept running when idle; if not specified, threads will run until the executor is shut down.",
                "expressions-allowed" => false,
                "nillable" => true,
                "value-type" => {
                    "time" => {
                        "type" => LONG,
                        "required" => true,
                        "description" => "The time"
                    },
                    "unit" => {
                        "type" => STRING,
                        "required" => true,
                        "description" => "The time unit"
                    }
                },
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "no-services"
            },
            "thread-factory" => {
                "type" => STRING,
                "description" => "Specifies the name of a specific thread factory to use to create worker threads. If not defined an appropriate default thread factory will be used.",
                "expressions-allowed" => false,
                "nillable" => true,
                "min-length" => 1L,
                "max-length" => 2147483647L,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "all-services"
            },
            "queue-length" => {
                "type" => INT,
                "description" => "The queue length.",
                "expressions-allowed" => true,
                "nillable" => false,
                "min" => 0L,
                "max" => 2147483647L,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "all-services"
            },
            "max-threads" => {
                "type" => INT,
                "description" => "The maximum thread pool size.",
                "expressions-allowed" => true,
                "nillable" => false,
                "min" => 0L,
                "max" => 2147483647L,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "no-services"
            },
            "name" => {
                "type" => STRING,
                "description" => "The name of the thread pool.",
                "expressions-allowed" => false,
                "nillable" => true,
                "min-length" => 1L,
                "max-length" => 2147483647L,
                "access-type" => "read-only",
                "storage" => "configuration"
            },
            "rejected-count" => {
                "type" => INT,
                "description" => "The number of tasks that have been passed to the handoff-executor (if one is specified) or discarded.",
                "expressions-allowed" => false,
                "nillable" => false,
                "access-type" => "metric",
                "storage" => "runtime"
            },
            "allow-core-timeout" => {
                "type" => BOOLEAN,
                "description" => "Whether core threads may time out.",
                "expressions-allowed" => false,
                "nillable" => true,
                "default" => false,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "no-services"
            }
        },
        "operations" => undefined,
        "children" => {}
    }
}
[standalone@192.168.1.66:9999 /] /subsystem=threads/blocking-bounded-queue-thread-pool=http-thread-pool:read-resource
{
    "outcome" => "success",
    "result" => {
        "allow-core-timeout" => false,
        "core-threads" => 50,
        "keepalive-time" => {
            "time" => 10L,
            "unit" => "SECONDS"
        },
        "max-threads" => 50,
        "name" => "http-thread-pool",
        "queue-length" => 50,
        "thread-factory" => undefined
    }
}

Data source

A note up front is in order. When installing and configuring an Oracle database on, for example, a Windows machine, it usually is bound to localhost. To change this, we have to edit listener.ora and tnsnames.ora (both are located in ${ORACLE_HOME}/product/11.2.0/dbhome_1/NETWORK/ADMIN). For example,

# listener.ora Network Configuration File: c:\oracle\product\11.2.0\dbhome_1\network\admin\listener.ora
# Generated by Oracle configuration tools.

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = CLRExtProc)
      (ORACLE_HOME = c:\oracle\product\11.2.0\dbhome_1)
      (PROGRAM = extproc)
      (ENVS = "EXTPROC_DLLS=ONLY:c:\oracle\product\11.2.0\dbhome_1\bin\oraclr11.dll")
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.65)(PORT = 1521))
    )
  )

ADR_BASE_LISTENER = c:\oracle

# tnsnames.ora Network Configuration File: c:\oracle\product\11.2.0\dbhome_1\network\admin\tnsnames.ora
# Generated by Oracle configuration tools.

ORACLR_CONNECTION_DATA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
    (CONNECT_DATA =
      (SID = CLRExtProc)
      (PRESENTATION = RO)
    )
  )

ORCL11 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.65)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl11)
    )
  )

LISTENER_ORCL11 =
  (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.65)(PORT = 1521))

The procedure for installing a new module requires copying the .jar libraries in the appropriate modules path and adding a module.xml file, which declares the module and its dependencies. For the module path we are going to use ${JBOSS_HOME}/modules/${MODULE}/main, in which ${MODULE} will be com/oracle/database. We have the following directory structure

${JBOSS_HOME}/modules
	/com/oracle/database
		/main
			module.xml
			ojdbc6.jar

The contents of module.xml are the following

<module xmlns="urn:jboss:module:1.1" name="com.oracle.database">
    <resources>
        <resource-root path="ojdbc6.jar"/>
    </resources>
    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>

To configure a data source, find the entry subsystem xmlns="urn:jboss:domain:datasources:1.0" in standalone.xml, and add the following:

<subsystem xmlns="urn:jboss:domain:datasources:1.0">
	<datasources>
		<datasource jndi-name="java:/jdbc/OracleDS" pool-name="OracleDS" enabled="true" jta="true" use-java-context="true" use-ccm="true">
			<connection-url>jdbc:oracle:thin:@192.168.1.65:1521:orcl11</connection-url>
			<driver>oracle</driver>
			<pool>
				<min-pool-size>1</min-pool-size>
				<max-pool-size>15</max-pool-size>
				<prefill>true</prefill>
				<use-strict-min>true</use-strict-min>
		    </pool>
			<security>
				<user-name>example</user-name>
				<password>example</password>
			</security>
		    <statement>
				<prepared-statement-cache-size>10</prepared-statement-cache-size>
		    </statement>
			<timeout>
				<idle-timeout-minutes>0</idle-timeout-minutes>
				<query-timeout>600</query-timeout>
		    </timeout>
		</datasource>
		<drivers>
			<driver name="oracle" module="com.oracle.database">
				<driver-class>oracle.jdbc.OracleDriver</driver-class>
				<xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
		    </driver>
		</drivers>
	</datasources>
</subsystem>

When the server is started, we can check the data sources present by using the command-line interface

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] /subsystem=datasources:installed-drivers-list
{
    "outcome" => "success",
    "result" => [
        {
            "driver-name" => "h2",
            "deployment-name" => undefined,
            "driver-module-name" => "com.h2database.h2",
            "module-slot" => "main",
            "driver-datasource-class-name" => "",
            "driver-xa-datasource-class-name" => "org.h2.jdbcx.JdbcDataSource",
            "driver-class-name" => "org.h2.Driver",
            "driver-major-version" => 1,
            "driver-minor-version" => 3,
            "jdbc-compliant" => true
        },
        {
            "driver-name" => "oracle",
            "deployment-name" => undefined,
            "driver-module-name" => "com.oracle.database",
            "module-slot" => "main",
            "driver-datasource-class-name" => "",
            "driver-xa-datasource-class-name" => "oracle.jdbc.xa.client.OracleXADataSource",
            "driver-class-name" => "oracle.jdbc.OracleDriver",
            "driver-major-version" => 11,
            "driver-minor-version" => 2,
            "jdbc-compliant" => true
        }
    ]
}

To test a connection

[standalone@192.168.1.66:9999 /] /subsystem=datasources/data-source=OracleDS:test-connection-in-pool
{
    "outcome" => "success",
    "result" => [true]
}

To obtain the configuration of the data source

[standalone@192.168.1.66:9999 /] /subsystem=datasources/data-source=OracleDS:read-resource
{
    "outcome" => "success",
    "result" => {
        "allocation-retry" => undefined,
        "allocation-retry-wait-millis" => undefined,
        "background-validation" => undefined,
        "background-validation-millis" => undefined,
        "blocking-timeout-wait-millis" => undefined,
        "check-valid-connection-sql" => undefined,
        "connection-properties" => undefined,
        "connection-url" => "jdbc:oracle:thin:@192.168.1.65:1521:orcl11",
        "datasource-class" => undefined,
        "driver-class" => undefined,
        "driver-name" => "oracle",
        "enabled" => true,
        "exception-sorter-class-name" => undefined,
        "exception-sorter-properties" => undefined,
        "flush-strategy" => undefined,
        "idle-timeout-minutes" => 0L,
        "jndi-name" => "java:/jdbc/OracleDS",
        "jta" => true,
        "max-pool-size" => 15,
        "min-pool-size" => 1,
        "new-connection-sql" => undefined,
        "password" => "example",
        "pool-prefill" => true,
        "pool-use-strict-min" => true,
        "prepared-statements-cache-size" => 10L,
        "query-timeout" => 600L,
        "reauth-plugin-class-name" => undefined,
        "reauth-plugin-properties" => undefined,
        "security-domain" => undefined,
        "set-tx-query-timeout" => "false",
        "share-prepared-statements" => "false",
        "spy" => "false",
        "stale-connection-checker-class-name" => undefined,
        "stale-connection-checker-properties" => undefined,
        "track-statements" => "\"NOWARN\"",
        "transaction-isolation" => undefined,
        "url-delimiter" => undefined,
        "url-selector-strategy-class-name" => undefined,
        "use-ccm" => true,
        "use-fast-fail" => "false",
        "use-java-context" => true,
        "use-try-lock" => undefined,
        "user-name" => "example",
        "valid-connection-checker-class-name" => undefined,
        "valid-connection-checker-properties" => undefined,
        "validate-on-match" => "false",
        "statistics" => {
            "pool" => undefined,
            "jdbc" => undefined
        }
    }
}

JMS

The JMS server configuration is done through the messaging subsystem. Which is not enabled when the default standalone.xml configuration is used. To enable the messaging subsystem we add the following to the standalone.xml configuration file (note that this can be copied from standalone-full.xml)

<server xmlns="urn:jboss:domain:1.1">
    <extensions>
		...
        <extension module="org.jboss.as.messaging"/>
		...
    </extensions>
    <management>...</management>
    <profile>
		...
        <subsystem xmlns="urn:jboss:domain:messaging:1.1">
            <hornetq-server>
                <persistence-enabled>true</persistence-enabled>
                <journal-file-size>102400</journal-file-size>
                <journal-min-files>2</journal-min-files>
                <connectors>
                    <netty-connector name="netty" socket-binding="messaging"/>
                    <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                    </netty-connector>
                    <in-vm-connector name="in-vm" server-id="0"/>
                </connectors>
                <acceptors>
                    <netty-acceptor name="netty" socket-binding="messaging"/>
                    <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                        <param key="direct-deliver" value="false"/>
                    </netty-acceptor>
                    <in-vm-acceptor name="in-vm" server-id="0"/>
                </acceptors>
                <security-settings>
                    <security-setting match="#">
                        <permission type="send" roles="guest"/>
                        <permission type="consume" roles="guest"/>
                        <permission type="createNonDurableQueue" roles="guest"/>
                        <permission type="deleteNonDurableQueue" roles="guest"/>
                    </security-setting>
                </security-settings>
                <address-settings>
                    <address-setting match="#">
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <redelivery-delay>0</redelivery-delay>
                        <max-size-bytes>10485760</max-size-bytes>
                        <address-full-policy>BLOCK</address-full-policy>
                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                    </address-setting>
                </address-settings>
                <jms-connection-factories>
                    <connection-factory name="InVmConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/ConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <connection-factory name="RemoteConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="netty"/>
                        </connectors>
                        <entries>
                            <entry name="RemoteConnectionFactory"/>
                            <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <pooled-connection-factory name="hornetq-ra">
                        <transaction mode="xa"/>
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/JmsXA"/>
                        </entries>
                    </pooled-connection-factory>
                </jms-connection-factories>
                <jms-destinations>
                    <jms-queue name="testQueue">
                        <entry name="java:/queue/test"/>
                        <entry name="java:jboss/exported/jms/queue/test"/>
                    </jms-queue>
                    <jms-topic name="testTopic">
                        <entry name="java:/topic/test"/>
                        <entry name="java:jboss/exported/jms/topic/test"/>
                    </jms-topic>
                </jms-destinations>
            </hornetq-server>
        </subsystem>
    </profile>
    <interfaces>...</interfaces>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
	    ...
        <socket-binding name="messaging" port="5445"/>
        <socket-binding name="messaging-throughput" port="5455"/>
		...
    </socket-binding-group>
</server>

The transport of JMS messages is a key part of messaging system tuning. HornetQ uses Netty (a NIO client server framework) as its network library. The HornetQ transports configuration consist of acceptors and connectors. An acceptor defines which type of connection is accepted by the HornetQ server. A connector defines how to connect to a HornetQ server. HornetQ defines two the following types:

In-VM (Intra Virtual Machine) – can be used when both HornetQ client and server run in the same virtual machine.
netty – used when HornetQ client and server run in different virtual machines.

The JMS connection factories can be split into two kinds: In-VM connections and connections factories that can be used by remote clients. Each connection factory does reference a connector declaration, that is associated with a socket binding. The connection factory entry declaration specifies the JNDI name under which the factory will be exposed. Queues and topics are sub resources of the messaging subsystem. Each entry refers to a JNDI name of the queue or topic.

To check the configuration when the server is restarted, we can use

[standalone@192.168.1.66:9999 /] /subsystem=messaging/hornetq-server=default/connection-factory=InVmConnectionFactory:read-resource
{
    "outcome" => "success",
    "result" => {
        "auto-group" => false,
        "block-on-acknowledge" => false,
        "block-on-durable-send" => true,
        "block-on-non-durable-send" => false,
        "cache-large-message-client" => false,
        "call-timeout" => 30000L,
        "client-failure-check-period" => 30000L,
        "client-id" => undefined,
        "compress-large-messages" => false,
        "confirmation-window-size" => -1,
        "connection-load-balancing-policy-class-name" => "org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy",
        "connection-ttl" => 60000L,
        "connector" => {"in-vm" => undefined},
        "consumer-max-rate" => -1,
        "consumer-window-size" => 1048576,
        "discovery-group-name" => undefined,
        "discovery-initial-wait-timeout" => undefined,
        "dups-ok-batch-size" => 1048576,
        "entries" => ["java:/ConnectionFactory"],
        "failover-on-initial-connection" => false,
        "failover-on-server-shutdown" => undefined,
        "group-id" => undefined,
        "ha" => false,
        "max-retry-interval" => 2000L,
        "min-large-message-size" => 102400,
        "pre-acknowledge" => false,
        "producer-max-rate" => -1,
        "producer-window-size" => 65536,
        "reconnect-attempts" => 0,
        "retry-interval" => 2000L,
        "retry-interval-multiplier" => 1.0,
        "scheduled-thread-pool-max-size" => 5,
        "thread-pool-max-size" => -1,
        "transaction-batch-size" => 1048576,
        "use-global-pools" => true
    }
}

[standalone@192.168.1.66:9999 /] /subsystem=messaging/hornetq-server=default/jms-queue=testQueue:read-resource
{
    "outcome" => "success",
    "result" => {
        "durable" => true,
        "entries" => [
            "java:/queue/test",
            "java:jboss/exported/jms/queue/test"
        ],
        "selector" => undefined
    }
}

Enterprise beans

By default, the following configuration is present

<subsystem xmlns="urn:jboss:domain:ejb3:1.2">
	<session-bean>
		<stateless>
			<bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
		</stateless>
		<stateful default-access-timeout="5000" cache-ref="simple"/>
		<singleton default-access-timeout="5000"/>
	</session-bean>
	<mdb>
		<resource-adapter-ref resource-adapter-name="hornetq-ra"/>
		<bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
	</mdb>
	<pools>
		<bean-instance-pools>
			<strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
			<strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
		</bean-instance-pools>
	</pools>
	<caches>
		<cache name="simple" aliases="NoPassivationCache"/>
		<cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
	</caches>
	<passivation-stores>
		<file-passivation-store name="file"/>
	</passivation-stores>
	<async thread-pool-name="default"/>
	<timer-service thread-pool-name="default">
		<data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
	</timer-service>
	<remote connector-ref="remoting-connector" thread-pool-name="default"/>
	<thread-pools>
		<thread-pool name="default">
			<max-threads count="10"/>
			<keepalive-time time="100" unit="milliseconds"/>
		</thread-pool>
	</thread-pools>
</subsystem>

One of the key responsibilities of the EJB container is the management of EJB component lifecycles. Bean instances are pooled and reused by the container to reduce the number of object instantiations. For example the lifecycle of a stateless session bean is as follows:

The client obtains a reference to the stateless session bean. There are three options here. First, the reference can be injected into the client via an @EJB annotation. Second, the reference can be bound into the client’s local java:comp/env environment, using the @EJB annotation at class-level or a deployment descriptor entry, and the client can retrieve the reference with a JNDI lookup. Third, the client can look up the stateless session bean in the server’s global JNDI tree.
The client invokes a business method on the bean reference.
The container can reuse an existing instance of the stateless session bean from a pool, if available, or can instantiate a new bean instance. If a bean instance is newly created, the container will first perform dependency injection. If the stateless session bean implements the optional javax.ejb.SessionBean interface, the container will call setSessionContext. The session context will also be injected to any SessionContext field marked with the @Resource annotation. After completing injection of the session context and other dependencies, the container will call any methods marked with the @PostConstruct annotation.
The container starts a transaction, if appropriate. This is controlled by the @javax.ejb.TransactionManagement and @javax.ejb.TransactionAttribute annotations.
The container invokes the called business method on the bean instance, and the bean performs the desired operation.
The container commits the transaction, if appropriate.
The results of the business method call are returned to the client.
The client may invoke additional business methods on the bean reference, each of which may end up invoking methods on a different bean instance.
At some point, if the container decides to reduce the size of the bean instance pool, the container invokes any @PreDestroy methods on the bean instance, or ejbRemove if the bean implements the SessionBean interface. It is important to understand that this decision is not related to any client action.

There is a stronger link between the lifecycle of a stateful session bean and the client’s use of its reference:

The client obtains a reference to the stateful session bean.
The container instantiates a new bean instance. The container will perform dependency injection. If the stateful session bean implements the optional javax.ejb.SessionBean interface, the container will call setSessionContext. The session context will also be injected to any SessionContext field marked with the @Resource annotation. Any methods marked with the @PostConstruct will then be called.
The client invokes a business method on the bean reference.
The container starts a transaction, if appropriate.
The container invokes a business method on the bean instance, and the bean performs the desired operation.
The container commits the transaction, if appropriate.
The results of the business method call are returned to the client.
The client may invoke additional business methods on the bean reference and is assured that these additional calls will go to the same instance of the bean.
The client calls a method annotated with @javax.ejb.Remove when it is done with the stateful session bean.
The container invokes any @PreDestroy methods on the bean instance, or ejbRemove if the bean implements the SessionBean interface.

This introduction to the lifecycle of session EJB components represents a simplified view of the process. Additional complexities are introduced by limitations in the pool and cache sizes that we need to understand to configure the application properly.

EJB Pooling – JBoss maintains a pool of stateless session EJB instances for each stateless session bean deployment. This pool improves performance, because a client request can be handled immediately by any free initialized EJB instance. By default, the pool starts empty, and grows on demand. This can be controlled with the max-pool-size parameter. Within the pool configuration, there is also set a instance-acquisition-timeout of 5 minutes, which will come into play if the number of requests are larger than the maximum number of beans in the pool. If we set max-pool-size, we are choosing deliberately to throttle the application. If there are no idle bean instances, an thread making a new request will block until a bean becomes available or the transaction times out (so it must not be set too low). Message-driven beans are pooled in a manner very similar to stateless session beans. The number of message-driven bean instances can be controlled by using the max-pool-size parameter.

Stateful session bean instances are created as they are needed to service client requests. Between requests these instances reside in a bean-specific cache in the active state, ready for the next request. The size of the cache is limited by the max-size element (which is available for a passivation store). So long as the application never requires more than max-size instances of the stateful session bean at any given time to service all concurrent clients, there is no contention for the cache and performance is optimal. If we limit the number of beans in the cache, the managing of the cache is fairly active:

If the cache is full, bean instances that are not being used at that moment for client requests are subject to passivation.
If bean-managed transaction demarcation is used, a transaction may not be committed or rolled back at the end of a business method call. This leaves the bean instance associated with the transaction, pinned in the cache, and not eligible for passivation. Applications that keep transactions open between stateful session bean calls do not scale well and are difficult to manage.
Passivation logic is controlled by the cache type; least recently used (LRU), passivates based the max-size and the time when the bean has not been used; not recently used (NRU), passivates beans only when the number of active beans approaches the max-size setting. The NRU strategy is lazy; the LRU strategy is eager. Although the LRU setting can be a convenient way of enforcing idle timeouts on the resources the objects encapsulate, it requires the cache to keep track of the bean’s access time and maintain an ordered list that gets updated after each bean access. Unless we have a good reason to need idle timeouts strictly enforced, most applications should probably use the NRU algorithm.

Passivation of beans means serialization of non-transient data in the bean to disk storage to release the memory used by the bean. The next request for the passivated bean will require activation, the reverse process, where bean elements are read from the disk store and the active bean instance is recreated in memory. Needless to say, passivation and activation cycles can be extremely expensive. The application should always call a @Remove method to delete the active bean instance from the cache when a client is through using the instance. Failure to call a @Remove method leaves the bean instance in the active state and consumes one slot in the cache, requiring eventual passivation during cache management to make room for additional client beans.

By using the command-line interface we retrieve some configuration information, for example,

[standalone@192.168.1.66:9999 /] /subsystem=ejb3:read-resource
{
    "outcome" => "success",
    "result" => {
        "cluster-passivation-store" => undefined,
        "default-clustered-sfsb-cache" => undefined,
        "default-entity-bean-instance-pool" => undefined,
        "default-entity-bean-optimistic-locking" => undefined,
        "default-mdb-instance-pool" => "mdb-strict-max-pool",
        "default-resource-adapter-name" => "hornetq-ra",
        "default-sfsb-cache" => "simple",
        "default-singleton-bean-access-timeout" => 5000L,
        "default-slsb-instance-pool" => "slsb-strict-max-pool",
        "default-stateful-bean-access-timeout" => 5000L,
        "in-vm-remote-interface-invocation-pass-by-value" => "true",
        "cache" => {
            "simple" => undefined,
            "passivating" => undefined
        },
        "file-passivation-store" => {"file" => undefined},
        "service" => {
            "timer-service" => undefined,
            "remote" => undefined,
            "async" => undefined
        },
        "strict-max-bean-instance-pool" => {
            "slsb-strict-max-pool" => undefined,
            "mdb-strict-max-pool" => undefined
        },
        "thread-pool" => {"default" => undefined}
    }
}

To get insight in the file passivation configuration we can use

[standalone@192.168.1.66:9999 /] /subsystem=ejb3/file-passivation-store=file:read-resource
{
    "outcome" => "success",
    "result" => {
        "groups-path" => "ejb3/groups",
        "idle-timeout" => 300L,
        "idle-timeout-unit" => "SECONDS",
        "max-size" => 100000,
        "name" => "file",
        "relative-to" => "jboss.server.data.dir",
        "sessions-path" => "ejb3/sessions",
        "subdirectory-count" => 100
    }
}

Or the stateless session bean pool configuration

[standalone@192.168.1.66:9999 /] /subsystem=ejb3/strict-max-bean-instance-pool=slsb-strict-max-pool:read-resource
{
    "outcome" => "success",
    "result" => {
        "max-pool-size" => 20,
        "name" => "slsb-strict-max-pool",
        "timeout" => 5L,
        "timeout-unit" => "MINUTES"
    }
}

Application

The application consists of a servlet that calls a stateless enterprise bean, which in turn (based on the task to be performed); inserts data in the database and sends a message to the queue; updates data in the database; or removes data from the database and also sends a message to the queue. We have to following persistence configuration,

<persistence xmlns="http://java.sun.com/xml/ns/persistence" version="2.0">
    <persistence-unit name="PersonPersistenceUnit">
        <provider>org.hibernate.ejb.HibernatePersistence</provider>
        <jta-data-source>java:/jdbc/OracleDS</jta-data-source>
        <class>model.entities.Person</class>
    </persistence-unit>
</persistence>

Note that in JBoss, Hibernate is the default persistence provider and we do not need to include the classes as these are already present on the application server. The jta-data-source is mapped to the data source we have configured above with a JNDI name of java:/jdbc/OracleDS. The stateless enterprise bean looks as follows

package model.logic;

import model.entities.Person;

import javax.ejb.Remote;

@Remote
public interface Company {
    public void insertPerson(Person person);

    public void removePerson(Integer sofinummer);

    public void updatePerson(Person person);
}

and has the following implementation

package model.logic;

import model.entities.Person;

import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import javax.annotation.Resource;
import javax.ejb.Stateless;
import javax.jms.*;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;

@Stateless(name = "Company", mappedName = "Company")
public class CompanyBean implements Company {

    @PersistenceContext(unitName = "PersonPersistenceUnit")
    private EntityManager entityManager;

    @Resource(name = "java:/JmsXA")
    private ConnectionFactory connectionFactory;
    @Resource(name = "java:/queue/test")
    private Queue destination;

    private Connection connection = null;
    private Session session = null;
    private MessageProducer messageProducer = null;

    @PostConstruct
    public void init() {
        try {
            connection = connectionFactory.createConnection();
            session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
            messageProducer = session.createProducer(destination);
        } catch (JMSException e) {
            if (connection != null) {
                try {
                    connection.close();
                } catch (JMSException f) {
                    e.printStackTrace();
                }
            }
        }
    }

    @PreDestroy
    public void release() {
        try {
            if (connection != null) {
                connection.close();
            }
        } catch (JMSException e) {
            e.printStackTrace();
        }
    }

    public void insertPerson(Person person) {
        if (findPerson(person.getSofinummer()) == null) {
            entityManager.persist(person);
            sendMessage(person);
        } else {
            updatePerson(person);
        }
    }

    public void removePerson(Integer sofinummer) {
        Person person = findPerson(sofinummer);
        if (person != null) {
            entityManager.remove(entityManager.merge(person));
            sendMessage(person);
        }
    }

    public void updatePerson(Person person) {
        entityManager.merge(person);
    }

    private Person findPerson(Integer sofinummer) {
        return entityManager.find(Person.class, sofinummer);
    }

    private void sendMessage(Person person) {
        try {
            ObjectMessage message = session.createObjectMessage();
            message.setObject(person);
            messageProducer.send(message);
        } catch (JMSException e) {
            e.printStackTrace();
        }
    }
}

Here, we inject a JMS connection factory (with JNDI java:/JmsXA) and a JMS queue (with JNDI java:/queue/test). The enterprise bean is initialized by setting-up a message producer, that will be used in the insert and remove tasks. The message-driven bean looks as follows

@MessageDriven(name = "CompanyMDB", activationConfig = {
        @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
        @ActivationConfigProperty(propertyName = "destination", propertyValue = "java:/queue/test")
})
public class CompanyMDB implements MessageListener {

    public void onMessage(Message message) {
        if (message instanceof ObjectMessage) {
            ObjectMessage objectMessage = (ObjectMessage) message;
            try {
                System.out.println("RECEIVED OBJECT MESSAGE " + objectMessage.getObject());
            } catch (JMSException e) {
                e.printStackTrace();
            }
        }

        if (message instanceof TextMessage) {
            TextMessage textMessage = (TextMessage) message;
            try {
                System.out.println("RECEIVED TEXT MESSAGE " + textMessage.getText());
            } catch (JMSException e) {
                e.printStackTrace();
            }
        }
    }
}

The servlet looks as follows

package userinterface.servlets;

import model.entities.Person;
import model.logic.Company;

import javax.ejb.EJB;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Random;

@WebServlet(name = "TestServlet", urlPatterns = "/testservlet")
public class TestServlet extends HttpServlet {

    @EJB(name = "Company", mappedName = "java:global/LoadTest6/Model/Company")
    Company company;

    private Random generator = null;

    @Override
    public void init() throws ServletException {
        generator = new Random();
    }

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        Person person = createPerson();

        if (generator.nextDouble() &lt; 0.01) {
            company.removePerson(person.getSofinummer());
        } else {
            company.insertPerson(person);
        }
    }

    @Override
    public void destroy() {
        generator = null;
    }

    private Person createPerson() {
        Person person = new Person();
        person.setNaam(Long.toString(Math.abs(generator.nextLong()), 36));
        person.setSofinummer(generator.nextInt(10000));
        return person;
    }
}

When an enterprise bean is deployed is will be bound to JNDI as java:global/<deployment-name>/<module-name>/<mapped-name>.

Deployment

In the standalone mode JBoss comes with a deployment scanner. Its job is to monitor a directory for new files and to deploy those files. In the standalone.xml configuration file the following entry can be found

<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
	<deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>
</subsystem>

To auto-deploy an application we can put our .ear or .war file into the ${JBOSS_HOME}/deployments directory. We can also use the preferred command-line interface

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/LoadTest6.ear

The following is observed in the logging

[jboss@axis-into-ict bin]$ ./standalone.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/jboss-as-7.1.0.Final

  JAVA: /home/jboss/jdk1.6.0_31/bin/java

  JAVA_OPTS:  -XX:+UseCompressedOops -XX:+TieredCompilation -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

=========================================================================

13:45:06,255 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
13:45:06,420 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
13:45:06,464 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
13:45:07,295 INFO  [org.xnio] XNIO Version 3.0.3.GA
13:45:07,302 INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
13:45:07,312 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
13:45:07,320 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.2.GA
13:45:07,356 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
13:45:07,400 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 27) JBAS016200: Activating ConfigAdmin Subsystem
13:45:07,405 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 50) JBAS015537: Activating WebServices Extension
13:45:07,420 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 41) JBAS011940: Activating OSGi Subsystem
13:45:07,422 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 46) JBAS013101: Activating Security Subsystem
13:45:07,424 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 40) JBAS011800: Activating Naming Subsystem
13:45:07,442 INFO  [org.jboss.as.security] (MSC service thread 1-2) JBAS013100: Current PicketBox version=4.0.6.final
13:45:07,471 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 32) JBAS010280: Activating Infinispan subsystem.
13:45:07,523 INFO  [org.jboss.as.connector] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
13:45:07,546 INFO  [org.jboss.as.naming] (MSC service thread 1-4) JBAS011802: Starting Naming Service
13:45:07,645 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 28) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
13:45:07,730 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS015400: Bound mail session [java:jboss/mail/Default]
13:45:07,773 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-3) JBoss Web Services - Stack CXF Server 4.0.1.GA
13:45:07,822 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 28) JBAS010403: Deploying JDBC-compliant driver class oracle.jdbc.OracleDriver (version 11.2)
13:45:07,923 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-4) Starting Coyote HTTP/1.1 on http-axis-into-ict.nl-192.168.1.66-8080
13:45:07,938 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/home/jboss/jboss-as-7.1.0.Final/standalone/data/messagingjournal,bindingsDirectory=/home/jboss/jboss-as-7.1.0.Final/standalone/data/messagingbindings,largeMessagesDirectory=/home/jboss/jboss-as-7.1.0.Final/standalone/data/messaginglargemessages,pagingDirectory=/home/jboss/jboss-as-7.1.0.Final/standalone/data/messagingpaging)
13:45:07,946 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) Waiting to obtain live lock
13:45:08,014 INFO  [org.hornetq.core.persistence.impl.journal.JournalStorageManager] (MSC service thread 1-3) Using AIO Journal
13:45:08,217 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on /192.168.1.66:9999
13:45:08,250 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) JBAS015012: Started FileSystemDeploymentService for directory /home/jboss/jboss-as-7.1.0.Final/standalone/deployments
13:45:08,267 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on axis-into-ict.nl/192.168.1.66:4447
13:45:08,272 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-3) Waiting to obtain live lock
13:45:08,273 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-3) Live Server Obtained live lock
13:45:08,339 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
13:45:08,340 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:/jdbc/OracleDS]
13:45:08,649 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-3) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5445 for CORE protocol
13:45:08,651 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-3) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5455 for CORE protocol
13:45:08,652 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) Server is now live
13:45:08,653 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) HornetQ Server version 2.2.11.Final (HQ_2_2_11_FINAL_AS7, 122) [a0959f8b-5cbd-11e1-899c-000c2976c82d]) started
13:45:08,656 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) trying to deploy queue jms.topic.testTopic
13:45:08,682 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/topic/test
13:45:08,683 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/topic/test
13:45:08,684 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) trying to deploy queue jms.queue.testQueue
13:45:08,689 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/test
13:45:08,689 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:/queue/test
13:45:08,704 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
13:45:08,706 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
13:45:08,712 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
13:45:08,754 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010406: Registered connection factory java:/JmsXA
13:45:08,763 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-2) HornetQ resource adaptor started
13:45:08,763 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-2) IJ020002: Deployed: file://RaActivatorhornetq-ra
13:45:08,767 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-4) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
13:45:08,844 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 2784ms - Started 157 of 231 services (72 services are passive or on-demand)
13:46:30,698 INFO  [org.jboss.as.repository] (management-handler-threads - 6) JBAS014900: Content added at location /home/jboss/jboss-as-7.1.0.Final/standalone/data/content/16/1f51dde7f085c822cc4c68b306d57f1bee902d/content
13:46:30,714 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "LoadTest6.ear"
13:46:30,773 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "Web.war"
13:46:30,773 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "Model.jar"
13:46:30,838 INFO  [org.jboss.as.jpa] (MSC service thread 1-4) JBAS011401: Read persistence.xml for PersonPersistenceUnit
13:46:30,961 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-2) JNDI bindings for session bean named Company in deployment unit subdeployment "Model.jar" of deployment "LoadTest6.ear" are as follows:

        java:global/LoadTest6/Model/Company!model.logic.Company
        java:app/Model/Company!model.logic.Company
        java:module/Company!model.logic.Company
        java:jboss/exported/LoadTest6/Model/Company!model.logic.Company
        java:global/LoadTest6/Model/Company
        java:app/Model/Company
        java:module/Company

13:46:31,177 INFO  [org.jboss.as.jpa] (MSC service thread 1-2) JBAS011402: Starting Persistence Unit Service 'LoadTest6.ear/Model.jar#PersonPersistenceUnit'
13:46:31,234 INFO  [org.jboss.as.ejb3] (MSC service thread 1-4) JBAS014142: Started message driven bean 'CompanyMDB' with 'hornetq-ra' resource adapter
13:46:31,360 INFO  [org.jboss.web] (MSC service thread 1-3) JBAS018210: Registering web context: /LoadTest6
13:46:31,387 INFO  [org.hibernate.annotations.common.Version] (MSC service thread 1-2) HCANN000001: Hibernate Commons Annotations {4.0.1.Final}
13:46:31,393 INFO  [org.hibernate.Version] (MSC service thread 1-2) HHH000412: Hibernate Core {4.0.1.Final}
13:46:31,397 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-2) HHH000206: hibernate.properties not found
13:46:31,398 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-2) HHH000021: Bytecode provider name : javassist
13:46:31,414 INFO  [org.hibernate.ejb.Ejb3Configuration] (MSC service thread 1-2) HHH000204: Processing PersistenceUnitInfo [
        name: PersonPersistenceUnit
        ...]
13:46:31,532 INFO  [org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator] (MSC service thread 1-2) HHH000130: Instantiating explicit connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
13:46:31,639 INFO  [org.hibernate.dialect.Dialect] (MSC service thread 1-2) HHH000400: Using dialect: org.hibernate.dialect.Oracle10gDialect
13:46:31,656 INFO  [org.hibernate.engine.transaction.internal.TransactionFactoryInitiator] (MSC service thread 1-2) HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jta.CMTTransactionFactory
13:46:31,660 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (MSC service thread 1-2) HHH000397: Using ASTQueryTranslatorFactory
13:46:31,698 INFO  [org.hibernate.validator.util.Version] (MSC service thread 1-2) Hibernate Validator 4.2.0.Final
13:46:32,054 INFO  [org.jboss.as.server] (management-handler-threads - 6) JBAS018559: Deployed "LoadTest6.ear"

When we do a few tests (access the application by using http://hostname:8080/LoadTest6/testservlet) the following is observed in the logging:

13:50:37,002 INFO  [org.jboss.ejb.client] (http-axis-into-ict.nl-192.168.1.66-8080-1) JBoss EJB Client version 1.0.2.Final
13:50:37,484 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1a0100uhg5tiu 7907
13:50:39,692 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE iuhpgb6pcs13 9223
13:50:40,164 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 6m08798r5ehs 2500
13:50:40,621 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 12wx9zs77r9cx 4262
13:50:41,044 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 6dh6zc60tw1y 285
13:50:41,850 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE extyyn0pov1w 674
13:50:42,242 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE pv0ypyftodqy 8381
13:50:42,625 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1xh0e0cn7ggsa 5954
13:50:43,026 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1fiaxr5gsxkuo 4480
13:50:43,470 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1dnmh60s6fl47 7487
13:50:43,763 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 15mxq08abiozd 6698
13:50:44,183 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE hzgxdkqw1zo7 9573
13:50:44,433 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 4mvfve8o8gfb 1940
13:50:44,810 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1skgi6orrjiaw 3535
13:50:45,066 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE vb52wrkqgx81 5712
13:50:45,598 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1rxts2saz9fhi 7821
13:50:45,697 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE j8gonmtmsd8g 4627
13:50:45,824 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE da4b2qrqpt7b 9219
13:50:45,864 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE yqlwfar66vfk 4998
13:50:45,880 INFO  [stdout] (Thread-4 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE pzqok192zmnz 7551
13:50:45,882 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE jcyf4wttbwmy 1071
13:50:45,901 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1jg4yksazwtkx 5685
13:50:45,904 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1mv38p7onaidx 4399
13:50:45,915 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE wvsj6gq17e2i 586
13:50:45,936 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1c9ij29bk00ru 3746
13:50:45,973 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE ue08qs0k9unr 6877
13:50:46,008 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE hb372i5a3dai 3762
13:50:46,048 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE cg1pr42138dr 1066
13:50:46,086 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1k28o323a0k52 3510
13:50:46,121 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE fo0pn8x69hr9 2643
13:50:46,194 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1134xboronc8u 3323
13:50:46,235 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 16znhvr4pu6hy 5842

To obtain run-time information we can use the command-line interface, for example,

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] cd subsystem=messaging/hornetq-server=default/jms-queue=testQueue
[standalone@192.168.1.66:9999 jms-queue=testQueue] pwd
/subsystem=messaging/hornetq-server=default/jms-queue=testQueue
[standalone@192.168.1.66:9999 jms-queue=testQueue] read-attribute delivering-count
0
[standalone@192.168.1.66:9999 jms-queue=testQueue] read-attribute consumer-count
15
[standalone@192.168.1.66:9999 jms-queue=testQueue] read-attribute message-count
0L
[standalone@192.168.1.66:9999 jms-queue=testQueue] read-attribute messages-added
32L
[standalone@192.168.1.66:9999 jms-queue=testQueue] cd /
[standalone@192.168.1.66:9999 /] cd subsystem=datasources/data-source=OracleDS
[standalone@192.168.1.66:9999 data-source=OracleDS] ls
connection-properties                                       statistics                                                  allocation-retry=undefined
allocation-retry-wait-millis=undefined                      background-validation=undefined                             background-validation-millis=undefined
blocking-timeout-wait-millis=undefined                      check-valid-connection-sql=undefined                        connection-url=jdbc:oracle:thin:@192.168.1.65:1521:orcl11
datasource-class=undefined                                  driver-class=undefined                                      driver-name=oracle
enabled=true                                                exception-sorter-class-name=undefined                       exception-sorter-properties=undefined
flush-strategy=undefined                                    idle-timeout-minutes=0                                      jndi-name=java:/jdbc/OracleDS
jta=true                                                    max-pool-size=15                                            min-pool-size=1
new-connection-sql=undefined                                password=example                                            pool-prefill=true
pool-use-strict-min=true                                    prepared-statements-cache-size=10                           query-timeout=600
reauth-plugin-class-name=undefined                          reauth-plugin-properties=undefined                          security-domain=undefined
set-tx-query-timeout=false                                  share-prepared-statements=false                             spy=false
stale-connection-checker-class-name=undefined               stale-connection-checker-properties=undefined               track-statements="NOWARN"
transaction-isolation=undefined                             url-delimiter=undefined                                     url-selector-strategy-class-name=undefined
use-ccm=true                                                use-fast-fail=false                                         use-java-context=true
use-try-lock=undefined                                      user-name=example                                           valid-connection-checker-class-name=undefined
valid-connection-checker-properties=undefined               validate-on-match=false
[standalone@192.168.1.66:9999 data-source=OracleDS] cd statistics=pool
[standalone@192.168.1.66:9999 statistics=pool] read-attribute AverageCreationTime
93
[standalone@192.168.1.66:9999 statistics=pool] read-attribute AvailableCount
15
[standalone@192.168.1.66:9999 statistics=pool] read-attribute ActiveCount
5
[standalone@192.168.1.66:9999 statistics=pool] read-attribute MaxWaitTime
0
[standalone@192.168.1.66:9999 statistics=pool] cd ..
[standalone@192.168.1.66:9999 data-source=OracleDS] cd statistics=jdbc
[standalone@192.168.1.66:9999 statistics=jdbc] read-attribute PreparedStatementCacheAccessCount
70
[standalone@192.168.1.66:9999 statistics=jdbc] read-attribute PreparedStatementCacheHitCount
58
[standalone@192.168.1.66:9999 statistics=jdbc] read-attribute PreparedStatementCacheMissCount
0
[standalone@192.168.1.66:9999 statistics=jdbc] cd /
[standalone@192.168.1.66:9999 /] cd deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3
[standalone@192.168.1.66:9999 subsystem=ejb3] ls
entity-bean              message-driven-bean      singleton-bean           stateful-session-bean    stateless-session-bean
[standalone@192.168.1.66:9999 subsystem=ejb3] cd message-driven-bean=CompanyMDB
[standalone@192.168.1.66:9999 message-driven-bean=CompanyMDB] read-attribute pool-current-size
2
[standalone@192.168.1.66:9999 message-driven-bean=CompanyMDB] read-attribute pool-available-count
50
[standalone@192.168.1.66:9999 message-driven-bean=CompanyMDB] cd ..
[standalone@192.168.1.66:9999 subsystem=ejb3] cd stateless-session-bean=Company
[standalone@192.168.1.66:9999 stateless-session-bean=Company] read-attribute pool-current-size
5
[standalone@192.168.1.66:9999 stateless-session-bean=Company] read-attribute pool-available-count
50
[standalone@192.168.1.66:9999 stateless-session-bean=Company] cd /
[standalone@192.168.1.66:9999 /] cd deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=jpa
[standalone@192.168.1.66:9999 subsystem=jpa] ls
hibernate-persistence-unit
[standalone@192.168.1.66:9999 subsystem=jpa] cd hibernate-persistence-unit=LoadTest6.ear\/Model.jar#PersonPersistenceUnit
[standalone@192.168.1.66:9999 hibernate-persistence-unit=LoadTest6.ear/Model.jar#PersonPersistenceUnit] ls
collection                                        entity                                            entity-cache                                      query-cache
close-statement-count=0                           collection-fetch-count=0                          collection-load-count=0                           collection-recreated-count=0
collection-remove-count=0                         collection-update-count=0                         completed-transaction-count=35                    connect-count=70
enabled=false                                     entity-delete-count=0                             entity-fetch-count=0                              entity-insert-count=0
entity-load-count=0                               entity-update-count=0                             flush-count=0                                     optimistic-failure-count=0
prepared-statement-count=70                       query-cache-hit-count=0                           query-cache-miss-count=0                          query-cache-put-count=0
query-execution-count=0                           query-execution-max-time=0                        query-execution-max-time-query-string=undefined   second-level-cache-hit-count=0
second-level-cache-miss-count=0                   second-level-cache-put-count=0                    session-close-count=0                             session-open-count=0
successful-transaction-count=35
[standalone@192.168.1.66:9999 hibernate-persistence-unit=LoadTest6.ear/Model.jar#PersonPersistenceUnit] cd ../../../
[standalone@192.168.1.66:9999 deployment=LoadTest6.ear] cd subdeployment=Web.war/subsystem=web
[standalone@192.168.1.66:9999 subsystem=web] ls
servlet                     context-root=/LoadTest6     virtual-host=default-host
[standalone@192.168.1.66:9999 subsystem=web] cd servlet=TestServlet
[standalone@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
35
[standalone@192.168.1.66:9999 servlet=TestServlet] read-attribute processing-time
2002L

For the messaging we looked at the attributes; message-count that indicates how many messages are still in the queue. For a data source it important to know if requests (tasks) are waiting for a connection from the pool. This is indicated by the max-wait-time attribute, when this differs from zero it might be useful to adjust the connection pool settings, i.e., increase the maximum number of connections in the pool when possible. Other statistics we viewed was some run-time information from the EJB and servlet environment. An alternative way to view run-time statistics is by using an MBean browser, for example, jconsole or jvisualvm. Note that in the latter case we have to install a plug-in. In jvisualvm click tools, plug-ins, available plug-ins and select visualvm-mbeans. The MBean browser has an entry jboss.as that contains all the run-time information.

We can also put the commands in a script, for example,

connect 192.168.1.66:9999
cd subsystem=messaging/hornetq-server=default/jms-queue=testQueue
pwd
read-attribute delivering-count
read-attribute consumer-count
read-attribute message-count
read-attribute messages-added
cd /
cd subsystem=datasources/data-source=OracleDS
ls
cd statistics=pool
read-attribute AverageCreationTime
read-attribute AvailableCount
read-attribute ActiveCount
read-attribute MaxWaitTime
cd ..
cd statistics=jdbc
read-attribute PreparedStatementCacheAccessCount
read-attribute PreparedStatementCacheHitCount
read-attribute PreparedStatementCacheMissCount
cd /
cd deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3
ls
cd message-driven-bean=CompanyMDB
read-attribute pool-current-size
read-attribute pool-available-count
cd ..
cd stateless-session-bean=Company
read-attribute pool-current-size
read-attribute pool-available-count
cd /
cd deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=jpa
ls
cd hibernate-persistence-unit=LoadTest6.ear\/Model.jar#PersonPersistenceUnit
ls
cd ../../../
cd subdeployment=Web.war/subsystem=web
ls
cd servlet=TestServlet
read-attribute request-count
read-attribute processing-time

To execute the script, we can use (note that we called the script example-script.cli)

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh --file=/home/jboss/jboss-as-7.1.0.Final/scripts/example-script.cli

Using the Apache HTTP Server as a front-end

General information about the Apache HTTP Server can be found here. When we want to use the Apache HTTP Server as a front-end and in particular mod_jk we have to enable an AJP connector for the server in question. To create a new connector requires us to declare a new socket binding first

[standalone@192.168.1.66:9999 /] /socket-binding-group=standard-sockets/socket-binding=ajp:add(port=9080)

The created socket binding can then be used to create a new connector

[standalone@192.168.1.66:9999 /] /subsystem=web/connector=ajp:add(socket-binding=ajp, protocol="AJP/1.3", enabled=true)

With the new connector in place, we can continue with setting-up the Apache HTTP server. To install the Apache HTTP server we can follow these steps

Unpack the httpd-2.2.21.tar.gz file:
- gzip -d httpd-2.2.21.tar.gz
- tar xvf httpd-2.2.21.tar
- cd httpd-2.2.21
The next step is to configure:
- ./configure –prefix=/home/jboss/apache
Next, compile the various parts for the Apache HTTP Server by using:
- make
To install the Apache HTTP Server we use:
- make install
Open the httpd.conf (/home/jboss/apache/conf) file and adjust the following directives:
- Listen 8888
- ServerName ip-address
To test the set-up, start the Apache HTTP Server:
- Navigate to /home/jboss/apache/bin.
- Run: ./apachectl -k start (To stop the Apache HTTP Server we can use: ./apachectl -k stop).
- Open a browser en type the following URL: http://hostname:8888.

To install the mod_jk module we can follow these steps

Unpack the tomcat-connectors-1.2.32-src.tar.gz file:
- gzip -d tomcat-connectors-1.2.32-src.tar.gz
- tar xvf tomcat-connectors-1.2.32-src.tar
- cd tomcat-connectors-1.2.32-src/native
The next step is to configure the module:
- ./configure –with-apxs=/home/jboss/apache/bin/apxs
Compile:
- make
In the native directory, a directory apache-2.0 is created. Here a mod_jk.so file is present. Copy this file to the directory: /home/jboss/apache/modules.

To configure the mod_jk module, create a new file mod_jk.conf in the directory /home/jboss/apache/conf. An example configuration looks as follows

LoadModule jk_module /home/jboss/apache/modules/mod_jk.so

JkWorkersFile /home/jboss/apache/conf/jboss-workers.properties

# where to put the log files for the jk module
JkLogFile /home/jboss/apache/logs/mod_jk.log

# the log level [debug|info|error]
JkLogLevel info

# log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# options to indicate to send SSL KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

# set the request log format
JkRequestLogFormat "%w %V %T"

# send requests that contain the following to JBoss
JkMount /LoadTest6/* standalone-server-worker

Next, we need to define the worker file that is specified by JkWorkersFile directive. In this case, we create a file called jboss-workers.properties in the directory /home/jboss/apache/conf. An example worker configuration is the following

# define a worker that uses ajp13
worker.list=standalone-server-worker

# set properties for the worker
worker.standalone-server-worker.type=ajp13
worker.standalone-server-worker.host=192.168.1.66 (which is the IP-address where the JBoss server is running)
worker.standalone-server-worker.port=9080 (which is AJP listen port of the JBoss Server)

To let the Apache HTTP Server pick up the configuration we add the following to the httpd.conf file

# put it near the end of the file where all the other includes are present
# mod_jk configuration
Include conf/mod_jk.conf

Restart the HTTP Server (./apachectl -k stop and ./apachectl -k start). The application can now be reached at: http://hostname:8888/LoadTest6/testservlet. Note that by using the JkMount directive, we tell mod_jk to forward requests to a configured back-end server.

We can fine tune the web container, for example,

<extensions>
	...
	<extension module="org.jboss.as.web"/>
	...
</extensions>
<profile>
	...
	<subsystem xmlns="urn:jboss:domain:web:1.1" native="false" default-virtual-server="default-host">
		<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
		<connector name="ajp" protocol="AJP/1.3" socket-binding="ajp" enabled="true" enable-lookups="false" executor="ajp-thread-pool" max-connections="150" max-post-size="1048576"/>
		<virtual-server name="default-host" enable-welcome-root="true">
			<alias name="localhost"/>
			<alias name="example.com"/>
		</virtual-server>
	</subsystem>
	...
</profile>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
	<socket-binding name="http" port="8080"/>
	<socket-binding name="https" port="8443"/>
	...
	<socket-binding name="ajp" port="9080"/>
	...
</socket-binding-group>

The connector is a Java object, which provides an interface to Web server clients. The default configuration provides a connector configuration in which most attributes have their default values. For example, to configure a denial of service we add the following attributes:

max-connections – the maximum number of simultaneous requests that can be made. We set this equal to the MaxClients directive configured in the Apache HTTP server.
max-post-size – limits the maximum size of HTTP POST requests it accepts (the default is 2MB). We set this to 1048576 (1024*1024) bytes. Note that the value is based on what post size to expect from clients. Usually for a web application this is in the range of a few kilobytes. When using web service this value can be in the range of a few 100 kilobytes depending on the message sizes the web service defines, and if attachments can be send as well.
max-save-post-size – maximum size of a POST that will be saved by the container during authentication (the default is 4kB which we will leave as it is).

In general, a JBoss application server will be fronted by an Apache HTTP server. One thing to note is that, we have to make the thread pool size equal to the MaxClients directive configured on the Apache HTTP Server (usually it is something like MaxClients 150) otherwise threads will pile up in the JBoss Server. An example thread pool configuration is the following.

<subsystem xmlns="urn:jboss:domain:threads:1.1">
	<blocking-bounded-queue-thread-pool name="ajp-thread-pool">
		<queue-length count="200"/>
		<core-threads count="150"/>
		<max-threads count="150"/>
		<keepalive-time time="30" unit="seconds"/>
	</blocking-bounded-queue-thread-pool>
</subsystem>

Note that when using a small pool with a large queue, we tend to minimize CPU usage, operating system resources and context switching. This can potentially lead to a low throughput. By using small queues we need larger a larger thread pool. This keeps the CPU busier and can lead to an significant increase in scheduling overhead and thereby decreasing throughput. To use a certain thread pool configuration in a connector we can use the executor attribute.

The attribute enable-lookups when set to true, a call to request.getRemoteHost() returns the actual hostname of the remote client. When set to false the IP address is returned. In the latter case no DNS lookup is performed and thus improving performance.

To check the configuration we can use

[standalone@192.168.1.66:9999 /] /subsystem=threads/blocking-bounded-queue-thread-pool=ajp-thread-pool:read-resource(recursive=true)
{
    "outcome" => "success",
    "result" => {
        "allow-core-timeout" => false,
        "core-threads" => 150,
        "keepalive-time" => {
            "time" => 30L,
            "unit" => "SECONDS"
        },
        "max-threads" => 150,
        "name" => "ajp-thread-pool",
        "queue-length" => 200,
        "thread-factory" => undefined
    }
}

[standalone@192.168.1.66:9999 /] /subsystem=web/connector=ajp:read-resource(recursive=true)
{
    "outcome" => "success",
    "result" => {
        "enable-lookups" => false,
        "enabled" => true,
        "executor" => "ajp-thread-pool",
        "max-connections" => 150,
        "max-post-size" => 1048576,
        "max-save-post-size" => 4096,
        "protocol" => "AJP/1.3",
        "redirect-port" => 8443,
        "scheme" => "http",
        "secure" => false,
        "socket-binding" => "ajp",
        "ssl" => undefined,
        "virtual-server" => undefined
    }
}

To see web metrics by using the admin console, after making a number of requests to http://hostname:8888/LoadTest6/testservlet. Open the console (http://hostname:9990/console), click the run-time tab, web in subsystem metrics and click the ajp row. It shows the request count, error count, processing time and max time.

Clustering

To cluster standalone servers we create two configuration files, for example, server1.xml and server2.xml, which are based on the standalone.xml with which we have been working. We first undeploy the LoadTest6 application, by using the command-line interface command: undeploy LoadTest6.ear. Copy the standalone.xml file twice and name the copies to server1.xml and server2.xml. To enable clustering we set the log file name for the server, enable jgroups, and make the messaging system clusterable in the configuration files

<server xmlns="urn:jboss:domain:1.1">
    <extensions>
		...
        <extension module="org.jboss.as.clustering.jgroups"/>
		...
    </extensions>
    <profile>
		<subsystem xmlns="urn:jboss:domain:logging:1.1">
			...
            <periodic-rotating-file-handler name="FILE">
                <formatter>
                    <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="server1.log"/>
                <suffix value=".yyyy-MM-dd"/>
                <append value="true"/>
            </periodic-rotating-file-handler>
			...
        </subsystem>
		...
        <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">
            <stack name="udp">
                <transport type="UDP" socket-binding="jgroups-udp" diagnostics-socket-binding="jgroups-diagnostics"/>
                <protocol type="PING"/>
                <protocol type="MERGE2"/>
                <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                <protocol type="FD"/>
                <protocol type="VERIFY_SUSPECT"/>
                <protocol type="BARRIER"/>
                <protocol type="pbcast.NAKACK"/>
                <protocol type="UNICAST2"/>
                <protocol type="pbcast.STABLE"/>
                <protocol type="pbcast.GMS"/>
                <protocol type="UFC"/>
                <protocol type="MFC"/>
                <protocol type="FRAG2"/>
            </stack>
            <stack name="tcp">
                <transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>
                <protocol type="MPING" socket-binding="jgroups-mping"/>
                <protocol type="MERGE2"/>
                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                <protocol type="FD"/>
                <protocol type="VERIFY_SUSPECT"/>
                <protocol type="BARRIER"/>
                <protocol type="pbcast.NAKACK"/>
                <protocol type="UNICAST2"/>
                <protocol type="pbcast.STABLE"/>
                <protocol type="pbcast.GMS"/>
                <protocol type="UFC"/>
                <protocol type="MFC"/>
                <protocol type="FRAG2"/>
            </stack>
        </subsystem>
		...
		<subsystem xmlns="urn:jboss:domain:messaging:1.1">
            <hornetq-server>
                <clustered>true</clustered>
                <cluster-user>guest</cluster-user>
                <cluster-password>guest</cluster-password>
                <persistence-enabled>true</persistence-enabled>
                <journal-type>ASYNCIO</journal-type>
                <journal-file-size>102400</journal-file-size>
                <journal-min-files>2</journal-min-files>
                <paging-directory path="messagingpaging1"/>
                <bindings-directory path="messagingbindings1"/>
                <journal-directory path="messagingjournal1"/>
                <large-messages-directory path="messaginglargemessages1"/>
                <connectors>
                    <netty-connector name="netty" socket-binding="messaging"/>
                    <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                    </netty-connector>
                    <in-vm-connector name="in-vm" server-id="0"/>
                </connectors>
                <acceptors>
                    <netty-acceptor name="netty" socket-binding="messaging"/>
                    <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                        <param key="direct-deliver" value="false"/>
                    </netty-acceptor>
                    <in-vm-acceptor name="in-vm" server-id="0"/>
                </acceptors>
                <broadcast-groups>
                    <broadcast-group name="bg-group1">
                        <group-address>231.7.7.7</group-address>
                        <group-port>9876</group-port>
                        <broadcast-period>5000</broadcast-period>
                        <connector-ref>netty</connector-ref>
                    </broadcast-group>
                </broadcast-groups>
                <discovery-groups>
                    <discovery-group name="dg-group1">
                        <group-address>231.7.7.7</group-address>
                        <group-port>9876</group-port>
                        <refresh-timeout>10000</refresh-timeout>
                    </discovery-group>
                </discovery-groups>
                <cluster-connections>
                    <cluster-connection name="my-cluster">
                        <address>jms</address>
                        <connector-ref>netty</connector-ref>
                        <discovery-group-ref discovery-group-name="dg-group1"/>
                    </cluster-connection>
                </cluster-connections>
                <security-settings>
                    <security-setting match="#">
                        <permission type="send" roles="guest"/>
                        <permission type="consume" roles="guest"/>
                        <permission type="createNonDurableQueue" roles="guest"/>
                        <permission type="deleteNonDurableQueue" roles="guest"/>
                    </security-setting>
                </security-settings>
                <address-settings>
                    <address-setting match="#">
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <redelivery-delay>0</redelivery-delay>
                        <redistribution-delay>1000</redistribution-delay>
                        <max-size-bytes>10485760</max-size-bytes>
                        <address-full-policy>BLOCK</address-full-policy>
                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                    </address-setting>
                </address-settings>
                <jms-connection-factories>
                    <connection-factory name="InVmConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/ConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <connection-factory name="RemoteConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="netty"/>
                        </connectors>
                        <entries>
                            <entry name="RemoteConnectionFactory"/>
                            <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <pooled-connection-factory name="hornetq-ra">
                        <transaction mode="xa"/>
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/JmsXA"/>
                        </entries>
                    </pooled-connection-factory>
                </jms-connection-factories>
                <jms-destinations>
                    <jms-queue name="testQueue">
                        <entry name="java:/queue/test"/>
                        <entry name="java:jboss/exported/jms/queue/test"/>
                    </jms-queue>
                    <jms-topic name="testTopic">
                        <entry name="java:/topic/test"/>
                        <entry name="java:jboss/exported/jms/topic/test"/>
                    </jms-topic>
                </jms-destinations>
            </hornetq-server>
        </subsystem>
		...
    </profile>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
		...
        <socket-binding name="jgroups-diagnostics" port="0" multicast-address="224.0.75.75" multicast-port="7500"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" port="57600"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="54200"/>
		...
    </socket-binding-group>
</server>

For server2 we use the port-offset attribute (which we set to one) in the socket binding

<server xmlns="urn:jboss:domain:1.1">
    <extensions>
		...
        <extension module="org.jboss.as.clustering.jgroups"/>
		...
    </extensions>
    <profile>
		<subsystem xmlns="urn:jboss:domain:logging:1.1">
			...
            <periodic-rotating-file-handler name="FILE">
                <formatter>
                    <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="server2.log"/>
                <suffix value=".yyyy-MM-dd"/>
                <append value="true"/>
            </periodic-rotating-file-handler>
			...
        </subsystem>
		...
        <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">...</subsystem>
		...
		<subsystem xmlns="urn:jboss:domain:messaging:1.1">
            <hornetq-server>
                <clustered>true</clustered>
                <cluster-user>guest</cluster-user>
                <cluster-password>guest</cluster-password>
                <persistence-enabled>true</persistence-enabled>
                <journal-type>ASYNCIO</journal-type>
                <journal-file-size>102400</journal-file-size>
                <journal-min-files>2</journal-min-files>
                <paging-directory path="messagingpaging2"/>
                <bindings-directory path="messagingbindings2"/>
                <journal-directory path="messagingjournal2"/>
                <large-messages-directory path="messaginglargemessages2"/>
				...
            </hornetq-server>
        </subsystem>
		...
    </profile>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:1}">
		...
    </socket-binding-group>
</server>

To start the servers, we can use standalone.sh --server-config=server1.xml and standalone.sh --server-config=server2.xml. When the servers are started the following is shown when hornetq is connected successfully between both servers

# server1
17:24:06,129 INFO  [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-9 (HornetQ-server-HornetQServerImpl::serverUUID=d052af0e-5efa-11e1-ad54-000c2976c82d-37774788)) Bridge ClusterConnectionBridge@4552a64d [name=sf.my-cluster.c25598f7-5efa-11e1-8e93-000c2976c82d, queue=QueueImpl[name=sf.my-cluster.c25598f7-5efa-11e1-8e93-000c2976c82d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=d052af0e-5efa-11e1-ad54-000c2976c82d]]@35242cc9 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@4552a64d [name=sf.my-cluster.c25598f7-5efa-11e1-8e93-000c2976c82d, queue=QueueImpl[name=sf.my-cluster.c25598f7-5efa-11e1-8e93-000c2976c82d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=d052af0e-5efa-11e1-ad54-000c2976c82d]]@35242cc9 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5446&host=axis-into-ict-nl], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@762010908[nodeUUID=d052af0e-5efa-11e1-ad54-000c2976c82d, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=axis-into-ict-nl, address=jms, server=HornetQServerImpl::serverUUID=d052af0e-5efa-11e1-ad54-000c2976c82d])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5446&host=axis-into-ict-nl], discoveryGroupConfiguration=null]] is connected

# server2
17:24:06,123 INFO  [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-29 (HornetQ-server-HornetQServerImpl::serverUUID=c25598f7-5efa-11e1-8e93-000c2976c82d-1291383602)) Bridge ClusterConnectionBridge@4174b989 [name=sf.my-cluster.d052af0e-5efa-11e1-ad54-000c2976c82d, queue=QueueImpl[name=sf.my-cluster.d052af0e-5efa-11e1-ad54-000c2976c82d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c25598f7-5efa-11e1-8e93-000c2976c82d]]@5640f2f1 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@4174b989 [name=sf.my-cluster.d052af0e-5efa-11e1-ad54-000c2976c82d, queue=QueueImpl[name=sf.my-cluster.d052af0e-5efa-11e1-ad54-000c2976c82d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c25598f7-5efa-11e1-8e93-000c2976c82d]]@5640f2f1 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=axis-into-ict-nl], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@211830739[nodeUUID=c25598f7-5efa-11e1-8e93-000c2976c82d, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5446&host=axis-into-ict-nl, address=jms, server=HornetQServerImpl::serverUUID=c25598f7-5efa-11e1-8e93-000c2976c82d])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=axis-into-ict-nl], discoveryGroupConfiguration=null]] is connected

To deploy the application we can use the following

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/LoadTest6.ear
[standalone@192.168.1.66:9999 /] connect 192.168.1.66:10000
[standalone@192.168.1.66:10000 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/LoadTest6.ear11

Before, we configure the load balancing, test if the application is working. Either use the URL: http://hostname:8080/LoadTest6/testservlet or the URL: http://hostname:8081/LoadTest6/testservlet.

Configure load balancing

The load balancer decides how to dispatch requests to the back end server instances. A load balancer needs to do tasks such as distributing requests, health checking and session binding. As these are simple jobs it is rare that a load balancer will become a bottleneck. To load balance requests, we are going to use mod_jk. To configure mod_jk, open the mod_jk.conf file and add the following contents:

LoadModule jk_module /home/jboss/apache/modules/mod_jk.so

JkWorkersFile /home/jboss/apache/conf/jboss-workers.properties

# where to put the log files for the jk module
JkLogFile /home/jboss/apache/logs/mod_jk.log

# the log level [debug|info|error]
JkLogLevel info

# log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# options to indicate to send SSL KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

# set the request log format
JkRequestLogFormat "%w %V %T"

# send requests that contain the following to JBoss
JkMount /LoadTest6/* loadbalancer

The workers (jboss-workers.properties) are defined as follows:

# define a worker that uses ajp13
worker.list=server1-worker,server2-worker,loadbalancer

# set properties for the server1-worker
worker.server1-worker.type=ajp13
worker.server1-worker.host=192.168.1.66
worker.server1-worker.port=9080
worker.server1-worker.lbfactor=1
worker.server1-worker.socket_keepalive=1
worker.server1-worker.socket_timeout=300

# set properties for the server2-worker
worker.server2-worker.type=ajp13
worker.server2-worker.host=192.168.1.66
worker.server2-worker.port=9081
worker.server2-worker.lbfactor=1
worker.server2-worker.socket_keepalive=1
worker.server2-worker.socket_timeout=300

# set properties for the loadbalancer
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=server1-worker,server2-worker

With the configuration in place, start the Apache HTTP server (./apachectl -k start) and enter the following URL: http://hostname:8888/LoadTest6/testservlet a few times to see if the requests get send to the different servers. By using the following URL’s: http://hostname:9990/console/App.html#web-metrics and http://hostname:9991/console/App.html#web-metrics, we can check how the requests are being processed. We can also use the command-line interface

[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] cd /deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web/servlet=TestServlet
[standalone@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
69
[standalone@192.168.1.66:9999 servlet=TestServlet] connect 192.168.1.66:10000
[standalone@192.168.1.66:10000 servlet=TestServlet] read-attribute request-count
68
[standalone@192.168.1.66:10000 servlet=TestServlet] read-attribute request-count
118
[standalone@192.168.1.66:10000 servlet=TestServlet] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
119

As we can see the load is balanced like a charm.

Load test

We can test the environment by using the The Grinder. The following script calls the application:

from net.grinder.script.Grinder import grinder
from net.grinder.script import Test
from net.grinder.plugin.http import HTTPRequest

test1 = Test(1, "Request resource")
request1 = test1.wrap(HTTPRequest())

class TestRunner:
    def __call__(self):
        result = request1.GET("http://192.168.1.66:8888/LoadTest6/testservlet")

Edit the grinder.properties (${GRINDER_HOME}/examples) file such that the script is used, for example,

# The file name of the script to run. Relative paths are evaluated from the directory containing the properties file.
grinder.script = test.py

# The number of worker processes each agent should start.
grinder.processes = 1

# The number of worker threads each worker process should start.
grinder.threads = 1

# The number of runs each worker process will perform. When using the console set this to 0, i.e., the runs are controlled by the console.
grinder.runs = 0

Use the following scripts to start a test

setGrinderEnv.sh

#!/bin/sh
GRINDERPATH=/home/jboss/grinder-3.4
export GRINDERPATH

JAVA_HOME=/home/jboss/jdk1.6.0_31
export JAVA_HOME

USER_MEM_ARGS="-server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC"
export USER_MEM_ARGS
GRINDERPROPERTIES=${GRINDERPATH}/examples/grinder.properties
export GRINDERPROPERTIES
CLASSPATH=${GRINDERPATH}/lib/grinder.jar:${CLASSPATH}
export CLASSPATH

PATH=${JAVA_HOME}/bin:${PATH}
export PATH

startConsole.sh

#!/bin/sh
source setGrinderEnv.sh
java ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Console

startAgent.sh

#!/bin/sh
source setGrinderEnv.sh
java ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Grinder ${GRINDERPROPERTIES}

First start the startConsole.sh script and then start the startAgent.sh script. The agent will now wait for a start action from the console (click the action tab and then choose start processes)

By using jvisualvm or jconsole the JVM can be monitored and MBeans can browsed

We can clearly see when the load test was running. The test introduced no particular problems for the JVM. Run-time statistics from the command-line interface

server 1
subsystem=messaging/hornetq-server=default/jms-queue=testQueue
delivering-count (The number of messages that this queue is currently delivering to its consumers): 0
consumer-count (The number of consumers consuming messages from this queue): 15
message-count (The number of messages currently in this queue): 0L
messages-added (The number of messages added to this queue since it was created): 11987L

subsystem=datasources/data-source=OracleDS/statistics=pool
AverageCreationTime (The average time spent creating a physical connection): 2594
AvailableCount (The available count): 15
ActiveCount (The active count): 2
MaxWaitTime (The maximum wait time for a connection): 3

subsystem=datasources/data-source=OracleDS/statistics=jdbc
PreparedStatementCacheAccessCount (The number of times that the statement cache was accessed): 770567
PreparedStatementCacheHitCount (The number of times that statements from the cache were used): 770560
PreparedStatementCacheMissCount (The number of times that a statement request could not be satisfied with a statement from the cache): 0

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3/message-driven-bean=CompanyMDB
pool-current-size (The current size of the pool): 4
pool-available-count (The number of available (i.e. not in use) instances in the pool): 12003

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3/stateless-session-bean=Company
pool-current-size (The current size of the pool): 1
pool-available-count (The number of available (i.e. not in use) instances in the pool): 385348

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=jpa/hibernate-persistence-unit=LoadTest6.ear\/Model.jar#PersonPersistenceUnit
completed-transaction-count (Number of completed transactions): 385329L
successful-transaction-count (Number of successful transactions): 385329L

deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web/servlet=TestServlet
request-count (Number of requests processed by this servlet): 385329
processing-time (Total execution time of the servlet's service method): 926143L

server 2
subsystem=messaging/hornetq-server=default/jms-queue=testQueue
delivering-count (The number of messages that this queue is currently delivering to its consumers): 0
consumer-count (The number of consumers consuming messages from this queue): 15
message-count (The number of messages currently in this queue): 0L
messages-added (The number of messages added to this queue since it was created): 11987L

subsystem=datasources/data-source=OracleDS/statistics=pool
AverageCreationTime (The average time spent creating a physical connection): 184
AvailableCount (The available count): 15
ActiveCount (The active count): 2
MaxWaitTime (The maximum wait time for a connection): 1

subsystem=datasources/data-source=OracleDS/statistics=jdbc
PreparedStatementCacheAccessCount (The number of times that the statement cache was accessed): 770566
PreparedStatementCacheHitCount (The number of times that statements from the cache were used): 770559
PreparedStatementCacheMissCount (The number of times that a statement request could not be satisfied with a statement from the cache): 0

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3/message-driven-bean=CompanyMDB
pool-current-size (The current size of the pool): 5
pool-available-count (The number of available (i.e. not in use) instances in the pool): 12002

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3/stateless-session-bean=Company
pool-current-size (The current size of the pool): 1
pool-available-count (The number of available (i.e. not in use) instances in the pool): 385344

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=jpa/hibernate-persistence-unit=LoadTest6.ear\/Model.jar#PersonPersistenceUnit
completed-transaction-count (Number of completed transactions): 385325L
successful-transaction-count (Number of successful transactions): 385325L

deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web/servlet=TestServlet
request-count (Number of requests processed by this servlet): 385325
processing-time (Total execution time of the servlet's service method): 922760L

Overall the test was easily handled by the Apache HTTP server (load balancing) and the JBoss application servers present in the cluster.

Domains

One of the new features of JBoss AS 7 is the ability to manage multiple server instances from a single control point. A domain controller process is acting as the management point. All the server instances share a common management policy, with the domain controller acting to ensure that each server is configured according to that policy. Domains can span multiple machines, with server instances on a certain host under the control of a host controller process. One host controller instance is configured as the domain controller. The host controller on each host interacts with the domain controller to control the lifecycle of the server instances running on a particular host and to assist the domain controller in managing them. One thing to note is that the configuration of subsystems (such as messaging and data sources) within a profile does not differ from the standalone configuration (apart from the fact that a domain configuration can have multiple profiles whereas a standalone configuration can only have one). What we need to get the hang of when working with domains are the domain and host controller configurations.

When the ${JBOSS_HOME}/bin/domain.sh script is run a host controller process is started. The host controller starts and stops server instances on the particular host where it was started. Each host controller is configured by using the ${JBOSS_HOME}/domain/configuration/host.xml file. This file contains

The list of application server names that must run on the host.
The configuration how the host controller must contact the domain controller to register itself and access the domain configuration. This can either be how to contact a remote domain controller or telling the host controller to act itself as a domain controller.

As already mentioned one host controller must act as the domain controller (central management point). The domain controller maintains a central management policy and makes sure the host controllers are aware of the contents. It also make sure that each server instance is configured with respect to the central management policy. The central management policy is by default stored in the ${JBOSS_HOME}/domain/configuration/domain.xml file on the filesystem where the domain controller is present (it is not necessary to have the domain.xml on the filesystems where other host controllers are running).

When working with a domain we also need to introduce the concept of a server group, which is a set of server instances that will be managed and configured as one. In a managed domain each application server instance is a member of a server group. A domain can have multiple server groups. Different server groups can be configured with different profiles and deployments.

An example domain.xml look as follows:

<domain xmlns="urn:jboss:domain:1.1">
    <extensions>
		<extension module="org.jboss.as.clustering.infinispan"/>
        <extension module="org.jboss.as.clustering.jgroups"/>
		...
        <extension module="org.jboss.as.messaging"/>
		...
        <extension module="org.jboss.as.weld"/>
	</extensions>
    <system-properties>
        <property name="java.net.preferIPv4Stack" value="true"/>
    </system-properties>
    <profiles>
        <profile name="default">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
        <profile name="ha">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
        <profile name="full">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:messaging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
        <profile name="full-ha">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
			<subsystem xmlns="urn:jboss:domain:datasources:1.0">
                <datasources>
                    ...
                    <datasource jta="true" jndi-name="java:/jdbc/OracleDS" pool-name="OracleDS" enabled="true" use-java-context="true" use-ccm="true">
                        <connection-url>jdbc:oracle:thin:@192.168.1.65:1521:orcl11</connection-url>
                        <driver>oracle</driver>
                        <pool>
                            <min-pool-size>1</min-pool-size>
                            <max-pool-size>15</max-pool-size>
                            <prefill>true</prefill>
                            <use-strict-min>true</use-strict-min>
                        </pool>
                        <security>
                            <user-name>example</user-name>
                            <password>example</password>
                        </security>
                        <timeout>
                            <idle-timeout-minutes>0</idle-timeout-minutes>
                            <query-timeout>600</query-timeout>
                        </timeout>
                        <statement>
                            <prepared-statement-cache-size>10</prepared-statement-cache-size>
                        </statement>
                    </datasource>
                    <drivers>
					    ...
                        <driver name="oracle" module="com.oracle.database">
                            <driver-class>oracle.jdbc.OracleDriver</driver-class>
                            <xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
                        </driver>
                    </drivers>
                </datasources>
            </subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:messaging:1.1">...</subsystem>
			...
			<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
				...
                <connector name="ajp" protocol="AJP/1.3" socket-binding="ajp" enable-lookups="false" max-post-size="1048576" enabled="true" max-connections="150"/>
				...
            </subsystem>
			...
			<subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
    </profiles>
    <!--Named interfaces that can be referenced elsewhere in the configuration. The configuration
        for how to associate these logical names with an actual network interface can either
        be specified here or can be declared on a per-host basis in the equivalent element in host.xml.
        These default configurations require the binding specification to be done in host.xml.-->
    <interfaces>
        <interface name="management"/>
        <interface name="public"/>
        <interface name="unsecure"/>
    </interfaces>
    <socket-binding-groups>
        <socket-binding-group name="standard-sockets" default-interface="public">...</socket-binding-group>
        <socket-binding-group name="ha-sockets" default-interface="public">
			...
			<socket-binding name="ajp" port="9080"/>
			...
		</socket-binding-group>
    </socket-binding-groups>
	<deployments>
        <deployment name="LoadTest6.ear" runtime-name="LoadTest6.ear">
            <content sha1="161f51dde7f085c822cc4c68b306d57f1bee902d"/>
        </deployment>
    </deployments>
    <server-groups>
        <server-group name="standard-server-group" profile="full">
            <jvm name="default"/>
            <socket-binding-group ref="standard-sockets"/>
        </server-group>
        <server-group name="ha-server-group" profile="full-ha">
            <jvm name="default"/>
            <socket-binding-group ref="ha-sockets"/>
			<deployments>
                <deployment name="LoadTest6.ear" runtime-name="LoadTest6.ear"/>
            </deployments>
        </server-group>
    </server-groups>
</domain>

In the example above we have set-up different profiles:

default – a basic server set-up
ha – a basic cluster set-up
full – a server set-up with messaging enabled
full-ha – a cluster set-up with messaging enabled

Different socket-binding groups one for the default and full profiles and one for the ha and full-ha profiles. Further, we defined two server groups, each mapped to a different profile. Note that server groups can have their own settings, such as JVM, socket-binding-group and deployments. To map server instances to a server group, we use a host configuration, for example,

<host name="axis-into-ict.nl" xmlns="urn:jboss:domain:1.1">
    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <authentication>
                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
                </authentication>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <authentication>
                    <properties path="application-users.properties" relative-to="jboss.domain.config.dir" />
                </authentication>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket interface="management" port="${jboss.management.native.port:9999}"/>
            </native-interface>
            <http-interface security-realm="ManagementRealm">
                <socket interface="management" port="${jboss.management.http.port:9990}"/>
            </http-interface>
        </management-interfaces>
    </management>
    <domain-controller>
       <local/>
       <!-- remote domain controller configuration: remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" -->
    </domain-controller>
    <interfaces>
        <interface name="management">
            <nic name="eth0"/>
        </interface>
        <interface name="public">
           <nic name="eth0"/>
        </interface>
        <interface name="unsecure">
            <inet-address value="127.0.0.1"/>
        </interface>
    </interfaces>
    <jvms>
    	<jvm name="default">
            <heap size="512m" max-size="512m"/>
            <permgen size="256m" max-size="256m"/>
            <jvm-options>
                <option value="-server"/>
                <option value="-XX:NewRatio=2"/>
                <option value="-XX:+UseParallelGC"/>
                <option value="-XX:ParallelGCThreads=2"/>
                <option value="-XX:MaxGCPauseMillis=200"/>
                <option value="-XX:GCTimeRatio=19"/>
                <option value="-XX:+UseParallelOldGC"/>
            </jvm-options>
        </jvm>
    </jvms>
    <servers>
        <server name="server1" group="ha-server-group" auto-start="true">
        </server>
        <server name="server2" group="ha-server-group" auto-start="true">
            <socket-bindings port-offset="1"/>
        </server>
    </servers>
</host>

Configuring a host to act as the domain controller is done through the domain-controller declaration in host.xml. If it includes the local element, then this host will become the domain controller. A host acting as the domain controller must expose a native (i.e. non-HTTP) management interface on an address accessible to the other hosts in the domain. Exposing an HTTP management interface is not required, but is recommended as it allows the administration console to work. The configuration above also contains the network interfaces which we mapped to the network interface card eth0.

We can start the domain by using domain.sh located in the ${JBOSS_HOME}/bin directory (do not forget to define JAVA_HOME and PATH before running domain.sh)

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./domain.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/jboss-as-7.1.0.Final

  JAVA: /home/jboss/jdk1.6.0_31/bin/java

  JAVA_OPTS: -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml

=========================================================================

10:13:12,861 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
10:13:13,459 INFO  [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
[Host Controller] 10:13:14,908 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Host Controller] 10:13:15,288 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Host Controller] 10:13:15,603 INFO  [org.jboss.as] (MSC service thread 1-3) JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
[Host Controller] 10:13:17,216 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.3.GA
[Host Controller] 10:13:17,237 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.3.GA
[Host Controller] 10:13:17,764 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.2.2.GA
[Host Controller] 10:13:21,800 INFO  [org.jboss.as] (Controller Boot Thread) JBAS010902: Creating http management service using network interface (management) port (9990) securePort (-1)
[Host Controller] 10:13:21,829 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on /192.168.1.66:9999
[Host Controller] 10:13:22,129 INFO  [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010922: Starting server server1
[Host Controller] 10:13:22,197 INFO  [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010922: Starting server server2
10:13:22,199 INFO  [org.jboss.as.process.Server:server1.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server1'
[Host Controller] 10:13:22,216 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" (Host Controller) started in 8647ms - Started 11 of 11 services (0 services are passive or on-demand)
10:13:22,292 INFO  [org.jboss.as.process.Server:server2.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server2'
[Server:server1] 10:13:22,976 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Server:server2] 10:13:23,121 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Server:server1] 10:13:23,671 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Server:server2] 10:13:23,672 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Server:server1] 10:13:23,833 INFO  [org.jboss.as] (MSC service thread 1-4) JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
[Server:server2] 10:13:23,850 INFO  [org.jboss.as] (MSC service thread 1-4) JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
[Server:server1] 10:13:24,097 INFO  [org.xnio] (MSC service thread 1-3) XNIO Version 3.0.3.GA
[Server:server1] 10:13:24,127 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.0.3.GA
[Server:server2] 10:13:24,135 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.3.GA
[Server:server1] 10:13:24,161 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 3.2.2.GA
[Server:server2] 10:13:24,204 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.3.GA
[Server:server2] 10:13:24,236 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 3.2.2.GA
[Server:server1] 10:13:26,666 INFO  [org.jboss.as.logging] (MSC service thread 1-3) JBAS011502: Removing bootstrap log handlers
[Server:server2] 10:13:26,867 INFO  [org.jboss.as.logging] (MSC service thread 1-1) JBAS011502: Removing bootstrap log handlers
[Server:server2] 10:13:26,890 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 32) JBAS015537: Activating WebServices Extension
[Server:server2] 10:13:26,902 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 36) JBAS013101: Activating Security Subsystem
[Server:server2] 10:13:26,993 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 59) JBAS016200: Activating ConfigAdmin Subsystem
[Server:server2] 10:13:26,975 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 55) JBAS010280: Activating Infinispan subsystem.
[Server:server2] 10:13:26,974 INFO  [org.jboss.as.jacorb] (ServerService Thread Pool -- 54) JBAS016300: Activating JacORB Subsystem
[Server:server2] 10:13:26,964 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 49) JBAS010260: Activating JGroups subsystem.
[Server:server2] 10:13:26,950 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 41) JBAS011940: Activating OSGi Subsystem
[Server:server2] 10:13:26,914 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 42) JBAS011800: Activating Naming Subsystem
[Server:server2] 10:13:27,225 INFO  [org.jboss.as.security] (MSC service thread 1-1) JBAS013100: Current PicketBox version=4.0.6.final
[Server:server2] 10:13:27,568 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 58) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
[Server:server2] 10:13:27,719 INFO  [org.jboss.as.connector] (MSC service thread 1-4) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
[Server:server1] 10:13:27,837 INFO  [org.jboss.as.naming] (MSC service thread 1-4) JBAS011802: Starting Naming Service
[Server:server2] 10:13:27,973 INFO  [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: Starting Naming Service
[Server:server2] 10:13:27,998 INFO  [org.jboss.as.jaxr] (MSC service thread 1-1) Binding JAXR ConnectionFactory: java:jboss/jaxr/ConnectionFactory
[Server:server1] 10:13:27,858 INFO  [org.jboss.as.connector] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
[Server:server2] 10:13:28,163 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) JBAS015400: Bound mail session [java:jboss/mail/Default]
[Server:server1] 10:13:28,187 INFO  [org.jboss.as.jaxr] (MSC service thread 1-4) Binding JAXR ConnectionFactory: java:jboss/jaxr/ConnectionFactory
[Server:server2] 10:13:28,300 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 58) JBAS010403: Deploying JDBC-compliant driver class oracle.jdbc.OracleDriver (version 11.2)
[Server:server1] 10:13:28,297 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 58) JBAS010403: Deploying JDBC-compliant driver class oracle.jdbc.OracleDriver (version 11.2)
[Host Controller] 10:13:28,359 INFO  [org.jboss.as.domain.controller.mgmt] (proxy-threads - 1) JBAS010920: Server [Server:server1] registered using connection [Channel ID 6cdb4665 (inbound) of Remoting connection 71ee88dd to /192.168.1.66:59624]
[Host Controller] 10:13:28,443 INFO  [org.jboss.as.domain.controller.mgmt] (proxy-threads - 1) JBAS010920: Server [Server:server2] registered using connection [Channel ID 025391e5 (inbound) of Remoting connection 633a494d to /192.168.1.66:55081]
[Host Controller] 10:13:28,472 INFO  [org.jboss.as.host.controller] (proxy-threads - 1) JBAS010919: Registering server server1
[Server:server1] 10:13:28,862 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS015400: Bound mail session [java:jboss/mail/Default]
[Host Controller] 10:13:28,874 INFO  [org.jboss.as.host.controller] (proxy-threads - 2) JBAS010919: Registering server server2
[Server:server2] 10:13:29,228 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 55) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[Server:server1] 10:13:29,225 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 55) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[Server:server1] 10:13:29,387 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 55) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[Server:server2] 10:13:29,449 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 55) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[Server:server2] 10:13:29,672 INFO  [org.apache.coyote.ajp.AjpAprProtocol] (MSC service thread 1-2) Starting Coyote AJP/1.3 on ajp-axis-into-ict.nl-192.168.1.66-9081
[Server:server1] 10:13:29,687 INFO  [org.apache.coyote.ajp.AjpAprProtocol] (MSC service thread 1-1) Starting Coyote AJP/1.3 on ajp-axis-into-ict.nl-192.168.1.66-9080
[Server:server2] 10:13:29,815 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-4) JBoss Web Services - Stack CXF Server 4.0.1.GA
[Server:server2] 10:13:29,817 INFO  [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-1) Starting Coyote HTTP/1.1 on http-axis-into-ict.nl-192.168.1.66-8081
[Server:server1] 10:13:29,839 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-3) JBoss Web Services - Stack CXF Server 4.0.1.GA
[Server:server2] 10:13:29,846 INFO  [org.jboss.as.modcluster] (MSC service thread 1-3) JBAS011704: Mod_cluster uses default load balancer provider
[Server:server1] 10:13:29,853 INFO  [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-4) Starting Coyote HTTP/1.1 on http-axis-into-ict.nl-192.168.1.66-8080
[Server:server1] 10:13:29,879 INFO  [org.jboss.as.modcluster] (MSC service thread 1-2) JBAS011704: Mod_cluster uses default load balancer provider
[Server:server1] 10:13:30,055 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/data/messagingjournal,bindingsDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/data/messagingbindings,largeMessagesDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/data/messaginglargemessages,pagingDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/data/messagingpaging)
[Server:server2] 10:13:30,068 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/data/messagingjournal,bindingsDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/data/messagingbindings,largeMessagesDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/data/messaginglargemessages,pagingDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/data/messagingpaging)
[Server:server1] 10:13:30,090 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-2) Initializing mod_cluster 1.2.0.Final
[Server:server1] 10:13:30,091 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) Waiting to obtain live lock
[Server:server2] 10:13:30,102 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) Waiting to obtain live lock
[Server:server2] 10:13:30,131 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-3) Initializing mod_cluster 1.2.0.Final
[Server:server1] 10:13:30,208 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-2) Listening to proxy advertisements on 224.0.1.105:23364
[Server:server2] 10:13:30,264 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-3) Listening to proxy advertisements on 224.0.1.105:23364
[Server:server1] 10:13:30,338 INFO  [org.hornetq.core.persistence.impl.journal.JournalStorageManager] (MSC service thread 1-3) Using AIO Journal
[Server:server2] 10:13:30,386 INFO  [org.hornetq.core.persistence.impl.journal.JournalStorageManager] (MSC service thread 1-2) Using AIO Journal
[Server:server1] 10:13:30,816 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-3) Waiting to obtain live lock
[Server:server1] 10:13:30,817 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-3) Live Server Obtained live lock
[Server:server2] 10:13:30,954 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-2) Waiting to obtain live lock
[Server:server2] 10:13:30,954 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-2) Live Server Obtained live lock
[Server:server2] 10:13:31,115 INFO  [org.jboss.as.jacorb] (MSC service thread 1-4) JBAS016330: CORBA ORB Service started
[Server:server1] 10:13:31,108 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) JBAS016330: CORBA ORB Service started
[Server:server1] 10:13:31,341 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
[Server:server2] 10:13:31,438 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
[Server:server2] 10:13:31,439 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:/jdbc/OracleDS]
[Server:server1] 10:13:31,448 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010400: Bound data source [java:/jdbc/OracleDS]
[Server:server1] 10:13:31,449 INFO  [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on axis-into-ict.nl/192.168.1.66:4447
[Server:server2] 10:13:31,462 INFO  [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100: Listening on axis-into-ict.nl/192.168.1.66:4448
[Server:server2] 10:13:31,554 INFO  [org.jboss.as.jacorb] (MSC service thread 1-4) JBAS016328: CORBA Naming Service started
[Server:server1] 10:13:31,587 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) JBAS016328: CORBA Naming Service started
[Server:server2] 10:13:32,254 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-2) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5446 for CORE protocol
[Server:server2] 10:13:32,257 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-2) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5456 for CORE protocol
[Server:server1] 10:13:32,257 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-3) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5445 for CORE protocol
[Server:server2] 10:13:32,258 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) Server is now live
[Server:server2] 10:13:32,259 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) HornetQ Server version 2.2.11.Final (HQ_2_2_11_FINAL_AS7, 122) [2b25644c-61f8-11e1-a2e3-000c2976c82d]) started
[Server:server1] 10:13:32,260 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-3) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5455 for CORE protocol
[Server:server1] 10:13:32,262 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) Server is now live
[Server:server1] 10:13:32,263 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) HornetQ Server version 2.2.11.Final (HQ_2_2_11_FINAL_AS7, 122) [2b200d12-61f8-11e1-9621-000c2976c82d]) started
[Server:server1] 10:13:32,338 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
[Server:server2] 10:13:32,342 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
[Server:server2] 10:13:32,344 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
[Server:server1] 10:13:32,343 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) trying to deploy queue jms.queue.testQueue
[Server:server2] 10:13:32,347 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) trying to deploy queue jms.queue.testQueue
[Server:server2] 10:13:32,356 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:/queue/test
[Server:server2] 10:13:32,362 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/test
[Server:server2] 10:13:32,368 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-4) trying to deploy queue jms.topic.testTopic
[Server:server1] 10:13:32,368 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/queue/test
[Server:server1] 10:13:32,370 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/test
[Server:server1] 10:13:32,371 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) trying to deploy queue jms.topic.testTopic
[Server:server1] 10:13:32,442 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-1) JBAS010406: Registered connection factory java:/JmsXA
[Server:server2] 10:13:32,445 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010406: Registered connection factory java:/JmsXA
[Server:server2] 10:13:32,468 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-2) HornetQ resource adaptor started
[Server:server1] 10:13:32,470 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/topic/test
[Server:server1] 10:13:32,472 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/topic/test
[Server:server2] 10:13:32,469 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:/topic/test
[Server:server1] 10:13:32,473 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
[Server:server1] 10:13:32,474 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
[Server:server2] 10:13:32,475 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-2) IJ020002: Deployed: file://RaActivatorhornetq-ra
[Server:server2] 10:13:32,478 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/topic/test
[Server:server2] 10:13:32,480 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
[Server:server1] 10:13:32,482 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-1) HornetQ resource adaptor started
[Server:server2] 10:13:32,483 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
[Server:server1] 10:13:32,490 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-1) IJ020002: Deployed: file://RaActivatorhornetq-ra
[Server:server1] 10:13:32,497 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
[Server:server2] 10:13:32,504 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 10041ms - Started 169 of 292 services (121 services are passive or on-demand)
[Server:server1] 10:13:32,513 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 10204ms - Started 169 of 292 services (121 services are passive or on-demand)

When starting the domain a process controller is active is as well to start the host controller and the servers. The following shows the process controller log (that is located in the ${JBOSS_HOME}/domain/log directory; in this directory the log file of the host controller is present as well)

10:13:12,861 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
10:13:13,459 INFO  [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
10:13:22,199 INFO  [org.jboss.as.process.Server:server1.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server1'
10:13:22,292 INFO  [org.jboss.as.process.Server:server2.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server2'

In a domain set-up each server automatically gets its own directory, i.e.,

${JBOSS_HOME}/domain
	/configuration
		domain.xml
		host.xml
	/servers
		/server1
			/data
			/log
		/server2
			/data
			/log

Now the question arises: which use cases are appropriate for a domain and which are appropriate for standalone servers? A domain is about coordinated multi-server management, i.e., it offers a central point through which we can manage multiple servers, with the capability to keep the configuration consistent among the servers in a particular server group. The choice between using a domain or standalone server is about how server are managed. As we saw above we can set-up a group of standalone servers to form a cluster. One thing to note here, as we look at the directory structure provided out of the box by using a domain, is that the servers get there own directory structure, i.e., we get data directories that contain among others the HornetQ message bindings and journals and log directories. Recall that in the standalone case, we had to configure this ourselves. So from a management point of view a domain would be the obvious choice when multiple servers are involved.

Once the domain controller is configured we can configure the host controllers that should join the domain. A host controller configuration requires the following steps

Define a unique name for the host controller within the domain.
Configure the management interface and interface binding on an address that is accessible to the domain controller.
Tell the host controller how to find the domain controller so it can register itself with the domain.
Define the JVM and the servers to run on the host.

A basic example of a host controller configuration looks as follows:

<host name="hostname" xmlns="urn:jboss:domain:1.0">
	<management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <authentication>
                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
                </authentication>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket interface="management" port="9999"/>
            </native-interface>
        </management-interfaces>
    </management>
	<domain-controller>
       <remote host="hostname or IP-address of the domain controller host" port="9999"/>
    </domain-controller>
    <interfaces>
        <interface name="management">
            <nic name="eth0"/>
        </interface>
    </interfaces>
	<jvms>
    	<jvm name="default">
            <heap size="512m" max-size="512m"/>
            <permgen size="256m" max-size="256m"/>
            <jvm-options>
                <option value="-server"/>
                <option value="-XX:NewRatio=2"/>
                <option value="-XX:+UseParallelGC"/>
                <option value="-XX:ParallelGCThreads=2"/>
                <option value="-XX:MaxGCPauseMillis=200"/>
                <option value="-XX:GCTimeRatio=19"/>
                <option value="-XX:+UseParallelOldGC"/>
            </jvm-options>
        </jvm>
    </jvms>
    <servers>
        <server name="server3" group="ha-server-group" auto-start="true">
			<socket-bindings port-offset="2"/>
        </server>
        <server name="server4" group="ha-server-group" auto-start="true">
            <socket-bindings port-offset="3"/>
        </server>
		<server name="server5" group="ha-server-group" auto-start="true">
            <socket-bindings port-offset="4"/>
        </server>
    </servers>
</host>

Let us deploy the same application to our server-group (ha-server-group). To this end we need to configure a data source for the database. We also want to use the Apache HTTP server for the load balancing, which means we need to configure an AJP connector and socket binding. The domain.xml example presented above already contains the these configurations. From the start-up log we can see that AJP is bound to 192.168.1.66 and listening on ports 9080 for server1 and 9081 for server2. To deploy the application we can use the command-line interface. When doing this we first need to connect to domain controller which is listening on 192.168.1.66:9999 (see log output above)

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[domain@192.168.1.66:9999 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/LoadTest6.ear --server-groups=ha-server-group

When the application gets deployed the following is observed in the logging

[Host Controller] 10:31:14,061 INFO  [org.jboss.as.repository] (management-handler-threads -  JBAS014900: Content added at location /home/jboss/jboss-as-7.1.0.Final/domain/data/content/16/1f51dde7f085c822cc4c68b306d57f1bee902d/content
[Server:server1] 10:31:14,259 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "LoadTest6.ear"
[Server:server2] 10:31:14,262 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "LoadTest6.ear"
[Server:server1] 10:31:14,315 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "Model.jar"
[Server:server2] 10:31:14,315 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "Web.war"
[Server:server2] 10:31:14,316 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "Model.jar"
[Server:server1] 10:31:14,316 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "Web.war"
[Server:server2] 10:31:14,421 INFO  [org.jboss.as.jpa] (MSC service thread 1-4) JBAS011401: Read persistence.xml for PersonPersistenceUnit
[Server:server1] 10:31:14,423 INFO  [org.jboss.as.jpa] (MSC service thread 1-4) JBAS011401: Read persistence.xml for PersonPersistenceUnit
[Server:server1] 10:31:14,528 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-3) JNDI bindings for session bean named Company in deployment unit subdeployment "Model.jar" of deployment "LoadTest6.ear" are as follows:
[Server:server1]
[Server:server1]        java:global/LoadTest6/Model/Company!model.logic.Company
[Server:server1]        java:app/Model/Company!model.logic.Company
[Server:server1]        java:module/Company!model.logic.Company
[Server:server1]        java:jboss/exported/LoadTest6/Model/Company!model.logic.Company
[Server:server1]        java:global/LoadTest6/Model/Company
[Server:server1]        java:app/Model/Company
[Server:server1]        java:module/Company
[Server:server1]
[Server:server2] 10:31:14,576 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-2) JNDI bindings for session bean named Company in deployment unit subdeployment "Model.jar" of deployment "LoadTest6.ear" are as follows:
[Server:server2]
[Server:server2]        java:global/LoadTest6/Model/Company!model.logic.Company
[Server:server2]        java:app/Model/Company!model.logic.Company
[Server:server2]        java:module/Company!model.logic.Company
[Server:server2]        java:jboss/exported/LoadTest6/Model/Company!model.logic.Company
[Server:server2]        java:global/LoadTest6/Model/Company
[Server:server2]        java:app/Model/Company
[Server:server2]        java:module/Company
[Server:server2]
[Server:server2] 10:31:15,167 INFO  [org.jboss.as.jpa] (MSC service thread 1-4) JBAS011402: Starting Persistence Unit Service 'LoadTest6.ear/Model.jar#PersonPersistenceUnit'
[Server:server1] 10:31:15,182 INFO  [org.jboss.as.jpa] (MSC service thread 1-3) JBAS011402: Starting Persistence Unit Service 'LoadTest6.ear/Model.jar#PersonPersistenceUnit'
[Server:server2] 10:31:15,377 INFO  [org.jboss.as.ejb3] (MSC service thread 1-1) JBAS014142: Started message driven bean 'CompanyMDB' with 'hornetq-ra' resource adapter
[Server:server1] 10:31:15,381 INFO  [org.jboss.as.ejb3] (MSC service thread 1-4) JBAS014142: Started message driven bean 'CompanyMDB' with 'hornetq-ra' resource adapter
[Server:server2] 10:31:15,558 INFO  [org.jboss.web] (MSC service thread 1-3) JBAS018210: Registering web context: /LoadTest6
[Server:server1] 10:31:15,563 INFO  [org.jboss.web] (MSC service thread 1-2) JBAS018210: Registering web context: /LoadTest6
[Server:server2] 10:31:15,613 INFO  [org.hibernate.annotations.common.Version] (MSC service thread 1-4) HCANN000001: Hibernate Commons Annotations {4.0.1.Final}
[Server:server1] 10:31:15,613 INFO  [org.hibernate.annotations.common.Version] (MSC service thread 1-3) HCANN000001: Hibernate Commons Annotations {4.0.1.Final}
[Server:server1] 10:31:15,622 INFO  [org.hibernate.Version] (MSC service thread 1-3) HHH000412: Hibernate Core {4.0.1.Final}
[Server:server2] 10:31:15,622 INFO  [org.hibernate.Version] (MSC service thread 1-4) HHH000412: Hibernate Core {4.0.1.Final}
[Server:server1] 10:31:15,624 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-3) HHH000206: hibernate.properties not found
[Server:server2] 10:31:15,626 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-4) HHH000206: hibernate.properties not found
[Server:server1] 10:31:15,626 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-3) HHH000021: Bytecode provider name : javassist
[Server:server2] 10:31:15,628 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-4) HHH000021: Bytecode provider name : javassist
[Server:server1] 10:31:15,680 INFO  [org.hibernate.ejb.Ejb3Configuration] (MSC service thread 1-3) HHH000204: Processing PersistenceUnitInfo [
[Server:server1]        name: PersonPersistenceUnit
[Server:server1]        ...]
[Server:server2] 10:31:15,682 INFO  [org.hibernate.ejb.Ejb3Configuration] (MSC service thread 1-4) HHH000204: Processing PersistenceUnitInfo [
[Server:server2]        name: PersonPersistenceUnit
[Server:server2]        ...]
[Server:server2] 10:31:15,879 INFO  [org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator] (MSC service thread 1-4) HHH000130: Instantiating explicit connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
[Server:server1] 10:31:15,882 INFO  [org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator] (MSC service thread 1-3) HHH000130: Instantiating explicit connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
[Server:server1] 10:31:16,034 INFO  [org.hibernate.dialect.Dialect] (MSC service thread 1-3) HHH000400: Using dialect: org.hibernate.dialect.Oracle10gDialect
[Server:server2] 10:31:16,034 INFO  [org.hibernate.dialect.Dialect] (MSC service thread 1-4) HHH000400: Using dialect: org.hibernate.dialect.Oracle10gDialect
[Server:server2] 10:31:16,052 INFO  [org.hibernate.engine.transaction.internal.TransactionFactoryInitiator] (MSC service thread 1-4) HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jta.CMTTransactionFactory
[Server:server1] 10:31:16,052 INFO  [org.hibernate.engine.transaction.internal.TransactionFactoryInitiator] (MSC service thread 1-3) HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jta.CMTTransactionFactory
[Server:server1] 10:31:16,061 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (MSC service thread 1-3) HHH000397: Using ASTQueryTranslatorFactory
[Server:server2] 10:31:16,062 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (MSC service thread 1-4) HHH000397: Using ASTQueryTranslatorFactory
[Server:server2] 10:31:16,144 INFO  [org.hibernate.validator.util.Version] (MSC service thread 1-4) Hibernate Validator 4.2.0.Final
[Server:server1] 10:31:16,145 INFO  [org.hibernate.validator.util.Version] (MSC service thread 1-3) Hibernate Validator 4.2.0.Final
[Server:server1] 10:31:17,007 INFO  [org.jboss.as.server] (host-controller-connection-threads - 3) JBAS018559: Deployed "LoadTest6.ear"
[Server:server2] 10:31:17,007 INFO  [org.jboss.as.server] (host-controller-connection-threads - 3) JBAS018559: Deployed "LoadTest6.ear"

As the servers are listening on the same IP addresses and AJP ports as in the standalone case, we do not need to edit anything in the Apache HTTP server configuration. Start the Apache HTTP server and enter the URL: http://hostname:8888/LoadTest6/testservlet and make a few requests to see if everything is working. Open the admin console (http://192.168.1.66:9990/console) and enter the necessary credentials to log in (these we have created in the beginning by using add-user.sh). In the admin console, click server instances and select a particular server. Subsequently, click web to see how many requests were processed by the AJP connector. Let us perform the load test again to see how the environment is performing. When using the jvisualvm we need to know the process IDs of the servers within the domain. To this end we can use the ps command to get information on the running Java process

[jboss@axis-into-ict ~]$ ps -ef|grep java
jboss    7277  7263  0 10:13 pts/2    00:00:06 /home/jboss/jdk1.6.0_31/bin/java -D[Process Controller] -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/log/process-controller.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -jar /home/jboss/jboss-as-7.1.0.Final/jboss-modules.jar -mp /home/jboss/jboss-as-7.1.0.Final/modules org.jboss.as.process-controller -jboss-home /home/jboss/jboss-as-7.1.0.Final -jvm /home/jboss/jdk1.6.0_31/bin/java -- -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/log/host-controller.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -- -default-jvm /home/jboss/jdk1.6.0_31/bin/java
jboss    7294  7277  0 10:13 pts/2    00:00:11 /home/jboss/jdk1.6.0_31/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/log/host-controller.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -jar /home/jboss/jboss-as-7.1.0.Final/jboss-modules.jar -mp /home/jboss/jboss-as-7.1.0.Final/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller --pc-address localhost.localdomain --pc-port 60207 -default-jvm /home/jboss/jdk1.6.0_31/bin/java -Djboss.home.dir=/home/jboss/jboss-as-7.1.0.Final
jboss    7347  7277  9 10:13 pts/2    00:06:43 /home/jboss/jdk1.6.0_31/bin/java -D[Server:server1] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms512m -Xmx512m -server -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djboss.host.default.config=host.xml -D[Host=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.home.dir=/home/jboss/jboss-as-7.1.0.Final -Djboss.domain.default.config=domain.xml -Djava.net.preferIPv4Stack=true -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/log/boot.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -jar jboss-modules.jar -mp modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server
jboss    7364  7277  9 10:13 pts/2    00:06:44 /home/jboss/jdk1.6.0_31/bin/java -D[Server:server2] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms512m -Xmx512m -server -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djboss.host.default.config=host.xml -D[Host=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.home.dir=/home/jboss/jboss-as-7.1.0.Final -Djboss.domain.default.config=domain.xml -Djava.net.preferIPv4Stack=true -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/log/boot.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -jar jboss-modules.jar -mp modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server

As we can see the first one (7277) is the process controller; the second (7294) is the host controller; the third (7347) and fourth (7364) are respectively server1 and server2. The load test statistics are as follows

To get request count and processing time information we use the command-line interface

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[domain@192.168.1.66:9999 /] cd host=axis-into-ict.nl/server=server1
[domain@192.168.1.66:9999 server=server1] cd deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web
[domain@192.168.1.66:9999 subsystem=web] cd servlet=TestServlet
[domain@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
394063
[domain@192.168.1.66:9999 servlet=TestServlet] read-attribute processing-time
930637L
[domain@192.168.1.66:9999 servlet=TestServlet] cd /
[domain@192.168.1.66:9999 /] cd host=axis-into-ict.nl/server=server2
[domain@192.168.1.66:9999 server=server2] cd deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web
[domain@192.168.1.66:9999 subsystem=web] cd servlet=TestServlet
[domain@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
394056
[domain@192.168.1.66:9999 servlet=TestServlet] read-attribute processing-time
930302L

To see how the JVMs are doing during the load test we use jvisualvm

The test introduced no particular problems for the JVM.

References

[1] Marchioni, “JBoss AS7 Configuration, Deployment and Administration”, PACKT Publishing, Birmingham, UK, 2011. Essential reading material.
[2] JBoss Documentation.

configuration, installation, JBoss AS7 Leave a Comment more...

Feb/12

Bring Your Own Laptop Knowledge Sessions: WebLogic 12c

by René van Wijk under Cluster, Coherence, DataSource, Deploy, JMS, JRockit, JVM Tuning, Monitoring, NodeManager, Security, WLST, WorkManager

René van Wijk

Axis into ICT offers you the opportunity to increase your skills. We are organizing ‘Bring Your Own Laptop Knowledge Sessions’. In a small group of up to 8 people we are going discuss all the practical aspects of WebLogic Server you ever wanted to know.

This is not a standard course, but a training where applying the material in practice is of importance. All participants will receive their own virtual machine, which offers you to ability to continue afterwards with your own practice environment. By keeping the groups small we create an informal atmosphere with plenty of room for all your questions or to even discuss your specific situation. The approach is highly interactive; after all you are attending to increase your knowledge.

Topics that will be covered

Introduction
- WebLogic Server architecture (domain, admin server, managed server, node manager, cluster, machine)
- WebLogic Server internal architecture (listen threads, socket muxer, execute queue, execute threads)
JVM Tuning
- Code generation
- Memory management
- Garbage collection performance
- JRockit Mission Control
Deployment
- Packaging applications recommendations
- Deployment descriptors
- Deployment plans
- Work Manager configuration
Diagnostic Framework (logging, instrumentation, harvesting, watching and notifying)
Class Loading
- Bundled libraries
- Shared libraries
- Filtering class loader
Security
- JAAS introduction
- Embedded LDAP
- WebLogic Server security providers
- Secure communications (SSL and TLS)
Configure Resources
- Configure data sources
- Monitoring resources
- Configure JMS environment
- Migrating singleton services (leasing, migratable targets, service migration policy)
Clustering
- Unicast or multicast?
- Node manager and machine set-up
- Denial of service configuration
- Network channels
- Vertical and horizontal scaling
- Caching by using Coherence
- Apache HTTP Server and WebLogic plug-in set-up
Scripting (WLST)

Registration

If you are interested in attending the ‘Bring Your Own Laptop Knowledge Sessions: WebLogic 12c’, you can register here.

training WebLogic Server 3 Comments more...

Feb/12

WebLogic, SOA Suite and Access Manager

by René van Wijk under Cluster, NodeManager, Security, WLST

René van Wijk

Sort of a deployment guide to integrate the products WebLogic Server, Oracle SOA Suite and Oracle Access Manager. We start with preparing our operating system. Next, we create the repository needed for the SOA Suite. Before we install the software, we first set-up a directory structure that is easy to back-up. Next, we install the software for WebLogic Server and SOA Suite after which we configure an admin server and a cluster for the SOA Suite. Subsequently, we show the steps involved to test the environment and show how the JVM parameters can be set for individual servers. We show the steps involved how the SOA Suite cluster can be scaled out to different machines. To create start and stop scripts we use WLST. After creating the start and stop scripts, we set-up a load balancer by using the WebLogic Server proxy plug-in. For this purpose we install and configure Web Tier components. To secure the environment we will use access manager and set-up a WebGate instance on the configured Web Tier components.

Prepare the environment

We first create a user, for example, oracle. To create a user run the following commands under the root user:

useradd oracle
passwd magic11g

This creates a /home/oracle directory. To delete the user we can use: userdel -r oracle the option -r removes the /home/oracle directory.

Maximum number of open file descriptors. Most operating systems handle sockets as a form of file access and use file descriptors to keep track of which sockets are open. To contain the resources per process, the operating system restricts the number of file descriptors per process. Linux limits the number of open file descriptors per process, by default this is equal to 1024. It could be that the 1024 limit does not offer optimal performance. To obtain the current open file limit we can use the ulimit command. The ulimit -aS command shows the current limit, the ulimit -aH command shows the hard limit. To increase the total number of file descriptors for all users we can add the following to the /etc/sysctl.conf file: fs.file-max = 344030. To check the current value we can use cat /proc/sys/fs/file-max. If we want to make sure that a process (running under the oracle user) gets a maximum of 4096 open file descriptors, we have to edit the /etc/security/limits.conf file:

oracle soft nofile 4096
oracle hard nofile 4096

Note that this number applies across all processes running under the oracle user. To make the adjustments active, execute the ulimit -u 4096 command and log out and log in again.

Required libraries. During the installation a check is performed that concerns a library version check. When failures occur during this step, we must correct them before continuing. The log file /home/oracle/oraInventory/logs/install<date-time>.out shows which part of the check failed. Usually, it means we have to install the updated version, by using, for example,

cd /etc/yum.repos.d/
wget http://yum.hostname.com/repos/filename.repo
yum clean all
yum install gcc
yum install gcc-c++
yum install apr
yum install elfutils-libelf-devel
yum install libaio-devel
yum install sysstat
yum install glibc-devel

More information on the system requirements can be found here.

We use the following versions in the installation:

Repository Creation Utility 11.1.1.5.0 (ofm_rcu_linux_11.1.1.5.0_disk1_1of1.zip)
JRockit JDK 28.2.2 (jrockit-jdk1.6.0_29-R28.2.2-4.1.0-linux-x64.bin)
WebLogic 10.3.5 (wls1035_generic.jar)
SOA Suite 11.1.1.5.0 (ofm_soa_generic_11.1.1.5.0_disk1_1of2.zip and ofm_soa_generic_11.1.1.5.0_disk1_2of2.zip)
Access Manager 11.1.1.5.0 (ofm_iam_generic_11.1.1.5.0_disk1_1of1.zip)
Web Tier 11.1.1.2.0_64 (ofm_webtier_linux_11.1.1.2.0_64_disk1_1of1.zip)
Web Tier Patch 11.1.1.3.0_64 (ofm_webtier_linux_11.1.1.3.0_64_disk1_1of1.zip)
Web Gate 11.1.1.5.0 (ofm_webgates_generic_11.1.1.5.0_disk1_1of1.zip)

Setting up the SOA repository

Before we run the repository creation utility, make sure an 11.2 database or higher is installed:

Download the distribution of Oracle 11g database (for example, win32_11gR2_database_1of2.zip and win32_11gR2_database_2of2.zip, unzip the files and run the installer).
De-select the security update option if you do not wish to receive any updates.
Select create and configure a database and click next.
Select desktop class and click next.
Enter the following parameters:
- oracle base: /home/oracle/Oracle
- software location: /home/oracle/Oracle/product/11.2.0/dbhome_1
- database file location: /home/oracle/Oracle/oradata
- database edition: enterprise edition
- character set: unicode (al32utf8)
- global database name: orcl11
- administrative password: magic11g
- confirm password: magic11g
Click next and in the pop-up yes.
Click finish.

The steps for creating a SOA repository are as follows:

Unzip ofm_rcu_linux_11.1.1.5.0_disk1_1of1.zip.
Run the installer (./rcu), which is located in the /rcuHome/bin directory.
Click next on the welcome screen.
Choose create and click next.
Enter the following parameters:
- database type: Oracle Database
- host name: hostname or IP-address of the database server
- port: 1521
- service name: orcl11
- username: sys
- password: magic11g
- role: sysdba
Click next and click OK in the pop-up.
Enter the following parameters:
- create a new prefix: DEV
- open the soa and bpm infrastructure tree and select soa infrastructure (note that user messaging service and meta-data services are automatically selected)
Click next and OK in the pop-up.
Select use the same password for all schemas, for example, magic11g.
Click next.
Accept the defaults in the tablespace screen, click next and click OK in the pop-up.
Click OK, click create and when the creation is finished click close.

Installation

First, we define an installation directory structure, for example,

/home/oracle
	/jrockit-jdk1.6.0_29-R28.2.2-4.1.0 (created during the installation of JRockit)
	/oraInventory (created during the installation of Oracle SOA Suite)
	/soa
		/installation (directory which will contain all the software)
			/coherence_3.6 (created during the installation of WebLogic Server)
			/oracle_access_manager (created during the installation of Oracle Access Manager)
			/oracle_common (created during the installation of Oracle SOA Suite)
			/oracle_soa_suite (created during the installation of Oracle SOA Suite)
			/oracle_webgate (created during the installation of Oracle WebGate only on machine 192.168.1.66)
			/oracle_webtier (created during the installation of Oracle WebTier Components only on machine 192.168.1.66)
			/wlserver_10.3 (created during the installation of WebLogic Server)
		/configuration
			/applications
				/soa_domain (created during the configuration of Oracle SOA Suite)
			/domains
				/soa_domain (created during the configuration of Oracle SOA Suite)
			/instances
				/webtier_instance (created during the configuration of Oracle WebTier only on machine 192.168.1.66)
			/nodemanagers
				/soa_domain (created by the startNodeManager.sh script)

Here /home/oracle/soa/installation will be our middleware home in the installation of WebLogic. To back-up the configuration we can use tar -cvpf /home/oracle/temp/backup.tar /home/oracle/soa/configuration.

To install JRockit we follow these steps:

Change the execution right when necessary, for example, chmod ug+x jrockit-jdk1.6.0_29-R28.2.2-4.1.0-linux-x64.bin.
Run ./jrockit-jdk1.6.0_29-R28.2.2-4.1.0-linux-x64.bin .
Click next on the welcome screen.
Define the install directory /home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0 and click next.
Optionally select extra components and click next.
When the installation is finished click done.

To install WebLogic we follow these steps:

Navigate to the JDK’s bin directory (/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0).
Enter the following command: ./java -d64 -Xms1024m -Xmx1024m -jar wls1035_generic.jar.
Click next on the welcome screen.
Define the middleware home directory (/home/oracle/soa/installation) and click next.
Fill in support credentials if applicable and click next.
Select custom as installation type and click next.
De-select the evaluation database and click next.
Select the installed JDK (if the JDK is not shown, click browse and browse to the root of the JDK) and click next.
Accept the defaults in the directories screen and click next.
Click next to start the installation.
De-select run quickstart and click done.

To uninstall WebLogic we can run uninstall.sh located in the /home/oracle/soa/installation/wlserver_10.3/uninstall directory.

To install the SOA Suite we follow these steps:

Unzip ofm_soa_generic_11.1.1.5.0_disk1_1of2.zip and ofm_soa_generic_11.1.1.5.0_disk1_2of2.zip to /home/oracle/temp/soa.
Navigate to Disk1 and run the installer (./runInstaller).
Specify the inventory directory and click OK.
Open a new terminal and run createCentralInventory.sh as root.

[oracle@axis-into-ict ~]$ su root
Password:
[root@axis-into-ict oracle]# cd /home/oracle/oraInventory/
[root@axis-into-ict oraInventory]# ./createCentralInventory.sh
Setting the inventory to /home/oracle/oraInventory
Setting the group name to oracle
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /home/oracle/oraInventory to 770.
Changing groupname of /home/oracle/oraInventory to oracle.
The execution of the script is complete

Click OK in the pop-up
Click next in the welcome screen.
Enter the credentials of software updates when applicable and click next.
Prerequisite checks are run, click next when all the prerequisite succeed.
Enter the following parameters:
- Oracle middleware home: /home/oracle/soa/installation
- Oracle home directory: oracle_soa_suite
Click next and select the application server, in our case we choose WebLogic Server.
Click next.
Check the install summary and click install.
Click next when the installation finishes and click finish.

To uninstall a fusion middleware installation, such as SOA Suite, we can run ./runInstaller -deinstall.

Configuration

Create an admin server

Run the configuration wizard (./config.sh) located in the ${MIDDLEWARE_HOME}/oracle_common/common/bin directory.
On the welcome screen select create a new weblogic domain.
Click next and select the following products:
- basic weblogic server domain (automatically selected)
- oracle enterprise manager – 11.1.1.0 [oracle_common]
- oracle jrf – 11.1.1.0 [oracle_common] (automatically selected when oracle enterprise manager is selected)
Click next and enter the following parameters:
- domain name: soa_domain
- domain location: /home/oracle/soa/configuration/domains
- application location: /home/oracle/soa/configuration/applications
Click next and enter the admin username and password.
Click next, select production mode and select the JDK.
Click next, no need to select anything, click next again, check the configuration summary and click create.
Click done.

Create a SOA cluster

Run the configuration wizard again.
Select extend an existing weblogic domain and click next.
Select the domain directory: /home/oracle/soa/configuration/domains/soa_domain.
Click next and select the following products:
- oracle soa suite – 11.1.1.0 [oracle_soa_suite]
- oracle wsm policy manager – 11.1.1.0 [oracle_common] (automatically selected when oracle soa suite is selected)
Click next, select all schemas and enter the following parameters:
- vendor: Oracle
- driver: Oracle’s driver (thin) for service connections
- schema owner: varies among component schemas (the defaults are ok)
- schema password: magic11g
- dbms/service: orcl11
- hostname: hostname or IP-address of the database server
- port: 1521
Click next and when the test succeeds click next again.
Select the options: jms distributed destinations, managed servers clusters and machines, deployments and services, JMS file stores.
Click next.
Make sure the UDD (uniform distributed destination) option is selected for all JMS system resources.
Click next (click OK in the pop-up the cluster will be configured later).
Configure a managed server using the following parameters:
- name: soa_server1
- listen address: 192.168.1.66
- listen port: 8001
Click next.
Configure a cluster, click add and enter the following parameters:
- name: soa_cluster
- clustering messaging mode: unicast
Click next and add soa_server1 to the soa_cluster and click next
Configure a machine, delete the localmachine, click on the unix machine, click add and enter the following parameters:
- name: soa_machine1
- post bind UID enables: select this option
- post bind UID: oracle
- nodemanager listen address: 192.168.1.66
- nodemanager listen port: 5556
Click next and soa_server1 to soa_machine1 and click next.
Target deployments to clusters and servers:
- target oracle.wsm.*, oracle.soa.* and oracle.sdp.* only to the soa_cluster
Click next.
Target services to cluster or servers:
- target JOC-Startup and JOC-Shutdown only to the soa_cluster
Click next and configure the JMS file stores:
- name: UMSJMSFileStore1 – directory: /home/oracle/soa/configuration/application/soa_domain
- name: BPMJMSFileStore1 – directory: /home/oracle/soa/configuration/application/soa_domain
- name: SOAJMSFileStore1 – directory: /home/oracle/soa/configuration/application/soa_domain
Click next, check the summary and click extend.
Click done.

Test the environment

Add a boot.properties file in the ${DOMAIN_HOME}/servers/AdminServer/security directory (in which ${DOMAIN_HOME} is /home/oracle/soa/configuration/domains/soa_domain):

[oracle@axis-into-ict ~]$ cd soa/configuration/domains/soa_domain/servers/AdminServer/
[oracle@axis-into-ict AdminServer]$ mkdir security
[oracle@axis-into-ict AdminServer]$ vi boot.properties

Add the following name-value pairs to the boot.properties file:

username=weblogic
password=magic11g

Change the JVM parameters for the admin server and the managed servers. Edit the setSOADomainEnv.sh file (located in the ${DOMAIN_HOME}/bin directory, locate the DEFAULT_MEM_ARGS and PORT_MEM_ARGS and make the following alterations:

#DEFAULT_MEM_ARGS="-Xms512m -Xmx1024m"
#PORT_MEM_ARGS="-Xms768m -Xmx1536m"

# memory arguments for the admin server
if [ "${SERVER_NAME}" = "AdminServer" ] ; then
	DEFAULT_MEM_ARGS="-Xms1024m -Xmx1024m -Xns256m -Xgc:throughput"
	PORT_MEM_ARGS="-Xms1024m -Xmx1024m -Xns256m -Xgc:throughput"
fi

# memory argument for soa_server1
if [ "${SERVER_NAME}" = "soa_server1" ] ; then
	DEFAULT_MEM_ARGS="-Xms1024m -Xmx1024m -Xns256m -Xgc:throughput"
	PORT_MEM_ARGS="-Xms1024m -Xmx1024m -Xns256m -Xgc:throughput"
fi

# memory arguments for soa_server2 which will be added in a later stadium
if [ "${SERVER_NAME}" = "soa_server2" ] ; then
	DEFAULT_MEM_ARGS="-Xms1024m -Xmx1024m -Xns256m -Xgc:throughput"
	PORT_MEM_ARGS="-Xms1024m -Xmx1024m -Xns256m -Xgc:throughput"
fi

# memory arguments for oam_server1 which will be added in a later stadium
if [ "${SERVER_NAME}" = "oam_server1" ] ; then
	DEFAULT_MEM_ARGS="-Xms1024m -Xmx1024m -Xns256m -Xgc:throughput"
	PORT_MEM_ARGS="-Xms1024m -Xmx1024m -Xns256m -Xgc:throughput"
fi

Start the admin server by using ./startWebLogic.sh, which is located in the ${DOMAIN_HOME} directory. When the admin server is running open the admin console http://hostname:7001/console, edit the nodemanager username and password, and disable hostname verification. To change the nodemanager username and password

Click soa_domain, security, advanced link and enter the following parameters:
- NodeManager Username: weblogic
- NodeManager Password: magic11g
- Confirm NodeManager Password: magic11g

Click save and activate the changes.

To disable the hostname verification:

Click environment, servers, admin server.
Click on the SSL, configuration tab, subsequently on the advanced link and enter the following parameters:
- Hostname Verification: none
Click save and activate the changes.

In order for the changes to take effect the admin server must be restarted. Next, we create a new node manager home. To this end open the startNodeManager.sh file (located in the ${WL_HOME}/server/bin directory, in which ${WL_HOME} is /home/oracle/soa/installation/wlserver_10.3) and edit the NODEMGR_HOME property, i.e., set this property to:

WL_HOME="/home/oracle/soa/installation/wlserver_10.3"
. "${WL_HOME}/common/bin/commEnv.sh"

NODEMGR_HOME="/home/oracle/soa/configuration/nodemanagers/soa_domain"

# If NODEMGR_HOME does not exist, create it

Run ./startNodeManager.sh in order to create the nodemanager.properties file and stop the nodemanager again (ctrl+c). Open the nodemanager.properties file (located in the specified ${NODEMGR_HOME} directory) and set the property StartScriptEnabled to true. Create a nodemanager.domains file in the ${NODEMGR_HOME} directory and add the following key-value pair:

soa_domain=/home/oracle/soa/configuration/domains/soa_domain

Note that when the configuration wizard is run such a file is created in the ${WL_HOME}/common/nodemanager directory. Start the nodemanager by using ./startNodeManager.sh. Next, use to admin console to start the managed server soa_server1.

Scaling

Let us add an extra server on a different machine. Open the admin console and clone soa_server1 and set the server name to soa_server2, server listen address to 192.168.1.67 and server listen port to 9001. Delete soa_server2 from soa_machine1. Create a new machine:

Click machine, new and enter the following parameters:
- name: soa_machine2
- machine os: unix
Click next and enter the nodemanager parameters:
- type: ssl
- listen address: 192.168.1.67
- listen port: 5556
Click finish
Click soa_machine2, select enable post bind uid and set post bind uid to the user oracle
Click save
Click servers tab and add soa_server2

Create persistent stores for BPM, SOA and UMS, for example, name: BPMJMSFileStore2, target: soa_server2, directory: /home/oracle/soa/configuration/applications/soa_domain. Subsequently, create corresponding JMS Servers and target these to soa_server2. Next, we have to edit the subdeployments of the JMS modules and add the appropriate JMS server to the subdeployment. After the subdeployments have been adjusted, check if the targets of the uniform distributed queues show both the JMS servers.

Pack the domain by using pack.sh, for example,

[oracle@axis-into-ict ~]$ cd soa/installation/wlserver_10.3/common/bin/
[oracle@axis-into-ict bin]$ ./pack.sh -managed=true -domain=/home/oracle/soa/configuration/domains/soa_domain -template=/home/oracle/temp/soa_domain.jar -template_name=soa_domain
<< read domain from "/home/oracle/soa/configuration/domains/soa_domain"
>>  succeed: read domain from "/home/oracle/soa/configuration/domains/soa_domain"
<< set config option Managed to "true"
>>  succeed: set config option Managed to "true"
<< write template to "/home/oracle/temp/soa_domain.jar"
....................................................................................................
>>  succeed: write template to "/home/oracle/temp/soa_domain.jar"
<< close template
>>  succeed: close template

Copy the create template to the other machine by using

[oracle@axis-into-ict bin]$ scp oracle@192.168.1.66:/home/oracle/temp/soa_domain.jar oracle@192.168.1.67:/home/oracle/temp/
oracle@192.168.1.66's password:
oracle@192.168.1.67's password:
soa_domain.jar                                                   100%  512KB 512.3KB/s   00:00
Connection to 192.168.1.66 closed.

Install the software on the soa_machine2 (192.168.1.67) by using the same steps as presented in the Installation section.

Unpack the domain by using unpack.sh, for example,

[oracle@middleware-magic ~]$ cd soa/installation/wlserver_10.3/common/bin/
[oracle@middleware-magic bin]$ ./unpack.sh -domain=/home/oracle/soa/configuration/domains/soa_domain -template=/home/oracle/temp/soa_domain.jar -app_dir=/home/oracle/soa/configuration/applications/soa_domain
<< read template from "/home/oracle/temp/soa_domain.jar"
>>  succeed: read template from "/home/oracle/temp/soa_domain.jar"
<< set config option AppDir to "/home/oracle/soa/configuration/applications/soa_domain"
>>  succeed: set config option AppDir to "/home/oracle/soa/configuration/applications/soa_domain"
<< set config option DomainName to "soa_domain"
>>  succeed: set config option DomainName to "soa_domain"
<< write Domain to "/home/oracle/soa/configuration/domains/soa_domain"
...............................................................................................
>>  succeed: write Domain to "/home/oracle/soa/configuration/domains/soa_domain"
<< close template
>>  succeed: close template

Configure the nodemanager by using the nodemanager configuration steps presented in the configuration section. Test if soa_server2 can be started by using the admin console.

Start and stop scripts

To create start and stop scripts we will use WLST. First, we define a properties file (environment.properties) that contains the environment settings:

domain_name=soa_domain
domain_home=/home/oracle/soa/configuration/domains/soa_domain
listen_address_machine1=192.168.1.66
listen_address_machine2=192.168.1.67
node_manager_listen_port=5556
node_manager_listen_address=192.168.1.66 (for soa_machine2 this must be set to 192.168.1.67)
node_manager_home=/home/oracle/soa/configuration/nodemanagers/soa_domain
admin_username=weblogic
admin_password=magic11g
admin_server_listen_port=7001
admin_server_url=t3://192.168.1.66:7001

To start the nodemanager (startNodeManager.py) we can use:

startNodeManager(verbose='true', NodeManagerHome=node_manager_home, ListenPort=node_manager_listen_port, ListenAddress=node_manager_listen_address);

To stop the nodemanager (stopNodeManager.py) we can use:

nmConnect(admin_username, admin_password, node_manager_listen_address, node_manager_listen_port, domain_name, domain_home, 'ssl');
stopNodeManager();

To start all the servers present in the domain (startDomain.py) we can use:

print 'CONNECT TO NODE MANAGER ON MACHINE1';
nmConnect(admin_username, admin_password, listen_address_machine1, node_manager_listen_port, domain_name, domain_home, 'ssl');

print 'START ADMIN SERVER ON MACHINE1';
nmStart('AdminServer');

print 'CONNECT TO ADMIN SERVER';
connect(admin_username, admin_password, admin_server_url);

print 'START MANAGED SERVERS ON MACHINE1';
start('soa_server1','Server');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE1';
nmDisconnect();

print 'CONNECT TO NODE MANAGER ON MACHINE2';
nmConnect(admin_username, admin_password, listen_address_machine2, node_manager_listen_port, domain_name, domain_home, 'ssl');

print 'START MANAGED SERVERS ON MACHINE2';
start('soa_server2','Server');
start('oam_server1','Server');

print 'DISCONNECT FROM THE ADMIN SERVER';
disconnect();

print 'DISCONNECT FROM NODE MANAGER ON MACHINE2';
nmDisconnect();

To stop all the servers present in the domain (stopDomain.py) we can use:

print 'CONNECT TO NODE MANAGER ON MACHINE2';
nmConnect(admin_username, admin_password, listen_address_machine2, node_manager_listen_port, domain_name, domain_home, 'ssl');

print 'CONNECT TO ADMIN SERVER';
connect(admin_username, admin_password, admin_server_url);

print 'STOPPING MANAGED SERVERS ON MACHINE2';
shutdown('oam_server1','Server','true',1000,'true');
shutdown('soa_server2','Server','true',1000,'true');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE2';
nmDisconnect();

print 'CONNECT TO NODE MANAGER ON MACHINE1';
nmConnect(admin_username, admin_password, listen_address_machine1, node_manager_listen_port, domain_name, domain_home, 'ssl');

print 'STOPPING MANAGED SERVERS ON MACHINE1';
shutdown('soa_server1','Server','true',1000,'true');

print 'STOPPING ADMIN SERVER ON MACHINE1';
shutdown('AdminServer','Server','true',1000,'true');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE1';
nmDisconnect();

Next, we create shell scripts that run the created WLST scripts (note the use of the loadProperties option, which enables us to define properties that will become available automatically in the WLST script that will be run):

#!/bin/sh

SCRIPTS_HOME="/home/oracle/soa/configuration/applications/soa_domain/scripts"
export SCRIPTS_HOME

WL_HOME="/home/oracle/soa/installation/wlserver_10.3"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh -loadProperties ${SCRIPTS_HOME}/environment.properties ${SCRIPTS_HOME}/startDomain.py

To start-up the servers, we first start the nodemanager on both machines (below the output is shown for soa_machine1):

[oracle@axis-into-ict scripts]$ ./NodeManagerStartService.sh
CLASSPATH=/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:

PATH=/home/oracle/soa/installation/wlserver_10.3/server/bin:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/bin:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/bin:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/oracle/bin

Your environment has been set.

CLASSPATH=/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/soa/installation/utils/config/10.3/config-launch.jar::/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbynet.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbyclient.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Launching NodeManager ...
 Properties: {ListenAddress=192.168.1.66,ListenPort=5556,NodeManagerHome=/home/oracle/soa/configuration/nodemanagers/soa_domain,}
 Command: /home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/bin/java -classpath /home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/lib/rt.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/lib/i18n.jar:/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/home/oracle/soa/installation/utils/config/10.3/config-launch.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbynet.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbyclient.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbytools.jar weblogic.NodeManager -v
NMProcess: <Feb 07, 2012 3:44:54 PM> <INFO> <Loading domains file: /home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.domains>
NMProcess: Feb 07, 2012 3:44:54 PM weblogic.nodemanager.server.NMServerConfig initDomainsMap
NMProcess: INFO: Loading domains file: /home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.domains
NMProcess: <Feb 07, 2012 3:44:54 PM> <INFO> <Loading identity key store: FileName=/home/oracle/soa/installation/wlserver_10.3/server/lib/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
NMProcess: Feb 07, 2012 3:44:54 PM weblogic.nodemanager.server.SSLConfig loadKeyStoreConfig
NMProcess: INFO: Loading identity key store: FileName=/home/oracle/soa/installation/wlserver_10.3/server/lib/DemoIdentity.jks, Type=jks, PassPhraseUsed=true
NMProcess: <Feb 07, 2012 3:44:54 PM> <INFO> <Loaded node manager configuration properties from '/home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.properties'>
NMProcess: Feb 07, 2012 3:44:54 PM weblogic.nodemanager.server.NMServer <init>
NMProcess: INFO: Loaded node manager configuration properties from '/home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.properties'
NMProcess: Node manager v10.3
NMProcess:
NMProcess: Configuration settings:
NMProcess:
NMProcess: NodeManagerHome=/home/oracle/soa/configuration/nodemanagers/soa_domain
NMProcess: ListenAddress=192.168.1.66
NMProcess: ListenPort=5556
NMProcess: ListenBacklog=50
NMProcess: SecureListener=true
NMProcess: AuthenticationEnabled=true
NMProcess: NativeVersionEnabled=true
NMProcess: CrashRecoveryEnabled=false
NMProcess: JavaHome=/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre
NMProcess: StartScriptEnabled=true
NMProcess: StopScriptEnabled=false
NMProcess: StartScriptName=startWebLogic.sh
NMProcess: StopScriptName=
NMProcess: LogFile=/home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.log
NMProcess: LogLevel=INFO
NMProcess: LogLimit=0
NMProcess: LogCount=1
NMProcess: LogAppend=true
NMProcess: LogToStderr=true
NMProcess: LogFormatter=weblogic.nodemanager.server.LogFormatter
NMProcess: DomainsFile=/home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.domains
NMProcess: DomainsFileEnabled=true
NMProcess: StateCheckInterval=500
NMProcess: QuitEnabled=true
NMProcess: UseMACBroadcast=false
NMProcess: DomainRegistrationEnabled=false
NMProcess: DomainsDirRemoteSharingEnabled=false
NMProcess:
NMProcess: Domain name mappings:
NMProcess:
NMProcess: soa_domain -> /home/oracle/soa/configuration/domains/soa_domain
NMProcess:
NMProcess: <Feb 07, 2012 3:44:55 PM> <INFO> <soa_domain> <soa_server1> <Startup configuration properties loaded from "/home/oracle/soa/configuration/domains/soa_domain/servers/soa_server1/data/nodemanager/startup.properties">
NMProcess: Feb 07, 2012 3:44:55 PM weblogic.nodemanager.server.AbstractServerManager log
NMProcess: INFO: Startup configuration properties loaded from "/home/oracle/soa/configuration/domains/soa_domain/servers/soa_server1/data/nodemanager/startup.properties"
NMProcess: <Feb 07, 2012 3:44:55 PM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
NMProcess: <Feb 07, 2012 3:44:55 PM> <INFO> <Secure socket listener started on port 5556, host /192.168.1.66>
NMProcess: Feb 07, 2012 3:44:55 PM weblogic.nodemanager.server.SSLListener run
NMProcess: INFO: Secure socket listener started on port 5556, host /192.168.1.66
Successfully launched the Node Manager.
The Node Manager process is running independent of the WLST process.
Exiting WLST will not stop the Node Manager process. Please refer
to the Node Manager logs for more information.
The Node Manager logs will be under /home/oracle/soa/configuration/nodemanagers/soa_domain
Node Manager starting in the background

When the nodemanagers are running we can start the servers:

[oracle@axis-into-ict scripts]$ ./DomainStartService.sh
CLASSPATH=/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:

PATH=/home/oracle/soa/installation/wlserver_10.3/server/bin:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/bin:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/bin:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/oracle/bin

Your environment has been set.

CLASSPATH=/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/soa/installation/utils/config/10.3/config-launch.jar::/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbynet.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbyclient.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

CONNECT TO NODE MANAGER ON MACHINE1
Connecting to Node Manager ...
Successfully Connected to Node Manager.
START ADMIN SERVER ON MACHINE1
Starting server AdminServer ...
Successfully started server AdminServer ...
CONNECT TO ADMIN SERVER
Connecting to t3://192.168.1.66:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'soa_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

START MANAGED SERVERS ON MACHINE1

Starting server soa_server1 ..........................................................................................................................................................
Server with name soa_server1 started successfully
DISCONNECT FROM NODE MANAGER ON MACHINE1
Successfully disconnected from Node Manager.
CONNECT TO NODE MANAGER ON MACHINE2
Connecting to Node Manager ...
Successfully Connected to Node Manager.
START MANAGED SERVERS ON MACHINE2

Starting server soa_server2 ...........................................................................................................................
Server with name soa_server2 started successfully
DISCONNECT FROM THE ADMIN SERVER
Disconnected from weblogic server: AdminServer
DISCONNECT FROM NODE MANAGER ON MACHINE2
Successfully disconnected from Node Manager.

Load balancer

To install the Oracle Web Tier we can follow the following steps:

Unzip ofm_webtier_linux_11.1.1.2.0_64_disk1_1of1.zip.
Navigate to the directory Disk1 and run ./runInstaller.
Click next in the welcome screen.
Select install software – do not configure and click next.
Prerequisite checks are run, when these succeed click next.
Specificy the install directories:
- oracle middleware home: /home/oracle/soa/installation
- oracle home directory: oracle_webtier
Click next and enter credentials for security updates when applicable.
Click next and click install.
Click next and click finish.

Next, we patch the installation to 11.1.1.3 or higher (which is needed for WebGate not for the WebLogic plug-in):

Unzip ofm_webtier_linux_11.1.1.3.0_64_disk1_1of1.zip.
Navigate to the directory Disk1 and run ./runInstaller.
Click next in the welcome screen.
Specificy the install directories:
- oracle middleware home: /home/oracle/soa/installation
- oracle home directory: oracle_webtier
Click next and enter credentials for security updates when applicable.
Click next and click install.
Click next and click finish.

To configure the HTTP server, we can follow these steps:

Run ./config.sh (/home/oracle/soa/oracle_webtier/bin).
Click next in the welcome screen.
Only select the Oracle HTTP Server and click next.
Specify the following component details:
- instance home location: /home/oracle/soa/configuration/instances/webtier_instance
- instance name: webtier_instance
- ohs component name: http_server
Click next and select auto port configuration.
Click next and enter credentials for security updates when applicable.
Click configure.
Click next and click finish.

Next, we create a shell script (ohs.sh) to start and stop the HTTP server, for example,

#!/bin/sh

ORACLE_INSTANCE=/home/oracle/soa/configuration/instances/webtier_instance
export ORACLE_INSTANCE
PATH=${PATH}:${ORACLE_INSTANCE}/bin/

if [ "$1" == "status" ] ; then
	opmnctl status -l
fi

if [ "$1" == "startopmn" ] ; then
	opmnctl start
fi

if [ "$1" == "stopohs" ] ; then
	opmnctl stopproc process-type=OHS
fi

if [ "$1" == "startohs" ] ; then
	opmnctl startproc process-type=OHS
fi

if [ "$1" == "stopopmn" ] ; then
	opmnctl shutdown
fi

Edit mod_wl_ohs.conf (located in the ${ORACLE_INSTANCE}/config/OHS/http_server directory) such that requests to the HTTP server are forwarded to the appropriate WebLogic server, for example,

LoadModule weblogic_module   "${ORACLE_HOME}/ohs/modules/mod_wl_ohs.so"

<IfModule weblogic_module>
	ConnectTimeoutSecs 10
	ConnectRetrySecs 2
	DebugConfigInfo ON
	WLSocketTimeoutSecs 2
	WLIOTimeoutSecs 300
	Idempotent ON
	FileCaching ON
	KeepAliveSecs 20
	KeepAliveEnabled ON
	DynamicServerList ON
	WLProxySSL OFF
</IfModule>

<Location /b2bconsole>
	SetHandler weblogic-handler
	WebLogicCluster 192.168.1.66:8001,192.168.1.67:9001
</Location>

<Location /integration>
	SetHandler weblogic-handler
	WebLogicCluster 192.168.1.66:8001,192.168.1.67:9001
</Location>

<Location /soa-infra>
	SetHandler weblogic-handler
	WebLogicCluster 192.168.1.66:8001,192.168.1.67:9001
</Location>

We can also edit the listen port of the HTTP server. Open the httpd.conf (${ORACLE_INSTANCE}/config/OHS/http_server) file and adjust the Listen directive, for example, Listen 7779.

Restart the HTTP Server (./ohs.sh stopohs and ./ohs.sh startohs). Test if the configuration is working by using the URLs:

http://192.168.1.66:7779/b2bconsole
http://192.168.1.66:7779/integration/worklistapp
http://192.168.1.66:7779/soa-infra

Access Manager

At one point in time, we decide to use access manager for our security. First, we need to set-up the necessary data environment by running the repository creation utility:

Stop the servers and the Apache HTTP server in the environment.
Navigate to /rcuHome/bin and run ./rcu
Click next on the welcome screen
Choose create and click next
Enter the database parameters:
- hostname: hostname of the machine where the database is running
- port: 1521
- service name: service name of the database
- username: sys username
- password: sys password
- role: sysdba
Click next, select an existing prefix (DEV) and select the following components:
- identity management, oracle access manager (DEV_OAM)
Click next and select use same passwords for all schemas and enter the password and confirm password fields
Click next and check the tablespaces
Click next such that the tablespaces are created
Check the summary and click create

Install access manager

Unzip ofm_iam_generic_11.1.1.5.0_disk1_1of1.zip.
Navigate to Disk1 and run ./runInstaller.
Specify the JDK location: /home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0.
Click next in the welcome screen.
Enter the credentials of software updates when applicable and click next.
The prerequisite checks are run, click next when these succeed.
Specify the install directory:
- oracle middleware home: /home/oracle/soa/installation
- oracle home directory: oracle_access_manager
Click next and click install.
Click next and click finish.

Configure access manager

Start the configuration wizard. Navigate to the directory /home/oracle/soa/oracle_access_manager/common/bin and run ./config.sh.
Select extend an existing domain and click next.
Select soa_domain (/home/oracle/soa/configuration/domains/) and click next.
Select Oracle Access Manager with Database Policy Store – 11.1.1.3 [oracle_iam].
Click next and enter the following parameters:
- vendor: Oracle
- driver: Oracle’s driver (thin) for service connections
- schema owner: DEV_OAM
- schema password: magic11g
- dbms/service: service name of the database
- host name: host name or IP address of the machine where the database is located
- port: 1521
Click next.
Click next when the tests are succeeded.
Select managed servers, clusters and machines:
- configure managed servers: set the listen address of oam_server1 to 192.168.1.67 and click next.
- configure clusters: click add and set name to oam_cluster and clustering messaging mode to unicast, and click next.
- assign servers to clusters: add oam_server1 to the oam_cluster and click next.
- create HTTP proxy: accept the defaults and click next.
- configure machines (unix machine): accept the defaults and click next.
- assign servers to machines: add the oam_server1 to the soa_machine2 and click next.
Click extend to start the configuration.
Click done.

When we extend a WebLogic domain in which the managed servers are distributed (by using the pack and unpack commands) to remote machines, the startup scripts on the remote machines are not updated automatically. To ensure that startup scripts on the remote machines are updated, we have to perform the following steps after extending the domain:

Delete the managed server directories (/home/oracle/soa/configuration/domains/soa_domain) on the remote machines.
Create a managed server template from the extended domain, by using the pack command with the managed=true option.
Create managed servers on the remote machines, by using the unpack command to unpack the managed server template.

Before we continue let us test the environment. Start the servers in the domain by using the start scripts:

[oracle@axis-into-ict scripts]$ ./NodeManagerStartService.sh
CLASSPATH=/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:

PATH=/home/oracle/soa/installation/wlserver_10.3/server/bin:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/bin:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/bin:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/oracle/bin

Your environment has been set.

CLASSPATH=/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/soa/installation/utils/config/10.3/config-launch.jar::/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbynet.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbyclient.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Launching NodeManager ...
 Properties: {ListenAddress=192.168.1.66,ListenPort=5556,NodeManagerHome=/home/oracle/soa/configuration/nodemanagers/soa_domain,}
 Command: /home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/bin/java -classpath /home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/lib/rt.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/lib/i18n.jar:/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/home/oracle/soa/installation/utils/config/10.3/config-launch.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbynet.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbyclient.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbytools.jar weblogic.NodeManager -v
NMProcess: <Feb 08, 2012 5:49:50 PM> <INFO> <Loading domains file: /home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.domains>
NMProcess: Feb 08, 2012 5:49:50 PM weblogic.nodemanager.server.NMServerConfig initDomainsMap
NMProcess: INFO: Loading domains file: /home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.domains
NMProcess: <Feb 08, 2012 5:49:51 PM> <INFO> <Loading identity key store: FileName=/home/oracle/soa/installation/wlserver_10.3/server/lib/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
NMProcess: Feb 08, 2012 5:49:51 PM weblogic.nodemanager.server.SSLConfig loadKeyStoreConfig
NMProcess: INFO: Loading identity key store: FileName=/home/oracle/soa/installation/wlserver_10.3/server/lib/DemoIdentity.jks, Type=jks, PassPhraseUsed=true
NMProcess: <Feb 08, 2012 5:49:51 PM> <INFO> <Loaded node manager configuration properties from '/home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.properties'>
NMProcess: Feb 08, 2012 5:49:51 PM weblogic.nodemanager.server.NMServer <init>
NMProcess: INFO: Loaded node manager configuration properties from '/home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.properties'
NMProcess: Node manager v10.3
NMProcess:
NMProcess: Configuration settings:
NMProcess:
NMProcess: NodeManagerHome=/home/oracle/soa/configuration/nodemanagers/soa_domain
NMProcess: ListenAddress=192.168.1.66
NMProcess: ListenPort=5556
NMProcess: ListenBacklog=50
NMProcess: SecureListener=true
NMProcess: AuthenticationEnabled=true
NMProcess: NativeVersionEnabled=true
NMProcess: CrashRecoveryEnabled=false
NMProcess: JavaHome=/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre
NMProcess: StartScriptEnabled=true
NMProcess: StopScriptEnabled=false
NMProcess: StartScriptName=startWebLogic.sh
NMProcess: StopScriptName=
NMProcess: LogFile=/home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.log
NMProcess: LogLevel=INFO
NMProcess: LogLimit=0
NMProcess: LogCount=1
NMProcess: LogAppend=true
NMProcess: LogToStderr=true
NMProcess: LogFormatter=weblogic.nodemanager.server.LogFormatter
NMProcess: DomainsFile=/home/oracle/soa/configuration/nodemanagers/soa_domain/nodemanager.domains
NMProcess: DomainsFileEnabled=true
NMProcess: StateCheckInterval=500
NMProcess: QuitEnabled=true
NMProcess: UseMACBroadcast=false
NMProcess: DomainRegistrationEnabled=false
NMProcess: DomainsDirRemoteSharingEnabled=false
NMProcess:
NMProcess: Domain name mappings:
NMProcess:
NMProcess: soa_domain -> /home/oracle/soa/configuration/domains/soa_domain
NMProcess:
NMProcess: <Feb 08, 2012 5:49:51 PM> <INFO> <soa_domain> <soa_server1> <Startup configuration properties loaded from "/home/oracle/soa/configuration/domains/soa_domain/servers/soa_server1/data/nodemanager/startup.properties">
NMProcess: Feb 08, 2012 5:49:51 PM weblogic.nodemanager.server.AbstractServerManager log
NMProcess: INFO: Startup configuration properties loaded from "/home/oracle/soa/configuration/domains/soa_domain/servers/soa_server1/data/nodemanager/startup.properties"
NMProcess: <Feb 08, 2012 5:49:52 PM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
NMProcess: <Feb 08, 2012 5:49:52 PM> <INFO> <Secure socket listener started on port 5556, host /192.168.1.66>
NMProcess: Feb 08, 2012 5:49:52 PM weblogic.nodemanager.server.SSLListener run
NMProcess: INFO: Secure socket listener started on port 5556, host /192.168.1.66
Successfully launched the Node Manager.
The Node Manager process is running independent of the WLST process.
Exiting WLST will not stop the Node Manager process. Please refer
to the Node Manager logs for more information.
The Node Manager logs will be under /home/oracle/soa/configuration/nodemanagers/soa_domain
Node Manager starting in the background

[oracle@axis-into-ict scripts]$ ./DomainStartService.sh
CLASSPATH=/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:

PATH=/home/oracle/soa/installation/wlserver_10.3/server/bin:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/bin:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/jre/bin:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/oracle/bin

Your environment has been set.

CLASSPATH=/home/oracle/soa/installation/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/soa/installation/patch_ocp360/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0/lib/tools.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/soa/installation/modules/features/weblogic.server.modules_10.3.5.0.jar:/home/oracle/soa/installation/wlserver_10.3/server/lib/webservices.jar:/home/oracle/soa/installation/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/soa/installation/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/soa/installation/utils/config/10.3/config-launch.jar::/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbynet.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbyclient.jar:/home/oracle/soa/installation/wlserver_10.3/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

CONNECT TO NODE MANAGER ON MACHINE1
Connecting to Node Manager ...
Successfully Connected to Node Manager.
START ADMIN SERVER ON MACHINE1
Starting server AdminServer ...
Successfully started server AdminServer ...
CONNECT TO ADMIN SERVER
Connecting to t3://192.168.1.66:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'soa_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

START MANAGED SERVERS ON MACHINE1

Starting server soa_server1 ............................................................................................................................................................
Server with name soa_server1 started successfully
DISCONNECT FROM NODE MANAGER ON MACHINE1
Successfully disconnected from Node Manager.
CONNECT TO NODE MANAGER ON MACHINE2
Connecting to Node Manager ...
Successfully Connected to Node Manager.
START MANAGED SERVERS ON MACHINE2

Starting server soa_server2 ..........................................................................................................................
Server with name soa_server2 started successfully

Starting server oam_server1 ...............................................................................
Server with name oam_server1 started successfully
DISCONNECT FROM THE ADMIN SERVER
Disconnected from weblogic server: AdminServer
DISCONNECT FROM NODE MANAGER ON MACHINE2
Successfully disconnected from Node Manager.

Install WebGate

Unzip ofm_webgates_generic_11.1.1.5.0_disk1_1of1.zip.
Navigate to the directory Disk1 and run: ./runInstaller -jreLoc /home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0.
Click next in the welcome screen.
Prerequisite checks are run, when these succeed click next.
Click next and specify the install directories:
- oracle middleware home: /home/oracle/soa/installation
- oracle home directory: oracle_webgate
Click next and specify the location of the gcc libraries. In general the locations are as follows:
- libgcc_s.so.1 – /lib64
- libstdc++.so.6 – /usr/lib64
As we can specify only one directory we need to copy one library to the other directory, for example, open a new command shell and login as root (su root) and run the following commands:
- cd /usr/lib64
- cp libstdc++.so.6 /lib64
- cp libstdc++.so.6.0.8 /lib64
Click next and click install.
Click next and click finish.

Next, we need to install WebGate on the HTTP server. To this end run the following commands:

[oracle@axis-into-ict ~]$ cd /home/oracle/soa/installation/oracle_webgate/webgate/ohs/tools/deployWebGate

[oracle@axis-into-ict deployWebGate]$ ./deployWebGateInstance.sh -w /home/oracle/soa/configuration/instances/webtier_instance/config/OHS/http_server -oh /home/oracle/soa/installation/oracle_webgate
Copying files from WebGate Oracle Home to WebGate Instancedir

[oracle@axis-into-ict deployWebGate]$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/oracle/soa/installation/oracle_webtier/lib
[oracle@axis-into-ict deployWebGate]$ echo ${LD_LIBRARY_PATH}
:/home/oracle/soa/installation/oracle_webtier/lib

[oracle@axis-into-ict deployWebGate]$ cd ../setup/InstallTools/

[oracle@axis-into-ict InstallTools]$ ./EditHttpConf -w /home/oracle/soa/configuration/instances/webtier_instance/config/OHS/http_server -oh /home/oracle/soa/installation/oracle_webgate -o /home/oracle/temp/output.log
The web server configuration file was successfully updated
/home/oracle/soa/configuration/instances/webtier_instance/config/OHS/http_server/httpd.conf has been backed up as /home/oracle/soa/configuration/instances/webtier_instance/config/OHS/http_server/httpd.conf.ORIG

To register a WebGate agent we need to create the appropriate artifacts, to this end

Unzip the file RREG.tar.gz to /home/oracle/temp. The file RREG.tar.gz is located in the directory /home/oracle/soa/installation/oracle_access_manager/oam/server/rreg/client.
Adjust the oamreg.sh file privileges (chmod ug=rwx oamreg.sh). The file oamreg.sh is located in the directory /home/oracle/temp/rreg/bin.
Edit oamreg.sh such that the OAM_REG_HOME and JAVA_HOME variables are pointing to the appropriate directories:
- OAM_REG_HOME=/home/oracle/temp/rreg
- JAVA_HOME=/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0
Open the file OAM11GRequest.xml (/home/oracle/temp/rreg/input) and adjust the following values:
- server address: http://192.168.1.66:7001
- agent name: RREG_OAM11G (can be any name)
- agentBaseUrl: http://192.168.1.66:7779
- security: open
Execute the command: ./oamreg.sh inband /home/oracle/temp/rreg/input/OAM11GRequest.xml.

[oracle@axis-into-ict bin]$ ./oamreg.sh inband /home/oracle/temp/rreg/input/OAM11GRequest.xml
JAVA_HOME=/home/oracle/jrockit-jdk1.6.0_29-R28.2.2-4.1.0
CLASSPATH=/home/oracle/temp/rreg/lib/rreg.jar:/home/oracle/temp/rreg/lib:/home/oracle/temp/rreg/lib/RequestResponse.jar:/home/oracle/temp/rreg/lib/commons-codec-1.3.jar:/home/oracle/temp/rreg/lib/commons-httpclient-3.1.jar:/home/oracle/temp/rreg/lib/commons-logging-1.1.1.jar:/home/oracle/temp/rreg/lib/ojmisc.jar:/home/oracle/temp/rreg/lib/jps-api.jar:/home/oracle/temp/rreg/lib/jps-internal.jar:/home/oracle/temp/rreg/lib/jps-common.jar:/home/oracle/temp/rreg/lib/identitystore.jar:/home/oracle/temp/rreg/lib/identityutils.jar:/home/oracle/temp/rreg/lib/ldapjclnt11.jar:/home/oracle/temp/rreg/lib/dms.jar:/home/oracle/temp/rreg/lib/fmw_audit.jar:/home/oracle/temp/rreg/lib/ojdl.jar:/home/oracle/temp/rreg/lib/oraclepki.jar:/home/oracle/temp/rreg/lib/osdt_cert.jar:/home/oracle/temp/rreg/lib/osdt_core.jar:/home/oracle/temp/rreg/lib/osdt_jce.jar:/home/oracle/temp/rreg/lib/osdt_saml.jar:/home/oracle/temp/rreg/lib/osdt_xmlsec.jar:/home/oracle/temp/rreg/lib/xmlparserv2.jar:/home/oracle/temp/rreg/lib/jps-unsupported-api.jar:/home/oracle/temp/rreg/lib/nap-api.jar:/home/oracle/temp/rreg/lib/utilities.jar:.
OAM_REG_HOME=/home/oracle/temp/rreg
------------------------------------------------
Welcome to OAM Remote Registration Tool!
Parameters passed to the registration tool are:
Mode: inband
Filename: /home/oracle/temp/rreg/input/OAM11GRequest.xml
Enter admin username:weblogic
Username: weblogic
Enter admin password:
Do you want to enter a Webgate password?(y/n):
n
Do you want to import an URIs file?(y/n):
n

----------------------------------------
Request summary:
OAM11G Agent Name:RREG_OAM11G
Base URL:http://192.168.1.66:7779
URL String:RREG_HostId11G
Registering in Mode:inband
Your registration request is being sent to the Admin server at: http://192.168.1.66:7001
----------------------------------------

Inband registration process completed successfully! Output artifacts are created in the output folder.

The artifacts are created in the directory /home/oracle/temp/rreg/output/RREG_OAM11G.
Copy the artifacts to the directory /home/oracle/soa/configuration/instances/webtier_instance/config/OHS/http_server/webgate/config.

Start the HTTP Server:

[oracle@axis-into-ict ~]$ cd soa/configuration/applications/soa_domain/scripts/
[oracle@axis-into-ict scripts]$ ./ohs.sh startopmn
[oracle@axis-into-ict scripts]$ ./ohs.sh startohs
opmnctl startproc: starting opmn managed processes...
[oracle@axis-into-ict scripts]$ ./ohs.sh status

Processes in Instance: webtier_instance
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
http_server                      | OHS                |   13235 | Alive    | 1047755169 |   449744 |   0:00:05 | https:10000,https:4444,http:7779

To check if the set-up works, enter the following URL: http://192.168.1.66:7779/, and enter the WebLogic admin user credentials.

References

[1] WebLogic Documentation.
[2] SOA Suite Documentation.
[3] Identity and Access Management Documentation.
[4] Web Tier Components Documentation.

access manager, SOA Suite, weblogic 1 Comment more...

Feb/12

Technical Session: Setting up a high available tuned environment using WebLogic server

by René van Wijk under Cluster, Coherence, DataSource, Deploy, JDBC, JMS, JRockit, JVM Tuning, Monitoring, NodeManager, WorkManager

René van Wijk

Once applications are deployed, the problems start: Multiple users connect to the application and complain about the speed. Where do you start to locate the problem? Is it the operating system? What about the garbage collections? Is the architecture of the application a problem? In this session we give a starting point in how to analyze a problem and act accordingly.

Axis into ICT (the company I now work for) is pleased to invite you for a free technical session on March 27, in which we outline best practices for high availability and tuning:

Deployment – determine a cluster structure and choose the number and types of machines.
System performance – principles and strategies for scalable Java EE systems.
Optimizing performance – design patterns and best practices that affect performance and scalability for a WebLogic environment.
Troubleshoot performance problems – steps and techniques to improve performance and solve scalability issues (for example by using Coherence).

Why should you attend?

You will increase your knowledge on high availability and tuning.
The seminar is a great networking opportunity: you can talk with your peers and industry experts to share knowledge and experience.

Registration

Date: March 27, 2012
Time: 12.00PM – 17:00PM (lunch is included)
Location: Headquarters Oracle Netherlands, Utrecht

You are in and around The Netherlands on March 27, then join us for this unique technical event to learn about some great technology. You can register here.

technical session 2 Comments more...

Jan/12

Deploy WebLogic12c to Multiple Machines

by René van Wijk under Cluster, Deploy, EJB, JDBC, JMS, JRockit, NodeManager, WLST

René van Wijk

In this post, we set-up a WebLogic cluster that spans multiple machines. We start with installing the WebLogic software on the machines involved. Next, we create the WebLogic domain by using the WebLogic Scripting Tool (WLST). Use pack and unpack to create managed server directories on remote machines. Create start and stop scripts for the environment. Deploy an application and tune it by using deployment override descriptors. Set-up load balancing by using the Apache HTTP Server and show the steps involved in scaling the environment. Finally, we perform a load test by using The Grinder and use JRockit Mission Control to obtain insight in the performance of the cluster.

Install WebLogic

First, we choose an installation directory, for example /home/oracle/weblogic12.1.1. This will be our middleware home in the installation of WebLogic. To keep things simple use the same directory structure on both machines.

To install JRockit we follow these steps:

Run the file jrrt-4.0.1-1.6.0-linux-x64.bin (JRockit can be downloaded here).
Click next on the welcome screen.
Define the install directory /home/oracle/jrrt-4.0.1-1.6.0 and click next.
Optionally select extra components and click next.
When the installation is finished click done.

To WebLogic we follow these steps:

Navigate to the JDK’s bin directory (/home/oracle/jrrt-4.0.1-1.6.0/bin).
Enter the following command: ./java -d64 -Xms1024m -Xmx1024m -jar wls1211_generic.jar (WebLogic can be downloaded here).
Click next on the welcome screen.
Define the Middleware-Home directory and click next.
Fill in support credentials if applicable and click next.
Select custom as installation type and click next.
De-select the evaluation database and click next.
Select the installed JDK (if the JDK is not shown, click browse and browse to the root of the JDK) and click next.
Accept the defaults in the directories screen and click next.
Click next to start the installation.
De-select run quickstart and click done.

Create domain

To create our domain, we are going to use WLST. The following script, creates the domain in production mode, disables hostname verification, creates two managed server that are clustered and are present on different machines and creates the artifacts needed to run the application, such as JMS and a data source.

beahome = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
adminusername = 'weblogic';
adminpassword = 'magic12c';
adminservername='AdminServer';
adminserverurl='t3://172.31.0.113:7001';
domainname = 'base_domain';
domaindirectory = beahome + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domainname;
domaintemplate = beahome + pathseparator + 'wlserver_12.1' + pathseparator + 'common' + pathseparator + 'templates' + pathseparator + 'domains' + pathseparator + 'wls.jar';
jvmdirectory = '/home/oracle/jrrt-4.0.1-1.6.0';

print 'CREATE DOMAIN';
readTemplate(domaintemplate);
setOption('DomainName', domainname);
setOption('OverwriteDomain', 'true');
setOption('ServerStartMode', 'prod');
cd('/Security/base_domain/User/weblogic');
cmo.setName(adminusername);
cmo.setUserPassword(adminpassword);
cd('/');
writeDomain(domaindirectory);

# When using startServer command on WebLogic12c, we have to perform the following steps (1 and 2):
# Starting weblogic server ...
# WLST-WLS-1326447719560: Exception in thread "Main Thread" java.lang.AssertionError: JAX-WS 2.2 API is required, but an older version was found in the JDK.
# WLST-WLS-1326447719560: Use the endorsed standards override mechanism (http://java.sun.com/javase/6/docs/technotes/guides/standards/).
# WLST-WLS-1326447719560: 1) locate the bundled Java EE 6 endorsed directory in $WL_HOME/endorsed.
# WLST-WLS-1326447719560: 2) copy those JAR files to $JAVA_HOME/jre/lib/endorsed OR add the endorsed directory to the value specified by system property java.endorsed.dirs.
print 'START ADMIN SERVER';
startServer(adminservername, domainname, adminserverurl, adminusername, adminpassword, domaindirectory);

print 'CONNECT TO ADMIN SERVER';
connect(adminusername, adminpassword, adminserverurl);

print 'START EDIT MODE';
edit();
startEdit();

#print 'CHANGE NODEMANAGER USERNAME AND PASSWORD';
#cd('/SecurityConfiguration/' + domainname);
#cmo.setNodeManagerUsername(adminusername);
#set('NodeManagerPasswordEncrypted', encrypt(adminpassword, domaindirectory));
#cd('/');

print 'DISABLE HOSTNAME VERIFICATION';
cd('/Servers/' + adminservername + '/SSL/' + adminservername);
cmo.setHostnameVerificationIgnored(true);
cmo.setHostnameVerifier(None);
cmo.setTwoWaySSLEnabled(false);
cmo.setClientCertificateEnforced(false);
cd('/');

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'SHUTDOWN THE ADMIN SERVER';
shutdown(block='true');

print 'START ADMIN SERVER';
startServer(adminservername, domainname, adminserverurl, adminusername, adminpassword, domaindirectory);

print 'CONNECT TO ADMIN SERVER';
connect(adminusername, adminpassword, adminserverurl);

print 'START EDIT MODE';
edit();
startEdit();

print 'CREATE MACHINE: machine1';
machine1 = cmo.createUnixMachine('machine1');
machine1.setPostBindUIDEnabled(true);
machine1.setPostBindUID('oracle');
machine1.getNodeManager().setListenAddress('172.31.0.175');
machine1.getNodeManager().setNMType('ssl');

print 'CREATE MACHINE: machine2';
machine2 = cmo.createUnixMachine('machine2');
machine2.setPostBindUIDEnabled(true);
machine2.setPostBindUID('oracle');
machine2.getNodeManager().setListenAddress('172.31.0.113');
machine2.getNodeManager().setNMType('ssl');

print 'CREATE CLUSTER: CLUSTER';
cluster = cmo.createCluster('cluster');
cluster.setClusterMessagingMode('unicast');

print 'CREATE MANAGED SERVER: server1';
server1 = cmo.createServer('server1');
server1.setListenPort(7002);
server1.setListenAddress('172.31.0.175');
server1.setAutoRestart(true);
server1.setAutoKillIfFailed(true);
server1.setRestartMax(2);
server1.setRestartDelaySeconds(10);
server1.getServerStart().setJavaHome(jvmdirectory);
server1.getServerStart().setJavaVendor('Oracle');
server1.getServerStart().setArguments('-jrockit -Xms1024m -Xmx1024m -Xns256m -Xgc:throughput');

print 'CREATE MANAGED SERVER: server2';
server2 = cmo.createServer('server2');
server2.setListenPort(7003);
server2.setListenAddress('172.31.0.113');
server2.setAutoRestart(true);
server2.setAutoKillIfFailed(true);
server2.setRestartMax(2);
server2.setRestartDelaySeconds(10);
server2.getServerStart().setJavaHome(jvmdirectory);
server2.getServerStart().setJavaVendor('Oracle');
server2.getServerStart().setArguments('-jrockit -Xms1024m -Xmx1024m -Xns256m -Xgc:throughput');

print 'ADD MANAGED SERVERS TO CLUSTER';
server1.setCluster(cluster);
server2.setCluster(cluster);

print 'ADD MANAGED SERVERS TO MACHINE';
server1.setMachine(machine1);
server2.setMachine(machine2);

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'START EDIT MODE';
startEdit();

print 'CREATE FILESTORE FOR SERVER1';
filestore1 = cmo.createFileStore('FileStore1');
filestore1.setDirectory(beahome + pathseparator + 'deploy');
targets = filestore1.getTargets();
targets.append(server1);
filestore1.setTargets(targets);

print 'CREATE JMS SERVER FOR SERVER1';
jmsserver1 = cmo.createJMSServer('JMSServer1');
jmsserver1.setPersistentStore(filestore1);
jmsserver1.setTargets(targets);

targets.remove(server1);
targets.append(server2);

print 'CREATE FILESTORE FOR SERVER2';
filestore2 = cmo.createFileStore('FileStore2');
filestore2.setDirectory(beahome + pathseparator + 'deploy');
filestore2.setTargets(targets);

print 'CREATE JMS SERVER FOR SERVER2';
jmsserver2 = cmo.createJMSServer('JMSServer2');
jmsserver2.setPersistentStore(filestore2);
jmsserver2.setTargets(targets);

targets.remove(server2);
targets.append(cluster);

print 'CREATE JMS SYSTEM MODULE';
module = cmo.createJMSSystemResource('SystemModule');
module.setTargets(targets);

print 'CREATE SUBDEPLOYMENT';
module.createSubDeployment('SubDeployment');
cd('/JMSSystemResources/SystemModule/SubDeployments/SubDeployment');
set('Targets',jarray.array([ObjectName('com.bea:Name=JMSServer1,Type=JMSServer'), ObjectName('com.bea:Name=JMSServer2,Type=JMSServer')], ObjectName));
cd('/');

resource = module.getJMSResource();

print 'CREATE CONNECTION FACTORY';
resource.createConnectionFactory('ConnectionFactory');
connectionfactory = resource.lookupConnectionFactory('ConnectionFactory');
connectionfactory.setJNDIName('jms/ConnectionFactory');
connectionfactory.setDefaultTargetingEnabled(true);
connectionfactory.getTransactionParams().setTransactionTimeout(3600);
connectionfactory.getTransactionParams().setXAConnectionFactoryEnabled(true);

print 'CREATE UNIFORM DISTRIBUTED QUEUE';
resource.createUniformDistributedQueue('DistributedQueue');
distributedqueue = resource.lookupUniformDistributedQueue('DistributedQueue');
distributedqueue.setJNDIName('jms/CompanyQueue');
distributedqueue.setLoadBalancingPolicy('Round-Robin');
distributedqueue.setSubDeploymentName('SubDeployment');

print 'CREATE DATA SOURCE';
datasource = cmo.createJDBCSystemResource('DataSource');
datasource.setTargets(targets);
jdbcResource = datasource.getJDBCResource();
jdbcResource.setName('DataSource');
names = ['jdbc/exampleDS'];
dataSourceParams = jdbcResource.getJDBCDataSourceParams();
dataSourceParams.setJNDINames(names);
dataSourceParams.setGlobalTransactionsProtocol('LoggingLastResource');
driverParams = jdbcResource.getJDBCDriverParams();
driverParams.setUrl('jdbc:oracle:thin:@hostname:1521:sid');
driverParams.setDriverName('oracle.jdbc.OracleDriver');
driverParams.setPassword('password');
driverProperties = driverParams.getProperties();
driverProperties.createProperty('user');
userProperty = driverProperties.lookupProperty('user');
userProperty.setValue('username');
connectionPoolParams = jdbcResource.getJDBCConnectionPoolParams();
connectionPoolParams.setTestTableName('SQL SELECT 1 FROM DUAL');
connectionPoolParams.setConnectionCreationRetryFrequencySeconds(100);

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'SHUTDOWN THE ADMIN SERVER';
shutdown(block='true');

To run the script we can use the following (which also shows the output logging):

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/
[oracle@edu-wls-rh bin]$ ./wlst.sh /home/oracle/weblogic12.1.1/deploy/scripts/create_environment.py 

CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/weblogic12.1.1/utils/config/10.3/config-launch.jar::/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbynet.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbyclient.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

CREATE DOMAIN
START ADMIN SERVER
Starting weblogic server ...
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:51 AM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:51 AM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:52 AM CET> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.0.1-21-133393-1.6.0_20-20100512-2126-linux-x86_64 from Oracle Corporation.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:52 AM CET> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.1.0  Wed Dec 7 08:40:57 PST 2011 1445491 >
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:54 AM CET> <Notice> <Log Management> <BEA-170019> <The server log file /home/oracle/weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:11:57 AM CET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:00 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:00 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 172.31.0.113:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:05 AM CET> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>
Server started successfully.
CONNECT TO ADMIN SERVER
Connecting to t3://172.31.0.113:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

START EDIT MODE
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit(). 

For more help, use help(edit)

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
DISABLE HOSTNAME VERIFICATION
SAVE AND ACTIVATE CHANGES
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141239> <The non-dynamic attribute HostnameVerificationIgnored on weblogic.management.configuration.SSLMBeanImpl@4c9421a2([base_domain]/Servers[AdminServer]/SSL[AdminServer]) has been changed. This may require redeploying or rebooting configured entities.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141238> <A non-dynamic change has been made which affects the server AdminServer. This server must be rebooted in order to consume this change.> 

The following non-dynamic attribute(s) have been changed on MBeans
that require server re-start:
MBean Changed : com.bea:Name=AdminServer,Type=SSL,Server=AdminServer
Attributes changed : HostnameVerificationIgnored

Activation completed
SHUTDOWN THE ADMIN SERVER
Shutting down the server AdminServer with force=false while connected to AdminServer ...
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by weblogic.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SUSPENDING.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <HTTP> <BEA-101278> <There are no active sessions. The Web service is ready to suspend.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SHUTTING_DOWN.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[3]", listening on 0:0:0:0:0:0:0:1:7001, was shut down.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default", listening on 172.31.0.113:7001, was shut down.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[2]", listening on 127.0.0.1:7001, was shut down.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[1]", listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001, was shut down.>
<Jan 16, 2012 9:12:11 AM CET> <Warning> <JNDI> <BEA-050001> <WLContext.close() was called in a different thread than the one in which it was created.>
WLST-WLS-1326701511109: <Jan 16, 2012 9:12:11 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:11 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>
WLST-WLS-1326701511109: Stopped draining WLST-WLS-1326701511109
WLST-WLS-1326701511109: Stopped draining WLST-WLS-1326701511109
WLST lost connection to the WebLogic Server that you were
connected to, this may happen if the server was shutdown or
partitioned. You will have to re-connect to the server once the
server is available.
Disconnected from weblogic server: AdminServer
Disconnected from weblogic server:
START ADMIN SERVER
Starting weblogic server ...
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:15 AM CET> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.0.1-21-133393-1.6.0_20-20100512-2126-linux-x86_64 from Oracle Corporation.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:16 AM CET> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.1.0  Wed Dec 7 08:40:57 PST 2011 1445491 >
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:18 AM CET> <Notice> <Log Management> <BEA-170019> <The server log file /home/oracle/weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:21 AM CET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:24 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:24 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 172.31.0.113:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:28 AM CET> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>
Server started successfully.
CONNECT TO ADMIN SERVER
Connecting to t3://172.31.0.113:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

START EDIT MODE

Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
CREATE MACHINE: machine1
CREATE MACHINE: machine2
CREATE CLUSTER: CLUSTER
CREATE MANAGED SERVER: server1
CREATE MANAGED SERVER: server2
ADD MANAGED SERVERS TO CLUSTER
ADD MANAGED SERVERS TO MACHINE
SAVE AND ACTIVATE CHANGES
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed
START EDIT MODE
Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
CREATE FILESTORE FOR SERVER1
CREATE JMS SERVER FOR SERVER1
CREATE FILESTORE FOR SERVER2
CREATE JMS SERVER FOR SERVER2
CREATE JMS SYSTEM MODULE
CREATE SUBDEPLOYMENT
CREATE CONNECTION FACTORY
CREATE UNIFORM DISTRIBUTED QUEUE
CREATE DATA SOURCE
SAVE AND ACTIVATE CHANGES
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed
SHUTDOWN THE ADMIN SERVER
Shutting down the server AdminServer with force=false while connected to AdminServer ...
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by weblogic.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SUSPENDING.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <HTTP> <BEA-101278> <There are no active sessions. The Web service is ready to suspend.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to SHUTTING_DOWN.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default", listening on 172.31.0.113:7001, was shut down.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[2]", listening on 127.0.0.1:7001, was shut down.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[3]", listening on 0:0:0:0:0:0:0:1:7001, was shut down.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Notice> <Server> <BEA-002607> <Channel "Default[1]", listening on fe80:0:0:0:20c:29ff:feb9:e2cc:7001, was shut down.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:34 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>
WLST-WLS-1326701534609: <Jan 16, 2012 9:12:34 AM CET> <Warning> <Management> <BEA-141269> <The temporary bean tree updateDeploymentContext(Mon Jan 16 09:12:32 CET 2012) was allocated for an undo, get, or activate operation, but has not been garbage collected.>
WLST-WLS-1326701534609: Stopped draining WLST-WLS-1326701534609
WLST-WLS-1326701534609: Stopped draining WLST-WLS-1326701534609
WLST lost connection to the WebLogic Server that you were
connected to, this may happen if the server was shutdown or
partitioned. You will have to re-connect to the server once the
server is available.
Disconnected from weblogic server: AdminServer
Disconnected from weblogic server:

Next, we add a boot.properties file in the ${DOMAIN_HOME}/servers/AdminServer/security directory:

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/user_projects/domains/base_domain/servers/AdminServer/
[oracle@edu-wls-rh AdminServer]$ mkdir security
[oracle@edu-wls-rh AdminServer]$ vi boot.properties

Add the following name-value pairs to the boot.properties file:

username=weblogic
password=magic12c

Note that when the admin server is started these values will be encrypted. The node manager requires authentication to start and stop managed servers. The first time the node manager is started it communicates with the admin server to obtain a username and password that will be used by the admin server to authenticate the node manager. As we created the domain in production mode, random node manager credits are created. When we want to use, for example, nmConnect we have to set the node manager username and password to known values. To this end start the admin server open the admin console:

Click on base_domain, security, general and click on the advanced link.
Edit the NodeManager Username and Password. In our case, we set these to respectively weblogic and magic12c.
Start the node manager by using the startNodeManager.sh script.

WebLogic provides two command-line utilities, pack and unpack, that helps to create new managed server directories on remote machines, for example on the 172.31.0.113 machine we do the following (first navigate to the ${WL_HOME}/common/bin directory):

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/
[oracle@edu-wls-rh bin]$ ./pack.sh -managed=true -domain=/home/oracle/weblogic12.1.1/user_projects/domains/base_domain -template=/home/oracle/temp/base_domain.jar -template_name=base_domain
<< read domain from "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"
>>  succeed: read domain from "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"
<< set config option Managed to "true"
>>  succeed: set config option Managed to "true"
<< write template to "/home/oracle/temp/base_domain.jar"
....................................................................................................
>>  succeed: write template to "/home/oracle/temp/base_domain.jar"
<< close template
>>  succeed: close template

Next, we copy the created base_domain.jar to the 172.31.0.175 machine, for example, by using:

[oracle@edu-wls-rh bin]$ scp oracle@172.31.0.113:/home/oracle/temp/base_domain.jar oracle@172.31.0.175:/home/oracle/temp/
oracle@172.31.0.113's password:
oracle@172.31.0.175's password:
base_domain.jar                                                                                  100%   45KB  44.9KB/s   00:00
Connection to 172.31.0.113 closed.

To use unpack on the 172.31.0.175 machine we can do the following (first navigate to the ${WL_HOME}/common/bin directory):

[oracle@middleware-magic ~]$ cd weblogic12.1.1/wlserver_12.1/common/bin/
[oracle@middleware-magic bin]$ ./unpack.sh -domain=/home/oracle/weblogic12.1.1/user_projects/domains/base_domain -template=/home/oracle/temp/base_domain.jar
<< read template from "/home/oracle/temp/base_domain.jar"
>>  succeed: read template from "/home/oracle/temp/base_domain.jar"
<< set config option DomainName to "base_domain"
>>  succeed: set config option DomainName to "base_domain"
<< write Domain to "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"
...............................................................................................
>>  succeed: write Domain to "/home/oracle/weblogic12.1.1/user_projects/domains/base_domain"
<< close template
>>  succeed: close template

To test the set-up, start the admin server (by using ./startWebLogic.sh located in the ${DOMAIN_HOME} directory). Subsequently, start the node managers on both machines (by using ./startNodeManager.sh located in the ${WL_HOME}/server/bin directory). When the node manager is started for the first time a nodemanager.properties file is created in the node manager home directory (which is ${WL_HOME}/common/nodemanager by default). We edit the nodemanager.properties and set the property StartScriptEnabled=false (the default is true). When doing this, we need to copy the ${WL_HOME}/endorsed directory to the jrrt-4.0.1-1.6.0/jre/lib/endorsed directory. When the changes are made the node manager needs to be restarted. Subsequently, use the admin console to the start the managed servers.

Start and stop scripts

In general, it is recommended to start the node manager when the machine boots. In this case, we need to know where to put our custom commands that will be called when the system boots. Note that RedHat Enterprise Linux has a /etc/rc.d/rc.local file, that can be used to put custom commands in. Let us do it in the recommended script-based way. Note that Unix-based systems specify so-called run levels, and that for each run level, scripts can be defined that start a certain service. These scripts are located in the /etc/rc.d/init.d directory. This allows for services to be started when the system boots or to be stopped on system shutdown. The different run levels are specified by a specific directory in the /etc/rc.d directory, i.e.,

rc0.d – contains scripts that are executed on system shutdown.
rc1.d – contains scripts for single-user mode.
rc2.d – contains scripts for multi-user mode.
rc3.d – contains scripts for multi-user mode and networking.
rc4.d – not used.
rc5.d – same as rc3.d plus some graphical stuff.
rc6.d – contains scripts that are executed on system reboot.

The boot sequence is as follows: in the /etc/inittab file the starting runlevel is defined, the script /etc/rc.d/rc.sysinit is called and /etc/rc.d/rc is run. The rc script looks in the /etc/rc.d/rc<start-runlevel>.d to execute the K**<script-name> scripts with the stop option. After this the S**<script-name> scripts are executed with the start option. Note that scripts are started in numerical order, i.e., the S10network script is executed before the S80sendmail script.

To create a node manager ‘service’, we first create basic start and stop scripts for the node manager. To this end, we will use WLST. To start the node manager by using WLST we can use the following:

bea_home = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
listen_address = '172.31.0.175';
nodemanager_listen_port = '5556';
domain_name='base_domain';

node_manager_home = bea_home + pathseparator + 'wlserver_12.1' + pathseparator + 'common' + pathseparator + 'nodemanager';

startNodeManager(verbose='true', NodeManagerHome=node_manager_home, ListenPort=nodemanager_listen_port, ListenAddress=listen_address);

To stop the node manager by using WLST we can use the following:

bea_home = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
admin_username = 'weblogic';
admin_password = 'magic12c';
listen_address = '172.31.0.175';
nodemanager_listen_port = '5556';
admin_server_listen_port = '7001';
domain_name = 'base_domain';

domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;
admin_server_url='t3://' + listen_address + ':' + admin_server_listen_port;

nmConnect(admin_username, admin_password, listen_address, nodemanager_listen_port, domain_name, domain_home, 'ssl');
stopNodeManager();

Create two shell scripts that respectively run the start and stop script, i.e.,

#!/bin/sh

BEA_HOME="/home/oracle/weblogic12.1.1"
export BEA_HOME

WL_HOME="${BEA_HOME}/wlserver_12.1"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/startNodeManager.py

#!/bin/sh

BEA_HOME="/home/oracle/weblogic12.1.1"
export BEA_HOME

WL_HOME="${BEA_HOME}/wlserver_12.1"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/stopNodeManager.py

Create a boot script (nodemanager) under the root user and place this script in the /etc/rc.d/init.d directory. An example boot script for the node manager looks as follows:

#!/bin/sh
#
# chkconfig: 235 91 35
# description: starts and stops the node manager
#
#
. /etc/rc.d/init.d/functions

RETVAL=0
SERVICE="nodemanager"

start() {
	echo "Starting Node Manager"
	su - oracle -c "/home/oracle/weblogic12.1.1/deploy/scripts/NodeManagerStartService.sh" >/dev/null 2>&1
	RETVAL=$?
	[ $RETVAL -eq 0 ] && success || failure
	echo
	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/${SERVICE}
	return $RETVAL
} 

stop() {
	echo "Stopping Node Manager"
	su - oracle -c "/home/oracle/weblogic12.1.1/deploy/scripts/NodeManagerStopService.sh" >/dev/null 2>&1
	RETVAL=$?
	[ $RETVAL -eq 0 ] && success || failure
	echo
	[ $RETVAL -eq 0 ] && rm -r /var/lock/subsys/${SERVICE}
	return $RETVAL
} 

restart() {
	stop
	start
} 

case "$1" in
	start)
		start
		;;
	stop)
		stop
		;;
	restart)
		restart
		;;
	*)
		echo $"Usage: $0 {start|stop|restart}"
		exit 1
esac 

exit $?

By using the chkconfig command we can update the runlevel information for system services, for example, chkconfig --add nodemanager. To test the set-up shut the system down and start it again. To check if the nodemanager is running we can use either ps -ef|grep java or netstat -anp|grep :5556 which assumes the node manager is listening on port 5556.

By using WLST we can also create start and stop scripts for the admin server and the managed servers, for example,

bea_home = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
admin_username = 'weblogic';
admin_password = 'magic12c';
listen_address_machine1 = '172.31.0.175';
listen_address_machine2 = '172.31.0.113';
nodemanager_listen_port = '5556';
admin_server_listen_port = '7001';
domain_name = 'base_domain';

domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;
admin_server_url='t3://' + listen_address_machine2 + ':' + admin_server_listen_port;

print 'CONNECT TO NODE MANAGER ON MACHINE2';
nmConnect(admin_username, admin_password, listen_address_machine2, nodemanager_listen_port, domain_name, domain_home, 'ssl');

print 'START ADMIN SERVER ON MACHINE2';
nmStart('AdminServer');

print 'CONNECT TO ADMIN SERVER';
connect(admin_username, admin_password, admin_server_url);

print 'START MANAGED SERVERS ON MACHINE2';
start('server2','Server');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE2';
nmDisconnect();

print 'CONNECT TO NODE MANAGER ON MACHINE1';
nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');

print 'START MANAGED SERVERS ON MACHINE1';
start('server1','Server');
# server3 will be added in a later stadium when we scale the cluster
#start('server3','Server');

print 'DISCONNECT FROM THE ADMIN SERVER';
disconnect();

print 'DISCONNECT FROM NODE MANAGER ON MACHINE1';
nmDisconnect();

bea_home = '/home/oracle/weblogic12.1.1';
pathseparator = '/';
admin_username = 'weblogic';
admin_password = 'magic12c';
listen_address_machine1 = '172.31.0.175';
listen_address_machine2 = '172.31.0.113';
nodemanager_listen_port = '5556';
admin_server_listen_port = '7001';
domain_name = 'base_domain';

domain_home = bea_home + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domain_name;
admin_server_url='t3://' + listen_address_machine2 + ':' + admin_server_listen_port;

print 'CONNECT TO NODE MANAGER ON MACHINE1';
nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');

print 'CONNECT TO ADMIN SERVER';
connect(admin_username, admin_password, admin_server_url);

print 'STOPPING MANAGED SERVERS ON MACHINE1';
shutdown('server1','Server','true',1000,'true');
# server3 will be added in a later stadium when we scale the cluster
#shutdown('server3','Server','true',1000,'true');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE1';
nmDisconnect();

print 'CONNECT TO NODE MANAGER ON MACHINE2';
nmConnect(admin_username, admin_password, listen_address_machine1, nodemanager_listen_port, domain_name, domain_home, 'ssl');

print 'STOPPING MANAGED SERVERS ON MACHINE2';
shutdown('server2','Server','true',1000,'true');

print 'STOPPING ADMIN SERVER ON MACHINE2';
shutdown('AdminServer','Server','true',1000,'true');

print 'DISCONNECT FROM NODE MANAGER ON MACHINE2';
nmDisconnect();

For these WLST scripts, we create two shell scripts that respectively run the start and stop script, i.e.,

#!/bin/sh

BEA_HOME="/home/oracle/weblogic12.1.1"
export BEA_HOME

WL_HOME="${BEA_HOME}/wlserver_12.1"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/startDomain.py

#!/bin/sh

BEA_HOME="/home/oracle/weblogic12.1.1"
export BEA_HOME

WL_HOME="${BEA_HOME}/wlserver_12.1"
export WL_HOME

. ${WL_HOME}/common/bin/wlst.sh ${BEA_HOME}/deploy/scripts/stopDomain.py

To start the domain we can use:

[oracle@edu-wls-rh scripts]$ ./DomainStartService.sh

CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/home/oracle/weblogic12.1.1/utils/config/10.3/config-launch.jar::/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbynet.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbyclient.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/common/derby/lib/derbytools.jar::

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

CONNECT TO NODE MANAGER ON MACHINE2
Connecting to Node Manager ...
<Jan 16, 2012 1:59:54 PM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Jan 16, 2012 1:59:54 PM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
Successfully Connected to Node Manager.
START ADMIN SERVER ON MACHINE2
Starting server AdminServer ...
Successfully started server AdminServer ...
CONNECT TO ADMIN SERVER
Connecting to t3://172.31.0.113:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

START MANAGED SERVERS ON MACHINE2

Starting server server2 ...............................................................
Server with name server2 started successfully
DISCONNECT FROM NODE MANAGER ON MACHINE2
Successfully disconnected from Node Manager.
CONNECT TO NODE MANAGER ON MACHINE1
Connecting to Node Manager ...
Successfully Connected to Node Manager.
START MANAGED SERVERS ON MACHINE1

Starting server server1 ..................................................................................
Server with name server1 started successfully
DISCONNECT FROM THE ADMIN SERVER
Disconnected from weblogic server: AdminServer
DISCONNECT FROM NODE MANAGER ON MACHINE1
Successfully disconnected from Node Manager.

To stop the domain we can use:

[oracle@edu-wls-rh scripts]$ ./DomainStopService.sh

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

CONNECT TO NODE MANAGER ON MACHINE1
Connecting to Node Manager ...
<Jan 16, 2012 2:05:40 PM CET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Jan 16, 2012 2:05:40 PM CET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
Successfully Connected to Node Manager.
CONNECT TO ADMIN SERVER
Connecting to t3://172.31.0.113:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

STOPPING MANAGED SERVERS ON MACHINE1
Shutting down the server server1 with force=true while connected to AdminServer ...
DISCONNECT FROM NODE MANAGER ON MACHINE1
Successfully disconnected from Node Manager.
CONNECT TO NODE MANAGER ON MACHINE2
Connecting to Node Manager ...
Successfully Connected to Node Manager.
STOPPING MANAGED SERVERS ON MACHINE2
Shutting down the server server2 with force=true while connected to AdminServer ...
STOPPING ADMIN SERVER ON MACHINE2
Shutting down the server AdminServer with force=true while connected to AdminServer ...
WLST lost connection to the WebLogic Server that you were
connected to, this may happen if the server was shutdown or
partitioned. You will have to re-connect to the server once the
server is available.
Disconnected from weblogic server: AdminServer
Disconnected from weblogic server:
DISCONNECT FROM NODE MANAGER ON MACHINE2
Successfully disconnected from Node Manager.

Deployment

The application can be found in the post WebLogic12c in Action (in the ‘My first JavaEE6 application’ section). First, we create the following directory structure in /home/oracle/weblogic12.1.1/deploy:

/loadtest
	/app
		LoadTest6.ear
	/plan

By using weblogic.PlanGenerator, we generate a deployment plan:

[oracle@edu-wls-rh ~]$ cd weblogic12.1.1/wlserver_12.1/server/bin/
[oracle@edu-wls-rh bin]$ . ./setWLSEnv.sh
CLASSPATH=/home/oracle/weblogic12.1.1/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/weblogic12.1.1/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic_sp.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/weblogic.jar:/home/oracle/weblogic12.1.1/modules/features/weblogic.server.modules_12.1.1.0.jar:/home/oracle/weblogic12.1.1/wlserver_12.1/server/lib/webservices.jar:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/weblogic12.1.1/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:

PATH=/home/oracle/weblogic12.1.1/wlserver_12.1/server/bin:/home/oracle/weblogic12.1.1/modules/org.apache.ant_1.7.1/bin:/home/oracle/jrrt-4.0.1-1.6.0/jre/bin:/home/oracle/jrrt-4.0.1-1.6.0/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/oracle/bin

Your environment has been set.
[oracle@edu-wls-rh bin]$ java weblogic.PlanGenerator -plan /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Plan.xml -root /home/oracle/weblogic12.1.1/deploy/loadtest/ /home/oracle/weblogic12.1.1/deploy/loadtest/app/LoadTest6.ear
Generating plan for application /home/oracle/weblogic12.1.1/deploy/loadtest/app/LoadTest6.ear
Export option is: dependencies
Exporting properties...
Saving plan to /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Plan.xml...
<Jan 17, 2012 1:56:43 PM CET> <Info> <J2EE Deployment SPI> <BEA-260086> <The descriptor information at /home/oracle/weblogic12.1.1/deploy/loadtest/plan/Web.war/WEB-INF/weblogic.xml was saved to the configuration area.>
<Jan 17, 2012 1:56:43 PM CET> <Info> <J2EE Deployment SPI> <BEA-260072> <Saved configuration for application, LoadTest6.ear>

When the deployment plan has been generated, the files Plan.xml and weblogic.xml are created automatically. The other files, weblogic-application.xml and weblogic-ejb-jar.xml, we have to create ourselves according to the below directory structure:

/loadtest
	/app
		LoadTest6.ear
	/plan
		/LoadTest6.ear
			/META-INF
				weblogic-application.xml
		/Model.jar
			/META-INF
				weblogic-ejb-jar.xml
		/Web.war
			/WEB-INF
				weblogic.xml
		Plan.xml

in which the deployment overrides (weblogic-application.xml, weblogic-ejb-jar.xml and weblogic.xml) have the following contents:

<weblogic-application xmlns="http://xmlns.oracle.com/weblogic/weblogic-application" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-application http://xmlns.oracle.com/weblogic/weblogic-application/1.4/weblogic-application.xsd">
</weblogic-application>

<weblogic-ejb-jar xmlns="http://xmlns.oracle.com/weblogic/weblogic-ejb-jar" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-ejb-jar http://xmlns.oracle.com/weblogic/weblogic-ejb-jar/1.1/weblogic-ejb-jar.xsd">
	<weblogic-enterprise-bean>
		<ejb-name>Company</ejb-name>
		<enable-call-by-reference>True</enable-call-by-reference>
	</weblogic-enterprise-bean>
</weblogic-ejb-jar>

<weblogic-web-app xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.4/weblogic-web-app.xsd">
</weblogic-web-app>

The Java EE specification requires that EJB components invoked through their remote interfaces must use pass-by-value semantics, meaning that method parameters are copied during the invocation. Changes made to a parameter in the bean method are not reflected in the caller’s version of the object. Copying method parameters is required in the case of a true remote invocation, of course, because the parameters are serialized by the underlying RMI infrastructure before being provided to the bean method. Pass-by-value semantics are also required between components located in different enterprise applications in the same Java virtual machine due to classloader constraints. EJB components located in the same enterprise application archive (.ear) file are loaded by the same classloader and have the option of using pass-by-reference semantics for all invocations, eliminating the unnecessary copying of parameters passed during the invocation and improving performance. By setting the enable-call-by-reference parameter to true in weblogic-ejb-jar.xml, we enable this feature a specific bean in the application. Local references always use pass-by-reference semantics and are unaffected by the enable-call-by-reference setting. When we deploy an EJB with a remote interface and do not enable call by reference, WebLogic will issue a warning of the performance cost, i.e.,

<Jan 16, 2012 5:01:56 PM CET> <Warning> <EJB> <BEA-010202> <Call-by-reference is not enabled for EJB Company. The server will have better performance if it is enabled. To enable call-by-reference, set the enable-call-by-reference element to True in the weblogic-ejb-jar.xml deployment descriptor or corresponding annotation for this EJB.>

Load balancing

To install the Apache HTTP Server we can follow these steps:

Unpack the httpd-2.2.21.tar.gz file:
- gzip -d httpd-2.2.21.tar.gz
- tar xvf httpd-2.2.21.tar
- cd httpd-2.2.21
The next step is to configure:
- ./configure --prefix=/home/oracle/weblogic12.1.1/apache
Next, compile the various parts for the Apache HTTP Server by using:
- make
To install the Apache HTTP Server we use:
- make install
Open the httpd.conf (/home/oracle/weblogic12.1.1/apache/conf) file and adjust the following directives:
- Listen 7777
- ServerName 172.31.0.113
To test the set-up, start the Apache HTTP Server:
- Navigate to /home/oracle/weblogic12.1.1/apache/bin.
- Run: ./apachectl -k start (To stop the Apache HTTP Server we can use: ./apachectl -k stop).
- Open a browser en type the following URL: http://172.31.0.113:7777.

To install the the WebLogic plugin we can follow these steps:

The WebLogic plugins can be downloaded here.
Unzip the distribution that is appropriate for the operating system, for example, WLSPlugin11g-64bitApache2.2-linux64-x86_64.zip.
Copy the libclntsh.so.11.1, libnnz11.so and libwlssl.so files to the ${APACHE_HOME}/lib directory. Note in our case ${APACHE_HOME} is /home/oracle/weblogic12.1.1/apache.
Copy the mod_wl.so file to the ${APACHE_HOME}/modules directory.
Create a configuration file (for example, mod_wl.conf) for the WebLogic plugin and place this file in the ${APACHE_HOME}/conf directory.

The contents of the mod_wl.conf look as follows:

LoadModule weblogic_module   "/home/oracle/weblogic12.1.1/apache/modules/mod_wl.so"

<IfModule weblogic_module>
	ConnectTimeoutSecs 10
	ConnectRetrySecs 2
	DebugConfigInfo ON
	WLSocketTimeoutSecs 2
	WLIOTimeoutSecs 300
	Idempotent ON
	FileCaching ON
	KeepAliveSecs 20
	KeepAliveEnabled ON
	DynamicServerList ON
	WLProxySSL OFF
</IfModule>

<Location /LoadTest6>
	SetHandler weblogic-handler
	WebLogicCluster 172.31.0.175:7002,172.31.0.113:7003
</Location>

To let the Apache HTTP Server pick up the configuration we add the following to the httpd.conf file:

# put it near the end of the file where all the other includes are present
# mod_wl configuration
Include conf/mod_wl.conf

Restart the HTTP Server (./apachectl -k stop and ./apachectl -k start).

We are not happy with the load (do not know why, just a hypotheses) and need an extra server. We have two scaling options: vertical and horizontal scaling. Vertical scaling relates to adding more CPUs to a machine. To better utilize the server hardware we can add more WebLogic instances to the machine that could lead to increased application throughput. To determine if this is indeed the case we need to benchmark. Benchmarking for scalability is about measuring resource utilization. Good scalability means that service levels can be maintained while the workload is increased. If an application does not scale well, it is not fully utilizing the hardware. Consequently, throughput will degrade. Ideally, a linear load increase should lead to a linear degradation in service levels and performance. Linear scalability can be approached when so-called share nothing clusters are used. The nodes provide the same functionality and know nothing about other nodes in the cluster (no HTTP session replication). In this case, the computing ability of the cluster increases almost linearly as more nodes are added to the cluster, if the back-end information systems, such as a database, are powerful enough.

Applications that ‘share nothing’ are usually sharing state through the database. The application-tier can scale as far as when the database becomes a bottleneck. In general, relying on a single shared resource will eventually cause contention for that resource and thus limit the scalability. Caching is a good resolution. When we cache data at the application-tier we avoid calls to the database (and also avoid relational data to object data conversions). Caching solutions, such as Coherence, provide different kind of caching, i.e., replicated and partitioned. Replicated does not scale well when cache writes are involved as the data needs to be replicated across all the nodes in the grid. A partioned cache, on the other hand, scales very well when cache writes are involved as data ownership is spread throughout the cluster (the system automatically rebalances the data when the number of nodes in the grid changes – we do not need to decide on how to partition the data, it comes out of the box). Another plus is that access to the cache means at most one network trip, this in order to maintain linear scalability. An optimization on read-access can be made when data can be obtained locally (sticky access) in this case a hybrid solution such as the near cache can be applied.

Horizontal scaling relates to adding more machines to the environment, which gives a failover capability that we cannot get with vertical scaling. A good approach is to combine both scaling techniques to obtain better CPU utilization and failover capability.

In order to create a new server based on server1 open the admin console:

Click environment, servers, select server1 and click clone.
Change the server name to server3 and listen port to 7004.
Click OK.

Create a JMS server and a file store and target these to server3:

Click services, messaging, JMS servers.
Click new and enter the name as JMSServer3.
Click create a new store:
- Choose file store as type and click next.
- Enter the following parameters:
  - name: FileStore3
  - target: server3
  - directory: /home/oracle/weblogic12.1.1/deploy
- Click OK.
Click next and choose as target, server3.
Click finish.

Add the created JMS server to the subdeployment of the JMS module:

Click services, messaging, JMS modules and choose systemmodule.
Click the subdeployments tab and choose subdeployment.
Add JMSServer3 and click save.

Start server3 by using the admin console:

Click environment, servers.
Click the control tab.
Select server3 and click start.

To test the configuration we can use the URL: http://172.31.0.113:7777/LoadTest6/?__WebLogicBridgeConfig. When the URL is accessed the General Server List should show the servers in the cluster, i.e.,

WebLogic Server Plugin version 1.1, <WLSPLUGINS_11.1.1.4.0_LINUX.X64_101209.1115>
Query String: '?__WebLogicBridgeConfig'
WebLogic Cluster List:
General Server List:
Host: '172.31.0.175' Port: 7004 SecurePort: 0 Status: OK
Host: '172.31.0.113' Port: 7003 SecurePort: 0 Status: OK
Host: '172.31.0.175' Port: 7002 SecurePort: 0 Status: OK
ConnectRetrySecs: '2'
ConnectTimeoutSecs: '10'
WLCookieName: JSESSIONID
Debug: 'no debugging'
DebugConfigInfo: 'ON'
DefaultFileName: ''
DisableCookie2Server: OFF
DynamicServerList: 'ON'
ErrorPage: ''
FileCaching: ON
Idempotent: ON
KeepAliveEnabled: ON
KeepAliveSecs: 20
MaxPostSize: '0'
MaxSkipTime: '10'
PathPrepend: ''
PathTrim: ''
QueryFromRequest: OFF
WLForwardUriUnparsed: OFF
WLAllowDoubleEscapedURI: OFF
SecureProxy: 'OFF'
StatPath: 'false'
WLDNSRefreshInterval: '0'
WLIOTimeoutSecs(old name is HungServerRecoverSecs): '300'
WLLogFile: ''
WLSocketTimeoutSecs: '2'
WLProxySSL: OFF
WLProxyPassThrough: OFF
WLProxySSLPassThrough: OFF

Runtime statistics:
requests: 0
successful requests: 0
Exception objects created: 0
Exception Objects deleted: 0
URL Objects created: 1
URL Objects deleted: 0
connections recycled: 0
UNKNOWN_ERROR_CODE exceptions: 0
CONNECTION_REFUSED exceptions: 0
CONNECTION_TIMEOUT exceptions: 0
READ_ERROR_FROM_CLIENT exceptions: 0
READ_ERROR_FROM_SERVER exceptions: 0
READ_ERROR_FROM_FILE exceptions: 0
WRITE_ERROR_TO_CLIENT exceptions: 0
WRITE_ERROR_TO_SERVER exceptions: 0
WRITE_ERROR_TO_FILE exceptions: 0
READ_TIMEOUT exceptions: 0
WRITE_TIMEOUT exceptions: 0
UNKNOWN_HOST exceptions: 0
NO_RESOURCES exceptions: 0
PROTOCOL_ERROR exceptions: 0
CONFIG_ERROR exceptions: 0
FAILOVER_REQUIRED exceptions: 0
POST_TIMEOUT exceptions: 0
REQUEST_ENTITY_TOO_LARGE exceptions: 0
HALF_OPEN_SOCKET_RETRY exceptions: 0
BAD_REQUEST_FROM_CLIENT exceptions: 0
SSL_INIT_ERROR exceptions: 0
Build date/time: Dec 20 2010 22:11:08
Change Number: 1013

The runtime statistics depict results when a number of requests (http://172.31.0.113:7777/LoadTest6/testservlet) are made. The new server is added to the general server list (which is due to the used DynamicServerList configuration). As the application is stateless and no sessions are created (no users need to login), we have no session binding. To get an idea how the load is balanced, click in the admin console on deployments, LoadTest6 and subsequently click on the workload, monitoring tab.

Testing

We can test the environment by using the The Grinder. To set-up The Grinder, we create the following scripts:

A script to set the environment (setGrinderEnv.sh):

#!/bin/sh
GRINDERPATH=/home/oracle/grinder-3.4
export GRINDERPATH

JAVA_HOME=/home/oracle/jdk1.6.0_24
export JAVA_HOME

USER_MEM_ARGS="-server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC"
export USER_MEM_ARGS
GRINDERPROPERTIES=${GRINDERPATH}/examples/grinder.properties
export GRINDERPROPERTIES
CLASSPATH=${GRINDERPATH}/lib/grinder.jar:${CLASSPATH}
export CLASSPATH

PATH=${JAVA_HOME}/bin:${PATH}
export PATH

A script to start the console (startConsole.sh):

#!/bin/sh
source setGrinderEnv.sh
java ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Console

A script to start an agent that starts the test processes (startAgent.sh):

#!/bin/sh
source setGrinderEnv.sh
java ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Grinder ${GRINDERPROPERTIES}

The grinder.properties file has, among others, the following properties:

# The file name of the script to run.
#
# Relative paths are evaluated from the directory containing the
# properties file. The default is "grinder.py".
grinder.script = test.py

# The number of worker processes each agent should start. The default
# is 1.
grinder.processes = 1

# The number of worker threads each worker process should start. The
# default is 1.
grinder.threads = 1

# The number of runs each worker process will perform. When using the
# console this is usually set to 0, meaning "run until the console
# sneds a stop or reset signal". The default is 1.
grinder.runs = 0

The test.py script has the following contents:

from net.grinder.script.Grinder import grinder
from net.grinder.script import Test
from net.grinder.plugin.http import HTTPRequest

test1 = Test(1, "Request resource")
request1 = test1.wrap(HTTPRequest())

class TestRunner:
    def __call__(self):
        result = request1.GET("http://172.31.0.113:7777/LoadTest6/testservlet")

Start the test by running startConsole.sh and subsequently startAgent.sh:

During the loadtest we are interested in monitoring the paging. The Linux memory handler manages the allocation of physical memory by freeing portions of physical memory when possible. All processes use memory, but each process does not need all its allocated memory all the time. Taking advantage of this fact, the kernel frees up physical memory by writing some or all of a process’ memory to disk until it is needed again. The kernel uses paging and swapping to perform this memory management. Paging refers to writing portions (pages) of a process’ memory to disk. Swapping refers to writing the entire process to disk. When pages are written to disk, the event is called a page-out, and when pages are returned to physical memory, the event is called a page-in. A page fault occurs when the kernel needs a page, finds it does not exist in physical memory because it has been paged-out, and re-reads it in from disk. When the kernel detects that memory is running low, it attempts to free up memory by paging out. Though this may happen briefly from time to time, if page-outs are plentiful and constant, the kernel can reach a point where it is actually spending more time managing paging activity than running the applications, and system performance suffers. To monitor paging we can use, for example, vmstat 60 10 (which runs vmstat with ten updates, 60 seconds apart):

# Output machine1 (where server1 and server3 are running)
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 1185576  40744 521408    0    0    63    10  384  347  6  3 90  1  0
 2  0      0 1178368  40852 527824    0    0     0   132 2921 2509 21 10 68  0  0
 1  0      0 1174384  40924 530588    0    0     0    63 2442 2130 17  9 74  0  0
 0  0      0 1169796  41020 534336    0    0     0    77 2798 2408 19 10 71  0  0
 0  0      0 1162728  41124 540188    0    0     1   130 2941 2614 18 11 70  0  0
 1  0      0 1155028  41216 544020    0    0     0    76 2899 2549 21 10 69  0  0
 0  0      0 1150812  41316 548804    0    0     0   105 3130 2790 21 11 68  0  0
 0  0      0 1140256  41420 554224    0    0     0   120 2947 2595 19 11 69  0  0
 1  0      0 1121904  41528 561132    0    0     0   140 2896 2557 19 11 70  0  0
 3  0      0 1110760  41628 567212    0    0     0   123 2963 2657 18 13 68  0  0

Output machine2 (where the admin server and server2 are running)
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1622200  34556 574508    0    0    86    15  642  305  2  0 97  1  0
 0  0      0 1617552  34656 578236    0    0     0    87 2685 1967  5  2 93  0  0
 0  0      0 1615476  34748 580480    0    0     0    55 2309 1562  3  1 95  0  0
 0  0      0 1612872  34844 582992    0    0     0    64 2616 1898  4  2 95  0  0
 1  0      0 1609964  34948 585576    0    0     0    73 2735 2017  4  2 94  0  0
 1  0      0 1605508  35052 589116    0    0     0    81 2712 1985  4  2 94  0  0
 1  0      0 1602076  35152 592352    0    0     0    77 2913 2313  5  2 93  0  0
 0  0      0 1596448  35260 595104    0    0     0    69 2773 2095  4  2 94  0  0
 0  0      0 1590000  35360 599212    0    0     0    92 2693 2038  4  2 94  0  0
 0  0      0 1587164  35456 601748    0    0     0   106 2633 1861  5  2 93  0  0

 FIELD DESCRIPTION FOR VM MODE
   Procs
       r: The number of processes waiting for run time.
       b: The number of processes in uninterruptible sleep.

   Memory
       swpd: the amount of virtual memory used.
       free: the amount of idle memory.
       buff: the amount of memory used as buffers.
       cache: the amount of memory used as cache.
       inact: the amount of inactive memory. (-a option)
       active: the amount of active memory. (-a option)

   Swap
       si: Amount of memory swapped in from disk (/s).
       so: Amount of memory swapped to disk (/s).

   IO
       bi: Blocks received from a block device (blocks/s).
       bo: Blocks sent to a block device (blocks/s).

   System
       in: The number of interrupts per second, including the clock.
       cs: The number of context switches per second.

   CPU
       These are percentages of total CPU time.
       us: Time spent running non-kernel code. (user time, including nice time)
       sy: Time spent running kernel code. (system time)
	   id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
       wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
       st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

The values for si and so are both zero, indicating there are no page-ins and page-outs.

Let us look at some of the diagnostics collected by WebLogic. To this end, open the admin console, click deployments and click the monitoring tab. This page displays monitoring information for all applications deployed to the domain. The JMS tab displays monitoring information for all JMS destinations (note that the application only sends a message when a new person is added):

JMSServer1!JMSServer1.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40
JMSServer2!JMSServer2.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40
JMSServer3!JMSServer3.TemporaryQueue0 - Bytes Received Count: 1230, Messages Received Count: 40
SystemModule!JMSServer1@DistributedQueue - Bytes Received Count: 719274, Messages Received Count: 2347
SystemModule!JMSServer2@DistributedQueue - Bytes Received Count: 703990, Messages Received Count: 2297
SystemModule!JMSServer3@DistributedQueue - Bytes Received Count: 715648, Messages Received Count: 2335

The EJB (stateless and message-driven) tab displays monitoring information for all the Enterprise JavaBeans (EJBs):

Company	example - Pooled Beans Current Count: 3, Access Total Count: 348936, Transactions Committed Total Count: 348899, Transactions Rolled Back Total Count: 37
CompanyMDB example - Access Total Count: 6979, Processed Message Count: 2347, Transactions Committed Total Count: 6979

The JDBC tab displays monitoring information for all JDBC data sources:

DataSource server1 - Active Connections High Count: 2, Connection Delay Time: 155, PrepStmt Cache Access Count: 105396, Reserve Request Count (cummulative running count of requests for a connection): 116715, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0
DataSource server2 - Active Connections High Count: 2, Connection Delay Time: 127, PrepStmt Cache Access Count: 10173, Reserve Request Count (cummulative running count of requests for a connection): 116716, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0
DataSource server3 - Active Connections High Count: 2, Connection Delay Time: 143, PrepStmt Cache Access Count: 99600, Reserve Request Count (cummulative running count of requests for a connection): 116698, Waiting For Connection Total (cumulative running count of requests for a connection that had to wait before getting a connection) 0

The workload tab shows statistics for the Work Managers, constraints, and policies that are configured for application deployments:

default server1 - Pending Requests: 0, Completed Requests: 118664
default server2 - Pending Requests: 0, Completed Requests: 118616
default server3 - Pending Requests: 0, Completed Requests: 118635

We use the JRockit Mission Control Flight Recording, to see if there are any hick-ups due to garbage collections:

One thing to note is that the garbage collections did not run at same time:

server1 (172.31.0.175): 03:46:18 154ms – 03:46:18 295ms
server2 (172.31.0.113): 03:43:29 508ms – 03:43:29 751ms
server3 (172.31.0.175): 03:43:52 399ms – 03:43:52 650ms

In general, JVM instances running on the same machine will typically not run the garbage collection at the same time. This means that we will have a JVM available to process application requests on other available CPUs. This is an advantage of vertical scaling that leads to a higher application throughput.

To get some insight what effect the garbage collection had on the application we use the events environment of the flight recording. We enable the following events:

Java application – Java blocked
Java application – Java wait
Java virtual machine – GC – garbage collection
WebLogic Server – EJB – EJB business method invocation
WebLogic Server – JDBC – JDBC statement execute
WebLogic Server – Servlet – Servlet invocation

The thread group ‘thread group for queue: weblogic.socket.Muxer’ contain the muxer threads. As a native muxer is used and we have 2 CPUs available, the number of threads is equal to #CPUs + 1 = 3. These threads are always showing this behavior, i.e., one thread at a time is active; picking requests of the sockets and put it in the execute queue. Here it will be picked up by an execute thread from the thread group ‘pooled threads’, which will process it, i.e., the servlet invocation (light green), the EJB business method invocation (blue) and the JDBC statement execution (purple). Note that during the garbage collection the execute threads have to wait (yellow) and a pause in the processing of the work defined for the execute thread is introduced.

References

[1] WebLogic 12c Documentation Library.
[2] Monitoring Virtual Memory with vmstat.

load test, multiple machines, scaling, WebLogic cluster Leave a Comment more...

« Older Entries

References

Messaging bridge

Configuration

WLST

Test

Store-and-forward service

Configuration

WLST

Test

Performance considerations

References

Some notes up front

Developing the JMS client with Spring

References

Create domain

Application

References

Configure the operating system

Change the hostname and map it to a static ip address

Packet loss minimization

Maximum number of open file descriptors

TCP/IP

Network interface card (NIC)

Maximum transaction unit (MTU)

Swapping

Large pages

Start and stop scripts

Class loading

Security

Configure mod_cluster

Multiple machine set-up

References

Preparing the environment

Installation

Configuration

Thread pools

Data source

JMS

Enterprise beans

Application

Deployment

Using the Apache HTTP Server as a front-end

Clustering

Configure load balancing

Load test

Domains

References

Topics that will be covered

Registration

Prepare the environment

Setting up the SOA repository

Installation

Configuration

Test the environment

Scaling

Start and stop scripts

Load balancer

Access Manager

References

Why should you attend?

Registration

Install WebLogic

Create domain

Start and stop scripts

Deployment

Load balancing

Testing

References

Receive FREE Updates

Magic Account…

Top Magic Users

Magic Archives

Recent Tech. Discussion

Sitemeter Status

ClusterMap 7-Nov-2011 till Date

ClusterMap 6-Nov-2010 till 7-Nov-2011

Tags