In this post we show a step-by-step example of how to set-up a high available tuned Java EE environment. We start with installing a Java Virtual Machine (in this case HotSpot) and JBoss Application Server. We continue with configuring a standalone server, tune the operating system and the Java Virtual Machine. Subsequently, we show how to set-up data source and JMS resources that are used by an application (consisting of a servlet, persistence, a stateless enterprise bean and a message-driven bean). Have some words about thread-pooling and enterprise beans. Show how to use the command-line interface to obtain run-time statistics. Show the steps involved in setting up a cluster by using standalone servers and configure a load balancer. Finally, we take a look at how to configure a domain in order to manage multiple server instances from a single control point.

JBoss AS7 is designed around a new kernel, which is now based on:

  • JBoss Modules – handle class loading of resources in the container. Sort of a thin bootstrap wrapper for executing an application in a modular environment.
  • Modular Service Container – provides a way to install, uninstall, and manage services used by a container. It further enables resources injection into services and dependency management between services.

Preparing the environment

Some operating system tweaks are worth considering when we do not want to run against system restrictions:

  • Packet loss minimization – The operating system buffers must be large enough to handle incoming network traffic while the application is paused during garbage collection. Usually UDP (User Datagram Protocol) is used in order to transmit multicast messages to server instances in a cluster; to limit the need to retransmit UDP messages the size of the operating system buffers must be set appropriately to avoid excessive UDP datagram loss.
  • Maximum number of open file descriptors – Most operating systems handle sockets as a form of file access and use file descriptors to keep track of which sockets are open. To contain the resources per process, the operating system restricts the number of file descriptors per process. Linux limits the number of open file descriptors per process, by default this is equal to 1024. It could be that the 1024 limit does not offer optimal performance.
  • TCP/IP – On some systems the default value for the time wait interval is too high and needs to be adjusted. When the number approaches the maximum number of file descriptors per process, the application’s throughput will degrade, i.e., new connections have to wait for a free space in the application’s file descriptor table.
  • Timesources – Linux has several timesources to choose from; the fastest being TSC (time stamp counter) and is used by default. However, if during start-up inconsistencies are found Linux switches to a slower timesource. This can have a negative performance impact.
  • Network interface card (NIC) – Configure the network card at it’s maximum link speed and at full duplex.

In the post Tuning GlassFish for Performance all the tweaks are explained in detail and the steps involved in adjusting the operating system are explained as well. The post also contains suggestions in tuning the application server and Java Virtual Machine which are applicable to the JBoss application server as well.

Installation

  • Install a JDK (of which a distribution can be downloaded here).
  • Install JBoss (of which a distribution can be downloaded here).

We will use the versions: jdk-6u31-linux-x64.bin for the JDK and jboss-as-7.1.0.Final.zip for JBoss. The installation steps are straight forward, for the JDK we can just run the bin file. JBoss can be installed by unzipping the file. When we are done we have the following directory structure:

/home/jboss
	/jboss-as-7.1.0.Final (${JBOSS_HOME)
	/jdk1.6.0_31 (${JAVA_HOME})

After the set-up, we perform a startup test to validate that there are no major problems with the Java VM/operating system combination. To test the set-up navigate to ${JBOSS_HOME}/bin and run standalone.sh. Before running standalone.sh add the following lines:

#!/bin/sh

# These lines must be added
export JAVA_HOME=/home/jboss/jdk1.6.0_31
export PATH=${PATH}:${JAVA_HOME}/bin

To set JVM parameters we can edit standalone.conf, for example,

#
# Specify options to pass to the Java VM.
#
if [ "x$JAVA_OPTS" = "x" ]; then
   JAVA_OPTS="-server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
   JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
   JAVA_OPTS="$JAVA_OPTS -Djboss.server.default.config=standalone.xml"
else
   echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi

More information on tuning the JVM can be found here. Note that the default configuration is standalone.xml. This configuration does not by default include all subsystems, for example, messaging is not included. The configuration file standalone-full.xml has all the subsystems included. In the examples below we will work with standalone.xml and add specific subsystems when needed, hereby keeping the memory footprint of the application server to what is needed. When the standalone.sh is run the following is observed:

=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/jboss-as-7.1.0.Final

  JAVA: /home/jboss/jdk1.6.0_31/bin/java

  JAVA_OPTS:  -XX:+UseCompressedOops -XX:+TieredCompilation -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

=========================================================================

12:32:01,842 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
12:32:02,010 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
12:32:02,057 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
12:32:03,047 INFO  [org.xnio] XNIO Version 3.0.3.GA
12:32:03,063 INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
12:32:03,078 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
12:32:03,088 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.2.GA
12:32:03,145 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
12:32:03,151 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 26) JBAS016200: Activating ConfigAdmin Subsystem
12:32:03,212 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 44) JBAS013101: Activating Security Subsystem
12:32:03,216 INFO  [org.jboss.as.security] (MSC service thread 1-3) JBAS013100: Current PicketBox version=4.0.6.final
12:32:03,231 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
12:32:03,237 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 39) JBAS011940: Activating OSGi Subsystem
12:32:03,282 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 48) JBAS015537: Activating WebServices Extension
12:32:03,347 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 38) JBAS011800: Activating Naming Subsystem
12:32:03,399 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
12:32:03,406 INFO  [org.jboss.as.connector] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
12:32:03,478 INFO  [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: Starting Naming Service
12:32:03,605 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) JBAS015400: Bound mail session [java:jboss/mail/Default]
12:32:03,633 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-2) JBoss Web Services - Stack CXF Server 4.0.1.GA
12:32:03,716 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-4) Starting Coyote HTTP/1.1 on http--127.0.0.1-8080
12:32:03,976 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on /127.0.0.1:9999
12:32:03,984 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) JBAS015012: Started FileSystemDeploymentService for directory /home/jboss/jboss-as-7.1.0.Final/standalone/deployments
12:32:03,986 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on /127.0.0.1:4447
12:32:04,119 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
12:32:04,136 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 2726ms - Started 134 of 205 services (70 services are passive or on-demand)

To verify that the server is reachable, enter the following URL: http://localhost:8080.

To use the command line interface, we can start jboss-cli.sh (also add the JAVA_HOME and PATH variables to the script). To stop the started server, we can use

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect
[standalone@localhost:9999 /] :shutdown
{"outcome" => "success"}

We can also shuwdown the server by using:

[jboss@axis-into-ict bin]$ ./jboss-cli.sh --connect command=:shutdown
{"outcome" => "success"}

When browsing through the application server folders we see that its file system is divided into two parts: standalone servers (${JBOSS_HOME}/standalone) and domain servers (${JBOSS_HOME}/domain). An application server node which is not configured as part of a domain is qualified as a standalone server.

The ${JBOSS_HOME}/bin folder is where we start application server instances. As we already saw a standalone server can be started by using standalone.sh. When using a domain we will use domain.sh, which starts the application servers specified in the domain configuration file as we will see later on. The ${JBOSS_HOME}/modules folder contains the application server’s set of libraries, which are part of the server distribution. Each module is a pluggable unit. By using a static file system approach we can load modules, all we need to do is provide the location of the modules. When looking at the standalone.sh script (line 177 and on) we see an example of how to start the server and loading modules, for example, we could use

[jboss@axis-into-ict ~]$ cd jdk1.6.0_31/bin/
[jboss@axis-into-ict bin]$ ./java -jar /home/jboss/jboss-as-7.1.0.Final/jboss-modules.jar -mp /home/jboss/jboss-as-7.1.0.Final/modules/ org.jboss.as.standalone -Djboss.home.dir=/home/jboss/jboss-as-7.1.0.Final/

The module path (-mp) argument points to the root directory which will be searched by the module loader for module definitions. A module is defined by using an XML descriptor, for example,

<module xmlns="urn:jboss:module:1.1" name="javax.jms.api">
    <dependencies>
        <module name="javax.transaction.api" export="true"/>
    </dependencies>
    <resources>
        <resource-root path="jboss-jms-api_1.1_spec-1.0.0.Final.jar"/>
        <!-- Insert resources here -->
    </resources>
</module>

A module definition contains two main elements: the module dependencies and the resources defined in the module.

When we want to use the admin console, we first have to add a user. To this end run add-user.sh (${JBOSS_HOME}/bin) and follow the instructions given on the screen, for example,

[jboss@axis-into-ict bin]$ ./add-user.sh 

What type of user do you wish to add?
 a) Management User (mgmt-users.properties)
 b) Application User (application-users.properties)
(a): 

Enter the details of the new user to add.
Realm (ManagementRealm) :
Username : jboss
Password :
Re-enter Password :
About to add user 'jboss' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'jboss' to file '/home/jboss/jboss-as-7.1.0.Final/standalone/configuration/mgmt-users.properties'
Added user 'jboss' to file '/home/jboss/jboss-as-7.1.0.Final/domain/configuration/mgmt-users.properties'

Enter the following URL to access the admin console: http://hostname:9990/console and enter the log in credentials.

Configuration

The structure of the application server is maintained into a single file, which acts as a main reference for all server configurations. The default configuration files are named standalone.xml for standalone servers and domain.xml for an application server domain. An application server domain can be seen as a specialized server configuration, which also includes the domain and host controller set-up, which we will see later on. For now we will focus on the standalone configuration. The standalone.xml file is located in ${JBOSS_HOME}/standalone/configuration. An example looks as follows:

<server xmlns="urn:jboss:domain:1.1">
    <extensions>
        <extension module="org.jboss.as.clustering.infinispan"/>
        <extension module="org.jboss.as.configadmin"/>
		...
    </extensions>
    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <authentication>
                    <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <authentication>
                    <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket-binding native="management-native"/>
            </native-interface>
            <http-interface security-realm="ManagementRealm">
                <socket-binding http="management-http"/>
            </http-interface>
        </management-interfaces>
    </management>
    <profile>
        <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:configadmin:1.0"/>
        <subsystem xmlns="urn:jboss:domain:datasources:1.0">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:ee:1.0"/>
        <subsystem xmlns="urn:jboss:domain:ejb3:1.2">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:infinispan:1.1" default-cache-container="hibernate">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jca:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
        <subsystem xmlns="urn:jboss:domain:jmx:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:jpa:1.0">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:mail:1.0">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:naming:1.1"/>
        <subsystem xmlns="urn:jboss:domain:osgi:1.2" activation="lazy">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
        <subsystem xmlns="urn:jboss:domain:remoting:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:resource-adapters:1.0"/>
        <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
        <subsystem xmlns="urn:jboss:domain:security:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:threads:1.1"/>
        <subsystem xmlns="urn:jboss:domain:transactions:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:webservices:1.1">...</subsystem>
        <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
    </profile>
    <interfaces>
        <interface name="management">
            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
        </interface>
        <interface name="public">
            <inet-address value="${jboss.bind.address:127.0.0.1}"/>
        </interface>
    </interfaces>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="http" port="8080"/>
        <socket-binding name="https" port="8443"/>
        <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
        <socket-binding name="osgi-http" interface="management" port="8090"/>
        <socket-binding name="remoting" port="4447"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
    </socket-binding-group>
</server>

The application server contains a list of basic modules, called extensions, which are shared by all services. Extensions can be seen as a special type of module, which are used to extend the functionalities of the application server. They are stored in the ${JBOSS_HOME}/modules folder. Each extension is in turn picked up by the application server classloader at boot time, before any deployment. The application server has administration and management features that include a command-line interface (listens on port 9999) and an administration console (listens on port 9990). The profile can be seen as a collection of subsystems. Each subsystem contains a subset of functionalities used by the application server. For example, the web subsystem contains the definition of a set of connectors used by the container, the messaging subsystem defines the JMS configuration, and so on. The interfaces section contains the network interfaces and host names or IP addresses to which the application server is bound. Note that by default the server is bound to 127.0.0.1, which means it can only be reached locally. The IP address or host name can be changed at will, for example,

<interfaces>
	<interface name="management">
		<inet-address value="${jboss.bind.address.management:192.168.1.66}"/>
	</interface>
	<interface name="public">
		<inet-address value="${jboss.bind.address:192.168.1.66}"/>
	</interface>
</interfaces>

or even better gather the information from the network interface card, for example,

<interfaces>
	<interface name="management">
		<!--inet-address value="${jboss.bind.address.management:127.0.0.1}"/-->
		<nic name="eth0"/>
	</interface>
	<interface name="public">
		<!--inet-address value="${jboss.bind.address:127.0.0.1}"/-->
		<nic name="eth0"/>
	</interface>
</interfaces>

When we would now start the server the following is observed:

[jboss@axis-into-ict bin]$ ./standalone.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/jboss-as-7.1.0.Final

  JAVA: /home/jboss/jdk1.6.0_31/bin/java

  JAVA_OPTS:  -XX:+UseCompressedOops -XX:+TieredCompilation -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

=========================================================================

16:29:17,748 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
16:29:17,935 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
16:29:17,983 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
16:29:18,991 INFO  [org.xnio] XNIO Version 3.0.3.GA
16:29:18,998 INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
16:29:19,023 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
16:29:19,043 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.2.GA
16:29:19,070 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
16:29:19,086 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 26) JBAS016200: Activating ConfigAdmin Subsystem
16:29:19,167 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
16:29:19,210 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 48) JBAS015537: Activating WebServices Extension
16:29:19,214 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 44) JBAS013101: Activating Security Subsystem
16:29:19,224 INFO  [org.jboss.as.security] (MSC service thread 1-1) JBAS013100: Current PicketBox version=4.0.6.final
16:29:19,226 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 39) JBAS011940: Activating OSGi Subsystem
16:29:19,228 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 38) JBAS011800: Activating Naming Subsystem
16:29:19,290 INFO  [org.jboss.as.connector] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
16:29:19,351 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
16:29:19,367 INFO  [org.jboss.as.naming] (MSC service thread 1-2) JBAS011802: Starting Naming Service
16:29:19,371 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) JBAS015400: Bound mail session [java:jboss/mail/Default]
16:29:19,586 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-4) JBoss Web Services - Stack CXF Server 4.0.1.GA
16:29:19,694 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-3) Starting Coyote HTTP/1.1 on http--192.168.1.66-8080
16:29:19,915 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on /192.168.1.66:4447
16:29:19,916 INFO  [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100: Listening on /192.168.1.66:9999
16:29:19,949 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-3) JBAS015012: Started FileSystemDeploymentService for directory /home/jboss/jboss-as-7.1.0.Final/standalone/deployments
16:29:20,101 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
16:29:20,115 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 2615ms - Started 134 of 205 services (70 services are passive or on-demand)

The server is listening on another IP address than the 127.0.0.1, which is of course more useful. When using the command-line interface, we have to make sure we connect to the right host, for example,

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect
The controller is not available at localhost:9999
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /]

The command-line interface is a management tool for a domain or standalone server. It allows a user to connect to the domain controller or standalone server and execute management operations available through the de-typed management model. The native-management API is the entry point for management clients that rely on the application server’s native protocol to integrate with the management layer. It uses an open binary protocol and RPC style API based on a small number of Java types to describe and execute management operations against a domain or standalone server.

To get back at the configuration file. A socket binding makes up a named configuration of a socket. Here, we can configure the network ports, such as native management (command-line interface), which by default runs on port 9999.

Thread pools

Thread pools improve performance, and provide a means of resource management when a large number of tasks are executed asynchronously. By configuring the thread pool we can tune specific areas. The application server thread pool configuration can include:

  • A thread factory creates new threads when needed. The thread factory is not included by default in the server configuration as it relies on defaults, which in general do not need to be modified.
  • A bounded thread pool (or blocking bounded thread pool) is the most common kind of pool used by the application server, as it prevents resource exhaustion by defining a constraint on the thread pool size. This thread pool is also the most complex, i.e., it has to maintain a fixed-length queue and two pool sizes (core size and maximum size). When a new task is submitted, a new thread is created when the number of running threads is less than the core size. Otherwise, if there is room in the queue, the task is queued. When none of these options apply, the executor can create a new thread when the maximum size is not yet reached, otherwise the task can either be blocked (in the case of a blocking bounded thread pool) until there is room in the queue or it can be passed to a so-called hand-off executor (when one is specified, otherwise the task will be rejected).
  • A unbounded thread pool has a core size and a queue with no upper bound. When a task is submitted and the number of running threads is less than the core size, a new thread is created. Otherwise, the task is placed in a queue. If too many tasks are submitted, an out of memory error can occur.
  • A queueless thread pool (or blocking queueless thread pool) is a thread pool with no queue and has the same logic as the bounded thread pool (without the queue storage option).
  • A scheduled thread pool is used for activities on the server-side that require running periodically or with delays.

An example configuration looks as follows

<subsystem xmlns="urn:jboss:domain:threads:1.1">
	<blocking-bounded-queue-thread-pool name="http-thread-pool">
		<queue-length count="500" per-cpu="200"/>
		<core-threads count="5" per-cpu="2"/>
		<max-threads count="10" per-cpu="3"/>
		<keepalive-time time="30" unit="seconds"/>
	</blocking-bounded-queue-thread-pool>
</subsystem>

In the configuration above we did not define a thread-factory. By not defining a thread-factory an appropriate default thread factory will be used. To view the configuration we can use the following (note that by using read-resource-description we get a detailed description of all the elements the can be configured)

[standalone@192.168.1.66:9999 /] /subsystem=threads/blocking-bounded-queue-thread-pool=http-thread-pool:read-resource-description
{
    "outcome" => "success",
    "result" => {
        "description" => "A thread pool executor with a bounded queue where threads submittings tasks may block. Such a thread pool has a core and maximum size and a specified queue length.  When a task is submitted, if the number of running threads is less than the core size, a new thread is created.  Otherwise, if there is room in the queue, the task is enqueued. Otherwise, if the number of running threads is less than the maximum size, a new thread is created. Otherwise, the caller blocks until room becomes available in the queue.",
        "attributes" => {
            "core-threads" => {
                "type" => INT,
                "description" => "The core thread pool size which is smaller than the maximum pool size. If undefined, the core thread pool size is the same as the maximum thread pool size.",
                "expressions-allowed" => true,
                "nillable" => true,
                "min" => 0L,
                "max" => 2147483647L,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "no-services"
            },
            "current-thread-count" => {
                "type" => INT,
                "description" => "The current number of threads in the pool.",
                "expressions-allowed" => false,
                "nillable" => false,
                "access-type" => "metric",
                "storage" => "runtime"
            },
            "largest-thread-count" => {
                "type" => INT,
                "description" => "The largest number of threads that have ever simultaneously been in the pool.",
                "expressions-allowed" => false,
                "nillable" => false,
                "access-type" => "metric",
                "storage" => "runtime"
            },
            "keepalive-time" => {
                "type" => OBJECT,
                "description" => "Used to specify the amount of time that pool threads should be kept running when idle; if not specified, threads will run until the executor is shut down.",
                "expressions-allowed" => false,
                "nillable" => true,
                "value-type" => {
                    "time" => {
                        "type" => LONG,
                        "required" => true,
                        "description" => "The time"
                    },
                    "unit" => {
                        "type" => STRING,
                        "required" => true,
                        "description" => "The time unit"
                    }
                },
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "no-services"
            },
            "thread-factory" => {
                "type" => STRING,
                "description" => "Specifies the name of a specific thread factory to use to create worker threads. If not defined an appropriate default thread factory will be used.",
                "expressions-allowed" => false,
                "nillable" => true,
                "min-length" => 1L,
                "max-length" => 2147483647L,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "all-services"
            },
            "queue-length" => {
                "type" => INT,
                "description" => "The queue length.",
                "expressions-allowed" => true,
                "nillable" => false,
                "min" => 0L,
                "max" => 2147483647L,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "all-services"
            },
            "max-threads" => {
                "type" => INT,
                "description" => "The maximum thread pool size.",
                "expressions-allowed" => true,
                "nillable" => false,
                "min" => 0L,
                "max" => 2147483647L,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "no-services"
            },
            "name" => {
                "type" => STRING,
                "description" => "The name of the thread pool.",
                "expressions-allowed" => false,
                "nillable" => true,
                "min-length" => 1L,
                "max-length" => 2147483647L,
                "access-type" => "read-only",
                "storage" => "configuration"
            },
            "rejected-count" => {
                "type" => INT,
                "description" => "The number of tasks that have been passed to the handoff-executor (if one is specified) or discarded.",
                "expressions-allowed" => false,
                "nillable" => false,
                "access-type" => "metric",
                "storage" => "runtime"
            },
            "allow-core-timeout" => {
                "type" => BOOLEAN,
                "description" => "Whether core threads may time out.",
                "expressions-allowed" => false,
                "nillable" => true,
                "default" => false,
                "access-type" => "read-write",
                "storage" => "configuration",
                "restart-required" => "no-services"
            }
        },
        "operations" => undefined,
        "children" => {}
    }
}
[standalone@192.168.1.66:9999 /] /subsystem=threads/blocking-bounded-queue-thread-pool=http-thread-pool:read-resource
{
    "outcome" => "success",
    "result" => {
        "allow-core-timeout" => false,
        "core-threads" => 50,
        "keepalive-time" => {
            "time" => 10L,
            "unit" => "SECONDS"
        },
        "max-threads" => 50,
        "name" => "http-thread-pool",
        "queue-length" => 50,
        "thread-factory" => undefined
    }
}

Data source

A note up front is in order. When installing and configuring an Oracle database on, for example, a Windows machine, it usually is bound to localhost. To change this, we have to edit listener.ora and tnsnames.ora (both are located in ${ORACLE_HOME}/product/11.2.0/dbhome_1/NETWORK/ADMIN). For example,

# listener.ora Network Configuration File: c:\oracle\product\11.2.0\dbhome_1\network\admin\listener.ora
# Generated by Oracle configuration tools.

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = CLRExtProc)
      (ORACLE_HOME = c:\oracle\product\11.2.0\dbhome_1)
      (PROGRAM = extproc)
      (ENVS = "EXTPROC_DLLS=ONLY:c:\oracle\product\11.2.0\dbhome_1\bin\oraclr11.dll")
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.65)(PORT = 1521))
    )
  )

ADR_BASE_LISTENER = c:\oracle
# tnsnames.ora Network Configuration File: c:\oracle\product\11.2.0\dbhome_1\network\admin\tnsnames.ora
# Generated by Oracle configuration tools.

ORACLR_CONNECTION_DATA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
    (CONNECT_DATA =
      (SID = CLRExtProc)
      (PRESENTATION = RO)
    )
  )

ORCL11 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.65)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl11)
    )
  )

LISTENER_ORCL11 =
  (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.65)(PORT = 1521))

The procedure for installing a new module requires copying the .jar libraries in the appropriate modules path and adding a module.xml file, which declares the module and its dependencies. For the module path we are going to use ${JBOSS_HOME}/modules/${MODULE}/main, in which ${MODULE} will be com/oracle/database. We have the following directory structure

${JBOSS_HOME}/modules
	/com/oracle/database
		/main
			module.xml
			ojdbc6.jar

The contents of module.xml are the following

<module xmlns="urn:jboss:module:1.1" name="com.oracle.database">
    <resources>
        <resource-root path="ojdbc6.jar"/>
    </resources>
    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>

To configure a data source, find the entry subsystem xmlns="urn:jboss:domain:datasources:1.0" in standalone.xml, and add the following:

<subsystem xmlns="urn:jboss:domain:datasources:1.0">
	<datasources>
		<datasource jndi-name="java:/jdbc/OracleDS" pool-name="OracleDS" enabled="true" jta="true" use-java-context="true" use-ccm="true">
			<connection-url>jdbc:oracle:thin:@192.168.1.65:1521:orcl11</connection-url>
			<driver>oracle</driver>
			<pool>
				<min-pool-size>1</min-pool-size>
				<max-pool-size>15</max-pool-size>
				<prefill>true</prefill>
				<use-strict-min>true</use-strict-min>
		    </pool>
			<security>
				<user-name>example</user-name>
				<password>example</password>
			</security>
		    <statement>
				<prepared-statement-cache-size>10</prepared-statement-cache-size>
		    </statement>
			<timeout>
				<idle-timeout-minutes>0</idle-timeout-minutes>
				<query-timeout>600</query-timeout>
		    </timeout>
		</datasource>
		<drivers>
			<driver name="oracle" module="com.oracle.database">
				<driver-class>oracle.jdbc.OracleDriver</driver-class>
				<xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
		    </driver>
		</drivers>
	</datasources>
</subsystem>

When the server is started, we can check the data sources present by using the command-line interface

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] /subsystem=datasources:installed-drivers-list
{
    "outcome" => "success",
    "result" => [
        {
            "driver-name" => "h2",
            "deployment-name" => undefined,
            "driver-module-name" => "com.h2database.h2",
            "module-slot" => "main",
            "driver-datasource-class-name" => "",
            "driver-xa-datasource-class-name" => "org.h2.jdbcx.JdbcDataSource",
            "driver-class-name" => "org.h2.Driver",
            "driver-major-version" => 1,
            "driver-minor-version" => 3,
            "jdbc-compliant" => true
        },
        {
            "driver-name" => "oracle",
            "deployment-name" => undefined,
            "driver-module-name" => "com.oracle.database",
            "module-slot" => "main",
            "driver-datasource-class-name" => "",
            "driver-xa-datasource-class-name" => "oracle.jdbc.xa.client.OracleXADataSource",
            "driver-class-name" => "oracle.jdbc.OracleDriver",
            "driver-major-version" => 11,
            "driver-minor-version" => 2,
            "jdbc-compliant" => true
        }
    ]
}

To test a connection

[standalone@192.168.1.66:9999 /] /subsystem=datasources/data-source=OracleDS:test-connection-in-pool
{
    "outcome" => "success",
    "result" => [true]
}

To obtain the configuration of the data source

[standalone@192.168.1.66:9999 /] /subsystem=datasources/data-source=OracleDS:read-resource
{
    "outcome" => "success",
    "result" => {
        "allocation-retry" => undefined,
        "allocation-retry-wait-millis" => undefined,
        "background-validation" => undefined,
        "background-validation-millis" => undefined,
        "blocking-timeout-wait-millis" => undefined,
        "check-valid-connection-sql" => undefined,
        "connection-properties" => undefined,
        "connection-url" => "jdbc:oracle:thin:@192.168.1.65:1521:orcl11",
        "datasource-class" => undefined,
        "driver-class" => undefined,
        "driver-name" => "oracle",
        "enabled" => true,
        "exception-sorter-class-name" => undefined,
        "exception-sorter-properties" => undefined,
        "flush-strategy" => undefined,
        "idle-timeout-minutes" => 0L,
        "jndi-name" => "java:/jdbc/OracleDS",
        "jta" => true,
        "max-pool-size" => 15,
        "min-pool-size" => 1,
        "new-connection-sql" => undefined,
        "password" => "example",
        "pool-prefill" => true,
        "pool-use-strict-min" => true,
        "prepared-statements-cache-size" => 10L,
        "query-timeout" => 600L,
        "reauth-plugin-class-name" => undefined,
        "reauth-plugin-properties" => undefined,
        "security-domain" => undefined,
        "set-tx-query-timeout" => "false",
        "share-prepared-statements" => "false",
        "spy" => "false",
        "stale-connection-checker-class-name" => undefined,
        "stale-connection-checker-properties" => undefined,
        "track-statements" => "\"NOWARN\"",
        "transaction-isolation" => undefined,
        "url-delimiter" => undefined,
        "url-selector-strategy-class-name" => undefined,
        "use-ccm" => true,
        "use-fast-fail" => "false",
        "use-java-context" => true,
        "use-try-lock" => undefined,
        "user-name" => "example",
        "valid-connection-checker-class-name" => undefined,
        "valid-connection-checker-properties" => undefined,
        "validate-on-match" => "false",
        "statistics" => {
            "pool" => undefined,
            "jdbc" => undefined
        }
    }
}

JMS

The JMS server configuration is done through the messaging subsystem. Which is not enabled when the default standalone.xml configuration is used. To enable the messaging subsystem we add the following to the standalone.xml configuration file (note that this can be copied from standalone-full.xml)

<server xmlns="urn:jboss:domain:1.1">
    <extensions>
		...
        <extension module="org.jboss.as.messaging"/>
		...
    </extensions>
    <management>...</management>
    <profile>
		...
        <subsystem xmlns="urn:jboss:domain:messaging:1.1">
            <hornetq-server>
                <persistence-enabled>true</persistence-enabled>
                <journal-file-size>102400</journal-file-size>
                <journal-min-files>2</journal-min-files>
                <connectors>
                    <netty-connector name="netty" socket-binding="messaging"/>
                    <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                    </netty-connector>
                    <in-vm-connector name="in-vm" server-id="0"/>
                </connectors>
                <acceptors>
                    <netty-acceptor name="netty" socket-binding="messaging"/>
                    <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                        <param key="direct-deliver" value="false"/>
                    </netty-acceptor>
                    <in-vm-acceptor name="in-vm" server-id="0"/>
                </acceptors>
                <security-settings>
                    <security-setting match="#">
                        <permission type="send" roles="guest"/>
                        <permission type="consume" roles="guest"/>
                        <permission type="createNonDurableQueue" roles="guest"/>
                        <permission type="deleteNonDurableQueue" roles="guest"/>
                    </security-setting>
                </security-settings>
                <address-settings>
                    <address-setting match="#">
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <redelivery-delay>0</redelivery-delay>
                        <max-size-bytes>10485760</max-size-bytes>
                        <address-full-policy>BLOCK</address-full-policy>
                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                    </address-setting>
                </address-settings>
                <jms-connection-factories>
                    <connection-factory name="InVmConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/ConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <connection-factory name="RemoteConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="netty"/>
                        </connectors>
                        <entries>
                            <entry name="RemoteConnectionFactory"/>
                            <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <pooled-connection-factory name="hornetq-ra">
                        <transaction mode="xa"/>
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/JmsXA"/>
                        </entries>
                    </pooled-connection-factory>
                </jms-connection-factories>
                <jms-destinations>
                    <jms-queue name="testQueue">
                        <entry name="java:/queue/test"/>
                        <entry name="java:jboss/exported/jms/queue/test"/>
                    </jms-queue>
                    <jms-topic name="testTopic">
                        <entry name="java:/topic/test"/>
                        <entry name="java:jboss/exported/jms/topic/test"/>
                    </jms-topic>
                </jms-destinations>
            </hornetq-server>
        </subsystem>
    </profile>
    <interfaces>...</interfaces>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
	    ...
        <socket-binding name="messaging" port="5445"/>
        <socket-binding name="messaging-throughput" port="5455"/>
		...
    </socket-binding-group>
</server>

The transport of JMS messages is a key part of messaging system tuning. HornetQ uses Netty (a NIO client server framework) as its network library. The HornetQ transports configuration consist of acceptors and connectors. An acceptor defines which type of connection is accepted by the HornetQ server. A connector defines how to connect to a HornetQ server. HornetQ defines two the following types:

  • In-VM (Intra Virtual Machine) – can be used when both HornetQ client and server run in the same virtual machine.
  • netty – used when HornetQ client and server run in different virtual machines.

The JMS connection factories can be split into two kinds: In-VM connections and connections factories that can be used by remote clients. Each connection factory does reference a connector declaration, that is associated with a socket binding. The connection factory entry declaration specifies the JNDI name under which the factory will be exposed. Queues and topics are sub resources of the messaging subsystem. Each entry refers to a JNDI name of the queue or topic.

To check the configuration when the server is restarted, we can use

[standalone@192.168.1.66:9999 /] /subsystem=messaging/hornetq-server=default/connection-factory=InVmConnectionFactory:read-resource
{
    "outcome" => "success",
    "result" => {
        "auto-group" => false,
        "block-on-acknowledge" => false,
        "block-on-durable-send" => true,
        "block-on-non-durable-send" => false,
        "cache-large-message-client" => false,
        "call-timeout" => 30000L,
        "client-failure-check-period" => 30000L,
        "client-id" => undefined,
        "compress-large-messages" => false,
        "confirmation-window-size" => -1,
        "connection-load-balancing-policy-class-name" => "org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy",
        "connection-ttl" => 60000L,
        "connector" => {"in-vm" => undefined},
        "consumer-max-rate" => -1,
        "consumer-window-size" => 1048576,
        "discovery-group-name" => undefined,
        "discovery-initial-wait-timeout" => undefined,
        "dups-ok-batch-size" => 1048576,
        "entries" => ["java:/ConnectionFactory"],
        "failover-on-initial-connection" => false,
        "failover-on-server-shutdown" => undefined,
        "group-id" => undefined,
        "ha" => false,
        "max-retry-interval" => 2000L,
        "min-large-message-size" => 102400,
        "pre-acknowledge" => false,
        "producer-max-rate" => -1,
        "producer-window-size" => 65536,
        "reconnect-attempts" => 0,
        "retry-interval" => 2000L,
        "retry-interval-multiplier" => 1.0,
        "scheduled-thread-pool-max-size" => 5,
        "thread-pool-max-size" => -1,
        "transaction-batch-size" => 1048576,
        "use-global-pools" => true
    }
}

[standalone@192.168.1.66:9999 /] /subsystem=messaging/hornetq-server=default/jms-queue=testQueue:read-resource
{
    "outcome" => "success",
    "result" => {
        "durable" => true,
        "entries" => [
            "java:/queue/test",
            "java:jboss/exported/jms/queue/test"
        ],
        "selector" => undefined
    }
}

Enterprise beans

By default, the following configuration is present

<subsystem xmlns="urn:jboss:domain:ejb3:1.2">
	<session-bean>
		<stateless>
			<bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
		</stateless>
		<stateful default-access-timeout="5000" cache-ref="simple"/>
		<singleton default-access-timeout="5000"/>
	</session-bean>
	<mdb>
		<resource-adapter-ref resource-adapter-name="hornetq-ra"/>
		<bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
	</mdb>
	<pools>
		<bean-instance-pools>
			<strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
			<strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
		</bean-instance-pools>
	</pools>
	<caches>
		<cache name="simple" aliases="NoPassivationCache"/>
		<cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
	</caches>
	<passivation-stores>
		<file-passivation-store name="file"/>
	</passivation-stores>
	<async thread-pool-name="default"/>
	<timer-service thread-pool-name="default">
		<data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
	</timer-service>
	<remote connector-ref="remoting-connector" thread-pool-name="default"/>
	<thread-pools>
		<thread-pool name="default">
			<max-threads count="10"/>
			<keepalive-time time="100" unit="milliseconds"/>
		</thread-pool>
	</thread-pools>
</subsystem>

One of the key responsibilities of the EJB container is the management of EJB component lifecycles. Bean instances are pooled and reused by the container to reduce the number of object instantiations. For example the lifecycle of a stateless session bean is as follows:

  • The client obtains a reference to the stateless session bean. There are three options here. First, the reference can be injected into the client via an @EJB annotation. Second, the reference can be bound into the client’s local java:comp/env environment, using the @EJB annotation at class-level or a deployment descriptor entry, and the client can retrieve the reference with a JNDI lookup. Third, the client can look up the stateless session bean in the server’s global JNDI tree.
  • The client invokes a business method on the bean reference.
  • The container can reuse an existing instance of the stateless session bean from a pool, if available, or can instantiate a new bean instance. If a bean instance is newly created, the container will first perform dependency injection. If the stateless session bean implements the optional javax.ejb.SessionBean interface, the container will call setSessionContext. The session context will also be injected to any SessionContext field marked with the @Resource annotation. After completing injection of the session context and other dependencies, the container will call any methods marked with the @PostConstruct annotation.
  • The container starts a transaction, if appropriate. This is controlled by the @javax.ejb.TransactionManagement and @javax.ejb.TransactionAttribute annotations.
  • The container invokes the called business method on the bean instance, and the bean performs the desired operation.
  • The container commits the transaction, if appropriate.
  • The results of the business method call are returned to the client.
  • The client may invoke additional business methods on the bean reference, each of which may end up invoking methods on a different bean instance.
  • At some point, if the container decides to reduce the size of the bean instance pool, the container invokes any @PreDestroy methods on the bean instance, or ejbRemove if the bean implements the SessionBean interface. It is important to understand that this decision is not related to any client action.

There is a stronger link between the lifecycle of a stateful session bean and the client’s use of its reference:

  • The client obtains a reference to the stateful session bean.
  • The container instantiates a new bean instance. The container will perform dependency injection. If the stateful session bean implements the optional javax.ejb.SessionBean interface, the container will call setSessionContext. The session context will also be injected to any SessionContext field marked with the @Resource annotation. Any methods marked with the @PostConstruct will then be called.
  • The client invokes a business method on the bean reference.
  • The container starts a transaction, if appropriate.
  • The container invokes a business method on the bean instance, and the bean performs the desired operation.
  • The container commits the transaction, if appropriate.
  • The results of the business method call are returned to the client.
  • The client may invoke additional business methods on the bean reference and is assured that these additional calls will go to the same instance of the bean.
  • The client calls a method annotated with @javax.ejb.Remove when it is done with the stateful session bean.
  • The container invokes any @PreDestroy methods on the bean instance, or ejbRemove if the bean implements the SessionBean interface.

This introduction to the lifecycle of session EJB components represents a simplified view of the process. Additional complexities are introduced by limitations in the pool and cache sizes that we need to understand to configure the application properly.

EJB Pooling – JBoss maintains a pool of stateless session EJB instances for each stateless session bean deployment. This pool improves performance, because a client request can be handled immediately by any free initialized EJB instance. By default, the pool starts empty, and grows on demand. This can be controlled with the max-pool-size parameter. Within the pool configuration, there is also set a instance-acquisition-timeout of 5 minutes, which will come into play if the number of requests are larger than the maximum number of beans in the pool. If we set max-pool-size, we are choosing deliberately to throttle the application. If there are no idle bean instances, an thread making a new request will block until a bean becomes available or the transaction times out (so it must not be set too low). Message-driven beans are pooled in a manner very similar to stateless session beans. The number of message-driven bean instances can be controlled by using the max-pool-size parameter.

Stateful session bean instances are created as they are needed to service client requests. Between requests these instances reside in a bean-specific cache in the active state, ready for the next request. The size of the cache is limited by the max-size element (which is available for a passivation store). So long as the application never requires more than max-size instances of the stateful session bean at any given time to service all concurrent clients, there is no contention for the cache and performance is optimal. If we limit the number of beans in the cache, the managing of the cache is fairly active:

  • If the cache is full, bean instances that are not being used at that moment for client requests are subject to passivation.
  • If bean-managed transaction demarcation is used, a transaction may not be committed or rolled back at the end of a business method call. This leaves the bean instance associated with the transaction, pinned in the cache, and not eligible for passivation. Applications that keep transactions open between stateful session bean calls do not scale well and are difficult to manage.
  • Passivation logic is controlled by the cache type; least recently used (LRU), passivates based the max-size and the time when the bean has not been used; not recently used (NRU), passivates beans only when the number of active beans approaches the max-size setting. The NRU strategy is lazy; the LRU strategy is eager. Although the LRU setting can be a convenient way of enforcing idle timeouts on the resources the objects encapsulate, it requires the cache to keep track of the bean’s access time and maintain an ordered list that gets updated after each bean access. Unless we have a good reason to need idle timeouts strictly enforced, most applications should probably use the NRU algorithm.

Passivation of beans means serialization of non-transient data in the bean to disk storage to release the memory used by the bean. The next request for the passivated bean will require activation, the reverse process, where bean elements are read from the disk store and the active bean instance is recreated in memory. Needless to say, passivation and activation cycles can be extremely expensive. The application should always call a @Remove method to delete the active bean instance from the cache when a client is through using the instance. Failure to call a @Remove method leaves the bean instance in the active state and consumes one slot in the cache, requiring eventual passivation during cache management to make room for additional client beans.

By using the command-line interface we retrieve some configuration information, for example,

[standalone@192.168.1.66:9999 /] /subsystem=ejb3:read-resource
{
    "outcome" => "success",
    "result" => {
        "cluster-passivation-store" => undefined,
        "default-clustered-sfsb-cache" => undefined,
        "default-entity-bean-instance-pool" => undefined,
        "default-entity-bean-optimistic-locking" => undefined,
        "default-mdb-instance-pool" => "mdb-strict-max-pool",
        "default-resource-adapter-name" => "hornetq-ra",
        "default-sfsb-cache" => "simple",
        "default-singleton-bean-access-timeout" => 5000L,
        "default-slsb-instance-pool" => "slsb-strict-max-pool",
        "default-stateful-bean-access-timeout" => 5000L,
        "in-vm-remote-interface-invocation-pass-by-value" => "true",
        "cache" => {
            "simple" => undefined,
            "passivating" => undefined
        },
        "file-passivation-store" => {"file" => undefined},
        "service" => {
            "timer-service" => undefined,
            "remote" => undefined,
            "async" => undefined
        },
        "strict-max-bean-instance-pool" => {
            "slsb-strict-max-pool" => undefined,
            "mdb-strict-max-pool" => undefined
        },
        "thread-pool" => {"default" => undefined}
    }
}

To get insight in the file passivation configuration we can use

[standalone@192.168.1.66:9999 /] /subsystem=ejb3/file-passivation-store=file:read-resource
{
    "outcome" => "success",
    "result" => {
        "groups-path" => "ejb3/groups",
        "idle-timeout" => 300L,
        "idle-timeout-unit" => "SECONDS",
        "max-size" => 100000,
        "name" => "file",
        "relative-to" => "jboss.server.data.dir",
        "sessions-path" => "ejb3/sessions",
        "subdirectory-count" => 100
    }
}

Or the stateless session bean pool configuration

[standalone@192.168.1.66:9999 /] /subsystem=ejb3/strict-max-bean-instance-pool=slsb-strict-max-pool:read-resource
{
    "outcome" => "success",
    "result" => {
        "max-pool-size" => 20,
        "name" => "slsb-strict-max-pool",
        "timeout" => 5L,
        "timeout-unit" => "MINUTES"
    }
}

Application

The application consists of a servlet that calls a stateless enterprise bean, which in turn (based on the task to be performed); inserts data in the database and sends a message to the queue; updates data in the database; or removes data from the database and also sends a message to the queue. We have to following persistence configuration,

<persistence xmlns="http://java.sun.com/xml/ns/persistence" version="2.0">
    <persistence-unit name="PersonPersistenceUnit">
        <provider>org.hibernate.ejb.HibernatePersistence</provider>
        <jta-data-source>java:/jdbc/OracleDS</jta-data-source>
        <class>model.entities.Person</class>
    </persistence-unit>
</persistence>

Note that in JBoss, Hibernate is the default persistence provider and we do not need to include the classes as these are already present on the application server. The jta-data-source is mapped to the data source we have configured above with a JNDI name of java:/jdbc/OracleDS. The stateless enterprise bean looks as follows

package model.logic;

import model.entities.Person;

import javax.ejb.Remote;

@Remote
public interface Company {
    public void insertPerson(Person person);

    public void removePerson(Integer sofinummer);

    public void updatePerson(Person person);
}

and has the following implementation

package model.logic;

import model.entities.Person;

import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import javax.annotation.Resource;
import javax.ejb.Stateless;
import javax.jms.*;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;

@Stateless(name = "Company", mappedName = "Company")
public class CompanyBean implements Company {

    @PersistenceContext(unitName = "PersonPersistenceUnit")
    private EntityManager entityManager;

    @Resource(name = "java:/JmsXA")
    private ConnectionFactory connectionFactory;
    @Resource(name = "java:/queue/test")
    private Queue destination;

    private Connection connection = null;
    private Session session = null;
    private MessageProducer messageProducer = null;

    @PostConstruct
    public void init() {
        try {
            connection = connectionFactory.createConnection();
            session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
            messageProducer = session.createProducer(destination);
        } catch (JMSException e) {
            if (connection != null) {
                try {
                    connection.close();
                } catch (JMSException f) {
                    e.printStackTrace();
                }
            }
        }
    }

    @PreDestroy
    public void release() {
        try {
            if (connection != null) {
                connection.close();
            }
        } catch (JMSException e) {
            e.printStackTrace();
        }
    }

    public void insertPerson(Person person) {
        if (findPerson(person.getSofinummer()) == null) {
            entityManager.persist(person);
            sendMessage(person);
        } else {
            updatePerson(person);
        }
    }

    public void removePerson(Integer sofinummer) {
        Person person = findPerson(sofinummer);
        if (person != null) {
            entityManager.remove(entityManager.merge(person));
            sendMessage(person);
        }
    }

    public void updatePerson(Person person) {
        entityManager.merge(person);
    }

    private Person findPerson(Integer sofinummer) {
        return entityManager.find(Person.class, sofinummer);
    }

    private void sendMessage(Person person) {
        try {
            ObjectMessage message = session.createObjectMessage();
            message.setObject(person);
            messageProducer.send(message);
        } catch (JMSException e) {
            e.printStackTrace();
        }
    }
}

Here, we inject a JMS connection factory (with JNDI java:/JmsXA) and a JMS queue (with JNDI java:/queue/test). The enterprise bean is initialized by setting-up a message producer, that will be used in the insert and remove tasks. The message-driven bean looks as follows

@MessageDriven(name = "CompanyMDB", activationConfig = {
        @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
        @ActivationConfigProperty(propertyName = "destination", propertyValue = "java:/queue/test")
})
public class CompanyMDB implements MessageListener {

    public void onMessage(Message message) {
        if (message instanceof ObjectMessage) {
            ObjectMessage objectMessage = (ObjectMessage) message;
            try {
                System.out.println("RECEIVED OBJECT MESSAGE " + objectMessage.getObject());
            } catch (JMSException e) {
                e.printStackTrace();
            }
        }

        if (message instanceof TextMessage) {
            TextMessage textMessage = (TextMessage) message;
            try {
                System.out.println("RECEIVED TEXT MESSAGE " + textMessage.getText());
            } catch (JMSException e) {
                e.printStackTrace();
            }
        }
    }
}

The servlet looks as follows

package userinterface.servlets;

import model.entities.Person;
import model.logic.Company;

import javax.ejb.EJB;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Random;

@WebServlet(name = "TestServlet", urlPatterns = "/testservlet")
public class TestServlet extends HttpServlet {

    @EJB(name = "Company", mappedName = "java:global/LoadTest6/Model/Company")
    Company company;

    private Random generator = null;

    @Override
    public void init() throws ServletException {
        generator = new Random();
    }

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        Person person = createPerson();

        if (generator.nextDouble() &lt; 0.01) {
            company.removePerson(person.getSofinummer());
        } else {
            company.insertPerson(person);
        }
    }

    @Override
    public void destroy() {
        generator = null;
    }

    private Person createPerson() {
        Person person = new Person();
        person.setNaam(Long.toString(Math.abs(generator.nextLong()), 36));
        person.setSofinummer(generator.nextInt(10000));
        return person;
    }
}

When an enterprise bean is deployed is will be bound to JNDI as java:global/<deployment-name>/<module-name>/<mapped-name>.

Deployment

In the standalone mode JBoss comes with a deployment scanner. Its job is to monitor a directory for new files and to deploy those files. In the standalone.xml configuration file the following entry can be found

<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
	<deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>
</subsystem>

To auto-deploy an application we can put our .ear or .war file into the ${JBOSS_HOME}/deployments directory. We can also use the preferred command-line interface

[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/LoadTest6.ear

The following is observed in the logging

[jboss@axis-into-ict bin]$ ./standalone.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/jboss-as-7.1.0.Final

  JAVA: /home/jboss/jdk1.6.0_31/bin/java

  JAVA_OPTS:  -XX:+UseCompressedOops -XX:+TieredCompilation -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

=========================================================================

13:45:06,255 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
13:45:06,420 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
13:45:06,464 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
13:45:07,295 INFO  [org.xnio] XNIO Version 3.0.3.GA
13:45:07,302 INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
13:45:07,312 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
13:45:07,320 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.2.GA
13:45:07,356 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
13:45:07,400 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 27) JBAS016200: Activating ConfigAdmin Subsystem
13:45:07,405 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 50) JBAS015537: Activating WebServices Extension
13:45:07,420 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 41) JBAS011940: Activating OSGi Subsystem
13:45:07,422 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 46) JBAS013101: Activating Security Subsystem
13:45:07,424 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 40) JBAS011800: Activating Naming Subsystem
13:45:07,442 INFO  [org.jboss.as.security] (MSC service thread 1-2) JBAS013100: Current PicketBox version=4.0.6.final
13:45:07,471 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 32) JBAS010280: Activating Infinispan subsystem.
13:45:07,523 INFO  [org.jboss.as.connector] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
13:45:07,546 INFO  [org.jboss.as.naming] (MSC service thread 1-4) JBAS011802: Starting Naming Service
13:45:07,645 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 28) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
13:45:07,730 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS015400: Bound mail session [java:jboss/mail/Default]
13:45:07,773 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-3) JBoss Web Services - Stack CXF Server 4.0.1.GA
13:45:07,822 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 28) JBAS010403: Deploying JDBC-compliant driver class oracle.jdbc.OracleDriver (version 11.2)
13:45:07,923 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-4) Starting Coyote HTTP/1.1 on http-axis-into-ict.nl-192.168.1.66-8080
13:45:07,938 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/home/jboss/jboss-as-7.1.0.Final/standalone/data/messagingjournal,bindingsDirectory=/home/jboss/jboss-as-7.1.0.Final/standalone/data/messagingbindings,largeMessagesDirectory=/home/jboss/jboss-as-7.1.0.Final/standalone/data/messaginglargemessages,pagingDirectory=/home/jboss/jboss-as-7.1.0.Final/standalone/data/messagingpaging)
13:45:07,946 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) Waiting to obtain live lock
13:45:08,014 INFO  [org.hornetq.core.persistence.impl.journal.JournalStorageManager] (MSC service thread 1-3) Using AIO Journal
13:45:08,217 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on /192.168.1.66:9999
13:45:08,250 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) JBAS015012: Started FileSystemDeploymentService for directory /home/jboss/jboss-as-7.1.0.Final/standalone/deployments
13:45:08,267 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on axis-into-ict.nl/192.168.1.66:4447
13:45:08,272 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-3) Waiting to obtain live lock
13:45:08,273 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-3) Live Server Obtained live lock
13:45:08,339 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
13:45:08,340 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:/jdbc/OracleDS]
13:45:08,649 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-3) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5445 for CORE protocol
13:45:08,651 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-3) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5455 for CORE protocol
13:45:08,652 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) Server is now live
13:45:08,653 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) HornetQ Server version 2.2.11.Final (HQ_2_2_11_FINAL_AS7, 122) [a0959f8b-5cbd-11e1-899c-000c2976c82d]) started
13:45:08,656 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) trying to deploy queue jms.topic.testTopic
13:45:08,682 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/topic/test
13:45:08,683 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/topic/test
13:45:08,684 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) trying to deploy queue jms.queue.testQueue
13:45:08,689 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/test
13:45:08,689 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:/queue/test
13:45:08,704 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
13:45:08,706 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
13:45:08,712 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
13:45:08,754 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010406: Registered connection factory java:/JmsXA
13:45:08,763 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-2) HornetQ resource adaptor started
13:45:08,763 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-2) IJ020002: Deployed: file://RaActivatorhornetq-ra
13:45:08,767 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-4) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
13:45:08,844 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 2784ms - Started 157 of 231 services (72 services are passive or on-demand)
13:46:30,698 INFO  [org.jboss.as.repository] (management-handler-threads - 6) JBAS014900: Content added at location /home/jboss/jboss-as-7.1.0.Final/standalone/data/content/16/1f51dde7f085c822cc4c68b306d57f1bee902d/content
13:46:30,714 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "LoadTest6.ear"
13:46:30,773 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "Web.war"
13:46:30,773 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "Model.jar"
13:46:30,838 INFO  [org.jboss.as.jpa] (MSC service thread 1-4) JBAS011401: Read persistence.xml for PersonPersistenceUnit
13:46:30,961 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-2) JNDI bindings for session bean named Company in deployment unit subdeployment "Model.jar" of deployment "LoadTest6.ear" are as follows:

        java:global/LoadTest6/Model/Company!model.logic.Company
        java:app/Model/Company!model.logic.Company
        java:module/Company!model.logic.Company
        java:jboss/exported/LoadTest6/Model/Company!model.logic.Company
        java:global/LoadTest6/Model/Company
        java:app/Model/Company
        java:module/Company

13:46:31,177 INFO  [org.jboss.as.jpa] (MSC service thread 1-2) JBAS011402: Starting Persistence Unit Service 'LoadTest6.ear/Model.jar#PersonPersistenceUnit'
13:46:31,234 INFO  [org.jboss.as.ejb3] (MSC service thread 1-4) JBAS014142: Started message driven bean 'CompanyMDB' with 'hornetq-ra' resource adapter
13:46:31,360 INFO  [org.jboss.web] (MSC service thread 1-3) JBAS018210: Registering web context: /LoadTest6
13:46:31,387 INFO  [org.hibernate.annotations.common.Version] (MSC service thread 1-2) HCANN000001: Hibernate Commons Annotations {4.0.1.Final}
13:46:31,393 INFO  [org.hibernate.Version] (MSC service thread 1-2) HHH000412: Hibernate Core {4.0.1.Final}
13:46:31,397 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-2) HHH000206: hibernate.properties not found
13:46:31,398 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-2) HHH000021: Bytecode provider name : javassist
13:46:31,414 INFO  [org.hibernate.ejb.Ejb3Configuration] (MSC service thread 1-2) HHH000204: Processing PersistenceUnitInfo [
        name: PersonPersistenceUnit
        ...]
13:46:31,532 INFO  [org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator] (MSC service thread 1-2) HHH000130: Instantiating explicit connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
13:46:31,639 INFO  [org.hibernate.dialect.Dialect] (MSC service thread 1-2) HHH000400: Using dialect: org.hibernate.dialect.Oracle10gDialect
13:46:31,656 INFO  [org.hibernate.engine.transaction.internal.TransactionFactoryInitiator] (MSC service thread 1-2) HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jta.CMTTransactionFactory
13:46:31,660 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (MSC service thread 1-2) HHH000397: Using ASTQueryTranslatorFactory
13:46:31,698 INFO  [org.hibernate.validator.util.Version] (MSC service thread 1-2) Hibernate Validator 4.2.0.Final
13:46:32,054 INFO  [org.jboss.as.server] (management-handler-threads - 6) JBAS018559: Deployed "LoadTest6.ear"

When we do a few tests (access the application by using http://hostname:8080/LoadTest6/testservlet) the following is observed in the logging:

13:50:37,002 INFO  [org.jboss.ejb.client] (http-axis-into-ict.nl-192.168.1.66-8080-1) JBoss EJB Client version 1.0.2.Final
13:50:37,484 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1a0100uhg5tiu 7907
13:50:39,692 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE iuhpgb6pcs13 9223
13:50:40,164 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 6m08798r5ehs 2500
13:50:40,621 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 12wx9zs77r9cx 4262
13:50:41,044 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 6dh6zc60tw1y 285
13:50:41,850 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE extyyn0pov1w 674
13:50:42,242 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE pv0ypyftodqy 8381
13:50:42,625 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1xh0e0cn7ggsa 5954
13:50:43,026 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1fiaxr5gsxkuo 4480
13:50:43,470 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1dnmh60s6fl47 7487
13:50:43,763 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 15mxq08abiozd 6698
13:50:44,183 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE hzgxdkqw1zo7 9573
13:50:44,433 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 4mvfve8o8gfb 1940
13:50:44,810 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1skgi6orrjiaw 3535
13:50:45,066 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE vb52wrkqgx81 5712
13:50:45,598 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1rxts2saz9fhi 7821
13:50:45,697 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE j8gonmtmsd8g 4627
13:50:45,824 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE da4b2qrqpt7b 9219
13:50:45,864 INFO  [stdout] (Thread-6 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE yqlwfar66vfk 4998
13:50:45,880 INFO  [stdout] (Thread-4 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE pzqok192zmnz 7551
13:50:45,882 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE jcyf4wttbwmy 1071
13:50:45,901 INFO  [stdout] (Thread-5 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1jg4yksazwtkx 5685
13:50:45,904 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1mv38p7onaidx 4399
13:50:45,915 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE wvsj6gq17e2i 586
13:50:45,936 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1c9ij29bk00ru 3746
13:50:45,973 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE ue08qs0k9unr 6877
13:50:46,008 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE hb372i5a3dai 3762
13:50:46,048 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE cg1pr42138dr 1066
13:50:46,086 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1k28o323a0k52 3510
13:50:46,121 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE fo0pn8x69hr9 2643
13:50:46,194 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 1134xboronc8u 3323
13:50:46,235 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1977385357)) RECEIVED OBJECT MESSAGE 16znhvr4pu6hy 5842

To obtain run-time information we can use the command-line interface, for example,

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] cd subsystem=messaging/hornetq-server=default/jms-queue=testQueue
[standalone@192.168.1.66:9999 jms-queue=testQueue] pwd
/subsystem=messaging/hornetq-server=default/jms-queue=testQueue
[standalone@192.168.1.66:9999 jms-queue=testQueue] read-attribute delivering-count
0
[standalone@192.168.1.66:9999 jms-queue=testQueue] read-attribute consumer-count
15
[standalone@192.168.1.66:9999 jms-queue=testQueue] read-attribute message-count
0L
[standalone@192.168.1.66:9999 jms-queue=testQueue] read-attribute messages-added
32L
[standalone@192.168.1.66:9999 jms-queue=testQueue] cd /
[standalone@192.168.1.66:9999 /] cd subsystem=datasources/data-source=OracleDS
[standalone@192.168.1.66:9999 data-source=OracleDS] ls
connection-properties                                       statistics                                                  allocation-retry=undefined
allocation-retry-wait-millis=undefined                      background-validation=undefined                             background-validation-millis=undefined
blocking-timeout-wait-millis=undefined                      check-valid-connection-sql=undefined                        connection-url=jdbc:oracle:thin:@192.168.1.65:1521:orcl11
datasource-class=undefined                                  driver-class=undefined                                      driver-name=oracle
enabled=true                                                exception-sorter-class-name=undefined                       exception-sorter-properties=undefined
flush-strategy=undefined                                    idle-timeout-minutes=0                                      jndi-name=java:/jdbc/OracleDS
jta=true                                                    max-pool-size=15                                            min-pool-size=1
new-connection-sql=undefined                                password=example                                            pool-prefill=true
pool-use-strict-min=true                                    prepared-statements-cache-size=10                           query-timeout=600
reauth-plugin-class-name=undefined                          reauth-plugin-properties=undefined                          security-domain=undefined
set-tx-query-timeout=false                                  share-prepared-statements=false                             spy=false
stale-connection-checker-class-name=undefined               stale-connection-checker-properties=undefined               track-statements="NOWARN"
transaction-isolation=undefined                             url-delimiter=undefined                                     url-selector-strategy-class-name=undefined
use-ccm=true                                                use-fast-fail=false                                         use-java-context=true
use-try-lock=undefined                                      user-name=example                                           valid-connection-checker-class-name=undefined
valid-connection-checker-properties=undefined               validate-on-match=false
[standalone@192.168.1.66:9999 data-source=OracleDS] cd statistics=pool
[standalone@192.168.1.66:9999 statistics=pool] read-attribute AverageCreationTime
93
[standalone@192.168.1.66:9999 statistics=pool] read-attribute AvailableCount
15
[standalone@192.168.1.66:9999 statistics=pool] read-attribute ActiveCount
5
[standalone@192.168.1.66:9999 statistics=pool] read-attribute MaxWaitTime
0
[standalone@192.168.1.66:9999 statistics=pool] cd ..
[standalone@192.168.1.66:9999 data-source=OracleDS] cd statistics=jdbc
[standalone@192.168.1.66:9999 statistics=jdbc] read-attribute PreparedStatementCacheAccessCount
70
[standalone@192.168.1.66:9999 statistics=jdbc] read-attribute PreparedStatementCacheHitCount
58
[standalone@192.168.1.66:9999 statistics=jdbc] read-attribute PreparedStatementCacheMissCount
0
[standalone@192.168.1.66:9999 statistics=jdbc] cd /
[standalone@192.168.1.66:9999 /] cd deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3
[standalone@192.168.1.66:9999 subsystem=ejb3] ls
entity-bean              message-driven-bean      singleton-bean           stateful-session-bean    stateless-session-bean
[standalone@192.168.1.66:9999 subsystem=ejb3] cd message-driven-bean=CompanyMDB
[standalone@192.168.1.66:9999 message-driven-bean=CompanyMDB] read-attribute pool-current-size
2
[standalone@192.168.1.66:9999 message-driven-bean=CompanyMDB] read-attribute pool-available-count
50
[standalone@192.168.1.66:9999 message-driven-bean=CompanyMDB] cd ..
[standalone@192.168.1.66:9999 subsystem=ejb3] cd stateless-session-bean=Company
[standalone@192.168.1.66:9999 stateless-session-bean=Company] read-attribute pool-current-size
5
[standalone@192.168.1.66:9999 stateless-session-bean=Company] read-attribute pool-available-count
50
[standalone@192.168.1.66:9999 stateless-session-bean=Company] cd /
[standalone@192.168.1.66:9999 /] cd deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=jpa
[standalone@192.168.1.66:9999 subsystem=jpa] ls
hibernate-persistence-unit
[standalone@192.168.1.66:9999 subsystem=jpa] cd hibernate-persistence-unit=LoadTest6.ear\/Model.jar#PersonPersistenceUnit
[standalone@192.168.1.66:9999 hibernate-persistence-unit=LoadTest6.ear/Model.jar#PersonPersistenceUnit] ls
collection                                        entity                                            entity-cache                                      query-cache
close-statement-count=0                           collection-fetch-count=0                          collection-load-count=0                           collection-recreated-count=0
collection-remove-count=0                         collection-update-count=0                         completed-transaction-count=35                    connect-count=70
enabled=false                                     entity-delete-count=0                             entity-fetch-count=0                              entity-insert-count=0
entity-load-count=0                               entity-update-count=0                             flush-count=0                                     optimistic-failure-count=0
prepared-statement-count=70                       query-cache-hit-count=0                           query-cache-miss-count=0                          query-cache-put-count=0
query-execution-count=0                           query-execution-max-time=0                        query-execution-max-time-query-string=undefined   second-level-cache-hit-count=0
second-level-cache-miss-count=0                   second-level-cache-put-count=0                    session-close-count=0                             session-open-count=0
successful-transaction-count=35
[standalone@192.168.1.66:9999 hibernate-persistence-unit=LoadTest6.ear/Model.jar#PersonPersistenceUnit] cd ../../../
[standalone@192.168.1.66:9999 deployment=LoadTest6.ear] cd subdeployment=Web.war/subsystem=web
[standalone@192.168.1.66:9999 subsystem=web] ls
servlet                     context-root=/LoadTest6     virtual-host=default-host
[standalone@192.168.1.66:9999 subsystem=web] cd servlet=TestServlet
[standalone@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
35
[standalone@192.168.1.66:9999 servlet=TestServlet] read-attribute processing-time
2002L

For the messaging we looked at the attributes; message-count that indicates how many messages are still in the queue. For a data source it important to know if requests (tasks) are waiting for a connection from the pool. This is indicated by the max-wait-time attribute, when this differs from zero it might be useful to adjust the connection pool settings, i.e., increase the maximum number of connections in the pool when possible. Other statistics we viewed was some run-time information from the EJB and servlet environment. An alternative way to view run-time statistics is by using an MBean browser, for example, jconsole or jvisualvm. Note that in the latter case we have to install a plug-in. In jvisualvm click tools, plug-ins, available plug-ins and select visualvm-mbeans. The MBean browser has an entry jboss.as that contains all the run-time information.

We can also put the commands in a script, for example,

connect 192.168.1.66:9999
cd subsystem=messaging/hornetq-server=default/jms-queue=testQueue
pwd
read-attribute delivering-count
read-attribute consumer-count
read-attribute message-count
read-attribute messages-added
cd /
cd subsystem=datasources/data-source=OracleDS
ls
cd statistics=pool
read-attribute AverageCreationTime
read-attribute AvailableCount
read-attribute ActiveCount
read-attribute MaxWaitTime
cd ..
cd statistics=jdbc
read-attribute PreparedStatementCacheAccessCount
read-attribute PreparedStatementCacheHitCount
read-attribute PreparedStatementCacheMissCount
cd /
cd deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3
ls
cd message-driven-bean=CompanyMDB
read-attribute pool-current-size
read-attribute pool-available-count
cd ..
cd stateless-session-bean=Company
read-attribute pool-current-size
read-attribute pool-available-count
cd /
cd deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=jpa
ls
cd hibernate-persistence-unit=LoadTest6.ear\/Model.jar#PersonPersistenceUnit
ls
cd ../../../
cd subdeployment=Web.war/subsystem=web
ls
cd servlet=TestServlet
read-attribute request-count
read-attribute processing-time

To execute the script, we can use (note that we called the script example-script.cli)

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh --file=/home/jboss/jboss-as-7.1.0.Final/scripts/example-script.cli

Using the Apache HTTP Server as a front-end

General information about the Apache HTTP Server can be found here. When we want to use the Apache HTTP Server as a front-end and in particular mod_jk we have to enable an AJP connector for the server in question. To create a new connector requires us to declare a new socket binding first

[standalone@192.168.1.66:9999 /] /socket-binding-group=standard-sockets/socket-binding=ajp:add(port=9080)

The created socket binding can then be used to create a new connector

[standalone@192.168.1.66:9999 /] /subsystem=web/connector=ajp:add(socket-binding=ajp, protocol="AJP/1.3", enabled=true)

With the new connector in place, we can continue with setting-up the Apache HTTP server. To install the Apache HTTP server we can follow these steps

  • Unpack the httpd-2.2.21.tar.gz file:
    • gzip -d httpd-2.2.21.tar.gz
    • tar xvf httpd-2.2.21.tar
    • cd httpd-2.2.21
  • The next step is to configure:
    • ./configure –prefix=/home/jboss/apache
  • Next, compile the various parts for the Apache HTTP Server by using:
    • make
  • To install the Apache HTTP Server we use:
    • make install
  • Open the httpd.conf (/home/jboss/apache/conf) file and adjust the following directives:
    • Listen 8888
    • ServerName ip-address
  • To test the set-up, start the Apache HTTP Server:
    • Navigate to /home/jboss/apache/bin.
    • Run: ./apachectl -k start (To stop the Apache HTTP Server we can use: ./apachectl -k stop).
    • Open a browser en type the following URL: http://hostname:8888.

To install the mod_jk module we can follow these steps

  • Unpack the tomcat-connectors-1.2.32-src.tar.gz file:
    • gzip -d tomcat-connectors-1.2.32-src.tar.gz
    • tar xvf tomcat-connectors-1.2.32-src.tar
    • cd tomcat-connectors-1.2.32-src/native
  • The next step is to configure the module:
    • ./configure –with-apxs=/home/jboss/apache/bin/apxs
  • Compile:
    • make
  • In the native directory, a directory apache-2.0 is created. Here a mod_jk.so file is present. Copy this file to the directory: /home/jboss/apache/modules.

To configure the mod_jk module, create a new file mod_jk.conf in the directory /home/jboss/apache/conf. An example configuration looks as follows

LoadModule jk_module /home/jboss/apache/modules/mod_jk.so

JkWorkersFile /home/jboss/apache/conf/jboss-workers.properties

# where to put the log files for the jk module
JkLogFile /home/jboss/apache/logs/mod_jk.log

# the log level [debug|info|error]
JkLogLevel info

# log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# options to indicate to send SSL KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

# set the request log format
JkRequestLogFormat "%w %V %T"

# send requests that contain the following to JBoss
JkMount /LoadTest6/* standalone-server-worker

Next, we need to define the worker file that is specified by JkWorkersFile directive. In this case, we create a file called jboss-workers.properties in the directory /home/jboss/apache/conf. An example worker configuration is the following

# define a worker that uses ajp13
worker.list=standalone-server-worker

# set properties for the worker
worker.standalone-server-worker.type=ajp13
worker.standalone-server-worker.host=192.168.1.66 (which is the IP-address where the JBoss server is running)
worker.standalone-server-worker.port=9080 (which is AJP listen port of the JBoss Server)

To let the Apache HTTP Server pick up the configuration we add the following to the httpd.conf file

# put it near the end of the file where all the other includes are present
# mod_jk configuration
Include conf/mod_jk.conf

Restart the HTTP Server (./apachectl -k stop and ./apachectl -k start). The application can now be reached at: http://hostname:8888/LoadTest6/testservlet. Note that by using the JkMount directive, we tell mod_jk to forward requests to a configured back-end server.

We can fine tune the web container, for example,

<extensions>
	...
	<extension module="org.jboss.as.web"/>
	...
</extensions>
<profile>
	...
	<subsystem xmlns="urn:jboss:domain:web:1.1" native="false" default-virtual-server="default-host">
		<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
		<connector name="ajp" protocol="AJP/1.3" socket-binding="ajp" enabled="true" enable-lookups="false" executor="ajp-thread-pool" max-connections="150" max-post-size="1048576"/>
		<virtual-server name="default-host" enable-welcome-root="true">
			<alias name="localhost"/>
			<alias name="example.com"/>
		</virtual-server>
	</subsystem>
	...
</profile>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
	<socket-binding name="http" port="8080"/>
	<socket-binding name="https" port="8443"/>
	...
	<socket-binding name="ajp" port="9080"/>
	...
</socket-binding-group>

The connector is a Java object, which provides an interface to Web server clients. The default configuration provides a connector configuration in which most attributes have their default values. For example, to configure a denial of service we add the following attributes:

  • max-connections – the maximum number of simultaneous requests that can be made. We set this equal to the MaxClients directive configured in the Apache HTTP server.
  • max-post-size – limits the maximum size of HTTP POST requests it accepts (the default is 2MB). We set this to 1048576 (1024*1024) bytes. Note that the value is based on what post size to expect from clients. Usually for a web application this is in the range of a few kilobytes. When using web service this value can be in the range of a few 100 kilobytes depending on the message sizes the web service defines, and if attachments can be send as well.
  • max-save-post-size – maximum size of a POST that will be saved by the container during authentication (the default is 4kB which we will leave as it is).

In general, a JBoss application server will be fronted by an Apache HTTP server. One thing to note is that, we have to make the thread pool size equal to the MaxClients directive configured on the Apache HTTP Server (usually it is something like MaxClients 150) otherwise threads will pile up in the JBoss Server. An example thread pool configuration is the following.

<subsystem xmlns="urn:jboss:domain:threads:1.1">
	<blocking-bounded-queue-thread-pool name="ajp-thread-pool">
		<queue-length count="200"/>
		<core-threads count="150"/>
		<max-threads count="150"/>
		<keepalive-time time="30" unit="seconds"/>
	</blocking-bounded-queue-thread-pool>
</subsystem>

Note that when using a small pool with a large queue, we tend to minimize CPU usage, operating system resources and context switching. This can potentially lead to a low throughput. By using small queues we need larger a larger thread pool. This keeps the CPU busier and can lead to an significant increase in scheduling overhead and thereby decreasing throughput. To use a certain thread pool configuration in a connector we can use the executor attribute.

The attribute enable-lookups when set to true, a call to request.getRemoteHost() returns the actual hostname of the remote client. When set to false the IP address is returned. In the latter case no DNS lookup is performed and thus improving performance.

To check the configuration we can use

[standalone@192.168.1.66:9999 /] /subsystem=threads/blocking-bounded-queue-thread-pool=ajp-thread-pool:read-resource(recursive=true)
{
    "outcome" => "success",
    "result" => {
        "allow-core-timeout" => false,
        "core-threads" => 150,
        "keepalive-time" => {
            "time" => 30L,
            "unit" => "SECONDS"
        },
        "max-threads" => 150,
        "name" => "ajp-thread-pool",
        "queue-length" => 200,
        "thread-factory" => undefined
    }
}

[standalone@192.168.1.66:9999 /] /subsystem=web/connector=ajp:read-resource(recursive=true)
{
    "outcome" => "success",
    "result" => {
        "enable-lookups" => false,
        "enabled" => true,
        "executor" => "ajp-thread-pool",
        "max-connections" => 150,
        "max-post-size" => 1048576,
        "max-save-post-size" => 4096,
        "protocol" => "AJP/1.3",
        "redirect-port" => 8443,
        "scheme" => "http",
        "secure" => false,
        "socket-binding" => "ajp",
        "ssl" => undefined,
        "virtual-server" => undefined
    }
}

To see web metrics by using the admin console, after making a number of requests to http://hostname:8888/LoadTest6/testservlet. Open the console (http://hostname:9990/console), click the run-time tab, web in subsystem metrics and click the ajp row. It shows the request count, error count, processing time and max time.

Clustering

To cluster standalone servers we create two configuration files, for example, server1.xml and server2.xml, which are based on the standalone.xml with which we have been working. We first undeploy the LoadTest6 application, by using the command-line interface command: undeploy LoadTest6.ear. Copy the standalone.xml file twice and name the copies to server1.xml and server2.xml. To enable clustering we set the log file name for the server, enable jgroups, and make the messaging system clusterable in the configuration files

<server xmlns="urn:jboss:domain:1.1">
    <extensions>
		...
        <extension module="org.jboss.as.clustering.jgroups"/>
		...
    </extensions>
    <profile>
		<subsystem xmlns="urn:jboss:domain:logging:1.1">
			...
            <periodic-rotating-file-handler name="FILE">
                <formatter>
                    <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="server1.log"/>
                <suffix value=".yyyy-MM-dd"/>
                <append value="true"/>
            </periodic-rotating-file-handler>
			...
        </subsystem>
		...
        <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">
            <stack name="udp">
                <transport type="UDP" socket-binding="jgroups-udp" diagnostics-socket-binding="jgroups-diagnostics"/>
                <protocol type="PING"/>
                <protocol type="MERGE2"/>
                <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                <protocol type="FD"/>
                <protocol type="VERIFY_SUSPECT"/>
                <protocol type="BARRIER"/>
                <protocol type="pbcast.NAKACK"/>
                <protocol type="UNICAST2"/>
                <protocol type="pbcast.STABLE"/>
                <protocol type="pbcast.GMS"/>
                <protocol type="UFC"/>
                <protocol type="MFC"/>
                <protocol type="FRAG2"/>
            </stack>
            <stack name="tcp">
                <transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>
                <protocol type="MPING" socket-binding="jgroups-mping"/>
                <protocol type="MERGE2"/>
                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                <protocol type="FD"/>
                <protocol type="VERIFY_SUSPECT"/>
                <protocol type="BARRIER"/>
                <protocol type="pbcast.NAKACK"/>
                <protocol type="UNICAST2"/>
                <protocol type="pbcast.STABLE"/>
                <protocol type="pbcast.GMS"/>
                <protocol type="UFC"/>
                <protocol type="MFC"/>
                <protocol type="FRAG2"/>
            </stack>
        </subsystem>
		...
		<subsystem xmlns="urn:jboss:domain:messaging:1.1">
            <hornetq-server>
                <clustered>true</clustered>
                <cluster-user>guest</cluster-user>
                <cluster-password>guest</cluster-password>
                <persistence-enabled>true</persistence-enabled>
                <journal-type>ASYNCIO</journal-type>
                <journal-file-size>102400</journal-file-size>
                <journal-min-files>2</journal-min-files>
                <paging-directory path="messagingpaging1"/>
                <bindings-directory path="messagingbindings1"/>
                <journal-directory path="messagingjournal1"/>
                <large-messages-directory path="messaginglargemessages1"/>
                <connectors>
                    <netty-connector name="netty" socket-binding="messaging"/>
                    <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                    </netty-connector>
                    <in-vm-connector name="in-vm" server-id="0"/>
                </connectors>
                <acceptors>
                    <netty-acceptor name="netty" socket-binding="messaging"/>
                    <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                        <param key="direct-deliver" value="false"/>
                    </netty-acceptor>
                    <in-vm-acceptor name="in-vm" server-id="0"/>
                </acceptors>
                <broadcast-groups>
                    <broadcast-group name="bg-group1">
                        <group-address>231.7.7.7</group-address>
                        <group-port>9876</group-port>
                        <broadcast-period>5000</broadcast-period>
                        <connector-ref>netty</connector-ref>
                    </broadcast-group>
                </broadcast-groups>
                <discovery-groups>
                    <discovery-group name="dg-group1">
                        <group-address>231.7.7.7</group-address>
                        <group-port>9876</group-port>
                        <refresh-timeout>10000</refresh-timeout>
                    </discovery-group>
                </discovery-groups>
                <cluster-connections>
                    <cluster-connection name="my-cluster">
                        <address>jms</address>
                        <connector-ref>netty</connector-ref>
                        <discovery-group-ref discovery-group-name="dg-group1"/>
                    </cluster-connection>
                </cluster-connections>
                <security-settings>
                    <security-setting match="#">
                        <permission type="send" roles="guest"/>
                        <permission type="consume" roles="guest"/>
                        <permission type="createNonDurableQueue" roles="guest"/>
                        <permission type="deleteNonDurableQueue" roles="guest"/>
                    </security-setting>
                </security-settings>
                <address-settings>
                    <address-setting match="#">
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <redelivery-delay>0</redelivery-delay>
                        <redistribution-delay>1000</redistribution-delay>
                        <max-size-bytes>10485760</max-size-bytes>
                        <address-full-policy>BLOCK</address-full-policy>
                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                    </address-setting>
                </address-settings>
                <jms-connection-factories>
                    <connection-factory name="InVmConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/ConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <connection-factory name="RemoteConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="netty"/>
                        </connectors>
                        <entries>
                            <entry name="RemoteConnectionFactory"/>
                            <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <pooled-connection-factory name="hornetq-ra">
                        <transaction mode="xa"/>
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/JmsXA"/>
                        </entries>
                    </pooled-connection-factory>
                </jms-connection-factories>
                <jms-destinations>
                    <jms-queue name="testQueue">
                        <entry name="java:/queue/test"/>
                        <entry name="java:jboss/exported/jms/queue/test"/>
                    </jms-queue>
                    <jms-topic name="testTopic">
                        <entry name="java:/topic/test"/>
                        <entry name="java:jboss/exported/jms/topic/test"/>
                    </jms-topic>
                </jms-destinations>
            </hornetq-server>
        </subsystem>
		...
    </profile>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
		...
        <socket-binding name="jgroups-diagnostics" port="0" multicast-address="224.0.75.75" multicast-port="7500"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" port="57600"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="54200"/>
		...
    </socket-binding-group>
</server>

For server2 we use the port-offset attribute (which we set to one) in the socket binding

<server xmlns="urn:jboss:domain:1.1">
    <extensions>
		...
        <extension module="org.jboss.as.clustering.jgroups"/>
		...
    </extensions>
    <profile>
		<subsystem xmlns="urn:jboss:domain:logging:1.1">
			...
            <periodic-rotating-file-handler name="FILE">
                <formatter>
                    <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="server2.log"/>
                <suffix value=".yyyy-MM-dd"/>
                <append value="true"/>
            </periodic-rotating-file-handler>
			...
        </subsystem>
		...
        <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">...</subsystem>
		...
		<subsystem xmlns="urn:jboss:domain:messaging:1.1">
            <hornetq-server>
                <clustered>true</clustered>
                <cluster-user>guest</cluster-user>
                <cluster-password>guest</cluster-password>
                <persistence-enabled>true</persistence-enabled>
                <journal-type>ASYNCIO</journal-type>
                <journal-file-size>102400</journal-file-size>
                <journal-min-files>2</journal-min-files>
                <paging-directory path="messagingpaging2"/>
                <bindings-directory path="messagingbindings2"/>
                <journal-directory path="messagingjournal2"/>
                <large-messages-directory path="messaginglargemessages2"/>
				...
            </hornetq-server>
        </subsystem>
		...
    </profile>
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:1}">
		...
    </socket-binding-group>
</server>

To start the servers, we can use standalone.sh --server-config=server1.xml and standalone.sh --server-config=server2.xml. When the servers are started the following is shown when hornetq is connected successfully between both servers

# server1
17:24:06,129 INFO  [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-9 (HornetQ-server-HornetQServerImpl::serverUUID=d052af0e-5efa-11e1-ad54-000c2976c82d-37774788)) Bridge ClusterConnectionBridge@4552a64d [name=sf.my-cluster.c25598f7-5efa-11e1-8e93-000c2976c82d, queue=QueueImpl[name=sf.my-cluster.c25598f7-5efa-11e1-8e93-000c2976c82d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=d052af0e-5efa-11e1-ad54-000c2976c82d]]@35242cc9 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@4552a64d [name=sf.my-cluster.c25598f7-5efa-11e1-8e93-000c2976c82d, queue=QueueImpl[name=sf.my-cluster.c25598f7-5efa-11e1-8e93-000c2976c82d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=d052af0e-5efa-11e1-ad54-000c2976c82d]]@35242cc9 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5446&host=axis-into-ict-nl], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@762010908[nodeUUID=d052af0e-5efa-11e1-ad54-000c2976c82d, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=axis-into-ict-nl, address=jms, server=HornetQServerImpl::serverUUID=d052af0e-5efa-11e1-ad54-000c2976c82d])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5446&host=axis-into-ict-nl], discoveryGroupConfiguration=null]] is connected

# server2
17:24:06,123 INFO  [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-29 (HornetQ-server-HornetQServerImpl::serverUUID=c25598f7-5efa-11e1-8e93-000c2976c82d-1291383602)) Bridge ClusterConnectionBridge@4174b989 [name=sf.my-cluster.d052af0e-5efa-11e1-ad54-000c2976c82d, queue=QueueImpl[name=sf.my-cluster.d052af0e-5efa-11e1-ad54-000c2976c82d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c25598f7-5efa-11e1-8e93-000c2976c82d]]@5640f2f1 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@4174b989 [name=sf.my-cluster.d052af0e-5efa-11e1-ad54-000c2976c82d, queue=QueueImpl[name=sf.my-cluster.d052af0e-5efa-11e1-ad54-000c2976c82d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c25598f7-5efa-11e1-8e93-000c2976c82d]]@5640f2f1 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=axis-into-ict-nl], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@211830739[nodeUUID=c25598f7-5efa-11e1-8e93-000c2976c82d, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5446&host=axis-into-ict-nl, address=jms, server=HornetQServerImpl::serverUUID=c25598f7-5efa-11e1-8e93-000c2976c82d])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=axis-into-ict-nl], discoveryGroupConfiguration=null]] is connected

To deploy the application we can use the following

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/LoadTest6.ear
[standalone@192.168.1.66:9999 /] connect 192.168.1.66:10000
[standalone@192.168.1.66:10000 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/LoadTest6.ear11

Before, we configure the load balancing, test if the application is working. Either use the URL: http://hostname:8080/LoadTest6/testservlet or the URL: http://hostname:8081/LoadTest6/testservlet.

Configure load balancing

The load balancer decides how to dispatch requests to the back end server instances. A load balancer needs to do tasks such as distributing requests, health checking and session binding. As these are simple jobs it is rare that a load balancer will become a bottleneck. To load balance requests, we are going to use mod_jk. To configure mod_jk, open the mod_jk.conf file and add the following contents:

LoadModule jk_module /home/jboss/apache/modules/mod_jk.so

JkWorkersFile /home/jboss/apache/conf/jboss-workers.properties

# where to put the log files for the jk module
JkLogFile /home/jboss/apache/logs/mod_jk.log

# the log level [debug|info|error]
JkLogLevel info

# log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# options to indicate to send SSL KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

# set the request log format
JkRequestLogFormat "%w %V %T"

# send requests that contain the following to JBoss
JkMount /LoadTest6/* loadbalancer

The workers (jboss-workers.properties) are defined as follows:

# define a worker that uses ajp13
worker.list=server1-worker,server2-worker,loadbalancer

# set properties for the server1-worker
worker.server1-worker.type=ajp13
worker.server1-worker.host=192.168.1.66
worker.server1-worker.port=9080
worker.server1-worker.lbfactor=1
worker.server1-worker.socket_keepalive=1
worker.server1-worker.socket_timeout=300

# set properties for the server2-worker
worker.server2-worker.type=ajp13
worker.server2-worker.host=192.168.1.66
worker.server2-worker.port=9081
worker.server2-worker.lbfactor=1
worker.server2-worker.socket_keepalive=1
worker.server2-worker.socket_timeout=300

# set properties for the loadbalancer
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=server1-worker,server2-worker

With the configuration in place, start the Apache HTTP server (./apachectl -k start) and enter the following URL: http://hostname:8888/LoadTest6/testservlet a few times to see if the requests get send to the different servers. By using the following URL’s: http://hostname:9990/console/App.html#web-metrics and http://hostname:9991/console/App.html#web-metrics, we can check how the requests are being processed. We can also use the command-line interface

[disconnected /] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 /] cd /deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web/servlet=TestServlet
[standalone@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
69
[standalone@192.168.1.66:9999 servlet=TestServlet] connect 192.168.1.66:10000
[standalone@192.168.1.66:10000 servlet=TestServlet] read-attribute request-count
68
[standalone@192.168.1.66:10000 servlet=TestServlet] read-attribute request-count
118
[standalone@192.168.1.66:10000 servlet=TestServlet] connect 192.168.1.66:9999
[standalone@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
119

As we can see the load is balanced like a charm.

Load test

We can test the environment by using the The Grinder. The following script calls the application:

from net.grinder.script.Grinder import grinder
from net.grinder.script import Test
from net.grinder.plugin.http import HTTPRequest

test1 = Test(1, "Request resource")
request1 = test1.wrap(HTTPRequest())

class TestRunner:
    def __call__(self):
        result = request1.GET("http://192.168.1.66:8888/LoadTest6/testservlet")

Edit the grinder.properties (${GRINDER_HOME}/examples) file such that the script is used, for example,

# The file name of the script to run. Relative paths are evaluated from the directory containing the properties file.
grinder.script = test.py

# The number of worker processes each agent should start.
grinder.processes = 1

# The number of worker threads each worker process should start.
grinder.threads = 1

# The number of runs each worker process will perform. When using the console set this to 0, i.e., the runs are controlled by the console.
grinder.runs = 0

Use the following scripts to start a test

setGrinderEnv.sh

#!/bin/sh
GRINDERPATH=/home/jboss/grinder-3.4
export GRINDERPATH

JAVA_HOME=/home/jboss/jdk1.6.0_31
export JAVA_HOME

USER_MEM_ARGS="-server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC"
export USER_MEM_ARGS
GRINDERPROPERTIES=${GRINDERPATH}/examples/grinder.properties
export GRINDERPROPERTIES
CLASSPATH=${GRINDERPATH}/lib/grinder.jar:${CLASSPATH}
export CLASSPATH

PATH=${JAVA_HOME}/bin:${PATH}
export PATH

startConsole.sh

#!/bin/sh
source setGrinderEnv.sh
java ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Console

startAgent.sh

#!/bin/sh
source setGrinderEnv.sh
java ${USER_MEM_ARGS} -cp ${CLASSPATH} net.grinder.Grinder ${GRINDERPROPERTIES}

First start the startConsole.sh script and then start the startAgent.sh script. The agent will now wait for a start action from the console (click the action tab and then choose start processes)

By using jvisualvm or jconsole the JVM can be monitored and MBeans can browsed

We can clearly see when the load test was running. The test introduced no particular problems for the JVM. Run-time statistics from the command-line interface

server 1
subsystem=messaging/hornetq-server=default/jms-queue=testQueue
delivering-count (The number of messages that this queue is currently delivering to its consumers): 0
consumer-count (The number of consumers consuming messages from this queue): 15
message-count (The number of messages currently in this queue): 0L
messages-added (The number of messages added to this queue since it was created): 11987L

subsystem=datasources/data-source=OracleDS/statistics=pool
AverageCreationTime (The average time spent creating a physical connection): 2594
AvailableCount (The available count): 15
ActiveCount (The active count): 2
MaxWaitTime (The maximum wait time for a connection): 3

subsystem=datasources/data-source=OracleDS/statistics=jdbc
PreparedStatementCacheAccessCount (The number of times that the statement cache was accessed): 770567
PreparedStatementCacheHitCount (The number of times that statements from the cache were used): 770560
PreparedStatementCacheMissCount (The number of times that a statement request could not be satisfied with a statement from the cache): 0

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3/message-driven-bean=CompanyMDB
pool-current-size (The current size of the pool): 4
pool-available-count (The number of available (i.e. not in use) instances in the pool): 12003

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3/stateless-session-bean=Company
pool-current-size (The current size of the pool): 1
pool-available-count (The number of available (i.e. not in use) instances in the pool): 385348

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=jpa/hibernate-persistence-unit=LoadTest6.ear\/Model.jar#PersonPersistenceUnit
completed-transaction-count (Number of completed transactions): 385329L
successful-transaction-count (Number of successful transactions): 385329L

deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web/servlet=TestServlet
request-count (Number of requests processed by this servlet): 385329
processing-time (Total execution time of the servlet's service method): 926143L

server 2
subsystem=messaging/hornetq-server=default/jms-queue=testQueue
delivering-count (The number of messages that this queue is currently delivering to its consumers): 0
consumer-count (The number of consumers consuming messages from this queue): 15
message-count (The number of messages currently in this queue): 0L
messages-added (The number of messages added to this queue since it was created): 11987L

subsystem=datasources/data-source=OracleDS/statistics=pool
AverageCreationTime (The average time spent creating a physical connection): 184
AvailableCount (The available count): 15
ActiveCount (The active count): 2
MaxWaitTime (The maximum wait time for a connection): 1

subsystem=datasources/data-source=OracleDS/statistics=jdbc
PreparedStatementCacheAccessCount (The number of times that the statement cache was accessed): 770566
PreparedStatementCacheHitCount (The number of times that statements from the cache were used): 770559
PreparedStatementCacheMissCount (The number of times that a statement request could not be satisfied with a statement from the cache): 0

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3/message-driven-bean=CompanyMDB
pool-current-size (The current size of the pool): 5
pool-available-count (The number of available (i.e. not in use) instances in the pool): 12002

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=ejb3/stateless-session-bean=Company
pool-current-size (The current size of the pool): 1
pool-available-count (The number of available (i.e. not in use) instances in the pool): 385344

deployment=LoadTest6.ear/subdeployment=Model.jar/subsystem=jpa/hibernate-persistence-unit=LoadTest6.ear\/Model.jar#PersonPersistenceUnit
completed-transaction-count (Number of completed transactions): 385325L
successful-transaction-count (Number of successful transactions): 385325L

deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web/servlet=TestServlet
request-count (Number of requests processed by this servlet): 385325
processing-time (Total execution time of the servlet's service method): 922760L

Overall the test was easily handled by the Apache HTTP server (load balancing) and the JBoss application servers present in the cluster.

Domains

One of the new features of JBoss AS 7 is the ability to manage multiple server instances from a single control point. A domain controller process is acting as the management point. All the server instances share a common management policy, with the domain controller acting to ensure that each server is configured according to that policy. Domains can span multiple machines, with server instances on a certain host under the control of a host controller process. One host controller instance is configured as the domain controller. The host controller on each host interacts with the domain controller to control the lifecycle of the server instances running on a particular host and to assist the domain controller in managing them. One thing to note is that the configuration of subsystems (such as messaging and data sources) within a profile does not differ from the standalone configuration (apart from the fact that a domain configuration can have multiple profiles whereas a standalone configuration can only have one). What we need to get the hang of when working with domains are the domain and host controller configurations.

When the ${JBOSS_HOME}/bin/domain.sh script is run a host controller process is started. The host controller starts and stops server instances on the particular host where it was started. Each host controller is configured by using the ${JBOSS_HOME}/domain/configuration/host.xml file. This file contains

  • The list of application server names that must run on the host.
  • The configuration how the host controller must contact the domain controller to register itself and access the domain configuration. This can either be how to contact a remote domain controller or telling the host controller to act itself as a domain controller.

As already mentioned one host controller must act as the domain controller (central management point). The domain controller maintains a central management policy and makes sure the host controllers are aware of the contents. It also make sure that each server instance is configured with respect to the central management policy. The central management policy is by default stored in the ${JBOSS_HOME}/domain/configuration/domain.xml file on the filesystem where the domain controller is present (it is not necessary to have the domain.xml on the filesystems where other host controllers are running).

When working with a domain we also need to introduce the concept of a server group, which is a set of server instances that will be managed and configured as one. In a managed domain each application server instance is a member of a server group. A domain can have multiple server groups. Different server groups can be configured with different profiles and deployments.

An example domain.xml look as follows:

<domain xmlns="urn:jboss:domain:1.1">
    <extensions>
		<extension module="org.jboss.as.clustering.infinispan"/>
        <extension module="org.jboss.as.clustering.jgroups"/>
		...
        <extension module="org.jboss.as.messaging"/>
		...
        <extension module="org.jboss.as.weld"/>
	</extensions>
    <system-properties>
        <property name="java.net.preferIPv4Stack" value="true"/>
    </system-properties>
    <profiles>
        <profile name="default">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
        <profile name="ha">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
        <profile name="full">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:messaging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
        <profile name="full-ha">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
			<subsystem xmlns="urn:jboss:domain:datasources:1.0">
                <datasources>
                    ...
                    <datasource jta="true" jndi-name="java:/jdbc/OracleDS" pool-name="OracleDS" enabled="true" use-java-context="true" use-ccm="true">
                        <connection-url>jdbc:oracle:thin:@192.168.1.65:1521:orcl11</connection-url>
                        <driver>oracle</driver>
                        <pool>
                            <min-pool-size>1</min-pool-size>
                            <max-pool-size>15</max-pool-size>
                            <prefill>true</prefill>
                            <use-strict-min>true</use-strict-min>
                        </pool>
                        <security>
                            <user-name>example</user-name>
                            <password>example</password>
                        </security>
                        <timeout>
                            <idle-timeout-minutes>0</idle-timeout-minutes>
                            <query-timeout>600</query-timeout>
                        </timeout>
                        <statement>
                            <prepared-statement-cache-size>10</prepared-statement-cache-size>
                        </statement>
                    </datasource>
                    <drivers>
					    ...
                        <driver name="oracle" module="com.oracle.database">
                            <driver-class>oracle.jdbc.OracleDriver</driver-class>
                            <xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
                        </driver>
                    </drivers>
                </datasources>
            </subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:messaging:1.1">...</subsystem>
			...
			<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
				...
                <connector name="ajp" protocol="AJP/1.3" socket-binding="ajp" enable-lookups="false" max-post-size="1048576" enabled="true" max-connections="150"/>
				...
            </subsystem>
			...
			<subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
    </profiles>
    <!--Named interfaces that can be referenced elsewhere in the configuration. The configuration
        for how to associate these logical names with an actual network interface can either
        be specified here or can be declared on a per-host basis in the equivalent element in host.xml.
        These default configurations require the binding specification to be done in host.xml.-->
    <interfaces>
        <interface name="management"/>
        <interface name="public"/>
        <interface name="unsecure"/>
    </interfaces>
    <socket-binding-groups>
        <socket-binding-group name="standard-sockets" default-interface="public">...</socket-binding-group>
        <socket-binding-group name="ha-sockets" default-interface="public">
			...
			<socket-binding name="ajp" port="9080"/>
			...
		</socket-binding-group>
    </socket-binding-groups>
	<deployments>
        <deployment name="LoadTest6.ear" runtime-name="LoadTest6.ear">
            <content sha1="161f51dde7f085c822cc4c68b306d57f1bee902d"/>
        </deployment>
    </deployments>
    <server-groups>
        <server-group name="standard-server-group" profile="full">
            <jvm name="default"/>
            <socket-binding-group ref="standard-sockets"/>
        </server-group>
        <server-group name="ha-server-group" profile="full-ha">
            <jvm name="default"/>
            <socket-binding-group ref="ha-sockets"/>
			<deployments>
                <deployment name="LoadTest6.ear" runtime-name="LoadTest6.ear"/>
            </deployments>
        </server-group>
    </server-groups>
</domain>

In the example above we have set-up different profiles:

  • default – a basic server set-up
  • ha – a basic cluster set-up
  • full – a server set-up with messaging enabled
  • full-ha – a cluster set-up with messaging enabled

Different socket-binding groups one for the default and full profiles and one for the ha and full-ha profiles. Further, we defined two server groups, each mapped to a different profile. Note that server groups can have their own settings, such as JVM, socket-binding-group and deployments. To map server instances to a server group, we use a host configuration, for example,

<host name="axis-into-ict.nl" xmlns="urn:jboss:domain:1.1">
    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <authentication>
                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
                </authentication>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <authentication>
                    <properties path="application-users.properties" relative-to="jboss.domain.config.dir" />
                </authentication>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket interface="management" port="${jboss.management.native.port:9999}"/>
            </native-interface>
            <http-interface security-realm="ManagementRealm">
                <socket interface="management" port="${jboss.management.http.port:9990}"/>
            </http-interface>
        </management-interfaces>
    </management>
    <domain-controller>
       <local/>
       <!-- remote domain controller configuration: remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" -->
    </domain-controller>
    <interfaces>
        <interface name="management">
            <nic name="eth0"/>
        </interface>
        <interface name="public">
           <nic name="eth0"/>
        </interface>
        <interface name="unsecure">
            <inet-address value="127.0.0.1"/>
        </interface>
    </interfaces>
    <jvms>
    	<jvm name="default">
            <heap size="512m" max-size="512m"/>
            <permgen size="256m" max-size="256m"/>
            <jvm-options>
                <option value="-server"/>
                <option value="-XX:NewRatio=2"/>
                <option value="-XX:+UseParallelGC"/>
                <option value="-XX:ParallelGCThreads=2"/>
                <option value="-XX:MaxGCPauseMillis=200"/>
                <option value="-XX:GCTimeRatio=19"/>
                <option value="-XX:+UseParallelOldGC"/>
            </jvm-options>
        </jvm>
    </jvms>
    <servers>
        <server name="server1" group="ha-server-group" auto-start="true">
        </server>
        <server name="server2" group="ha-server-group" auto-start="true">
            <socket-bindings port-offset="1"/>
        </server>
    </servers>
</host>

Configuring a host to act as the domain controller is done through the domain-controller declaration in host.xml. If it includes the local element, then this host will become the domain controller. A host acting as the domain controller must expose a native (i.e. non-HTTP) management interface on an address accessible to the other hosts in the domain. Exposing an HTTP management interface is not required, but is recommended as it allows the administration console to work. The configuration above also contains the network interfaces which we mapped to the network interface card eth0.

We can start the domain by using domain.sh located in the ${JBOSS_HOME}/bin directory (do not forget to define JAVA_HOME and PATH before running domain.sh)

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./domain.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/jboss-as-7.1.0.Final

  JAVA: /home/jboss/jdk1.6.0_31/bin/java

  JAVA_OPTS: -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml

=========================================================================

10:13:12,861 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
10:13:13,459 INFO  [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
[Host Controller] 10:13:14,908 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Host Controller] 10:13:15,288 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Host Controller] 10:13:15,603 INFO  [org.jboss.as] (MSC service thread 1-3) JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
[Host Controller] 10:13:17,216 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.3.GA
[Host Controller] 10:13:17,237 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.3.GA
[Host Controller] 10:13:17,764 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.2.2.GA
[Host Controller] 10:13:21,800 INFO  [org.jboss.as] (Controller Boot Thread) JBAS010902: Creating http management service using network interface (management) port (9990) securePort (-1)
[Host Controller] 10:13:21,829 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on /192.168.1.66:9999
[Host Controller] 10:13:22,129 INFO  [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010922: Starting server server1
[Host Controller] 10:13:22,197 INFO  [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010922: Starting server server2
10:13:22,199 INFO  [org.jboss.as.process.Server:server1.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server1'
[Host Controller] 10:13:22,216 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" (Host Controller) started in 8647ms - Started 11 of 11 services (0 services are passive or on-demand)
10:13:22,292 INFO  [org.jboss.as.process.Server:server2.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server2'
[Server:server1] 10:13:22,976 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Server:server2] 10:13:23,121 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Server:server1] 10:13:23,671 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Server:server2] 10:13:23,672 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Server:server1] 10:13:23,833 INFO  [org.jboss.as] (MSC service thread 1-4) JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
[Server:server2] 10:13:23,850 INFO  [org.jboss.as] (MSC service thread 1-4) JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
[Server:server1] 10:13:24,097 INFO  [org.xnio] (MSC service thread 1-3) XNIO Version 3.0.3.GA
[Server:server1] 10:13:24,127 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.0.3.GA
[Server:server2] 10:13:24,135 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.3.GA
[Server:server1] 10:13:24,161 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 3.2.2.GA
[Server:server2] 10:13:24,204 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.3.GA
[Server:server2] 10:13:24,236 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 3.2.2.GA
[Server:server1] 10:13:26,666 INFO  [org.jboss.as.logging] (MSC service thread 1-3) JBAS011502: Removing bootstrap log handlers
[Server:server2] 10:13:26,867 INFO  [org.jboss.as.logging] (MSC service thread 1-1) JBAS011502: Removing bootstrap log handlers
[Server:server2] 10:13:26,890 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 32) JBAS015537: Activating WebServices Extension
[Server:server2] 10:13:26,902 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 36) JBAS013101: Activating Security Subsystem
[Server:server2] 10:13:26,993 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 59) JBAS016200: Activating ConfigAdmin Subsystem
[Server:server2] 10:13:26,975 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 55) JBAS010280: Activating Infinispan subsystem.
[Server:server2] 10:13:26,974 INFO  [org.jboss.as.jacorb] (ServerService Thread Pool -- 54) JBAS016300: Activating JacORB Subsystem
[Server:server2] 10:13:26,964 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 49) JBAS010260: Activating JGroups subsystem.
[Server:server2] 10:13:26,950 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 41) JBAS011940: Activating OSGi Subsystem
[Server:server2] 10:13:26,914 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 42) JBAS011800: Activating Naming Subsystem
[Server:server2] 10:13:27,225 INFO  [org.jboss.as.security] (MSC service thread 1-1) JBAS013100: Current PicketBox version=4.0.6.final
[Server:server2] 10:13:27,568 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 58) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
[Server:server2] 10:13:27,719 INFO  [org.jboss.as.connector] (MSC service thread 1-4) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
[Server:server1] 10:13:27,837 INFO  [org.jboss.as.naming] (MSC service thread 1-4) JBAS011802: Starting Naming Service
[Server:server2] 10:13:27,973 INFO  [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: Starting Naming Service
[Server:server2] 10:13:27,998 INFO  [org.jboss.as.jaxr] (MSC service thread 1-1) Binding JAXR ConnectionFactory: java:jboss/jaxr/ConnectionFactory
[Server:server1] 10:13:27,858 INFO  [org.jboss.as.connector] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.7.Final)
[Server:server2] 10:13:28,163 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) JBAS015400: Bound mail session [java:jboss/mail/Default]
[Server:server1] 10:13:28,187 INFO  [org.jboss.as.jaxr] (MSC service thread 1-4) Binding JAXR ConnectionFactory: java:jboss/jaxr/ConnectionFactory
[Server:server2] 10:13:28,300 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 58) JBAS010403: Deploying JDBC-compliant driver class oracle.jdbc.OracleDriver (version 11.2)
[Server:server1] 10:13:28,297 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 58) JBAS010403: Deploying JDBC-compliant driver class oracle.jdbc.OracleDriver (version 11.2)
[Host Controller] 10:13:28,359 INFO  [org.jboss.as.domain.controller.mgmt] (proxy-threads - 1) JBAS010920: Server [Server:server1] registered using connection [Channel ID 6cdb4665 (inbound) of Remoting connection 71ee88dd to /192.168.1.66:59624]
[Host Controller] 10:13:28,443 INFO  [org.jboss.as.domain.controller.mgmt] (proxy-threads - 1) JBAS010920: Server [Server:server2] registered using connection [Channel ID 025391e5 (inbound) of Remoting connection 633a494d to /192.168.1.66:55081]
[Host Controller] 10:13:28,472 INFO  [org.jboss.as.host.controller] (proxy-threads - 1) JBAS010919: Registering server server1
[Server:server1] 10:13:28,862 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS015400: Bound mail session [java:jboss/mail/Default]
[Host Controller] 10:13:28,874 INFO  [org.jboss.as.host.controller] (proxy-threads - 2) JBAS010919: Registering server server2
[Server:server2] 10:13:29,228 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 55) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[Server:server1] 10:13:29,225 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 55) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[Server:server1] 10:13:29,387 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 55) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[Server:server2] 10:13:29,449 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 55) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[Server:server2] 10:13:29,672 INFO  [org.apache.coyote.ajp.AjpAprProtocol] (MSC service thread 1-2) Starting Coyote AJP/1.3 on ajp-axis-into-ict.nl-192.168.1.66-9081
[Server:server1] 10:13:29,687 INFO  [org.apache.coyote.ajp.AjpAprProtocol] (MSC service thread 1-1) Starting Coyote AJP/1.3 on ajp-axis-into-ict.nl-192.168.1.66-9080
[Server:server2] 10:13:29,815 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-4) JBoss Web Services - Stack CXF Server 4.0.1.GA
[Server:server2] 10:13:29,817 INFO  [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-1) Starting Coyote HTTP/1.1 on http-axis-into-ict.nl-192.168.1.66-8081
[Server:server1] 10:13:29,839 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-3) JBoss Web Services - Stack CXF Server 4.0.1.GA
[Server:server2] 10:13:29,846 INFO  [org.jboss.as.modcluster] (MSC service thread 1-3) JBAS011704: Mod_cluster uses default load balancer provider
[Server:server1] 10:13:29,853 INFO  [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-4) Starting Coyote HTTP/1.1 on http-axis-into-ict.nl-192.168.1.66-8080
[Server:server1] 10:13:29,879 INFO  [org.jboss.as.modcluster] (MSC service thread 1-2) JBAS011704: Mod_cluster uses default load balancer provider
[Server:server1] 10:13:30,055 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/data/messagingjournal,bindingsDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/data/messagingbindings,largeMessagesDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/data/messaginglargemessages,pagingDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/data/messagingpaging)
[Server:server2] 10:13:30,068 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/data/messagingjournal,bindingsDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/data/messagingbindings,largeMessagesDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/data/messaginglargemessages,pagingDirectory=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/data/messagingpaging)
[Server:server1] 10:13:30,090 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-2) Initializing mod_cluster 1.2.0.Final
[Server:server1] 10:13:30,091 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) Waiting to obtain live lock
[Server:server2] 10:13:30,102 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) Waiting to obtain live lock
[Server:server2] 10:13:30,131 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-3) Initializing mod_cluster 1.2.0.Final
[Server:server1] 10:13:30,208 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-2) Listening to proxy advertisements on 224.0.1.105:23364
[Server:server2] 10:13:30,264 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-3) Listening to proxy advertisements on 224.0.1.105:23364
[Server:server1] 10:13:30,338 INFO  [org.hornetq.core.persistence.impl.journal.JournalStorageManager] (MSC service thread 1-3) Using AIO Journal
[Server:server2] 10:13:30,386 INFO  [org.hornetq.core.persistence.impl.journal.JournalStorageManager] (MSC service thread 1-2) Using AIO Journal
[Server:server1] 10:13:30,816 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-3) Waiting to obtain live lock
[Server:server1] 10:13:30,817 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-3) Live Server Obtained live lock
[Server:server2] 10:13:30,954 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-2) Waiting to obtain live lock
[Server:server2] 10:13:30,954 INFO  [org.hornetq.core.server.impl.AIOFileLockNodeManager] (MSC service thread 1-2) Live Server Obtained live lock
[Server:server2] 10:13:31,115 INFO  [org.jboss.as.jacorb] (MSC service thread 1-4) JBAS016330: CORBA ORB Service started
[Server:server1] 10:13:31,108 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) JBAS016330: CORBA ORB Service started
[Server:server1] 10:13:31,341 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
[Server:server2] 10:13:31,438 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
[Server:server2] 10:13:31,439 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:/jdbc/OracleDS]
[Server:server1] 10:13:31,448 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010400: Bound data source [java:/jdbc/OracleDS]
[Server:server1] 10:13:31,449 INFO  [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on axis-into-ict.nl/192.168.1.66:4447
[Server:server2] 10:13:31,462 INFO  [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100: Listening on axis-into-ict.nl/192.168.1.66:4448
[Server:server2] 10:13:31,554 INFO  [org.jboss.as.jacorb] (MSC service thread 1-4) JBAS016328: CORBA Naming Service started
[Server:server1] 10:13:31,587 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) JBAS016328: CORBA Naming Service started
[Server:server2] 10:13:32,254 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-2) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5446 for CORE protocol
[Server:server2] 10:13:32,257 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-2) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5456 for CORE protocol
[Server:server1] 10:13:32,257 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-3) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5445 for CORE protocol
[Server:server2] 10:13:32,258 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) Server is now live
[Server:server2] 10:13:32,259 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) HornetQ Server version 2.2.11.Final (HQ_2_2_11_FINAL_AS7, 122) [2b25644c-61f8-11e1-a2e3-000c2976c82d]) started
[Server:server1] 10:13:32,260 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-3) Started Netty Acceptor version 3.2.5.Final-a96d88c axis-into-ict.nl:5455 for CORE protocol
[Server:server1] 10:13:32,262 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) Server is now live
[Server:server1] 10:13:32,263 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) HornetQ Server version 2.2.11.Final (HQ_2_2_11_FINAL_AS7, 122) [2b200d12-61f8-11e1-9621-000c2976c82d]) started
[Server:server1] 10:13:32,338 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
[Server:server2] 10:13:32,342 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
[Server:server2] 10:13:32,344 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
[Server:server1] 10:13:32,343 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-1) trying to deploy queue jms.queue.testQueue
[Server:server2] 10:13:32,347 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-3) trying to deploy queue jms.queue.testQueue
[Server:server2] 10:13:32,356 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:/queue/test
[Server:server2] 10:13:32,362 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/test
[Server:server2] 10:13:32,368 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-4) trying to deploy queue jms.topic.testTopic
[Server:server1] 10:13:32,368 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/queue/test
[Server:server1] 10:13:32,370 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/queue/test
[Server:server1] 10:13:32,371 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) trying to deploy queue jms.topic.testTopic
[Server:server1] 10:13:32,442 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-1) JBAS010406: Registered connection factory java:/JmsXA
[Server:server2] 10:13:32,445 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010406: Registered connection factory java:/JmsXA
[Server:server2] 10:13:32,468 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-2) HornetQ resource adaptor started
[Server:server1] 10:13:32,470 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/topic/test
[Server:server1] 10:13:32,472 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/topic/test
[Server:server2] 10:13:32,469 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:/topic/test
[Server:server1] 10:13:32,473 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
[Server:server1] 10:13:32,474 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
[Server:server2] 10:13:32,475 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-2) IJ020002: Deployed: file://RaActivatorhornetq-ra
[Server:server2] 10:13:32,478 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:jboss/exported/jms/topic/test
[Server:server2] 10:13:32,480 INFO  [org.jboss.as.messaging] (MSC service thread 1-1) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
[Server:server1] 10:13:32,482 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-1) HornetQ resource adaptor started
[Server:server2] 10:13:32,483 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
[Server:server1] 10:13:32,490 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-1) IJ020002: Deployed: file://RaActivatorhornetq-ra
[Server:server1] 10:13:32,497 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-2) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
[Server:server2] 10:13:32,504 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 10041ms - Started 169 of 292 services (121 services are passive or on-demand)
[Server:server1] 10:13:32,513 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.0.Final "Thunder" started in 10204ms - Started 169 of 292 services (121 services are passive or on-demand)

When starting the domain a process controller is active is as well to start the host controller and the servers. The following shows the process controller log (that is located in the ${JBOSS_HOME}/domain/log directory; in this directory the log file of the host controller is present as well)

10:13:12,861 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
10:13:13,459 INFO  [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
10:13:22,199 INFO  [org.jboss.as.process.Server:server1.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server1'
10:13:22,292 INFO  [org.jboss.as.process.Server:server2.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server2'

In a domain set-up each server automatically gets its own directory, i.e.,

${JBOSS_HOME}/domain
	/configuration
		domain.xml
		host.xml
	/servers
		/server1
			/data
			/log
		/server2
			/data
			/log

Now the question arises: which use cases are appropriate for a domain and which are appropriate for standalone servers? A domain is about coordinated multi-server management, i.e., it offers a central point through which we can manage multiple servers, with the capability to keep the configuration consistent among the servers in a particular server group. The choice between using a domain or standalone server is about how server are managed. As we saw above we can set-up a group of standalone servers to form a cluster. One thing to note here, as we look at the directory structure provided out of the box by using a domain, is that the servers get there own directory structure, i.e., we get data directories that contain among others the HornetQ message bindings and journals and log directories. Recall that in the standalone case, we had to configure this ourselves. So from a management point of view a domain would be the obvious choice when multiple servers are involved.

Once the domain controller is configured we can configure the host controllers that should join the domain. A host controller configuration requires the following steps

  • Define a unique name for the host controller within the domain.
  • Configure the management interface and interface binding on an address that is accessible to the domain controller.
  • Tell the host controller how to find the domain controller so it can register itself with the domain.
  • Define the JVM and the servers to run on the host.

A basic example of a host controller configuration looks as follows:

<host name="hostname" xmlns="urn:jboss:domain:1.0">
	<management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <authentication>
                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
                </authentication>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket interface="management" port="9999"/>
            </native-interface>
        </management-interfaces>
    </management>
	<domain-controller>
       <remote host="hostname or IP-address of the domain controller host" port="9999"/>
    </domain-controller>
    <interfaces>
        <interface name="management">
            <nic name="eth0"/>
        </interface>
    </interfaces>
	<jvms>
    	<jvm name="default">
            <heap size="512m" max-size="512m"/>
            <permgen size="256m" max-size="256m"/>
            <jvm-options>
                <option value="-server"/>
                <option value="-XX:NewRatio=2"/>
                <option value="-XX:+UseParallelGC"/>
                <option value="-XX:ParallelGCThreads=2"/>
                <option value="-XX:MaxGCPauseMillis=200"/>
                <option value="-XX:GCTimeRatio=19"/>
                <option value="-XX:+UseParallelOldGC"/>
            </jvm-options>
        </jvm>
    </jvms>
    <servers>
        <server name="server3" group="ha-server-group" auto-start="true">
			<socket-bindings port-offset="2"/>
        </server>
        <server name="server4" group="ha-server-group" auto-start="true">
            <socket-bindings port-offset="3"/>
        </server>
		<server name="server5" group="ha-server-group" auto-start="true">
            <socket-bindings port-offset="4"/>
        </server>
    </servers>
</host>

Let us deploy the same application to our server-group (ha-server-group). To this end we need to configure a data source for the database. We also want to use the Apache HTTP server for the load balancing, which means we need to configure an AJP connector and socket binding. The domain.xml example presented above already contains the these configurations. From the start-up log we can see that AJP is bound to 192.168.1.66 and listening on ports 9080 for server1 and 9081 for server2. To deploy the application we can use the command-line interface. When doing this we first need to connect to domain controller which is listening on 192.168.1.66:9999 (see log output above)

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin/
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[domain@192.168.1.66:9999 /] deploy /home/jboss/jboss-as-7.1.0.Final/deploy/LoadTest6.ear --server-groups=ha-server-group

When the application gets deployed the following is observed in the logging

[Host Controller] 10:31:14,061 INFO  [org.jboss.as.repository] (management-handler-threads - 8) JBAS014900: Content added at location /home/jboss/jboss-as-7.1.0.Final/domain/data/content/16/1f51dde7f085c822cc4c68b306d57f1bee902d/content
[Server:server1] 10:31:14,259 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "LoadTest6.ear"
[Server:server2] 10:31:14,262 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "LoadTest6.ear"
[Server:server1] 10:31:14,315 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "Model.jar"
[Server:server2] 10:31:14,315 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "Web.war"
[Server:server2] 10:31:14,316 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "Model.jar"
[Server:server1] 10:31:14,316 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "Web.war"
[Server:server2] 10:31:14,421 INFO  [org.jboss.as.jpa] (MSC service thread 1-4) JBAS011401: Read persistence.xml for PersonPersistenceUnit
[Server:server1] 10:31:14,423 INFO  [org.jboss.as.jpa] (MSC service thread 1-4) JBAS011401: Read persistence.xml for PersonPersistenceUnit
[Server:server1] 10:31:14,528 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-3) JNDI bindings for session bean named Company in deployment unit subdeployment "Model.jar" of deployment "LoadTest6.ear" are as follows:
[Server:server1]
[Server:server1]        java:global/LoadTest6/Model/Company!model.logic.Company
[Server:server1]        java:app/Model/Company!model.logic.Company
[Server:server1]        java:module/Company!model.logic.Company
[Server:server1]        java:jboss/exported/LoadTest6/Model/Company!model.logic.Company
[Server:server1]        java:global/LoadTest6/Model/Company
[Server:server1]        java:app/Model/Company
[Server:server1]        java:module/Company
[Server:server1]
[Server:server2] 10:31:14,576 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-2) JNDI bindings for session bean named Company in deployment unit subdeployment "Model.jar" of deployment "LoadTest6.ear" are as follows:
[Server:server2]
[Server:server2]        java:global/LoadTest6/Model/Company!model.logic.Company
[Server:server2]        java:app/Model/Company!model.logic.Company
[Server:server2]        java:module/Company!model.logic.Company
[Server:server2]        java:jboss/exported/LoadTest6/Model/Company!model.logic.Company
[Server:server2]        java:global/LoadTest6/Model/Company
[Server:server2]        java:app/Model/Company
[Server:server2]        java:module/Company
[Server:server2]
[Server:server2] 10:31:15,167 INFO  [org.jboss.as.jpa] (MSC service thread 1-4) JBAS011402: Starting Persistence Unit Service 'LoadTest6.ear/Model.jar#PersonPersistenceUnit'
[Server:server1] 10:31:15,182 INFO  [org.jboss.as.jpa] (MSC service thread 1-3) JBAS011402: Starting Persistence Unit Service 'LoadTest6.ear/Model.jar#PersonPersistenceUnit'
[Server:server2] 10:31:15,377 INFO  [org.jboss.as.ejb3] (MSC service thread 1-1) JBAS014142: Started message driven bean 'CompanyMDB' with 'hornetq-ra' resource adapter
[Server:server1] 10:31:15,381 INFO  [org.jboss.as.ejb3] (MSC service thread 1-4) JBAS014142: Started message driven bean 'CompanyMDB' with 'hornetq-ra' resource adapter
[Server:server2] 10:31:15,558 INFO  [org.jboss.web] (MSC service thread 1-3) JBAS018210: Registering web context: /LoadTest6
[Server:server1] 10:31:15,563 INFO  [org.jboss.web] (MSC service thread 1-2) JBAS018210: Registering web context: /LoadTest6
[Server:server2] 10:31:15,613 INFO  [org.hibernate.annotations.common.Version] (MSC service thread 1-4) HCANN000001: Hibernate Commons Annotations {4.0.1.Final}
[Server:server1] 10:31:15,613 INFO  [org.hibernate.annotations.common.Version] (MSC service thread 1-3) HCANN000001: Hibernate Commons Annotations {4.0.1.Final}
[Server:server1] 10:31:15,622 INFO  [org.hibernate.Version] (MSC service thread 1-3) HHH000412: Hibernate Core {4.0.1.Final}
[Server:server2] 10:31:15,622 INFO  [org.hibernate.Version] (MSC service thread 1-4) HHH000412: Hibernate Core {4.0.1.Final}
[Server:server1] 10:31:15,624 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-3) HHH000206: hibernate.properties not found
[Server:server2] 10:31:15,626 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-4) HHH000206: hibernate.properties not found
[Server:server1] 10:31:15,626 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-3) HHH000021: Bytecode provider name : javassist
[Server:server2] 10:31:15,628 INFO  [org.hibernate.cfg.Environment] (MSC service thread 1-4) HHH000021: Bytecode provider name : javassist
[Server:server1] 10:31:15,680 INFO  [org.hibernate.ejb.Ejb3Configuration] (MSC service thread 1-3) HHH000204: Processing PersistenceUnitInfo [
[Server:server1]        name: PersonPersistenceUnit
[Server:server1]        ...]
[Server:server2] 10:31:15,682 INFO  [org.hibernate.ejb.Ejb3Configuration] (MSC service thread 1-4) HHH000204: Processing PersistenceUnitInfo [
[Server:server2]        name: PersonPersistenceUnit
[Server:server2]        ...]
[Server:server2] 10:31:15,879 INFO  [org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator] (MSC service thread 1-4) HHH000130: Instantiating explicit connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
[Server:server1] 10:31:15,882 INFO  [org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator] (MSC service thread 1-3) HHH000130: Instantiating explicit connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
[Server:server1] 10:31:16,034 INFO  [org.hibernate.dialect.Dialect] (MSC service thread 1-3) HHH000400: Using dialect: org.hibernate.dialect.Oracle10gDialect
[Server:server2] 10:31:16,034 INFO  [org.hibernate.dialect.Dialect] (MSC service thread 1-4) HHH000400: Using dialect: org.hibernate.dialect.Oracle10gDialect
[Server:server2] 10:31:16,052 INFO  [org.hibernate.engine.transaction.internal.TransactionFactoryInitiator] (MSC service thread 1-4) HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jta.CMTTransactionFactory
[Server:server1] 10:31:16,052 INFO  [org.hibernate.engine.transaction.internal.TransactionFactoryInitiator] (MSC service thread 1-3) HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jta.CMTTransactionFactory
[Server:server1] 10:31:16,061 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (MSC service thread 1-3) HHH000397: Using ASTQueryTranslatorFactory
[Server:server2] 10:31:16,062 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (MSC service thread 1-4) HHH000397: Using ASTQueryTranslatorFactory
[Server:server2] 10:31:16,144 INFO  [org.hibernate.validator.util.Version] (MSC service thread 1-4) Hibernate Validator 4.2.0.Final
[Server:server1] 10:31:16,145 INFO  [org.hibernate.validator.util.Version] (MSC service thread 1-3) Hibernate Validator 4.2.0.Final
[Server:server1] 10:31:17,007 INFO  [org.jboss.as.server] (host-controller-connection-threads - 3) JBAS018559: Deployed "LoadTest6.ear"
[Server:server2] 10:31:17,007 INFO  [org.jboss.as.server] (host-controller-connection-threads - 3) JBAS018559: Deployed "LoadTest6.ear"

As the servers are listening on the same IP addresses and AJP ports as in the standalone case, we do not need to edit anything in the Apache HTTP server configuration. Start the Apache HTTP server and enter the URL: http://hostname:8888/LoadTest6/testservlet and make a few requests to see if everything is working. Open the admin console (http://192.168.1.66:9990/console) and enter the necessary credentials to log in (these we have created in the beginning by using add-user.sh). In the admin console, click server instances and select a particular server. Subsequently, click web to see how many requests were processed by the AJP connector. Let us perform the load test again to see how the environment is performing. When using the jvisualvm we need to know the process IDs of the servers within the domain. To this end we can use the ps command to get information on the running Java process

[jboss@axis-into-ict ~]$ ps -ef|grep java
jboss    7277  7263  0 10:13 pts/2    00:00:06 /home/jboss/jdk1.6.0_31/bin/java -D[Process Controller] -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/log/process-controller.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -jar /home/jboss/jboss-as-7.1.0.Final/jboss-modules.jar -mp /home/jboss/jboss-as-7.1.0.Final/modules org.jboss.as.process-controller -jboss-home /home/jboss/jboss-as-7.1.0.Final -jvm /home/jboss/jdk1.6.0_31/bin/java -- -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/log/host-controller.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -- -default-jvm /home/jboss/jdk1.6.0_31/bin/java
jboss    7294  7277  0 10:13 pts/2    00:00:11 /home/jboss/jdk1.6.0_31/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/log/host-controller.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -server -Xms512m -Xmx512m -XX:NewRatio=2 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -jar /home/jboss/jboss-as-7.1.0.Final/jboss-modules.jar -mp /home/jboss/jboss-as-7.1.0.Final/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller --pc-address localhost.localdomain --pc-port 60207 -default-jvm /home/jboss/jdk1.6.0_31/bin/java -Djboss.home.dir=/home/jboss/jboss-as-7.1.0.Final
jboss    7347  7277  9 10:13 pts/2    00:06:43 /home/jboss/jdk1.6.0_31/bin/java -D[Server:server1] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms512m -Xmx512m -server -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djboss.host.default.config=host.xml -D[Host=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.home.dir=/home/jboss/jboss-as-7.1.0.Final -Djboss.domain.default.config=domain.xml -Djava.net.preferIPv4Stack=true -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server1/log/boot.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -jar jboss-modules.jar -mp modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server
jboss    7364  7277  9 10:13 pts/2    00:06:44 /home/jboss/jdk1.6.0_31/bin/java -D[Server:server2] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms512m -Xmx512m -server -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djboss.host.default.config=host.xml -D[Host=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.home.dir=/home/jboss/jboss-as-7.1.0.Final -Djboss.domain.default.config=domain.xml -Djava.net.preferIPv4Stack=true -Dorg.jboss.boot.log.file=/home/jboss/jboss-as-7.1.0.Final/domain/servers/server2/log/boot.log -Dlogging.configuration=file:/home/jboss/jboss-as-7.1.0.Final/domain/configuration/logging.properties -jar jboss-modules.jar -mp modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server

As we can see the first one (7277) is the process controller; the second (7294) is the host controller; the third (7347) and fourth (7364) are respectively server1 and server2. The load test statistics are as follows

To get request count and processing time information we use the command-line interface

[jboss@axis-into-ict ~]$ cd jboss-as-7.1.0.Final/bin
[jboss@axis-into-ict bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect 192.168.1.66:9999
[domain@192.168.1.66:9999 /] cd host=axis-into-ict.nl/server=server1
[domain@192.168.1.66:9999 server=server1] cd deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web
[domain@192.168.1.66:9999 subsystem=web] cd servlet=TestServlet
[domain@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
394063
[domain@192.168.1.66:9999 servlet=TestServlet] read-attribute processing-time
930637L
[domain@192.168.1.66:9999 servlet=TestServlet] cd /
[domain@192.168.1.66:9999 /] cd host=axis-into-ict.nl/server=server2
[domain@192.168.1.66:9999 server=server2] cd deployment=LoadTest6.ear/subdeployment=Web.war/subsystem=web
[domain@192.168.1.66:9999 subsystem=web] cd servlet=TestServlet
[domain@192.168.1.66:9999 servlet=TestServlet] read-attribute request-count
394056
[domain@192.168.1.66:9999 servlet=TestServlet] read-attribute processing-time
930302L

To see how the JVMs are doing during the load test we use jvisualvm

The test introduced no particular problems for the JVM.

References

[1] Marchioni, “JBoss AS7 Configuration, Deployment and Administration”, PACKT Publishing, Birmingham, UK, 2011. Essential reading material.
[2] JBoss Documentation.