In this post we show a step-by-step example of how to set-up a high available tuned Java EE environment. We start with tuning the operating system. Next, we install the Java Virtual Machine and JBoss EAP6. We continue with configuring a domain and tune the Java Virtual Machine. Next, we show the steps involved in setting up a cluster. Subsequently, we show how to create resources needed for Java messaging and database communication. Once the resources are configured, we deploy a Spring Hibernate application that uses these resources.

Operating System Tuning

We start by adding a user under which we will install the necessary software

[root@machine1 ~]# groupadd javainstall -g 500
[root@machine1 ~]# useradd -g javainstall jboss
[root@machine1 ~]# passwd jboss
Changing password for user jboss.
New password: jbosseap6
Retype new password: jbosseap6
passwd: all authentication tokens updated successfully.

Some operating system tweaks are worth considering when we do not want to run against system restrictions:

• Packet loss minimization – The operating system buffers must be large enough to handle incoming network traffic while the application is paused during garbage collection. Usually UDP (User Datagram Protocol) is used in order to transmit multicast messages to server instances in a cluster; to limit the need to retransmit UDP messages the size of the operating system buffers must be set appropriately to avoid excessive UDP datagram loss.
• TCP/IP – On some systems the default value for the time wait interval is too high and needs to be adjusted. When the number approaches the maximum number of file descriptors per process, the application’s throughput will degrade, i.e., new connections have to wait for a free space in the application’s file descriptor table.
• Swapping – Swapping, also known as paging, is the use of secondary storage to store and retrieve data for use in RAM. Swapping is automatically performed by the operating system and typically occurs when the available RAM memory is depleted. Swapping can have a significant impact on the performance and should thus be avoided.
• Network interface card (NIC) – Configure the network card at it’s maximum link speed and at full duplex.
• Maximum number of open file descriptors – Most operating systems handle sockets as a form of file access and use file descriptors to keep track of which sockets are open. To contain the resources per process, the operating system restricts the number of file descriptors per process. Linux limits the number of open file descriptors per process, by default this is equal to 1024. It could be that the 1024 limit does not offer optimal performance.
• Large pages – Large pages are essentially blocks of contiguous physical memory addresses that are reserved for a process. Large pages improve performance of applications that access memory frequently. When large pages are used the application uses the translation look-aside buffer (TLB) in the processor more effectively. The TLB is a cache of recently used virtual-to-physical address space translations stored in the processor memory. To obtain data from memory, the processor looks up the TLB to find out the physical addresses (RAM or hard disk) that hold the required data. In the case of large pages, a single entry in the TLB could represent a large contiguous address space and thereby potentially reducing the TLB look-up frequency and avoiding frequent look-ups in the hierarchical page table stored in-memory.

The contents of /etc/sysctl.conf look as follows

# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.
#net.bridge.bridge-nf-call-ip6tables = 0
#net.bridge.bridge-nf-call-iptables = 0
#net.bridge.bridge-nf-call-arptables = 0

# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536

# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

# increase TCP max buffer size (depending on the type of NIC and the round-trip time these values can be changed)
# Maximum TCP Receive Window
net.core.rmem_max = 8388608
net.core.rmem_default = 8388608
# Maximum TCP Send Window
net.core.wmem_max = 8388608
net.core.wmem_default = 8388608
#  memory reserved for TCP receive buffers (vector of 3 integers: [min, default, max])
net.ipv4.tcp_rmem = 4096 87380 8388608
# memory reserved for TCP send buffers (vector of 3 integers: [min, default, max])
net.ipv4.tcp_wmem = 4096 87380 8388608

# increase the length of the processor input queue
net.core.netdev_max_backlog = 30000
# maximum amount of memory buffers (could be set equal to net.core.rmem_max and net.core.wmem_max)
net.core.optmem_max = 20480
# socket of the listen backlog
net.core.somaxconn = 1024

# tcp selective acknowledgements (disable them on high-speed networks)
net.ipv4.tcp_sack = 1
net.ipv4.tcp_dsack = 1
# Timestamps add 12 bytes to the TCP header
net.ipv4.tcp_timestamps = 1
# Support for large TCP Windows - Needs to be set to 1 if the Max TCP Window is over 65535
net.ipv4.tcp_window_scaling = 1
# The interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe
net.ipv4.tcp_keepalive_time = 1800
# The interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime
net.ipv4.tcp_keepalive_intvl = 30
# The number of unacknowledged probes to send before considering the connection dead and notifying the application layer
net.ipv4.tcp_keepalive_probes = 5
# The time that must elapse before TCP/IP can release a closed connection and reuse its resources.
net.ipv4.tcp_fin_timeout = 30
# Size of the backlog connections queue.
net.ipv4.tcp_max_syn_backlog=4096
# The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers.
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1

# The percentage of how aggressively memory pages are swapped to disk
vm.swappiness = 0
# The percentage of main memory the pdflush daemon should write data out to the disk.
vm.dirty_background_ratio=25
# The percentage of main memory the actual disk writes will take place.
vm.dirty_ratio=20

# set the number of huge pages based on the Hugepagesize, i.e., 2048kB
# when we want to reserve 1GB in huge pages we have to set the number of huge pages to:
# (1024*1024*1024*1)/(1024*1024*2) = 1073741824/2097152 = 512
vm.nr_hugepages = 512

# give permission to the group that runs the process to access the shared memory segment
# to this end open the /etc/group file and retrieve the group-id (javainstall:x:500:)
vm.hugetlb_shm_group = 500

The contents of /etc/security/limits.conf look as follows

# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# open file descriptors
@javainstall soft nofile 8192
@javainstall hard nofile 8192

# memlock - maximum locked in-memory address space (kB), we set this equal to:
# number_of_huge_pages * huge_page_size = 512 * 2048 = 1048576
@javainstall soft memlock 1048576
@javainstall hard memlock 1048576
# End of file

Configure networking

To set-up a static ip-address we can follow these steps. Edit /etc/sysconfig/network, the following gives an example

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=machine1.com

To have a static ip address we edit /etc/sysconfig/network-scripts/ifcfg-eth0 as follows:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.1.155
NETMASK=255.255.255.0
TYPE=Ethernet
GATEWAY=192.168.1.1
IPV6INIT=no
USERCTL=no
DNS1=192.168.1.1

When one or more DNS servers are available, enter DN2, DNS3 etcetera. To see if the servers are resolved check the /etc/resolv.conf file

# Generated by NetworkManager
search com
nameserver 192.168.1.1

Note that the settings can be changed by using system-config-network. Finally, map the hostname to the IP-address. To this end, edit the /etc/hosts, for example,

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.155    machine1.com machine1

Restart the network service and check the configuration

[root@machine1 ~]# service network restart
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:  Active connection state: activated
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/2
                                                           [  OK  ]
[root@machine1 ~]# ifconfig
eth1      Link encap:Ethernet  HWaddr 00:0C:29:9D:E0:3B
          inet addr:192.168.1.155  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe9d:e03b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1665 errors:0 dropped:0 overruns:0 frame:0
          TX packets:963 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1826803 (1.7 MiB)  TX bytes:72314 (70.6 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:960 (960.0 b)  TX bytes:960 (960.0 b)

Verify the static ip configuration by using, for example,

[root@machine1 ~]# ping -c 3 -M do -s 1468 192.168.1.60
PING 192.168.1.60 (192.168.1.60) 1468(1496) bytes of data.
1476 bytes from 192.168.1.60: icmp_seq=1 ttl=128 time=0.412 ms
1476 bytes from 192.168.1.60: icmp_seq=2 ttl=128 time=0.426 ms
1476 bytes from 192.168.1.60: icmp_seq=3 ttl=128 time=0.558 ms

--- 192.168.1.60 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.412/0.465/0.558/0.068 ms

Install the Java Virtual Machine

Before we start with the installation we first define our directory structure

/home/jboss
	/deploy
		/springhibernate
			/app
			/config
			/modules
	/eap/jboss-eap-6.0 (${JBOSS_HOME})
		/bin (contains scripts to run the domain controller and command-line interface)
		/domain/configuration
			domain.xml (domain configuration file)
			host.xml (host configuration file)
			mgmt-users.properties (this file contains the admin user and host users)
		/modules
			/com/oracle/database/main
			/org/springframework/main
	/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64 (${JAVA_HOME})
	/scripts (contains script to start the environment)

We are going to use openjdk. To install openjdk, we can use yum install java-1.7.0-openjdk-devel. Unfortunately, this is installed in the /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.version.x86_64 directory. As we want to control the location (and update process) we are going to tar the mentioned directory, and extract it in our preferred location. When the openjdk distribution has been extracted to the preferred location, edit the bash profile (vi ~/.bash_profile) and set the JAVA_HOME and PATH variable

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

JBOSS_HOME="/home/jboss/eap/jboss-eap-6.0"
export JBOSS_HOME

APACHE_HOME="/home/jboss/apache"
export APACHE_HOME

JAVA_HOME="/home/jboss/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64"
export JAVA_HOME

PATH=$JAVA_HOME/bin:$JBOSS_HOME/bin:$PATH:$HOME/bin
export PATH

Log out and in again to make the changes effective.

Install JBoss EAP

To install JBoss EAP run the following command java -Xms512m -Xmx512m -jar jboss-eap-6.0.1-installer.jar. This opens the graphical installer

• Choose your language, for example, English and click ok
• Accept the licence agreement, and click next
• Select the installation path, for example, /home/jboss/eap and click next (click ok in the pop-up)
• Create an admin user, for example, admin with password jbosseap6! and click next
• We choose not to install quickstarts and click next
• Select all the packages (default) and click next
• Choose default as the socket binding set-up and click next
• Choose not to start the server and click next
• Click next on the installation summary
• Click next twice and choose not to create shortcuts and click next
• Generate an automatic install file and click done

The automatic install file has the following contents

<AutomatedInstallation langpack="eng">
	<com.izforge.izpack.panels.HTMLLicencePanel id="HTMLLicencePanel"/>
	<com.izforge.izpack.panels.TargetPanel id="DirectoryPanel">
		<installpath>/home/jboss/eap</installpath>
	</com.izforge.izpack.panels.TargetPanel>
	<com.izforge.izpack.panels.UserInputPanel id="Userinputpanel.0">
		<userInput>
			<entry key="adminPassword" value="4f0d0c202a4aadd1b8c25276f18be512"/>
			<entry key="adminUser" value="admin"/>
		</userInput>
	</com.izforge.izpack.panels.UserInputPanel>
	<com.izforge.izpack.panels.UserInputPanel id="UserInputPanel.1">
		<userInput>
			<entry key="installQuickStarts" value="false"/>
		</userInput>
	</com.izforge.izpack.panels.UserInputPanel>
	<com.izforge.izpack.panels.JDKCheckPanel id="JDKCheckPanel"/>
	<com.izforge.izpack.panels.MavenRepoCheckPanel id="MavenRepoCheckPanel"/>
	<com.izforge.izpack.panels.TreePacksPanel id="TreePacksPanel">
		<pack index="0" name="eap-core" selected="true"/>
		<pack index="1" name="JBossEAP AppClient" selected="true"/>
		<pack index="2" name="JBossEAP Bin" selected="true"/>
		<pack index="3" name="JBossEAP Bundles" selected="true"/>
		<pack index="4" name="JBossEAP Docs" selected="true"/>
		<pack index="5" name="JBossEAP Domain" selected="true"/>
		<pack index="6" name="JBossAS Domain Shell Scripts" selected="true"/>
		<pack index="7" name="JBossEAP Modules" selected="true"/>
		<pack index="8" name="JBossEAP Standalone" selected="true"/>
		<pack index="9" name="JBossAS Standalone Shell Scripts" selected="true"/>
		<pack index="10" name="JBossEAP Welcome Content" selected="true"/>
		<pack index="11" name="JBossEAP Quickstarts" selected="false"/>
		<pack index="12" name="icons" selected="true"/>
		<pack index="13" name="Native Zips" selected="true"/>
		<pack index="14" name="Native RHEL6-x86_64" selected="true"/>
		<pack index="15" name="Native Utils RHEL6-x86_64" selected="true"/>
		<pack index="16" name="Native Webserver RHEL6-x86_64" selected="true"/>
	</com.izforge.izpack.panels.TreePacksPanel>
	<com.izforge.izpack.panels.UserInputPanel id="UserInputPanel.2">
		<userInput>
			<entry key="portDecision" value="off"/>
		</userInput>
	</com.izforge.izpack.panels.UserInputPanel>
	<com.izforge.izpack.panels.UserInputPanel id="UserInputPanel.3"/>
	<com.izforge.izpack.panels.UserInputPanel id="UserInputPanel.4"/>
	<com.izforge.izpack.panels.UserInputPanel id="UserInputPanel.5"/>
	<com.izforge.izpack.panels.UserInputPanel id="UserInputPanel.6"/>
	<com.izforge.izpack.panels.UserInputPanel id="UserInputPanel.7"/>
	<com.izforge.izpack.panels.UserInputPanel id="UserInputPanel.8">
		<userInput>
			<entry key="serverStartup" value="none"/>
		</userInput>
	</com.izforge.izpack.panels.UserInputPanel>
	<com.izforge.izpack.panels.SummaryPanel id="SummaryPanel"/>
	<com.izforge.izpack.panels.InstallPanel id="InstallPanel"/>
	<com.izforge.izpack.panels.ProcessPanel id="ProcessPanel"/>
	<com.izforge.izpack.panels.ShortcutPanel id="ShortcutPanel"/>
	<com.izforge.izpack.panels.FinishPanel id="FinishPanel"/>
</AutomatedInstallation>

To install JBoss on other machines we can use

[jboss@machine2 temp]$ java -Xms512m -Xmx512m -jar jboss-eap-6.0.1-installer.jar installscript.xml
[ Starting automated installation ]
Read pack list from xml definition.
Try to add to selection [Name: eap-core and Index: 0]
Try to add to selection [Name: JBossEAP AppClient and Index: 1]
Try to add to selection [Name: JBossEAP Bin and Index: 2]
Try to add to selection [Name: JBossEAP Bundles and Index: 3]
Try to add to selection [Name: JBossEAP Docs and Index: 4]
Try to add to selection [Name: JBossEAP Domain and Index: 5]
Try to add to selection [Name: JBossAS Domain Shell Scripts and Index: 6]
Try to add to selection [Name: JBossEAP Modules and Index: 7]
Try to add to selection [Name: JBossEAP Standalone and Index: 8]
Try to add to selection [Name: JBossAS Standalone Shell Scripts and Index: 9]
Try to add to selection [Name: JBossEAP Welcome Content and Index: 10]
Try to remove from selection [Name: JBossEAP Quickstarts and Index: 11]
Try to add to selection [Name: icons and Index: 12]
Try to add to selection [Name: Native Zips and Index: 13]
Try to add to selection [Name: Native RHEL6-x86_64 and Index: 14]
Try to add to selection [Name: Native Utils RHEL6-x86_64 and Index: 15]
Try to add to selection [Name: Native Webserver RHEL6-x86_64 and Index: 16]
Modify pack selection.
Pack [Name: JBossEAP Quickstarts and Index: 11] must be installed because it is required!
[ Starting to unpack ]
[ Processing package: eap-core (1/17) ]
[ Processing package: JBossEAP AppClient (2/17) ]
[ Processing package: JBossEAP Bin (3/17) ]
[ Processing package: JBossEAP Bundles (4/17) ]
[ Processing package: JBossEAP Docs (5/17) ]
[ Processing package: JBossEAP Domain (6/17) ]
[ Processing package: JBossAS Domain Shell Scripts (7/17) ]
[ Processing package: JBossEAP Modules (8/17) ]
[ Processing package: JBossEAP Standalone (9/17) ]
[ Processing package: JBossAS Standalone Shell Scripts (10/17) ]
[ Processing package: JBossEAP Welcome Content (11/17) ]
[ Processing package: icons (13/17) ]
[ Processing package: Native Zips (14/17) ]
[ Processing package: Native RHEL6-x86_64 (15/17) ]
[ Processing package: Native Utils RHEL6-x86_64 (16/17) ]
[ Processing package: Native Webserver RHEL6-x86_64 (17/17) ]
[ Unpacking finished ]
[ Starting processing ]
Starting process Running Password Script (1/2)
Starting process Unpacking natives (2/2)
[ Processing finished ]
[ Creating shortcuts  done. ]
[ Add shortcuts to uninstaller  done. ]
[ Writing the uninstaller data ... ]
[ Automated installation done ]

Test the installation by running domain.sh (located in ${JBOSS_HOME}/bin/).

JBoss Configuration

To configure the domain we can use the following

<domain xmlns="urn:jboss:domain:1.3">

    <extensions>
        <extension module="org.jboss.as.clustering.infinispan"/>
        <extension module="org.jboss.as.clustering.jgroups"/>
        <extension module="org.jboss.as.cmp"/>
        <extension module="org.jboss.as.configadmin"/>
        <extension module="org.jboss.as.connector"/>
        <extension module="org.jboss.as.ee"/>
        <extension module="org.jboss.as.ejb3"/>
        <extension module="org.jboss.as.jacorb"/>
        <extension module="org.jboss.as.jaxr"/>
        <extension module="org.jboss.as.jaxrs"/>
        <extension module="org.jboss.as.jdr"/>
        <extension module="org.jboss.as.jmx"/>
        <extension module="org.jboss.as.jpa"/>
        <extension module="org.jboss.as.jsr77"/>
        <extension module="org.jboss.as.logging"/>
        <extension module="org.jboss.as.mail"/>
        <extension module="org.jboss.as.messaging"/>
        <extension module="org.jboss.as.modcluster"/>
        <extension module="org.jboss.as.naming"/>
        <extension module="org.jboss.as.osgi"/>
        <extension module="org.jboss.as.pojo"/>
        <extension module="org.jboss.as.remoting"/>
        <extension module="org.jboss.as.sar"/>
        <extension module="org.jboss.as.security"/>
        <extension module="org.jboss.as.threads"/>
        <extension module="org.jboss.as.transactions"/>
        <extension module="org.jboss.as.web"/>
        <extension module="org.jboss.as.webservices"/>
        <extension module="org.jboss.as.weld"/>
    </extensions>

    <system-properties>
        <property name="java.net.preferIPv4Stack" value="true"/>
    </system-properties>

    <profiles>
        <profile name="cluster">
            <subsystem xmlns="urn:jboss:domain:logging:1.1">...</subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:datasources:1.1">
                <datasources>
                    <datasource jta="true" jndi-name="java:/jdbc/OracleDS" pool-name="OracleDS" enabled="true" use-java-context="true" use-ccm="true">
                        <connection-url>jdbc:oracle:thin:@192.168.1.60:1521:orcl11</connection-url>
                        <driver>oracle</driver>
                        <pool>
                            <min-pool-size>1</min-pool-size>
                            <max-pool-size>15</max-pool-size>
                            <prefill>true</prefill>
                            <use-strict-min>true</use-strict-min>
                        </pool>
                        <security>
                            <user-name>example</user-name>
                            <password>example</password>
                        </security>
                        <timeout>
                            <idle-timeout-minutes>0</idle-timeout-minutes>
                            <query-timeout>600</query-timeout>
                        </timeout>
                        <statement>
                            <prepared-statement-cache-size>10</prepared-statement-cache-size>
                        </statement>
                    </datasource>
                    <drivers>
                        <driver name="oracle" module="com.oracle.database">
                            <driver-class>oracle.jdbc.OracleDriver</driver-class>
                            <xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
                        </driver>
                    </drivers>
                </datasources>
            </subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:messaging:1.2">
                <hornetq-server>
                    <persistence-enabled>true</persistence-enabled>
                    <security-enabled>false</security-enabled>
                    <create-journal-dir>true</create-journal-dir>
                    <journal-type>ASYNCIO</journal-type>
                    <journal-buffer-timeout>500000</journal-buffer-timeout>
                    <journal-buffer-size>1048576</journal-buffer-size>
                    <journal-sync-transactional>true</journal-sync-transactional>
                    <journal-sync-non-transactional>true</journal-sync-non-transactional>
                    <journal-file-size>10485760</journal-file-size>
                    <journal-min-files>2</journal-min-files>
                    <journal-compact-min-files>10</journal-compact-min-files>
                    <journal-compact-percentage>30</journal-compact-percentage>
                    <journal-max-io>1024</journal-max-io>

                    <connectors>
                        <netty-connector name="netty" socket-binding="messaging"/>
                        <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                            <param key="batch-delay" value="50"/>
                        </netty-connector>
                        <in-vm-connector name="in-vm" server-id="0"/>
                    </connectors>

                    <acceptors>
                        <netty-acceptor name="netty" socket-binding="messaging"/>
                        <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                            <param key="batch-delay" value="50"/>
                            <param key="direct-deliver" value="false"/>
                        </netty-acceptor>
                        <in-vm-acceptor name="in-vm" server-id="0"/>
                    </acceptors>

                    <broadcast-groups>
                        <broadcast-group name="bg-group1">
                            <socket-binding>messaging-group</socket-binding>
                            <broadcast-period>5000</broadcast-period>
                            <connector-ref>
                                netty
                            </connector-ref>
                        </broadcast-group>
                    </broadcast-groups>

                    <discovery-groups>
                        <discovery-group name="dg-group1">
                            <socket-binding>messaging-group</socket-binding>
                            <refresh-timeout>10000</refresh-timeout>
                        </discovery-group>
                    </discovery-groups>

                    <cluster-connections>
                        <cluster-connection name="my-cluster">
                            <address>jms</address>
                            <connector-ref>netty</connector-ref>
                            <discovery-group-ref discovery-group-name="dg-group1"/>
                        </cluster-connection>
                    </cluster-connections>

                    <security-settings>
                        <security-setting match="#">
                            <permission type="send" roles="guest"/>
                            <permission type="consume" roles="guest"/>
                            <permission type="createNonDurableQueue" roles="guest"/>
                            <permission type="deleteNonDurableQueue" roles="guest"/>
                        </security-setting>
                    </security-settings>

                    <address-settings>
                        <address-setting match="#">
                            <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                            <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                            <redelivery-delay>0</redelivery-delay>
                            <max-size-bytes>10485760</max-size-bytes>
                            <address-full-policy>BLOCK</address-full-policy>
                            <message-counter-history-day-limit>10</message-counter-history-day-limit>
                            <redistribution-delay>1000</redistribution-delay>
                        </address-setting>
                        <address-setting match="jms.queue.testQueue">
                            <dead-letter-address>jms.queue.deadLetterQueue</dead-letter-address>
                            <expiry-address>jms.queue.expiryQueue</expiry-address>
                            <redelivery-delay>5000</redelivery-delay>
                            <max-delivery-attempts>3</max-delivery-attempts>
                            <max-size-bytes>104857600</max-size-bytes>
                            <address-full-policy>BLOCK</address-full-policy>
                            <last-value-queue>true</last-value-queue>
                            <redistribution-delay>0</redistribution-delay>
                            <send-to-dla-on-no-route>true</send-to-dla-on-no-route>
                        </address-setting>
                    </address-settings>

                    <jms-connection-factories>
                        <connection-factory name="InVmConnectionFactory">
                            <connectors>
                                <connector-ref connector-name="in-vm"/>
                            </connectors>
                            <entries>
                                <entry name="java:/ConnectionFactory"/>
                            </entries>
                        </connection-factory>
                        <connection-factory name="RemoteConnectionFactory">
                            <connectors>
                                <connector-ref connector-name="netty"/>
                            </connectors>
                            <entries>
                                <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                            </entries>
                        </connection-factory>
                        <pooled-connection-factory name="hornetq-ra">
                            <transaction mode="xa"/>
                            <connectors>
                                <connector-ref connector-name="in-vm"/>
                            </connectors>
                            <entries>
                                <entry name="java:/JmsXA"/>
                            </entries>
                        </pooled-connection-factory>
                    </jms-connection-factories>

                    <jms-destinations>
                        <jms-queue name="testQueue">
                            <entry name="java:/queue/test"/>
                            <entry name="java:jboss/exported/jms/queue/test"/>
                        </jms-queue>
                        <jms-queue name="deadLetterQueue">
                            <entry name="java:/queue/deadLetterQueue"/>
                            <entry name="java:jboss/exported/jms/queue/deadLetterQueue"/>
                            <durable>false</durable>
                        </jms-queue>
                        <jms-queue name="expiryQueue">
                            <entry name="java:/queue/expiryQueue"/>
                            <entry name="java:jboss/exported/jms/queue/expiryQueue"/>
                            <durable>false</durable>
                        </jms-queue>
                        <jms-topic name="testTopic">
                            <entry name="java:/topic/test"/>
                            <entry name="java:jboss/exported/jms/topic/test"/>
                        </jms-topic>
                    </jms-destinations>
                </hornetq-server>
            </subsystem>
			...
            <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
        </profile>
    </profiles>

    <interfaces>
        <interface name="management"/>
        <interface name="public"/>
        <interface name="unsecure"/>
    </interfaces>

    <socket-binding-groups>
        <socket-binding-group name="cluster-sockets" default-interface="public">
            <socket-binding name="ajp" port="8009"/>
            <socket-binding name="http" port="8080"/>
            <socket-binding name="https" port="8443"/>
            <socket-binding name="jacorb" interface="unsecure" port="3528"/>
            <socket-binding name="jacorb-ssl" interface="unsecure" port="3529"/>
            <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
            <socket-binding name="jgroups-tcp" port="7600"/>
            <socket-binding name="jgroups-tcp-fd" port="57600"/>
            <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
            <socket-binding name="jgroups-udp-fd" port="54200"/>
            <socket-binding name="messaging" port="5445"/>
            <socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9876}"/>
            <socket-binding name="messaging-throughput" port="5455"/>
            <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
            <socket-binding name="osgi-http" interface="management" port="8090"/>
            <socket-binding name="remoting" port="4447"/>
            <socket-binding name="txn-recovery-environment" port="4712"/>
            <socket-binding name="txn-status-manager" port="4713"/>
            <outbound-socket-binding name="mail-smtp">
                <remote-destination host="localhost" port="25"/>
            </outbound-socket-binding>
        </socket-binding-group>
    </socket-binding-groups>

    <server-groups>
        <server-group name="cluster-group" profile="cluster">
            <jvm name="default"/>
            <socket-binding-group ref="cluster-sockets"/>
        </server-group>
    </server-groups>

</domain>

In the example above, we have configured one profile (cluster), and a socket-binding-group. Finally, we have defined a server-group, that uses the profile and socket-binding-group. Note that the configuration above is default except for the messaging and the data-source. When using the Linux firewall (disabled in our case), we must configure the iptables appropriately. The servers that belong to the server-group, will be defined in the host configuration below.

Host Configuration

First, we create two management users called machine1 and machine2. To this end run the add-user.sh script located in the ${JBOSS_HOME}/bin directory. Next, we create host.xml files for each host, i.e., on machine1 we have

<host name="machine1" xmlns="urn:jboss:domain:1.3">

    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
				<!-- Needs to be added to hosts that are not the domain controller -->
				<!--server-identities>
					<secret value="amJvc3NlYXA2IQ==" />
				</server-identities-->
                <authentication>
                    <local default-user="$local" />
                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
                </authentication>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <authentication>
                    <local default-user="$local" allowed-users="*" />
                    <properties path="application-users.properties" relative-to="jboss.domain.config.dir" />
                </authentication>
                <authorization>
                    <properties path="application-roles.properties" relative-to="jboss.domain.config.dir"/>
                </authorization>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket interface="management" port="${jboss.management.native.port:9999}"/>
            </native-interface>
            <http-interface security-realm="ManagementRealm">
                <socket interface="management" port="${jboss.management.http.port:9990}"/>
            </http-interface>
        </management-interfaces>
    </management>

    <domain-controller>
       <local/>
    </domain-controller>

    <interfaces>
        <interface name="management">
            <nic name="eth1"/>
        </interface>
        <interface name="public">
            <nic name="eth1"/>
        </interface>
        <interface name="unsecure">
            <inet-address value="127.0.0.1"/>
        </interface>
    </interfaces>

    <jvms>
    	<jvm name="default">
            <heap size="512m" max-size="512m"/>
            <permgen size="256m" max-size="256m"/>
            <jvm-options>
                <option value="-server"/>
                <option value="-XX:NewRatio=2"/>
				<option value="-XX:SurvivorRatio=128"/>
				<option value="-XX:MaxTenuringThreshold=0"/>
                <option value="-XX:+UseParallelGC"/>
                <option value="-XX:ParallelGCThreads=2"/>
                <option value="-XX:MaxGCPauseMillis=200"/>
                <option value="-XX:GCTimeRatio=19"/>
                <option value="-XX:+UseParallelOldGC"/>
                <option value="-XX:+UseTLAB"/>
                <option value="-XX:LargePageSizeInBytes=2048k"/>
                <option value="-XX:+UseLargePages"/>
            </jvm-options>
        </jvm>
    </jvms>

    <servers>
        <server name="cluster-server1" group="cluster-group" auto-start="false"/>
        <server name="cluster-server2" group="cluster-group" auto-start="false">
            <socket-bindings port-offset="1"/>
        </server>
    </servers>
</host>

Note the local for the domain controller, this means that machine1 will act as our domain controller. We have used the following JVM parameters

• -server – select the JIT compiler.
• -Xms – initial heap size.
• -Xmx – maximum heap size.
• -XX:PermSize and -XX:MaxPermSize – size of the permanent generation.
• -XX:NewRatio=N – sets the young generation to heap size / (1 + N).
• -XX:SurvivorRatio – ratio of eden/survivor space size.
• -XX:MaxTenuringThreshold – sets the maximum tenuring threshold (above we assume that objects that are not collected in the eden space are objects that will live for a long time).
• -XX:+UseParallelGC – select the parallel collector.
• -XX:ParallelGCThreads – sets the number of garbage collector threads, i.e., the number of CPUs to be used.
• -XX:MaxGCPauseMillis – sets the maximum pause time goal.
• -XX:GCTimeRatio=N – sets the throughput goal that is measured in terms of the time spent doing garbage collection versus the time spent outside of garbage collection. In the case above with N=19, the garbage collection time to application time is set to 1 / (1+N) = 1 / 20, i.e., 5% of the total time is spent in garbage collection.
• -XX:-UseParallelOldGC – selects the parallel collector for major collections.
• -XX:+UseTLAB – enables thread-local object allocation. More information on thread local allocation can be found in the ‘Compaction and Thread Local Area’ section of the Tune the JVM that runs Coherence post.
• -XX:LargePageSizeInBytes – sets the large page size used for the Java heap. We set this equal to the operating system parameter: Hugepagesize, which in our case is 2048kB
• -XX:+UseLargePages – use large page memory. The steps involved on how to configure large pages in the operating system can be found in the ‘Call profiling and large pages’ section of the Tune the JVM that runs Coherence post.

On the machine2 host we use the following host.xml file

<host name="machine2" xmlns="urn:jboss:domain:1.3">

    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <server-identities>
					<secret value="amJvc3NlYXA2IQ=="/>
                </server-identities>
                <authentication>
                    <local default-user="$local" />
                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
                </authentication>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <authentication>
                    <local default-user="$local" allowed-users="*" />
                    <properties path="application-users.properties" relative-to="jboss.domain.config.dir" />
                </authentication>
                <authorization>
                    <properties path="application-roles.properties" relative-to="jboss.domain.config.dir"/>
                </authorization>
            </security-realm>
        </security-realms>
        <management-interfaces>
            <native-interface security-realm="ManagementRealm">
                <socket interface="management" port="${jboss.management.native.port:9999}"/>
            </native-interface>
            <http-interface security-realm="ManagementRealm">
                <socket interface="management" port="${jboss.management.http.port:9990}"/>
            </http-interface>
        </management-interfaces>
    </management>

    <domain-controller>
       <remote host="machine1.com" port="9999" security-realm="ManagementRealm"/>
    </domain-controller>

    <interfaces>
        <interface name="management">
            <nic name="eth2"/>
        </interface>
        <interface name="public">
            <nic name="eth2"/>
        </interface>
        <interface name="unsecure">
            <inet-address value="127.0.0.1"/>
        </interface>
    </interfaces>

    <jvms>
    	<jvm name="default">
            <heap size="512m" max-size="512m"/>
            <permgen size="256m" max-size="256m"/>
            <jvm-options>
                <option value="-server"/>
                <option value="-XX:NewRatio=2"/>
				<option value="-XX:SurvivorRatio=128"/>
				<option value="-XX:MaxTenuringThreshold=0"/>
                <option value="-XX:+UseParallelGC"/>
                <option value="-XX:ParallelGCThreads=2"/>
                <option value="-XX:MaxGCPauseMillis=200"/>
                <option value="-XX:GCTimeRatio=19"/>
                <option value="-XX:+UseParallelOldGC"/>
                <option value="-XX:+UseTLAB"/>
                <option value="-XX:LargePageSizeInBytes=2048k"/>
                <option value="-XX:+UseLargePages"/>
            </jvm-options>
        </jvm>
    </jvms>

    <servers>
        <server name="cluster-server3" group="cluster-group" auto-start="false"/>
        <server name="cluster-server4" group="cluster-group" auto-start="false">
            <socket-bindings port-offset="1"/>
        </server>
    </servers>
</host>

The choice of the name of the host is very important, as a corresponding user name must be present in the mgmt-users.properties file on the host where the domain controller runs. Note that we tell this host where to find the domain controller, by using the remote element.

To test the set-up, we first start the domain controller on machine1

[jboss@machine1 bin]$ ./domain.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/eap/jboss-eap-6.0

  JAVA: /home/jboss/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/bin/java

  JAVA_OPTS: -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml

=========================================================================

10:03:24,645 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.3.GA-redhat-1
10:03:24,784 INFO  [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
[Host Controller] 10:03:25,189 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.3.GA-redhat-1
[Host Controller] 10:03:25,364 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA-redhat-2
[Host Controller] 10:03:25,455 INFO  [org.jboss.as] (MSC service thread 1-2) JBAS015899: JBoss EAP 6.0.1.GA (AS 7.1.3.Final-redhat-4) starting
[Host Controller] 10:03:25,979 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.7.GA-redhat-1
[Host Controller] 10:03:25,984 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.7.GA-redhat-1
[Host Controller] 10:03:25,990 INFO  [org.jboss.as] (Controller Boot Thread) JBAS010902: Creating http management service using network interface (management) port (9990) securePort (-1)
[Host Controller] 10:03:26,012 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 3.2.14.GA-redhat-1
[Host Controller] 10:03:26,084 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on 192.168.1.155:9999
[Host Controller] 10:03:27,326 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://192.168.1.155:9990/management
[Host Controller] 10:03:27,328 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://192.168.1.155:9990
[Host Controller] 10:03:27,331 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.0.1.GA (AS 7.1.3.Final-redhat-4) (Host Controller) started in 2457ms - Started 11 of 11 services (0 services are passive or on-demand)
[Host Controller] 10:04:07,114 INFO  [org.jboss.as.domain] (slave-request-threads - 1) JBAS010918: Registered remote slave host "machine2", JBoss EAP 6.0.1.GA (AS 7.1.3.Final-redhat-4)

Next, we start the domain controller on machine2 (note the last log line above that the host has been registered)

[jboss@machine2 bin]$ ./domain.sh
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/jboss/eap/jboss-eap-6.0

  JAVA: /home/jboss/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/bin/java

  JAVA_OPTS: -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml

=========================================================================

10:03:55,491 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.3.GA-redhat-1
10:03:56,433 INFO  [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
[Host Controller] 10:03:58,033 INFO  [org.jboss.modules] (main) JBoss Modules version 1.1.3.GA-redhat-1
[Host Controller] 10:03:58,431 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA-redhat-2
[Host Controller] 10:03:58,874 INFO  [org.jboss.as] (MSC service thread 1-1) JBAS015899: JBoss EAP 6.0.1.GA (AS 7.1.3.Final-redhat-4) starting
[Host Controller] 10:04:00,638 INFO  [org.xnio] (MSC service thread 1-3) XNIO Version 3.0.7.GA-redhat-1
[Host Controller] 10:04:00,655 INFO  [org.jboss.as] (Controller Boot Thread) JBAS010902: Creating http management service using network interface (management) port (9990) securePort (-1)
[Host Controller] 10:04:00,656 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.0.7.GA-redhat-1
[Host Controller] 10:04:00,727 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 3.2.14.GA-redhat-1
[Host Controller] 10:04:00,957 INFO  [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 192.168.1.160:9999
[Host Controller] 10:04:07,563 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://192.168.1.160:9990/management
[Host Controller] 10:04:07,565 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015954: Admin console is not enabled
[Host Controller] 10:04:07,567 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.0.1.GA (AS 7.1.3.Final-redhat-4) (Host Controller) started in 10950ms - Started 11 of 11 services (0 services are passive or on-demand)

Deployment

The application to be deployed, uses Spring and Hibernate (details can be found in the Spring and Hibernate4 post). To deploy the application we are going to use a custom module, i.e.,

${JBOSS_HOME}/modules
	/org/springframework
		/main
			aopalliance-1.0.jar
			cglib-2.2.2.jar
			commons-logging-1.0.4.jar
			module.xml
			org.springframework.aop-3.1.2.RELEASE.jar
			org.springframework.asm-3.1.2.RELEASE.jar
			org.springframework.beans-3.1.2.RELEASE.jar
			org.springframework.context-3.1.2.RELEASE.jar
			org.springframework.core-3.1.2.RELEASE.jar
			org.springframework.expression-3.1.2.RELEASE.jar
			org.springframework.jdbc-3.1.2.RELEASE.jar
			org.springframework.jms-3.1.2.RELEASE.jar
			org.springframework.orm-3.1.2.RELEASE.jar
			org.springframework.transaction-3.1.2.RELEASE.jar
			org.springframework.web-3.1.2.RELEASE.jar

where module.xml has the following contents

<module xmlns="urn:jboss:module:1.1" name="org.springframework">
    <resources>
		<resource-root path="aopalliance-1.0.jar"/>
		<resource-root path="cglib-2.2.2.jar"/>
		<resource-root path="commons-logging-1.0.4.jar"/>
		<resource-root path="org.springframework.aop-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.asm-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.beans-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.context-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.core-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.expression-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.jdbc-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.jms-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.orm-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.transaction-3.1.2.RELEASE.jar"/>
		<resource-root path="org.springframework.web-3.1.2.RELEASE.jar"/>
    </resources>
    <dependencies>
        <module name="javax.api" export="true"/>
        <module name="org.apache.log4j" export="true"/>
        <module name="org.antlr" export="true"/>
        <module name="org.dom4j" export="true"/>
        <module name="org.hibernate" export="true"/>
        <module name="javax.persistence.api" export="true"/>
		<module name="javax.servlet.api" export="true"/>
        <module name="org.javassist" export="true"/>
        <module name="org.jboss.logging" export="true"/>
        <module name="javax.transaction.api" export="true"/>
    </dependencies>
</module>

Note that we also have included a Hibernate dependency (as this is needed by Spring when using Hibernate with it). Next, we are going to start the servers by using the command-line interface

[jboss@machine1 ~]$ cd eap/jboss-eap-6.0/bin/
[jboss@machine1 bin]$ ./jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect machine1.com:9999
[domain@machine1.com:9999 /] /host=machine1/server-config=cluster-server1:start
{
    "outcome" => "success",
    "result" => "STARTING"
}
[domain@machine1.com:9999 /] /host=machine1/server-config=cluster-server2:start
{
    "outcome" => "success",
    "result" => "STARTING"
}
[domain@machine1.com:9999 /] /host=machine2/server-config=cluster-server3:start
{
    "outcome" => "success",
    "result" => "STARTING"
}
[domain@machine1.com:9999 /] /host=machine2/server-config=cluster-server4:start
{
    "outcome" => "success",
    "result" => "STARTING"
}
[domain@machine1.com:9999 /] deploy /home/jboss/deploy/springhibernate/SpringHibernate.war --server-groups=ccluster-group
[domain@machine1.com:9999 /]

Load balancing

We are going to use mod_cluster for the load balancing. First, we need to install the Apache HTTP server

• Unpack the httpd-2.2.21.tar.gz file:
  • gzip -d httpd-2.2.21.tar.gz
  • tar xvf httpd-2.2.21.tar
  • cd httpd-2.2.21
• The next step is to configure:
  • ./configure –with-mpm=worker –with-included-apr –prefix=/home/jboss/apache
• Next, compile the various parts for the Apache HTTP Server by using:
  • make
• To install the Apache HTTP Server we use:
  • make install
• Open the httpd.conf (/home/jboss/apache/conf) file and adjust the following directives:
  • Listen 8888
  • ServerName machine1.com
• To test the set-up, start the Apache HTTP Server:
  • Navigate to /home/jboss/apache/bin.
  • Run: ./apachectl -k start (To stop the Apache HTTP Server we can use: ./apachectl -k stop).
  • Open a browser en type the following URL: http://hostname:8888.

A mod_cluster distribution can be downloaded here. We will use mod_cluster 1.2.0.Final. Note that the binaries (for example, binaries linux2-x64) contain a full Apache distribution, while the dynamic libraries (for example, dynamic libraries linux2-x64) contain only the modules for mod_cluster (excluding mod_proxy.so, mod_proxy_ajp.so and mod_proxy_http.so). As we already have an Apache installed we will use the dynamic libraries distribution. To configure mod_cluster we first copy the libraries into ${APACHE_HOME}/modules, i.e.,

${APACHE_HOME}
	/conf
		httpd.conf
		jboss-workers.properties
		mod_jk.conf
		mod_cluster.conf
	/modules
		mod_advertise.so
		mod_jk.so
		mod_manager.so
		mod_proxy.so
		mod_proxy_ajp.so
		mod_proxy_http.so
		mod_proxy_cluster.so
		mod_slotmem.so

To configure mod_cluster, we create a mod_cluster configuration file, for example mod_cluster.conf. The configuration has the following contents

LoadModule proxy_module "/home/jboss/apache/modules/mod_proxy.so"
LoadModule proxy_ajp_module "/home/jboss/apache/modules/mod_proxy_ajp.so"
LoadModule proxy_http_module "/home/jboss/apache/modules/mod_proxy_http.so"
LoadModule slotmem_module "/home/jboss/apache/modules/mod_slotmem.so"
LoadModule manager_module "/home/jboss/apache/modules/mod_manager.so"
LoadModule proxy_cluster_module "/home/jboss/apache/modules/mod_proxy_cluster.so"
LoadModule advertise_module "/home/jboss/apache/modules/mod_advertise.so"

NameVirtualHost machine1.com:8888

<VirtualHost machine1.com:8888>
	EnableMCPMReceive
	ManagerBalancerName mycluster

	AllowDisplay On

	KeepAliveTimeout 60
	MaxKeepAliveRequests 0

	ServerAdvertise On
	AdvertiseGroup 224.0.1.105:23364
	AdvertiseFrequency 10

	<Directory />
		Order deny,allow
		Deny from all
	       Allow from 192.168.1.
	</Directory>

	<Location /mod_cluster-manager>
		SetHandler mod_cluster-manager
		Order deny,allow
   		Deny from all
   		Allow from 192.168.1.
	</Location>
</VirtualHost>

The used directives are explained here. Next, add the following Include directive to httpd.conf

# put it near the end of the file where all the other includes are present
# mod_cluster configuration
Include conf/mod_cluster.conf

The protocol used by mod_cluster to communicate with the servers in based on the connectors configured in domain.xml. When an AJP connector is configured it will be used, otherwise an HTTP connector will be used. While AJP is generally faster, an HTTP connector can optionally be secured via SSL.

<subsystem xmlns="urn:jboss:domain:web:1.2" default-virtual-server="default-host" native="false">
	<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
	<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>
	<virtual-server name="default-host" enable-welcome-root="true">
		<alias name="localhost"/>
		<alias name="example.com"/>
	</virtual-server>
</subsystem>

Restart the Apache HTTP Server. By accessing the URL: http://hostname:8888/mod_cluster-manager, we get an overview of the configuration.

Start and stop scripts

First, we are going to set-up ssh, such that we can execute commands from machine1 on machine2. Make sure sshd is running on machine2. To this end execute the following commands on machine2

[jboss@machine2 ~]$ su - root
Password:
[root@machine2 ~]# yum install openssh-server
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
 * base: mirror.proserve.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
Setting up Install Process
Package openssh-server-5.3p1-84.1.el6.x86_64 already installed and latest version
Nothing to do
[root@machine2 ~]# yum install openssh-clients
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
 * base: mirror.proserve.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
Setting up Install Process
Package openssh-clients-5.3p1-84.1.el6.x86_64 already installed and latest version
Nothing to do
[root@machine2 ~]# chkconfig sshd on
[root@machine2 ~]# service sshd start

When we want to execute commands from machine1 on machine2 without entering a password, we have to follow the steps below (these are executed on machine1)

[jboss@machine1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jboss/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/jboss/.ssh/id_rsa.
Your public key has been saved in /home/jboss/.ssh/id_rsa.pub.
The key fingerprint is:
03:58:ca:81:ad:0f:c6:45:2c:aa:ba:c1:d5:90:36:8b jboss@machine1.com
The key's randomart image is:
+--[ RSA 2048]----+
|   =o .          |
|  o.==           |
| o O+ .          |
|. B =  .         |
|.E = .  S        |
|o . .    .       |
|o.               |
|..               |
|..               |
+-----------------+
[jboss@machine1 ~]$ ssh jboss@machine2.com mkdir -p .ssh
jboss@machine2.com's password:
[jboss@machine1 ~]$ cat .ssh/id_rsa.pub | ssh jboss@machine2.com 'cat >> .ssh/authorized_keys'
jboss@machine2.com's password:
[jboss@machine1 ~]$ ssh jboss@machine2.com hostname
machine2.com

To start the environment we can use

#!/bin/bash

SCRIPT=$(readlink -f $0)
SCRIPT_PATH=$(dirname $SCRIPT)

. ${SCRIPT_PATH}/jboss-functions

start_all_domain_controllers

start_all_servers

start_apache

in which jboss-functions has the following contents

# -*-Shell-script-*-
#
# jboss-functions	This file contains functions to be used by shell scripts in the JBOSS_HOME/scripts directory
#

start_apache() {
	echo "starting Apache HTTP Server"
	${APACHE_HOME}/bin/apachectl -k start
}

stop_apache() {
	echo "stopping Apache HTTP Server"
	${APACHE_HOME}/bin/apachectl -k stop
}

# Starts the domain controller only on the machine where the script is run
start_domain_controller() {
	echo "starting process controller and host controller"
	${JBOSS_HOME}/bin/domain.sh >/dev/null 2>/dev/null &
}

# Stops the domain controller and servers only on the machine where the script is run
stop_domain_controller_and_local_servers() {
	echo "stopping process controller and host controler"
	${JBOSS_HOME}/bin/jboss-cli.sh --user=machine1 --password=jbosseap6! --file=${JBOSS_HOME}/scripts/stop-domain-controller-and-local-servers.cli
}

# This function starts domain controllers on multiple machines by using ssh
start_all_domain_controllers() {
	echo "starting process controller and host controller"
	${JBOSS_HOME}/bin/domain.sh >/dev/null 2>/dev/null &
	echo "waiting a little for the host controller to finish starting"
	sleep 15
	echo "starting process controller and host controller on other machines"
	ssh jboss@machine2.com ${JBOSS_HOME}/bin/domain.sh >/dev/null 2>/dev/null &
	echo "waiting a little for the host controller to finish starting"
	sleep 15
}

# When the start_all_domain_controllers has finished, this function can be called to start all the server
# Note that only the servers defined in start-all-server.cli are started
start_all_servers() {
	echo "starting servers"
	${JBOSS_HOME}/bin/jboss-cli.sh --file=${JBOSS_HOME}/scripts/start-all-servers.cli
}

# Based on what is defined in the stop-all-domain-controllers-and-servers.cli script, domain controllers and servers are stopped
stop_all_domain_controllers_and_servers() {
	echo "stopping JBoss processes"
	${JBOSS_HOME}/bin/jboss-cli.sh --file=${JBOSS_HOME}/scripts/stop-all-domain-controllers-and-servers.cli
}

The start-servers.cli script has the following contents

connect machine1.com:9999
/host=machine1/server-config=cluster-server1:start
/host=machine1/server-config=cluster-server2:start
/host=machine2/server-config=cluster-server3:start
/host=machine2/server-config=cluster-server4:start

When the script is executed the following is observed

[jboss@machine1 scripts]$ ./start-environment.sh
starting process controller and host controller
waiting a little for the host controller to finish starting
starting process controller and host controller on other machines
waiting a little for the host controller to finish starting
starting servers
{
    "outcome" => "success",
    "result" => "STARTING"
}
{
    "outcome" => "success",
    "result" => "STARTING"
}
{
    "outcome" => "success",
    "result" => "STARTING"
}
{
    "outcome" => "success",
    "result" => "STARTING"
}
starting Apache HTTP Server

To stop the environment, we can use

#!/bin/bash

SCRIPT=$(readlink -f $0)
SCRIPT_PATH=$(dirname $SCRIPT)

. ${SCRIPT_PATH}/jboss-functions

stop_apache

stop_all_domain_controllers_and_servers

in which stop-servers.cli has the following contents

connect machine1.com:9999
/host=machine1/server-config=cluster-server1:stop
/host=machine1/server-config=cluster-server2:stop
/host=machine2/server-config=cluster-server3:stop
/host=machine2/server-config=cluster-server4:stop
/host=machine2:shutdown
/host=machine1:shutdown

When the script is run the following is observed

[jboss@machine1 scripts]$ ./stop-environment.sh
stopping Apache HTTP Server
stopping JBoss processes
{
    "outcome" => "success",
    "result" => "STOPPING"
}
{
    "outcome" => "success",
    "result" => "STOPPING"
}
{
    "outcome" => "success",
    "result" => "STOPPING"
}
{
    "outcome" => "success",
    "result" => "STOPPING"
}
{
    "outcome" => "success",
    "result" => undefined
}
{"outcome" => "success"}

In general, it is recommended to start the domain controller when the machine boots. In this case, we need to know where to put our custom commands that will be called when the system boots. Note that Linux has a /etc/rc.d/rc.local file, that can be used to put custom commands in. Let us do it in the recommended script-based way. Note that Unix-based systems specify so-called run levels, and that for each run level, scripts can be defined that start a certain service. These scripts are located in the /etc/rc.d/init.d directory. This allows for services to be started when the system boots or to be stopped on system shutdown. The different run levels are specified by a specific directory in the /etc/rc.d directory, i.e.,

• rc0.d – contains scripts that are executed on system shutdown.
• rc1.d – contains scripts for single-user mode.
• rc2.d – contains scripts for multi-user mode.
• rc3.d – contains scripts for multi-user mode and networking.
• rc4.d – not used.
• rc5.d – same as rc3.d plus some graphical stuff.
• rc6.d – contains scripts that are executed on system reboot.

The boot sequence is as follows: in the /etc/inittab file the starting runlevel is defined, the script /etc/rc.d/rc.sysinit is called and /etc/rc.d/rc is run. The rc script looks in the /etc/rc.d/rc<start-runlevel>.d to execute the K**<script-name> scripts with the stop option. After this the S**<script-name> scripts are executed with the start option. Note that scripts are started in numerical order, i.e., the S10network script is executed before the S80sendmail script.

To create a domain controller ‘service’, we can use the following script

#!/bin/sh
#
# chkconfig: 235 91 35
# description: starts and stops the jboss domain controller
#
#
. /etc/rc.d/init.d/functions

RETVAL=0
SERVICE="jboss-domain-controller"

start() {
	echo "Starting JBoss Domain Controller"
	su - jboss -c "/home/jboss/scripts/start-domain-controller.sh" >/dev/null 2>&1
	RETVAL=$?
	[ $RETVAL -eq 0 ] && success || failure
	echo
	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/${SERVICE}
	return $RETVAL
} 

stop() {
	echo "Stopping JBoss Domain Controller and Local JBoss Server Instances"
	su - jboss -c "/home/jboss/scripts/stop-domain-controller-and-local-servers.sh" >/dev/null 2>&1
	RETVAL=$?
	[ $RETVAL -eq 0 ] && success || failure
	echo
	[ $RETVAL -eq 0 ] && rm -r /var/lock/subsys/${SERVICE}
	return $RETVAL
} 

restart() {
	stop
	start
} 

case "$1" in
	start)
		start
		;;
	stop)
		stop
		;;
	restart)
		restart
		;;
	*)
		echo $"Usage: $0 {start|stop|restart}"
		exit 1
esac 

exit $?

Place this script in the /etc/rc.d/init.d directory. By using the chkconfig command we can update the runlevel information for system services, for example, chkconfig --add jbossdomaincontroller. To test the set-up shut the system down and start it again. To check if the domain controller is running we can use ps -ef|grep java.

Next, we can start the configured servers, for example, by using

connect machine1.com:9999
/host=machine1/server-config=cluster-server1:start
/host=machine1/server-config=cluster-server2:start
/host=machine2/server-config=cluster-server3:start
/host=machine2/server-config=cluster-server4:start

To shutdown the environment we create host specific cli file, i.e., on machine1 we have

connect machine1.com:9999
/host=machine1/server-config=cluster-server1:stop
/host=machine1/server-config=cluster-server2:stop
/host=machine1:shutdown

and on machine2 we have

connect machine1.com:9999
/host=machine2/server-config=cluster-server3:stop
/host=machine2/server-config=cluster-server4:stop
/host=machine2:shutdown

The last set-up with the domain controllers starting when the servers boot, is the preferred way (as we can now bring down a specific host, start JBoss servers instances in a more controlled manner and we do not need ssh).

References

[1] Install Guide.
[2] Administration and Configuration Guide.
[3] JBoss Enterprise Application Platform Documentation.