In this post, we give an example on how to optimize the performance of the GlassFish Server. This post is a continuation of the Fun with GlassFish post.

Some notes up front

Some concepts that can affect performance are:

• load – the number of concurrent sessions, that are measured as, for example, the number of transactions and requests per a certain time span. We have to determine the performance capacity of a particular configuration and specify the resources that are required to sustain a specified load. A first step would be to determine the load a single processor can handle, by setting up a test environment that uses the application and load.
• scalability – transaction rate measured on a CPU
  • vertical scalability – percentage gain per additional CPU. By running a series of tests in which CPUs are added gives insight in the vertical scalability characteristics of the system. In this case the back-end systems (such as database resources) must be properly tuned as well such that they do not skew the results.
  • horizontal scalability – percentage gain per additional server (also provides fail-over capabilities).
• safety margins
  • high availability requirements – if the system must cope with failures, such as losing a server, the environment must be sized such that the load can be handled. Usually this means that servers must not run at their peak, there must still be a safety margin, when one of the servers fails.

When talking about tuning a first start would be the application itself. The book – Effective Java written by Joshua Bloch, gives the reader an excellent insight in how to make the most effective use of the Java programming language and its fundamental libraries. It consists of seventy-eight items, each of which conveys one rule. The rules capture practices generally held to be beneficial by the best and most experienced programmers. The items are loosely grouped into ten chapters, each concerning one broad aspect of software design.

When designing servlets some suggestions are:

• To minimize the use of synchronization and not to use the single thread model. In the default servlet multi-thread model, a single instance is created for each GlassFish Server instance, i.e., all requests share the same servlet instance. Note that this can lead to thread contention when synchronized blocks are used. The reason to use synchronized would be if we had some class variables that are shared and modified, so it is in general bad practice to use such variables in servlets.
• Initialize expensive resources by using the init method.
• HTTP Sessions
  • One thing to note it that session creation is not free. This, in general, means that if we do not need a session, we should not create one. Also when a created session is no longer needed we should invalidate the session. The reason for this is that session objects consume memory, i.e., they increase the number of live objects and as such lead to longer garbage collection (the computational complexity of a garbage collection is a function of both the amount of live data on the heap (mark) and the heap size (sweep)). See the post Fast, Faster, JRockit for more information on garbage collection algorithms.
  • When we do need to create HTTP session objects, we have to keep these objects small not only for the garbage collector, but also to keep response times small. With small we mean below 7 kilobytes.
  • Avoid large object graphs in an HttpSession. When a change is made in the object graph and setAttribute is called, the whole graph is serialized and replicated to a secondary server when we are using a cluster. This replication adds to the response time.

When using enterprise beans some suggestions are:

• Cache as many beans as possible. Stateful session bean instances are created as they are needed to service client requests. Between requests these instances reside in a bean-specific cache in the active state, ready for the next request.
• Stateless session bean instances are maintained in a pool. This pool improves performance, because a client request can be handled immediately by any free initialized EJB instance. The same rule applies as for the caching: pool as many beans as possible.

Beans in pools and caches consume memory, which can lead to a performance degradation for the garbage collector, so we have to find an optimum. In this case we have to weigh the number of requests against cache and pool demands. Tuning involves charting the behavior of the cache and pool over a period of time. For example, if too many passivations are happening in the case of a stateful bean and there is room on the JVM heap, we can increase the max-cache-size and the cache-idle-timeout-in-seconds by using the glassfish-ejb-jar.xml deployment override.

To improve performance of enterprise beans we have to use high-performance beans as much as possible. This means that we have to resort to stateless sessions beans or message driven beans and forget about the others (stateful session beans and entity beans). Another thing to note is that enterprise beans are not simple Java objects, i.e., they are components with semantics for remote call interfaces, security and transactions. So we have to keep in mind not to decompose an application into too many enterprise beans. Enterprise beans can either have remote or local interfaces. Calls to remote interfaces require marshalling arguments, transportation of the marshalled data across a network, unmarshall the arguments and dispatch at the receiving end, i.e., remote interfaces introduce some significant overhead. Local interfaces are more efficient as it does not require the argument marshalling etcetera. In general, pass-by-value semantics are used to call remote interfaces. When the clients are local to the enterprise bean we can use call-by-reference semantics. This can be accomplished by using pass-by-reference in the glassfish-ejb-jar.xml file.

When enterprise beans use transactions it is in general preferred to use container-managed transactions for better performance and consistency. When methods do not need any transactional requirements it is beneficial to declare those methods with NotSupported or Never transaction attributes. When multiple methods are called in a single transaction it is beneficial to use Required as the transactional attribute, this makes sure the same transaction will be used. When JDBC resources are used in a transaction that uses more than one resource (also JMS for example). It is beneficial to use last agent optimization, which allows one of the resources in a distributed transaction as a one phase commit. The overhead is much higher for JDBC resources when used in a distributed transaction than for a message queue. To take advantage of last agent optimization we configure the JDBC resource as a one phase commit resource and leave the JMS resources as they are. It is also advisable to only use XA type data source when needed, i.e., when two or more data sources are going to participate in a transaction.

Tuning the server

We can improve the performance of a server by adjusting a few deployment and server configuration settings. When using the Oracle GlassFish Server a performance tuner is provided to help to reach an optimal configuration. The performance tuner performs a static analysis of server resources and throughput requirements.

Deployment

Our first step is to disable auto-deployment. Note that when auto-deployment is enabled the reload poll interval can have an impact on performance. To disable auto-deployment, open the admin console and navigate to the domain node and click the applications configuration tab (disable the auto deploy option). Another option to disable is dynamic reloading. In this case the server periodically checks for changes in deployed applications and automatically reloads the application with the changes. To disable dynamic reloading of classes, open the admin console and navigate to the domain node and click the applications configuration tab (disable the reload option).

Logging

In general, writing to log files slows down performance and increases disk access. To change the log level for the server in the admin console, navigate to the configurations, configuration-name, logger settings page. For production systems a log level of warning suffices in most cases.

Web container

The web container settings can be adjusted (in the admin console, configurations, configuration-name, web container). The sesstion timeout determines how long a session is maintained by the server. The default is 1800 seconds. Setting a large value for the session timeout degrades performance by causing the server to maintain too many sessions in the session store.

EJB container

The EJB container caches and pools enterprise beans for better performance (in the admin console, configurations, configuration-name, EJB container). Tuning the cache and pool properties can provide performance enhancements. The difference between a pooled bean and a cached bean is that pooled beans are equivalent. Cached beans, on the other hand, contain state that is specific for its user.

By default, the pool grows on demand. This can be controlled with the steady-pool-size and max-pool-size deployment descriptor parameters. The stateless session bean pool can also shrink in size. A bean instance will be removed if it has been idle for more than the value of pool-idle-timeout-in-seconds which is 600 seconds by default. The default value of 0 for steady-pool-size is fine for most stateless session bean deployments. We may wish to set steady-pool-size higher if the stateless session bean is expensive to initialize so that we force the initialization of a number of bean instances at deployment time, and we ensure that pool shrinking does not discard these beans. The default value of 32 for max-pool-size could be somewhat low under heavy load. The number of beans to be created or deleted is controlled by the resize-quantity attribute. By default, the resize-quantity is set to 8 and needs to be adjusted when the maximum pool size changes.

Stateful session bean instances are created as they are needed to service client requests. Between requests these instances reside in a bean-specific cache in the active state, ready for the next request. The size of the cache is limited by the max-cache-size element in the glassfish-ejb-jar.xml deployment descriptor file. The default value is 512. So long as the application never requires more than max-cache-size instances of the stateful session bean at any given time to service all concurrent clients, there is no contention for the cache and performance is optimal. If we limit the number of beans in the cache, GlassFish may be forced to manage the cache in a fairly active manner using the following rules:

• If the cache is full, bean instances that are not being used at that moment for client requests are subject to passivation. Setting the cache-idle-timeout-in-seconds parameter has no effect on this rule because the server must make room for additional instances.
• If bean-managed transaction demarcation is used, a transaction may not be committed or rolled back at the end of a business method call. This leaves the bean instance associated with the transaction, pinned in the cache, and not eligible for passivation. Applications that keep transactions open between stateful session bean calls do not scale well and are difficult to manage.
• If the cache is full and all instances are currently pinned in the cache fulfilling client requests an exception is thrown, the server will not block and wait for an instance to become available for passivation. If container-managed transaction demarcation is used, this condition cannot occur if the max-cache-size setting is higher than the maximum number of request threads configured for the thread pool and the processing of each client request uses a single stateful session bean.
• Passivation logic is controlled by the victim-selection-policy and cache-idle-timeout-in-seconds elements in the descriptor. The default setting for victim-selection-policy, not recently used (NRU), passivates beans only when the number of active beans approaches the max-cache-size setting. An alternative victim-selection-policy value, least recently used (LRU), passivates based on both the maximum cache size and when the bean has not been used for longer than the value of the cache-idle-timeout-in-seconds setting. The NRU strategy is lazy; the LRU strategy is eager. Although the LRU setting can be a convenient way of enforcing idle timeouts on the resources the objects encapsulate, it requires the container to keep track of the bean’s access time and maintain an ordered list that gets updated after each bean access. Unless we have a good reason to need idle timeouts strictly enforced, most applications should retain the default NRU algorithm.
• The removal-timeout-in-seconds is the timeout value for passivated instances. Passivated instances that have been unused for longer than this timeout are subject to removal from the backup store. The default value is 5400 seconds.

Passivation of beans means serialization of non-transient data in the bean to a backup storage to release the memory used by the bean. The next request for the passivated bean will require activation, the reverse process, where bean elements are read from the backup store and the active bean instance is recreated in memory. Needless to say, passivation and activation cycles can be extremely expensive. We should monitor the amount of passivation activity occurring in our system and tune the max-cache-size setting to reduce or eliminate this activity to achieve high performance.

The application should always call a @Remove method to delete the active bean instance from the cache when a client is through using the instance. Failure to call a @Remove method leaves the bean instance in the active state and consumes one slot in the cache, requiring eventual passivation during cache management to make room for additional client beans.

The cache-idle-timeout-in-seconds setting is very important in cache management. The bean is subject to passivation once the timeout expires, assuming the LRU algorithm is being used, and may be removed from storage completely after the removal-timeout-in-seconds passes. The default timeout value, 600 seconds, may be too short if users are likely to pause between requests for a longer period of time. When using stateful session beans with a web application, it might make sense to set this timeout value equal to the HttpSession timeout value for the web application, for example, to be more consistent. Otherwise, review the business requirements and set the cache-idle-timeout-in-seconds to the lowest value possible that still meets the application’s requirements.

Message-driven beans provide a bridge between JMS and EJBs by listening on JMS destinations and invoking EJBs. Message-driven beans are pooled in a manner very similar to stateless session beans. The initial and maximum number of message-driven bean instances can be controlled by using the steady-pool-size and max-pool-size parameters in glassfish-ejb-jar.xml. Limiting the number of instances provides a simple mechanism to throttle the processing of incoming JMS messages. We may wish to do this to match the availability of resources used by a message-driven bean, for example, the number of JDBC connections in a connection pool.

Network listener

For machines that use only one network interface card, it proves beneficial to set the network address equal to the IP address of the machine, instead of using the default 0.0.0.0. If an IP address is specified the server will make one less system call per connection. If the machine has multiple network interface cards, we need to create a network listener for each network interface card.

The max connections (default 256) on the HTTP tab controls the number of requests that a client can make over a keep-alive connection. This value should be adjusted based on the number of requests a typical client makes. The number of connections specified is divided equally among the keep alive threads. When domain service name (DNS) lookup is enabled the server performs DNS lookups whenever a client accesses the server. If the server responds to many requests the load can be reduced by disabling the DNS lookup. The timeout option specifies the maximum time in seconds that a server holds an HTTP keep alive connection open. A client can keep a connection open so that multiple requests can be services by a single network connection. One thing to note is that the number of open connections is limited for a particular server, i.e., a high number of open connection prevents new clients from connecting. The default value for timeout is 30 seconds.

Thread pool

Two parameters are of interest max thread pool size and min thread pool size.

The max thread pool size parameter specifies the maximum number of simultaneous requests the server can handle. The default value is 5. When the server has reached the limit, it defers processing new requests until the number of active requests drops below the maximum amount. In practice, clients connect to the server and do not complete their requests. In these cases, the server waits the time specified by the timeout parameter. The thread count value must be adjusted to meet the load and the length of time for an average request. Suitable request max thread pool sizes range from 100 to 500, depending on the load. If the system has some extra CPU cycles to spare, we can increase the thread count until the performance saturates.

The min thread pool size parameter specifies the minimum number of thread the server initiates upon startup. The default value is 2. The min thread pool size parameter specifies a hard limit for the maximum number of threads that can run simultaneously. By specifying the same values for the minimum and maximum threads allows the server to use an optimized thread pool. This configuration should be considered unless the load on the server varies significantly.

Resources

When using database-intensive applications the JDBC connection pools must be tuned. Connection pools contain a number of live database connections that are reused in order to reduce the overhead of creating and destroying database connections. To obtain connection pool statistics we have to enable it first (configurations, configuration-name, monitoring). The following attributes are monitored:

• The number of connections that failed validation (numConnFailedValidation).
• The number of connections that have been used (numConnUsed).
• The number of connection free in the pool (numConnFree). When this value is constantly zero the number of connections in the pool must be increased as requests are waiting in order to fullfill their requirements, such as, for example obtained data from the database.
• The number of connections in the pool that have timed out (numConnTimedOut).

Parameters that can be tuned are:

• Pool size
  • Initial and minimum pool size – the size of the pool when it is created and the minimum number of connections in the pool.
  • Maximum pool size – the maximum number of connections in the pool.
  • Pool resize quantity – number of connections to be removed when the idle timeout expires.
• Timeout
  • Max wait time – the amount of time the code requesting a connection will wait before getting a connection timeout. The default is 60 seconds. A value of zero forces an in-definitive wait. To improve performance set max wait time to zero. This will block the thread until a connection becomes available. It also allows the server to track elapsed wait time such that performance can be increased.
  • Idle timeout – the time in seconds a connection can be idle in the pool. After this time the pool can close the connection. In general, the idle timeout should be kept shorter than the timeout configured configured on the database server, such that unusable connections are not accumulated in the pool.

Tuning the JVM

An in-depth discussion on the JVM can be found in the post Fast, Faster, JRockit. Steps and consideration involved are discussed in the posts: Tune the JVM that runs Coherence when using JRockit and Tune the JVM that runs Coherence Revisited when using HotSpot. When using GlassFish (or any other application server for that matter) it is beneficial to tune for application throughput, i.e., when using JRockit we could use something like:

-jrockit -Xms1024m -Xmx1024m -Xgc:throughput -XX:+UseCallProfiling -XX:+UseLargePagesForHeap

By choosing throughput as the optimization strategy the following defaults are present:

• The nursery size (-Xns) is automatically sized to 50% of free heap.
• The compaction is configured as -XXcompaction:abortable=false,percentage=6.25,heapParts=4096,maxReferences=299900.
• The thread local area size is configured as -XXtlasize:min=2k,preferred=16k,wastelimit=2k. Note that the preferred size depends on the heap size and lies between 16k and 64k.

Additional tuning may be necessary when compaction causes long garbage collection pauses. To find out the impact compaction has on the garbage collection pausetime, we can run a flight recording and examine the compaction pause parts of old garbage collections. In general, compaction pausetime depends on the compaction ratio (percentage or externalPercentage and internalPercentage) and the maximum number of references. In multi-threaded applications where threads allocate lots of objects, it might be beneficial to increase the TLA size. Caution must be taken, however, to not make the TLA size too large as this increases the fragmentation and as a result more garbage collections need to be run in order to allocate new objects.

When using HotSpot we could use something like:

-server -Xms1024m -Xmx1024m -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=19 -XX:+UseParallelOldGC -XX:+UseTLAB -XX:LargePageSizeInBytes=2048k -XX:+UseLargePages

In both configurations we have enabled large pages. Note that large pages must be configured in the operating system. Large pages are essentially blocks of contiguous physical memory addresses that are reserved for a process. Large pages improve performance of applications that access memory frequently. When large pages are used the application uses the translation look-aside buffer (TLB) in the processor more effectively. The TLB is a cache of recently used virtual-to-physical address space translations stored in the processor memory. To obtain data from memory, the processor looks up the TLB to find out the physical addresses (RAM or hard disk) that hold the required data. In the case of large pages, a single entry in the TLB could represent a large contiguous address space and thereby potentially reducing the TLB look-up frequency and avoiding frequent look-ups in the hierarchical page table stored in-memory.

The following command can be used to dynamically allocate/deallocate hugepages: echo 1024 > /proc/sys/vm/nr_hugepages. This command will try to configure 1024 hugepages in the system. The success or failure of allocation depends on the amount of physically contiguous memory that is present in the system at this time. At times, the JVM might not be able to reserve large pages because the pages might not be contiguous. To ensure that the JVM can use large pages, these must be enabled in the operating system as soon as possible after the system starts; otherwise the memory can become too fragmented which leads to a reduction in the number of large pages. In order to accomplish this, we can put the command in the /etc/rc.d/rc.local script, for example,

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

# set the sharable memory size - in this case we set the memory size to 4GB:
# 1024*1024*1024*4 = 4294967296
echo 4294967296 > /proc/sys/kernel/shmmax

# set the number of huge pages based on the Hugepagesize, i.e., 2048kB
# when we want to reserve 2GB in huge pages we have to set the number of huge pages to:
# (1024*1024*1024*2)/(1024*1024*2) = 2147483648/2097152 = 1024
echo 1024 > /proc/sys/vm/nr_hugepages

# give permission to the group that runs the process to access the shared memory segment
# to this end open the /etc/group file and retrieve the group-id (oracle:x:500:)
echo 500 > /proc/sys/vm/hugetlb_shm_group

Next, we set the locked in-memory address space for the oracle user in the file /etc/security/limits.conf, i.e.,

# memlock - maximum locked in-memory address space (kB), we set this equal to:
# number_of_huge_pages * huge_page_size = 1024 * 2048 = 2097152
oracle  soft    memlock 2097152
oracle  hard    memlock 2097152

When using JRockit we have to make a hugetblfs file system available in the directory /mnt/hugepages. This can be accomplished by using mount -t hugetlbfs nodev /mnt/hugepages/ (note that the /mnt/hugepages directory must exist). To make the /mnt/hugepages directory accessible for the oracle user, we can use chmod -R 777 /mnt/hugepages/.

Tuning the operating system

When we run our application we do not want to run against system restrictions. Some points are worth considering.

Packet loss minimization

The operating system buffers must be large enough to handle incoming network traffic while the application is paused during garbage collection. Also GlassFish uses UDP (User Datagram Protocol) in order to transmit multicast messages to server instances in a cluster; to limit the need to retransmit UDP messages the size of the operating system buffers must be set appropriately to avoid excessive UDP datagram loss. To configure Linux to allow for larger buffers we can follow these steps:

• Edit the /etc/sysctl.conf and add the following lines:

  		# increase TCP max buffer size
  		net.core.rmem_max = 2096304
  		net.core.wmem_max = 2096304
  		net.core.rmem_default=2096304
  		net.core.wmem_default=2096304
  		

• Run /sbin/sysctl -p to make the changes effective. By executing cat /proc/sys/net/core/rmem_max and cat /proc/sys/net/core/wmem_max we can check the current values.

Maximum number of open file descriptors

Most operating systems handle sockets as a form of file access and use file descriptors to keep track of which sockets are open. To contain the resources per process, the operating system restricts the number of file descriptors per process. Linux limits the number of open file descriptors per process, by default this is equal to 1024. It could be that the 1024 limit does not offer optimal performance. To obtain the current open file limit we can use the ulimit command. The ulimit -aS command shows the current limit, the ulimit -aH command shows the hard limit. To increase the total number of file descriptors for all users we can add the following to the /etc/sysctl.conf file: fs.file-max = 344030. To check the current value we can use cat /proc/sys/fs/file-max. If we want to make sure that a process (running under the oracle user) gets a maximum of 8192 open file descriptors, we have to edit the /etc/security/limits.conf file:

oracle soft nofile 8192
oracle hard nofile 8192

Note that this number applies across all processes running under the oracle user. To make the adjustments active, execute the ulimit -u 8192 command and log out and log in again.

TCP/IP

On some systems the default value for the time wait interval is too high and needs to be adjusted. To determine the number of sockets in TIME_WAIT we can use netstat -a | grep TIME_WAIT | wc -l. When the number approaches the maximum number of file descriptors per process, the application’s throughput will degrade, i.e., new connections have to wait for a free space in the application’s file descriptor table. To check the current value we can use cat /proc/sys/net/ipv4/tcp_keepalive_time and cat /proc/sys/net/ipv4/tcp_keepalive_intvl. The default time wait interval on newer Linux systems is 60 seconds and as such there is no need to change this. It the need arises that TCP/IP settings must be adjusted follow these steps:

• Edit the /etc/sysctl.conf and add the following lines:

  		net.ipv4.tcp_rmem = 4096    87380   4194304
  		net.ipv4.tcp_wmem = 4096    16384   4194304
  		net.ipv4.tcp_sack = 1
  		net.ipv4.tcp_timestamps = 1
  		net.ipv4.tcp_window_scaling = 1
  		net.ipv4.tcp_keepalive_time = 7200
  		net.ipv4.tcp_keepalive_intvl = 75
  		net.ipv4.tcp_fin_timeout = 60
  		

• Run /sbin/sysctl -p to make the changes effective. By executing, for example, cat /proc/sys/net/ipv4/tcp_keepalive_time we can check the current values.

Timesources

Linux has several timesources to choose from; the fastest being TSC (time stamp counter) and is used by default. However, if during start-up inconsistencies are found Linux switches to a slower timesource. This can have a negative performance impact. To check if TSC is used we have to check the system logs /var/log/dmesg. Look for an entry along the lines of:

time.c: Using tsc for timekeeping HZ 1000
time.c: Using 3.579545 MHz WALL PM GTOD TSC Timekeeping timer.
time.c: Detected 2799.204 MHz processor.

Network interface card (NIC)

Configure the network card at it’s maximum link speed and at full duplex. To determine the configuration of the network card we can use:

[root@edu-wls-rh oracle]# /sbin/ethtool eth0
Settings for eth0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
							100baseT/Half 100baseT/Full
							1000baseT/Full
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full
							100baseT/Half 100baseT/Full
							1000baseT/Full
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Supports Wake-on: d
	Wake-on: d
	Current message level: 0x00000007 (7)
	Link detected: yes

To change the maximum speed and duplex style we can use, for example, /sbin/ethtool -s eth0 speed 1000 duplex full.

Test the set-up

We are going to use the same set-up as was presented in the post Fun with GlassFish, in particular in the ‘Clustering’ section. First, we create a deployment plan for the following EAR:

LoadTest.ear
	META-INF
		application.xml
	PersonModel.jar
		META-INF
			ejb-jar.xml
			persistence.xml
	PersonUI.war
		WEB-INF
			web.xml

A corresponding deployment plan looks as follows:

LoadTestDeploymentPlan.jar
	glassfish-application.xml
	PersonModel.jar.glassfish-ejb-jar.xml
	PersonUI.war.glassfish-web.xml

in which glassfish-application.xml has the following contents:

<!DOCTYPE glassfish-application PUBLIC "-//GlassFish.org//DTD GlassFish Application Server 3.1 Java EE Application 6.0//EN" "http://glassfish.org/dtds/glassfish-application_6_0-1.dtd">
<glassfish-application></glassfish-application>

and PersonModel.jar.glassfish-ejb-jar.xml has the following contents:

<!DOCTYPE glassfish-ejb-jar PUBLIC "-//GlassFish.org//DTD GlassFish Application Server 3.1 EJB 3.1//EN" "http://glassfish.org/dtds/glassfish-ejb-jar_3_1-1.dtd">
<glassfish-ejb-jar>
	<display-name>PersonModel</display-name>
	<enterprise-beans>
		<ejb>
			<ejb-name>Company</ejb-name>
			<jndi-name>ejb/Company</jndi-name>
			<bean-pool>
				<steady-pool-size>10</steady-pool-size>
				<resize-quantity>10</resize-quantity>
				<max-pool-size>100</max-pool-size>
				<pool-idle-timeout-in-seconds>600</pool-idle-timeout-in-seconds>
			</bean-pool>
			<pass-by-reference>true</pass-by-reference>
		</ejb>
	</enterprise-beans>
</glassfish-ejb-jar>

and PersonUI.war.glassfish-web.xml has the following contents:

<!DOCTYPE glassfish-web-app PUBLIC "-//GlassFish.org//DTD GlassFish Application Server 3.1 Servlet 3.0//EN" "http://glassfish.org/dtds/glassfish-web-app_3_0-1.dtd">
<glassfish-web-app>
	<session-config>
		<session-manager persistence-type="replicated">
			<manager-properties>
				<property name="reapIntervalSeconds" value="60"/>
				<property name="maxSessions" value="-1"/>
				<property name="persistenceFrequency" value="web-method"/>
			</manager-properties>
			<store-properties>
				<property name="persistenceScope" value="modified-session"/>
			</store-properties>
		</session-manager>
		<session-properties>
			<property name="timeoutSeconds" value="1800"/>
			<property name="enableCookies" value="true"/>
			<property name="enableURLRewriting" value="true"/>
		</session-properties>
	</session-config>
</glassfish-web-app>

We can create a jar by using:

jar cvf LoadTestDeploymentPlan.jar *.xml

We deploy the application by using the following command:

./asadmin deploy --deploymentplan /home/oracle/glassfish3/deploy/loadtest/LoadTestDeploymentPlan.jar --enabled=true --target app-cluster /home/oracle/glassfish3/deploy/loadtest/LoadTest.ear

To analyze we are going to use the real-time visibility tool (RTView) from SL Corporation. The first step is to adjust the rtview.properties, for example,

# This file contains configuration settings for this instance of an RTView Data Agent

# RTVAPM Package(s) used by this application
rtvapm_package=rtvmgr
rtvapm_package=hostmon
rtvapm_package=gfmon

# Window Title
rtv_title=RTView Data Agent

# JVM settings for Data Agent process
sender.sl.rtview.jvm=-Xmx512m

sl.rtview.sub=$RTVHISTORY_DB:''
sl.rtview.sub=$ALERTDEFS_DB:''

# DATA COLLECTOR PROPERTIES

# JMX metrics query interval
collector.sl.rtview.jmx.jmx_metrics_period=10000

# Option: Enable autodiscovery of JVM processes on localhost
collector.sl.rtview.jmx.jmxautodiscover=true

# Option: Define a class filter for discovered connections
collector.sl.rtview.jmx.jmxautodiscoverfilter=net.grinder

# Option: Sample Named JMX connection to a JVM running on a specific machine
# Retrieve the JMX ports from the GlassFish admin console. Click configurations, configuration-name,
# system properties. The property JMX_SYSTEM_CONNECTOR_PORT holds the required value. Click on
# instance values to obtain the configured ports for the individual servers.
# Below glassfish is the admin username and magic11g the admin password.
collector.sl.rtview.jmx.jmxconn=admin-server 172.31.0.164 8686 URL:- glassfish magic11g false
collector.sl.rtview.jmx.jmxconn=app-server1 172.31.0.164 28686 URL:- glassfish magic11g false
collector.sl.rtview.jmx.jmxconn=app-cluster-server1 172.31.0.164 28688 URL:- glassfish magic11g false
collector.sl.rtview.jmx.jmxconn=app-cluster-server2 172.31.0.164 28687 URL:- glassfish magic11g false

# Options: Collect Host Data Via WMI (Windows)
collector.sl.rtview.wmi.conn=__name=localhost computer=localhost

# Rate of WMI data collection in seconds
collector.sl.rtview.wmi.__doup=15

# Types of data collected
collector.sl.rtview.cache.config=wmi_cpu_data.rtv 	# CPU/MEMORY

# AGENT SENDER PROPERTIES

# Configure Agent Target here (only applies to sender)
# Specify the specific Agent Data Receiver URL assigned to you by SL
sender.sl.rtview.sub=$rtvmgrAgentTarget:'http://sldemos.com/201510_appmon_rtvagent'

# Define a unique agent identifier for each running agent
sender.sl.rtview.sub=$rtvJmxSource:MyMachineName

To start the RTView agent we can use the following script:

#!/bin/sh

JAVA_HOME="/home/oracle/jdk1.6.0_24"
export JAVA_HOME

PATH="${JAVA_HOME}/bin:${PATH}"
export PATH

java -Dhttp.proxyHost=proxy.middleware-magic.com -Dhttp.proxyPort=3128 -cp .:${JAVA_HOME}/lib/tools.jar:/home/oracle/temp/RtvDataAgent/rtvdataplus.jar -Dcom.sl.rtview.customRtvAppManagerClassName=com.sl.gmsjrtvutils.RtvApmAppManager com.sl.gmsjrtvhistorian.GmsJRtViewHistorianDaemon -agent -daemon -dataout -ds:com.sl.gmsjjmxds.GmsRtViewJmxDs -ds:com.sl.gmsjcacheds.GmsRtViewCacheDs -ds:com.sl.gmsjagentds.GmsRtViewAgentDs -rtvapm_packages:"rtvmgr gfmon " -properties:rtvapm -properties:rtvapm.rtvmgr -properties:rtvapm.gfmon -properties:rtview -propfilter:collector -propfilter:sender $*

Start the load test as the described in the post Fun with GlassFish, open RTView in a browser, click GlassFish servers, click all server table and choose a server from the list to drill down on it to see what is happening:

To view application characteristics, click application summary and an application to be viewed:

To get an overview of the individual components such as servlets click on appl/components summary:

To view a particular JVM, click JVM views, choose all JVMs table, right click on a particular JVM and select drill down:

As can be seen from the data, no particular problems arise during the loadtest. Another thing to note is that RTView is great monitoring and visualization tool for among others GlassFish Server.

References

[1] GlassFish Performance Tuning Guide.
[2] GlassFish Application Deployment Guide