In this post, we will set-up a tuned environment (operating system, network and Java Virtual Machine) for Coherence. We start with installing our operating system (RedHat Enterprise Linux) on different machines. Next, we will present the steps involved in fine tuning the operating system for Coherence and conduct a network and multi-cast test. Once the operating system is set-up, we will tune the Java Virtual machine and present scripts to start individual nodes of the Coherence cluster. Finally, we will use the Oracle Coherence Monitor from SL-corporation to obtain insight in the network activity and the cache activity. The application that is used is presented here.
Install the operating system
We start with the steps involved in installing our operating system:
- Create a bootable USB by using for example iso2usb.
- Press enter to start the installation (note that various install options are available, for example linux mem=2048m noprobe, see the function buttons for more details).
- Choose the installation language, choose the keyboard type, and the media type that contains the packages to be installed (hard drive).
- Select the partition that contains the iso image for Red Hat Enterprise Linux (/dev/sda).
- After a few moments the welcome screen is shown, click next.
- Enter the installation number (can also be skipped) when applicable and click next.
- Choose install Red Hat Enterprise Linux Server and click next.
- The hard drive partitioning screen is shown:
- select – remove all partitions on selected drives and create default layout.
- de-select – the sda drive (which is the USB).
- select review and modify partition layout.
- Click next. A warning is shown that all partitions (ALL DATA) are going to be removed, click yes.
- The partition layout is shown and can be modified at will, which we are not going to do, click next
- The boot loader screen is shown, select configure advanced boot loader options and click next. Note that the default says that the boot loader will be installed on /dev/sda, which is the USB. This means we will need the USB in order to boot the machine.
- Click next.
- Click change drive order and make /dev/hda the first and click OK. Now the boot loader will be installed on /dev/hda Master Boot Record (MBR).
- Click next.
- Edit the network devices when necessary. We will accept the defaults, i.e., a network device eth0 that sets the hostname automatically by using DHCP. Click next.
- Choose the location that will be used for the time settings and click next
- Enter the password for the superuser (root) and click next.
- The option to customize the packages to be installed is offered. We choose to customize the software selection and click next.
- The option is shows to select different software packages:
- Desktop environments: choose GNOME or KDE (both are very nice).
- Applications: different kind of games, editors etcetera can be chosen, the defaults will suffice.
- Development: Can be installed afterwards, such as a Java JDK.
- Servers: The default has printing support selected, which will do in our case.
- Base system: add the system tools option.
- Languages: choose additional language support to be installed.
- Click next to begin the installation.
- When the installation is finished, remove the USB and click reboot.
On first start up, RedHat Enterprise Linux offers the system to be configured:
- Click forward on the welcome screen.
- The license agreement is shown, select yes and click forward.
- The option to configure the firewall is offered. Disable the firewall and click forward (click yes in the pop-up).
- Security Enhanced Linux. Set the SELinux setting to enforcing and click forward.
- No need to enable kdump, click forward.
- Set the date and time and click forward.
- Configure software updates and click forward.
- Create a user:
- username: oracle
- fullname: oracle
- password: magic11g
- confirm password: magic11g
- Click forward.
- Configure the sound card and click forward.
- Install additional CDs when needed and click finish.
Configure the operating system
Now, that we have our operating system in place, we start with the configuration.
Change the hostname
To change the hostname we can follow the following steps:
- Login as
root.
- Open the file
/etc/sysconfig/network and enter a hostname, i.e., HOSTNAME=middleware-magic.com.
- Retrieve the IP-address by using
/sbin/ifconfig.
- Open the file
/etc/hosts and add, for example, 172.31.0.175 middleware-magic.com.
- Use the
hostname utility to change the hostname: hostname middleware-magic.com.
- Run it again without any parameter to see if the hostname has been changed.
- Restart the network to apply the changes:
service network restart.
- Logout and login again.
Some information about hostname resolution. To check to what IP-address the hostname will be resolved we can use:
[oracle@middleware-magic ~]$ perl -e 'use Socket; $ip = gethostbyname($ARGV[0]); $name = inet_ntoa($ip); print "$name\n";' middleware-magic.com
172.31.0.175
[oracle@middleware-magic ~]$ perl -e 'use Socket; $name = gethostbyaddr(inet_aton($ARGV[0]),AF_INET); print "$name\n";' 172.31.0.175
middleware-magic.com
[oracle@edu-wls-rh ~]$ getent hosts middleware-magic.com
172.31.0.175 middleware-magic.com
[oracle@edu-wls-rh ~]$ getent ahosts middleware-magic.com
172.31.0.175 STREAM middleware-magic.com
172.31.0.175 DGRAM
172.31.0.175 RAW
The hostname is looked up based on the rules specified in /etc/nsswitch.conf usually there is something like hosts: files dns. In this case the resolver will first check /etc/hosts and if it cannot find anything it will check /etc/resolv.conf for the DNS servers specified, for example, search environment.local nameserver 203.104.22.22
Packet loss minimization
The operating system buffers must be large enough to handle incoming network traffic while the application is paused during garbage collection. Also when UDP (User Datagram Protocol) is used to transmit messages to server instances in a cluster we must limit the need to retransmit UDP messages. In this case the size of the operating system buffers must be set appropriately to avoid excessive UDP datagram loss. To configure Linux to allow for larger buffers we can follow these steps:
Maximum number of open file descriptors
Most operating systems handle sockets as a form of file access and use file descriptors to keep track of which sockets are open. To contain the resources per process, the operating system restricts the number of file descriptors per process. Linux limits the number of open file descriptors per process, by default this is equal to 1024. It could be that the 1024 limit does not offer optimal performance. To obtain the current open file limit we can use the ulimit command. The ulimit -aS command shows the current limit, the ulimit -aH command shows the hard limit. To increase the total number of file descriptors for all users we can add the following to the /etc/sysctl.conf file: fs.file-max = 344030. To check the current value we can use cat /proc/sys/fs/file-max. If we want to make sure that a process (running under the oracle user) gets a maximum of 8192 open file descriptors, we have to edit the /etc/security/limits.conf file:
oracle soft nofile 8192
oracle hard nofile 8192
Note that this number applies across all processes running under the oracle user. To make the adjustments active, execute the ulimit -u 8192 command and log out and log in again.
TCP/IP
On some systems the default value for the time wait interval is too high and needs to be adjusted. To determine the number of sockets in TIME_WAIT we can use netstat -a | grep TIME_WAIT | wc -l. When the number approaches the maximum number of file descriptors per process, the application’s throughput will degrade, i.e., new connections have to wait for a free space in the application’s file descriptor table. To check the current value we can use cat /proc/sys/net/ipv4/tcp_keepalive_time and cat /proc/sys/net/ipv4/tcp_keepalive_intvl. The default time wait interval on newer Linux systems is 60 seconds and as such there is no need to change this. If the need arises that TCP/IP settings must be adjusted follow these steps:
Network interface card (NIC)
Configure the network card at it’s maximum link speed and at full duplex. To determine the configuration of the network card we can use:
[root@middleware-magic oracle]# /sbin/ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes
To change the maximum speed and duplex style we can use, for example, /sbin/ethtool -s eth0 speed 1000 duplex full.
Maximum transaction unit (MTU)
MTU is the maximum packet size, in bytes, that can be transmitted across a link. By default, an MTU of 1500 bytes is assumed and a default packet size of 1500 – 32 = 1468 bytes is used. To determine the MTU we can use the ping utility, for example,
[oracle@middleware-magic ~]$ ping -c 3 -M do -s 1468 172.31.0.175
PING 172.31.0.175 (172.31.0.175) 1468(1496) bytes of data.
1476 bytes from 172.31.0.175: icmp_seq=1 ttl=64 time=0.250 ms
1476 bytes from 172.31.0.175: icmp_seq=2 ttl=64 time=0.075 ms
1476 bytes from 172.31.0.175: icmp_seq=3 ttl=64 time=0.069 ms
--- 172.31.0.175 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.069/0.131/0.250/0.084 ms
Swapping
Swapping, also known as paging, is the use of secondary storage to store and retrieve data for use in RAM. Swapping is automatically performed by the operating system and typically occurs when the available RAM memory is depleted. Swapping can have a significant impact on the performance of Coherence and should thus be avoided. One thing to avoid swapping is to have enough RAM available – a tool such as top can be used to monitor swap rates. The swappiness (swap out a process or some of its heap due to low resource usage) should also be avoided for Coherence. To obtain the current value for swappiness, we can use cat /proc/sys/vm/swappiness. For Coherence, a low value (0 if possible) should be set. To change the swappiness, open the /etc/sysctl.conf file and add the following:
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
# increase TCP max buffer size
net.core.rmem_max = 2096304
net.core.wmem_max = 2096304
net.core.rmem_default=2096304
net.core.wmem_default=2096304
# set the swappiness to zero
vm.swappiness = 0
Large pages
Large pages are essentially blocks of contiguous physical memory addresses that are reserved for a process. Large pages improve performance of applications that access memory frequently (as is the case when using Coherence). When large pages are used the application uses the translation look-aside buffer (TLB) in the processor more effectively. The TLB is a cache of recently used virtual-to-physical address space translations stored in the processor memory. To obtain data from memory, the processor looks up the TLB to find out the physical addresses (RAM or hard disk) that hold the required data. In the case of large pages, a single entry in the TLB could represent a large contiguous address space and thereby potentially reducing the TLB look-up frequency and avoiding frequent look-ups in the hierarchical page table stored in-memory.
The kernel built with hugepage support (CONFIG_HUGETLBFS and CONFIG_HUGETLB_PAGE selected) should show the number of configured hugepages in the system by running cat /proc/meminfo:
[root@middleware-magic oracle]# cat /proc/meminfo
MemTotal: 4044532 kB
MemFree: 3411928 kB
Buffers: 167968 kB
Cached: 195860 kB
SwapCached: 0 kB
Active: 262596 kB
Inactive: 247148 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 4044532 kB
LowFree: 3411928 kB
SwapTotal: 6088624 kB
SwapFree: 6088536 kB
Dirty: 232 kB
Writeback: 0 kB
AnonPages: 146020 kB
Mapped: 52732 kB
Slab: 79676 kB
PageTables: 19360 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 8110888 kB
Committed_AS: 426808 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 265976 kB
VmallocChunk: 34359471959 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
where,
HugePages_Total is the size of the pool of hugepages.HugePages_Free is the number of hugepages in the pool that are not yet allocated.HugePages_Rsvd is short for reserved, and is the number of hugepages for which a commitment to allocate from the pool has been made, but no allocation has yet been made.Hugepagesize is the configured hugepage size.
Also cat /proc/filesystems should show a filesystem of type hugetlbfs configured in the kernel:
[root@middleware-magic oracle]# cat /proc/filesystems
nodev sysfs
nodev rootfs
nodev bdev
nodev proc
nodev cpuset
nodev binfmt_misc
nodev debugfs
nodev securityfs
nodev sockfs
nodev usbfs
nodev pipefs
nodev anon_inodefs
nodev futexfs
nodev tmpfs
nodev inotifyfs
nodev eventpollfs
nodev devpts
ext2
nodev ramfs
nodev hugetlbfs
iso9660
nodev mqueue
ext3
nodev rpc_pipefs
nodev autofs
The following command can be used to dynamically allocate/deallocate hugepages: echo 1024 > /proc/sys/vm/nr_hugepages. This command will try to configure 1024 hugepages in the system. The success or failure of allocation depends on the amount of physically contiguous memory that is present in the system at this time. At times, the JVM might not be able to reserve large pages because the pages might not be contiguous. To ensure that the JVM can use large pages, these must be enabled in the operating system as soon as possible after the system starts; otherwise the memory can become too fragmented which leads to a reduction in the number of large pages. In order to accomplish this, we can put the command in the /etc/rc.d/rc.local script, for example,
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
# set the sharable memory size - in this case we set the memory size to 4GB:
# 1024*1024*1024*4 = 4294967296
echo 4294967296 > /proc/sys/kernel/shmmax
# set the number of huge pages based on the Hugepagesize, i.e., 2048kB
# when we want to reserve 2GB in huge pages we have to set the number of huge pages to:
# (1024*1024*1024*2)/(1024*1024*2) = 2147483648/2097152 = 1024
echo 1024 > /proc/sys/vm/nr_hugepages
# give permission to the group that runs the process to access the shared memory segment
# to this end open the /etc/group file and retrieve the group-id (oracle:x:500:)
echo 500 > /proc/sys/vm/hugetlb_shm_group
Next, we set the locked in-memory address space for the oracle user in the file /etc/security/limits.conf, i.e.,
# memlock - maximum locked in-memory address space (kB), we set this equal to:
# number_of_huge_pages * huge_page_size = 1024 * 2048 = 2097152
oracle soft memlock 2097152
oracle hard memlock 2097152
When using JRockit we have to make a hugetblfs file system available in the directory /mnt/hugepages. This can be accomplished by using mount -t hugetlbfs nodev /mnt/hugepages (note that the /mnt/hugepages directory must exist). To make the /mnt/hugepages directory accessible for the oracle user, we can use chmod -R 777 /mnt/hugepages and add it to /etc/rc.d/rc.local. To make the mount permanent we have to edit the /etc/fstab file, for example,
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0
nodev /mnt/hugepages hugetlbfs defaults 0 0
Network test
Coherence ships with a datagram test utility that can be used to test and tune the network performance between two or more machines. When the datagram test is run, a publisher transmits UDP packets to the listener who then measures the throughput, success rate etcetera. Based on the results the environment can then be tuned for performance. The datagram test can be run by using datagram-test.sh, which is located in the ${COHERENCE_HOME}/bin directory. Before the script can be run add the JAVA_HOME property, for example,
#!/bin/sh
JAVA_HOME="/home/oracle/jrrt-4.0.1-1.6.0"
export JAVA_HOME
# specify the Coherence installation directory
SCRIPT_PATH="${BASH_SOURCE[0]}";
...
To run the datagram test, start a listener on one server, by just using datagram-test.sh on the command-line. Then start a publisher on another server, for example, datagram-test.sh hostname_or_ip_of_the_listener_server.
The term multi-cast refers to the ability to send a packet of information from one server and to have that packet delivered in parallel by the network to many servers. Coherence supports both multi-cast and uni-cast clustering. Note that multi-cast is the default and is the recommended setting. In some cases however, multi-cast cannot be used, for example, multi-cast cannot operate over certain types of network equipment, such as WAN routers and some switches that do not support multi-cast. Before we deploy an application that uses multi-cast, we have to determine if multi-cast is enabled. To this end, we can use the multicast-test.sh script, which is also located in the ${COHERENCE_HOME}/bin directory. In this script the variable JAVA_HOME also has to be set, for example,
#!/bin/sh
JAVA_HOME="/home/oracle/jrrt-4.0.1-1.6.0"
export JAVA_HOME
JAVA_OPTIONS="-Dhttp.proxyHost=proxy.middleware-magic.com -Dhttp.proxyPort=3128 -Djava.net.preferIPv4Stack=true"
export JAVA_OPTIONS
# specify the Coherence installation directory
SCRIPT_PATH="${BASH_SOURCE[0]}";
...
The test can be run by starting multicast-test.sh on the servers. When multicast is not working properly, i.e., no packets are received from other servers, we have to resort to well-known-addresses (WKA). To define well-known-addresses we can use the tangosol-coherence-override.xml file, for example,
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">
<cluster-config>
<unicast-listener>
<well-known-addresses>
<socket-address id="1">
<address>172.31.0.175</address>
<port>8088</port>
</socket-address>
<socket-address id="2">
<address>172.31.0.113</address>
<port>8088</port>
</socket-address>
</well-known-addresses>
</unicast-listener>
</cluster-config>
</coherence>
Something about the load average that is displayed when using, for example, top:
top - 09:16:42 up 4 days, 17:43, 4 users, load average: 0.91, 0.87, 0.77
Tasks: 129 total, 2 running, 127 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.0%us, 6.8%sy, 0.0%ni, 69.0%id, 0.0%wa, 1.2%hi, 4.0%si, 0.0%st
Mem: 4044532k total, 3909728k used, 134804k free, 239356k buffers
Swap: 6088624k total, 0k used, 6088624k free, 577888k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27790 oracle 17 0 1535m 540m 10m S 31.3 13.7 10:04.21 java
27750 oracle 18 0 1531m 133m 10m S 22.6 3.4 7:14.01 java
3992 oracle 15 0 425m 33m 18m S 0.3 0.8 0:09.16 nautilus
1 root 15 0 10348 688 580 S 0.0 0.0 0:00.61 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.17 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.08 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.03 events/0
7 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/1
8 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
49 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
54 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
55 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/1
56 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
215 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
216 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/1
The mathematics behind it are clearly explained in the articles UNIX Load Average Part 1: How It Works and UNIX Load Average Part 2: Not Your Average Average. An example, say we have the following load average: 0.91 0.87 0.77. These numbers represent the number of processes waiting in the run-queue plus the ones running at the moment in periods of 1, 5 and 15 minutes. Looking at the load average number, we have a one minute load average of 0.91, a five minute load average of 0.87 and a fifteen minute load average of 0.77. From this we can draw the following conclusions:
- On average in the last minute 0.91 processes have been running or are waiting for a response.
- The trend is that the load is increasing, i.e., an average of 0.91 is higher than 0.87 and 0.77.
- The system is busy, but we can not conclude how busy based on the load. What can be concluded is that a system with a load average higher than the number of CPUs has too much to do. On the other hand, a system with a load average significantly lower than the number of CPUs is not running up to its capacity.
Tune the Java Virtual Machine
In the posts Tune the JVM that runs Coherence and Tune the JVM that runs Coherence Revisited comprehensive discussions are made for respectively the JRockit JVM and the HotSpot JVM. To incorporate the knowledge presented in the posts in scripts to start cluster nodes, we can use the following to start a default cache server that collects management data:
#!/bin/sh
# JMX configuration for JRockit
#JMX_OPTIONS="-Djava.rmi.server.hostname=172.31.0.113 -Xmanagement:ssl=false,authenticate=false,port=7091,autodiscovery=true"
# JMX configuration for HotSpot
JMX_OPTIONS="-Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
# Coherence options to collect management information
COHERENCE_MANAGEMENT_OPTIONS="-Dtangosol.coherence.management=all -Dtangosol.coherence.management.remote=true"
# Cache configuration to use
COHERENCE_OPTIONS="-Dtangosol.coherence.cacheconfig=tryout-cache-config.xml ${COHERENCE_MANAGEMENT_OPTIONS} ${JMX_OPTIONS}"
export COHERENCE_OPTIONS
# JRockit Home
#JAVA_HOME="/home/oracle/jrrt-4.0.1-1.6.0"
# HotSpot Home
JAVA_HOME="/home/oracle/jdk1.6.0_24"
export JAVA_HOME
# Memory arguments for JRockit
#MEM_ARGS="-jrockit -Xms1024m -Xmx1024m -Xns256m -XXkeepAreaRatio:25 -Xgc:pausetime -XpauseTarget:200ms -XX:+UseCallProfiling -XX:+UseLargePagesForHeap"
# Memory arguments for HotSpot
MEM_ARGS="-server -Xms1024m -Xmx1024m -XX:NewRatio=2 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:LargePageSizeInBytes=2048k -XX:+UseLargePages"
export MEM_ARGS
# Set the class path
CLASSPATH="/home/oracle/temp/TryOut/lib/Coherence/coherence-hibernate.jar:/home/oracle/temp/TryOut/lib/Coherence/commonj.jar:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar:/home/oracle/temp/TryOut/lib/Coherence/coherence-work.jar:/home/oracle/temp/TryOut/lib/Database/ojdbc6.jar:/home/oracle/temp/TryOut/lib/Hibernate/jta.jar:/home/oracle/temp/TryOut/lib/Hibernate/ehcache-1.2.3.jar:/home/oracle/temp/TryOut/lib/Hibernate/hibernate3.jar:/home/oracle/temp/TryOut/lib/Hibernate/commons-logging-1.0.4.jar:/home/oracle/temp/TryOut/lib/Hibernate/dom4j-1.6.1.jar:/home/oracle/temp/TryOut/lib/Hibernate/cglib-2.1.3.jar:/home/oracle/temp/TryOut/lib/Hibernate/commons-collections-2.1.1.jar:/home/oracle/temp/TryOut/lib/Hibernate/asm.jar:/home/oracle/temp/TryOut/lib/Hibernate/c3p0-0.9.1.jar:/home/oracle/temp/TryOut/lib/Hibernate/antlr-2.7.6.jar:/home/oracle/temp/TryOut/lib/Hibernate/asm-attrs.jar:/home/oracle/temp/TryOut/out/artifacts/test/test.jar"
export CLASSPATH
# start the default cache server
${JAVA_HOME}/bin/java ${MEM_ARGS} ${COHERENCE_OPTIONS} com.tangosol.net.DefaultCacheServer
To start other nodes we can use:
#!/bin/sh
# Cache configuration to use
COHERENCE_OPTIONS="-Dtangosol.coherence.cacheconfig=tryout-cache-config.xml"
export COHERENCE_OPTIONS
# JRockit Home
#JAVA_HOME="/home/oracle/jrrt-4.0.1-1.6.0"
# HotSpot Home
JAVA_HOME="/home/oracle/jdk1.6.0_24"
export JAVA_HOME
# Memory arguments for JRockit
#MEM_ARGS="-jrockit -Xms1024m -Xmx1024m -Xns256m -XXkeepAreaRatio:25 -Xgc:pausetime -XpauseTarget:200ms -XX:+UseCallProfiling -XX:+UseLargePagesForHeap"
# Memory arguments for HotSpot
MEM_ARGS="-server -Xms1024m -Xmx1024m -XX:NewRatio=2 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:LargePageSizeInBytes=2048k -XX:+UseLargePages"
export MEM_ARGS
# Set the class path
CLASSPATH="/home/oracle/temp/TryOut/lib/Coherence/coherence-hibernate.jar:/home/oracle/temp/TryOut/lib/Coherence/commonj.jar:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar:/home/oracle/temp/TryOut/lib/Coherence/coherence-work.jar:/home/oracle/temp/TryOut/lib/Database/ojdbc6.jar:/home/oracle/temp/TryOut/lib/Hibernate/jta.jar:/home/oracle/temp/TryOut/lib/Hibernate/ehcache-1.2.3.jar:/home/oracle/temp/TryOut/lib/Hibernate/hibernate3.jar:/home/oracle/temp/TryOut/lib/Hibernate/commons-logging-1.0.4.jar:/home/oracle/temp/TryOut/lib/Hibernate/dom4j-1.6.1.jar:/home/oracle/temp/TryOut/lib/Hibernate/cglib-2.1.3.jar:/home/oracle/temp/TryOut/lib/Hibernate/commons-collections-2.1.1.jar:/home/oracle/temp/TryOut/lib/Hibernate/asm.jar:/home/oracle/temp/TryOut/lib/Hibernate/c3p0-0.9.1.jar:/home/oracle/temp/TryOut/lib/Hibernate/antlr-2.7.6.jar:/home/oracle/temp/TryOut/lib/Hibernate/asm-attrs.jar:/home/oracle/temp/TryOut/out/artifacts/test/test.jar"
export CLASSPATH
# start the test
${JAVA_HOME}/bin/java ${MEM_ARGS} ${COHERENCE_OPTIONS} model.test.Test
To understand what cache size is required, we can determine how many JVMs, how much physical memory, and how many CPUs and servers are required. As a general rule, allocate at least three times the physical heap size as the data set size. This assumes, we are going to keep one backup copy. The cache memory requirement can be calculated as: cache memory requirement = ((size of cache entries + size of indexes) * 2 (for primary and backup)) + JVM work memory (approx 30% of 1GB JVM). A cache entry can be calculated as entry size = serialized form of key + serialized form of value + 150 bytes and the index size as index size = forward index map + backward index map + reference + value size. Note that sizing of a cache is not an exact science and usually assumptions on the size and maximum number of objects have to be made.
Looking at servers, it is recommended that servers be configured with a minimum of 4GB of RAM. Commodity x86 server RAM is available in a density of 2GB per DIMM, so a server with 8 memory slots supports 16GB. One thing to note is that a server with a large amount of RAM is likely to run more JVMs (Coherence nodes). In this case a larger amount of CPU cores will help. Applications that are data-heavy require a higher ratio of RAM to CPU, while applications that are processing-heavy require a lower ratio. This means that it may be sufficient to have two dual core Xeon CPUs in a 32GB server running 15 Coherence nodes that perform cache accesses and updates, while on the other hand an application that makes use of indexing, parallel queries, entry processors and parallel aggregation it is more effective to use two quad cores Xeon CPUs in a 16GB server.
The following rules from the documentation should be followed in determining how many servers are required for reliable high availability configuration and how to configure the number of storage-enabled JVMs:
- There must be more than two servers. When starting out with two servers with an equal number of JVMs, losing one JVM forces the grid out of machine-safe state. Four or more servers present the most stable topology, but deploying to three servers would work if the following rules are adhered to.
- For a server that has the largest number of JVMs in the cluster, that number of JVMs must not exceed the total number of JVMs on all the other servers in the cluster.
- A server with the smallest number of JVMs should run at least half the number of JVMs as the server with the largest number of JVMs.
- The margin of safety improves as the number JVMs tends toward equality on all computers in the cluster, which is the general practice.
The number of JVMs (nodes) to run per server depends on the server’s number of processors/cores and the amount of memory. As a start, plan to run one JVM for every two cores. This recommendation balances the following factors:
- Multiple JVMs per server allow Coherence to make more efficient use of network resources. Coherence’s packet-publisher and packet-receiver have a fixed number of threads per JVM (when cores are added, we can add JVMs to scale across the added cores).
- Too many JVMs increases contention and context switching on processors.
- Too few JVMs may not be able to handle available memory and may not fully use the network interface card (NIC).
- For large heap sizes, JVMs must have enough processing capacity available (1 core per 2GB is ideal) to avoid long garbage collection pauses.
- Pause times increase as the amount of live data increases. Staying below 70% is a good goal, this includes primary data, backup data, indexes and application data.
- High object allocation rates increase pause times. Note that Coherence applications can cause high object allocation as every network packet generates many objects.
- CPU-intensive computations increase contention and may result in higher pause times.
Now that we have put most things in place, let us turn our attention to the monitoring of the Coherence cluster.
Monitoring
The steps to set-up the Oracle Coherence Monitor from SL-corporation are the following:
- We set-up the Oracle Coherence Monitor on a separate Windows machine.
- Unzip the Oracle Coherence Monitor package
rtv_ocm_5.8.0.0.zip to rtvoc_5.8.0.0.
- Open a command prompt and check if Java is available, by using,
java -version.
- Run the set-up file
rtvoc_setup.bat (located in the rtvoc_5.8.0.0 directory).
- Run the
rtvoc_cmd.bat file and verify the environment variables that are set by using set RTV on the command prompt.
- Create a cluster property file. In the
rtvoc_5.8.0.0\conf directory, make a copy of the SampleCluster.properties file and call this, for example, KlantCluster.properties.
- Configure a connection to the Coherence cluster. In our case, we are going to use the JMX Remote Port option. Note that we added
-Djava.rmi.server.hostname=172.31.0.113 -Xmanagement:ssl=false,authenticate=false,port=7091,autodiscovery=true for JRockit and -Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false for HotSpot to the script that starts the default cache server. Open the KlantCluster.properties file, and adjust the following parameters:
- sl.rtview.ocm.node=false (Oracle Coherence Monitor does not connect as a node)
- tangosol.coherence.cluster=KlantCluster (not used when the Oracle Coherence Monitor does not connect as a node)
- sl.rtview.ocm.jmxhost=172.31.0.113 (host on which the default cache server, that collects management data, will be running)
- sl.rtview.ocm.jmxport=7091 (configured JMX port)
- Edit the
coherence_setup.bat file (located in the rtvoc_5.8.0.0\conf directory) and specify the path of the coherence jar files, for example,
- set COHERENCE_CLASSPATH=C:\documents\Coherence\Voorbeelden\TryOut\lib\Coherence\coherence.jar
- set COHERENCE_CLASSPATH=%COHERENCE_CLASSPATH%;%RTVOC_HOME%/lib/ocjmxtables.jar
- set COHERENCE_CLUSTER=KlantCluster
To test the set-up, start the default cache server on one of the Linux machines. On the Windows machine where the Oracle Coherence Monitor (OCM) is installed, perform the following steps to start the monitoring:
- After the set-up script is run, a shortcuts map is placed on the desktop. Open this map and start the OCM Database.
- Subsequently, start the OCM Data Server, the OCM Display Server (DS) and the OCM Historian (DS).
- Finally, start the OCM Monitor, which can be run either as a client or as a web application deployed, for example, on Apache Tomcat.
- To deploy it as a web application by using Apache Tomcat:
- Unzip the
apache-tomcat-6.0.18-sl.zip file to apache-tomcat-6.0.18-sl.
- Open the
startup.bat file (located in the apache-tomcat-6.0.18-sl\bin directory) and add the variables JAVA_HOME and JRE_HOME such they point to a valid JDK, for example, set JAVA_HOME=C:\Progra~1\Java\jdk1.6.0_21 and set JRE_HOME=C:\Progra~1\Java\jdk1.6.0_21.
- Start up Apache Tomcat by running
startup.bat.
- Deploy
myocm.war (located in the rtvoc_5.8.0.0\webconf directory) to Apache Tomcat, by copying it to the apache-tomcat-6.0.18-sl\webapps directory.
- The Oracle Coherence Monitor can be reached at
http://hostname:8068/myocm/. Login with username: demo and password: demo.
- To use the client, run the OCM Monitor (DS) in the shortcuts map. Note that flash must be available in order to view the graphs.
- In the OCM monitor click cluster views, overview and check if one node is present.
On the second machine start a test node, which will start putting data into the cache and getting data from the cache. Subsequently, start two other test nodes, one on each machine.
Default cache server (on machine 172.31.0.113) log:
2012-01-05 11:08:26.245/0.432 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/tangosol-coherence.xml"
2012-01-05 11:08:26.310/0.497 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/tangosol-coherence-override-dev.xml"
2012-01-05 11:08:26.375/0.562 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tangosol-coherence-override.xml"
2012-01-05 11:08:26.379/0.566 Oracle Coherence 3.7.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.7.1.0 Build 27797
Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
2012-01-05 11:08:26.974/1.161 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Loaded cache configuration from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tryout-cache-config.xml"
2012-01-05 11:08:27.029/1.216 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Loaded Reporter configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/reports/report-group.xml"
2012-01-05 11:08:27.389/1.576 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, member=n/a): TCMP bound to /172.31.0.113:8088 using SystemSocketProvider
2012-01-05 11:08:30.709/4.896 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster, member=n/a): Created a new cluster with Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) UID=0xAC1F007100000134AD574F23DBA31F98
2012-01-05 11:08:30.712/4.899 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Started cluster Name=n/a
WellKnownAddressList(Size=2,
WKA{Address=172.31.0.113, Port=8088}
WKA{Address=172.31.0.175, Port=8088}
)
MasterMemberSet(
ThisMember=Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1
Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2012-01-05 11:08:30.709|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
)
)
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
2012-01-05 11:08:30.732/4.919 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2012-01-05 11:08:31.018/5.205 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache:DistributedReadWriteCache, member=1): Loaded POF configuration from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tryout-pof-config.xml"
2012-01-05 11:08:31.041/5.228 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache:DistributedReadWriteCache, member=1): Loaded included POF configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/coherence-pof-config.xml"
2012-01-05 11:08:31.085/5.272 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=1): Service DistributedReadWriteCache joined the cluster with senior service member 1
2012-01-05 11:08:31.115/5.302 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=1):
Services
(
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
PartitionedCache{Name=DistributedReadWriteCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
)
Started DefaultCacheServer...
2012-01-05 11:17:37.822/552.009 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=2012-01-05 11:17:37.641, Address=172.31.0.175:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:8411, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:17:37.939/552.126 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 2 joined Service Management with senior member 1
2012-01-05 11:17:38.534/552.721 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 2 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:17:39.734/553.921 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=1): 1> Transferring vulnerable PartitionSet{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127} to member 2 requesting 128
2012-01-05 11:17:44.394/558.581 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=1): 1> Transferring 129 out of 129 partitions to a machine-safe backup 1 at member 2 (under 129)
2012-01-05 11:17:44.409/558.596 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=1): Transferring 0KB of backup[1] for PartitionSet{128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256} to member 2
2012-01-05 11:19:08.638/642.825 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=3, Timestamp=2012-01-05 11:19:08.622, Address=172.31.0.113:8090, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13350, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:19:08.913/643.100 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 3 joined Service Management with senior member 1
2012-01-05 11:19:10.519/644.706 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 3 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:19:10.528/644.715 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=1): 3> Transferring primary PartitionSet{128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170} to member 3 requesting 43
2012-01-05 11:19:28.348/662.535 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member(Id=4, Timestamp=2012-01-05 11:19:28.164, Address=172.31.0.175:8090, MachineId=65081, Location=site:,machine:middleware-magic,process:8480, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:19:28.686/662.873 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 4 joined Service Management with senior member 1
2012-01-05 11:19:30.817/665.004 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=1): Member 4 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:19:30.824/665.011 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=1): 3> Transferring primary PartitionSet{171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191} to member 4 requesting 21
2012-01-05 11:19:30.894/665.081 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=1): Transferring 230KB of backup[1] for PartitionSet{192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255} to member 4
2012-01-05 11:19:33.329/667.516 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=1): Transferring 76KB of backup[1] for PartitionSet{192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212} to member 2
Member two server (on machine 172.31.0.175) log:
2012-01-05 11:16:57.254/0.447 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/tangosol-coherence.xml"
2012-01-05 11:16:57.345/0.538 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/tangosol-coherence-override-dev.xml"
2012-01-05 11:16:57.500/0.693 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tangosol-coherence-override.xml"
2012-01-05 11:16:57.505/0.698 Oracle Coherence 3.7.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.7.1.0 Build 27797
Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
2012-01-05 11:16:58.234/1.427 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Loaded cache configuration from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tryout-cache-config.xml"
2012-01-05 11:16:58.886/2.079 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, member=n/a): TCMP bound to /172.31.0.175:8088 using SystemSocketProvider
2012-01-05 11:16:59.179/2.373 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2012-01-05 11:17:37.641, Address=172.31.0.175:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:8411, Role=ModelTestTest, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) joined cluster with senior Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2012-01-05 11:16:59.200/2.393 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2012-01-05 11:16:59.200/2.393 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:16:59.218/2.411 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Started cluster Name=n/a
WellKnownAddressList(Size=2,
WKA{Address=172.31.0.113, Port=8088}
WKA{Address=172.31.0.175, Port=8088}
)
MasterMemberSet(
ThisMember=Member(Id=2, Timestamp=2012-01-05 11:17:37.641, Address=172.31.0.175:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:8411, Role=ModelTestTest)
OldestMember=Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=2
Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
Member(Id=2, Timestamp=2012-01-05 11:17:37.641, Address=172.31.0.175:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:8411, Role=ModelTestTest)
)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2012-01-05 11:08:27.427|JOINED,
2|3.7.1|2012-01-05 11:17:37.831|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
)
)
TcpRing{Connections=[1]}
IpMonitor{AddressListSize=1}
2012-01-05 11:16:59.289/2.482 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:Management, member=2): Service Management joined the cluster with senior service member 1
2012-01-05 11:16:59.755/2.948 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache:DistributedReadWriteCache, member=2): Loaded POF configuration from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tryout-pof-config.xml"
2012-01-05 11:16:59.784/2.977 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache:DistributedReadWriteCache, member=2): Loaded included POF configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/coherence-pof-config.xml"
2012-01-05 11:16:59.848/3.041 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=2): Service DistributedReadWriteCache joined the cluster with senior service member 1
2012-01-05 11:16:59.895/3.088 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=2): Asking member 1 for 128 primary partitions
2012-01-05 11:18:29.998/93.191 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=2): Member(Id=3, Timestamp=2012-01-05 11:19:08.622, Address=172.31.0.113:8090, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13350, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:18:30.274/93.467 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=2): Member 3 joined Service Management with senior member 1
2012-01-05 11:18:31.878/95.071 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=2): Member 3 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:18:32.282/95.475 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=2): 3> Transferring primary PartitionSet{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41} to member 3 requesting 42
2012-01-05 11:18:32.572/95.765 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=2): Transferring 129KB of backup[1] for PartitionSet{42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83} to member 3
2012-01-05 11:18:49.710/112.903 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=2): Member(Id=4, Timestamp=2012-01-05 11:19:28.164, Address=172.31.0.175:8090, MachineId=65081, Location=site:,machine:middleware-magic,process:8480, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:18:50.040/113.233 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=2): Member 4 joined Service Management with senior member 1
2012-01-05 11:18:52.171/115.364 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=2): Member 4 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:18:52.899/116.092 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=2): 3> Transferring primary PartitionSet{42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62} to member 4 requesting 21
2012-01-05 11:18:55.278/118.471 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=2): 3> Transferring primary PartitionSet{63} to member 4 requesting 1
Member three server (on machine 172.31.0.113) log:
2012-01-05 11:19:06.954/0.372 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/tangosol-coherence.xml"
2012-01-05 11:19:07.016/0.434 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/tangosol-coherence-override-dev.xml"
2012-01-05 11:19:07.084/0.502 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tangosol-coherence-override.xml"
2012-01-05 11:19:07.088/0.506 Oracle Coherence 3.7.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.7.1.0 Build 27797
Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
2012-01-05 11:19:07.634/1.052 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Loaded cache configuration from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tryout-cache-config.xml"
2012-01-05 11:19:08.342/1.760 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, member=n/a): TCMP bound to /172.31.0.113:8090 using SystemSocketProvider
2012-01-05 11:19:08.648/2.066 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=3, Timestamp=2012-01-05 11:19:08.622, Address=172.31.0.113:8090, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13350, Role=ModelTestTest, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster with senior Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2012-01-05 11:19:08.842/2.260 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member(Id=2, Timestamp=2012-01-05 11:17:37.641, Address=172.31.0.175:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:8411, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:19:08.857/2.275 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2012-01-05 11:19:08.857/2.275 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:19:08.859/2.277 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 2 joined Service Management with senior member 1
2012-01-05 11:19:08.859/2.277 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 2 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:19:08.860/2.278 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Started cluster Name=n/a
WellKnownAddressList(Size=2,
WKA{Address=172.31.0.175, Port=8088}
WKA{Address=172.31.0.113, Port=8088}
)
MasterMemberSet(
ThisMember=Member(Id=3, Timestamp=2012-01-05 11:19:08.622, Address=172.31.0.113:8090, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13350, Role=ModelTestTest)
OldestMember=Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=3
Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
Member(Id=2, Timestamp=2012-01-05 11:17:37.641, Address=172.31.0.175:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:8411, Role=ModelTestTest)
Member(Id=3, Timestamp=2012-01-05 11:19:08.622, Address=172.31.0.113:8090, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13350, Role=ModelTestTest)
)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2012-01-05 11:08:27.427|JOINED,
2|3.7.1|2012-01-05 11:17:37.641|JOINED,
3|3.7.1|2012-01-05 11:19:08.853|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
)
)
TcpRing{Connections=[2]}
IpMonitor{AddressListSize=1}
2012-01-05 11:19:08.903/2.322 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:Management, member=3): Service Management joined the cluster with senior service member 1
2012-01-05 11:19:09.253/2.671 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache:DistributedReadWriteCache, member=3): Loaded POF configuration from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tryout-pof-config.xml"
2012-01-05 11:19:09.287/2.705 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache:DistributedReadWriteCache, member=3): Loaded included POF configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/coherence-pof-config.xml"
2012-01-05 11:19:09.347/2.765 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=3): Service DistributedReadWriteCache joined the cluster with senior service member 1
2012-01-05 11:19:10.521/3.939 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=3): Asking member 1 for 43 primary partitions
2012-01-05 11:19:10.896/4.314 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=3): Asking member 2 for 42 primary partitions
2012-01-05 11:19:11.222/4.640 Oracle Coherence GE 3.7.1.0 <D5> (thread=main, member=3): Repeating GetAllRequest for 2 out of 6 items due to the re-distribution of PartitionSet{11, 21}
2012-01-05 11:19:28.349/21.767 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=3): Member(Id=4, Timestamp=2012-01-05 11:19:28.164, Address=172.31.0.175:8090, MachineId=65081, Location=site:,machine:middleware-magic,process:8480, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:19:28.686/22.104 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=3): Member 4 joined Service Management with senior member 1
2012-01-05 11:19:30.817/24.235 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=3): Member 4 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:19:31.267/24.685 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=3): 1> Transferring vulnerable PartitionSet{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19} to member 4 requesting 20
2012-01-05 11:19:31.352/24.770 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=3): Transferring 75KB of backup[1] for PartitionSet{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41} to member 4
2012-01-05 11:19:34.033/27.451 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=3): 3> Transferring primary PartitionSet{128} to member 4 requesting 1
Member four server (on machine 172.31.0.175) log:
2012-01-05 11:18:47.021/0.657 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/tangosol-coherence.xml"
2012-01-05 11:18:47.220/0.856 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/tangosol-coherence-override-dev.xml"
2012-01-05 11:18:47.454/1.090 Oracle Coherence 3.7.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tangosol-coherence-override.xml"
2012-01-05 11:18:47.464/1.100 Oracle Coherence 3.7.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.7.1.0 Build 27797
Grid Edition: Development mode
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
2012-01-05 11:18:48.098/1.734 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Loaded cache configuration from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tryout-cache-config.xml"
2012-01-05 11:18:49.411/3.047 Oracle Coherence GE 3.7.1.0 <D4> (thread=main, member=n/a): TCMP bound to /172.31.0.175:8090 using SystemSocketProvider
2012-01-05 11:18:49.719/3.355 Oracle Coherence GE 3.7.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=4, Timestamp=2012-01-05 11:19:28.164, Address=172.31.0.175:8090, MachineId=65081, Location=site:,machine:middleware-magic,process:8480, Role=ModelTestTest, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) joined cluster with senior Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2012-01-05 11:18:49.911/3.547 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member(Id=2, Timestamp=2012-01-05 11:17:37.641, Address=172.31.0.175:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:8411, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:18:49.940/3.576 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member(Id=3, Timestamp=2012-01-05 11:19:08.622, Address=172.31.0.113:8090, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13350, Role=ModelTestTest) joined Cluster with senior member 1
2012-01-05 11:18:49.947/3.583 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 2 joined Service Management with senior member 2
2012-01-05 11:18:49.948/3.584 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 2 joined Service DistributedReadWriteCache with senior member 2
2012-01-05 11:18:49.948/3.584 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2012-01-05 11:18:49.948/3.585 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:18:49.949/3.585 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 3 joined Service Management with senior member 1
2012-01-05 11:18:49.949/3.585 Oracle Coherence GE 3.7.1.0 <D5> (thread=Cluster, member=n/a): Member 3 joined Service DistributedReadWriteCache with senior member 1
2012-01-05 11:18:49.952/3.588 Oracle Coherence GE 3.7.1.0 <Info> (thread=main, member=n/a): Started cluster Name=n/a
WellKnownAddressList(Size=2,
WKA{Address=172.31.0.113, Port=8088}
WKA{Address=172.31.0.175, Port=8088}
)
MasterMemberSet(
ThisMember=Member(Id=4, Timestamp=2012-01-05 11:19:28.164, Address=172.31.0.175:8090, MachineId=65081, Location=site:,machine:middleware-magic,process:8480, Role=ModelTestTest)
OldestMember=Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=4
Member(Id=1, Timestamp=2012-01-05 11:08:27.427, Address=172.31.0.113:8088, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13253, Role=CoherenceServer)
Member(Id=2, Timestamp=2012-01-05 11:17:37.641, Address=172.31.0.175:8088, MachineId=65081, Location=site:,machine:middleware-magic,process:8411, Role=ModelTestTest)
Member(Id=3, Timestamp=2012-01-05 11:19:08.622, Address=172.31.0.113:8090, MachineId=56227, Location=site:,machine:edu-wls-rh,process:13350, Role=ModelTestTest)
Member(Id=4, Timestamp=2012-01-05 11:19:28.164, Address=172.31.0.175:8090, MachineId=65081, Location=site:,machine:middleware-magic,process:8480, Role=ModelTestTest)
)
MemberId|ServiceVersion|ServiceJoined|MemberState
1|3.7.1|2012-01-05 11:08:27.427|JOINED,
2|3.7.1|2012-01-05 11:17:37.641|JOINED,
3|3.7.1|2012-01-05 11:19:08.622|JOINED,
4|3.7.1|2012-01-05 11:19:28.586|JOINED
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
)
)
TcpRing{Connections=[3]}
IpMonitor{AddressListSize=1}
2012-01-05 11:18:50.027/3.663 Oracle Coherence GE 3.7.1.0 <D5> (thread=Invocation:Management, member=4): Service Management joined the cluster with senior service member 1
2012-01-05 11:18:50.564/4.200 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache:DistributedReadWriteCache, member=4): Loaded POF configuration from "jar:file:/home/oracle/temp/TryOut/out/artifacts/test/test.jar!/tryout-pof-config.xml"
2012-01-05 11:18:50.601/4.237 Oracle Coherence GE 3.7.1.0 <Info> (thread=DistributedCache:DistributedReadWriteCache, member=4): Loaded included POF configuration from "jar:file:/home/oracle/temp/TryOut/lib/Coherence/coherence.jar!/coherence-pof-config.xml"
2012-01-05 11:18:50.675/4.311 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=4): Service DistributedReadWriteCache joined the cluster with senior service member 1
2012-01-05 11:18:52.174/5.810 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=4): Asking member 1 for 21 primary partitions
2012-01-05 11:18:52.600/6.236 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=4): Asking member 3 for 20 primary partitions
2012-01-05 11:18:52.882/6.519 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=4): Asking member 2 for 21 primary partitions
2012-01-05 11:18:53.151/6.787 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=4): Transferring 74KB of backup[1] for PartitionSet{171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191} to member 3
2012-01-05 11:18:55.275/8.911 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=4): Asking member 2 for 1 primary partitions
2012-01-05 11:18:55.386/9.022 Oracle Coherence GE 3.7.1.0 <D4> (thread=DistributedCache:DistributedReadWriteCache, member=4): Asking member 3 for 1 primary partitions
2012-01-05 11:18:55.508/9.144 Oracle Coherence GE 3.7.1.0 <D5> (thread=DistributedCache:DistributedReadWriteCache, member=4): Transferring 3KB of backup[1] for PartitionSet{128} to member 3
The reason for showing the logging is that it obtains insight in how the data partitioning in the Coherence cluster is going.
Results
In the cluster views, overview tab metrics and alerts are shown for the cluster.
The cluster views, all caches history tab, shows a heatmap that displays a summary of the recent activity on the caches running in the cluster.
The cluster views, all nodes history tab, shows a heatmap that displays a summary of the recent activity on the nodes running in the cluster. This tab can be used to identify which nodes are generating the most cluster activity (which gives us the possibility to do some pattern analysis in spotting hot spots in the cluster).
Coherence provides a lot of metrics through MBeans (when enabled), such as,
- Node metrics (memory usage, network packets, packet loss)
- Service metrics (CPU load, requests, messages, task backlog, requests pending)
- Cache metrics (total puts/getsm total hits/misses, store writes/reads)
To get an idea of network activity, i.e., the number of packets (unit of network data transfer), the number of messages (command from one node to another, which could be multiple packets depending on the data size) and the number of requests (a message that expects a response message that returns data). The cache services, cache service detail tab, shows a table view of attributes, such as CPU%, requests and messages.
The all nodes, all nodes detail tab, shows detailed information such as the number of packets send, the number of packets received, the number of duplicate packets received and the number of packets resend.
The single cache, cache summary tab, shows an overview of the size and activity such as puts, gets, hits, misses, backups, evictions etcetera.
Activity trends can be viewed in the single cache, activity trends tab.
The single cache, node/group distribution gives us an indication of how well we are scaling the application (As the data is showing a uniform distribution, we are actually doing very well as scaling is concerned).
The Oracle Coherence Monitor from SL corporation is just an incredible tool, that will come in very handy when troubleshooting or monitoring the Coherence cluster. Special thanks to Everett Williams for the great explanation – hoping for more conversations in the future.
References
[1] Oracle Coherence Documentation Library.
[2] Real-Time Monitoring and Management Application for Oracle Coherence In-Memory Data Grids.
Testimonials![\"René](\"http://middlewaremagic.com/weblogic/wp-content/uploads/2009/08/rwijk-foto-249x300.jpg\" "\\"rwijk-foto\\"")
![\"\"](\"http://middlewaremagic.com/weblogic/wp-content/uploads/2010/11/Ravish1-150x150.jpg\" "\\"Ravish\\"")
![\"Jay](\"http://middlewaremagic.com/weblogic/wp-content/uploads/2009/12/Joy_pool-150x150.jpg\" "\\"Joy_pool\\"")
