In this post, we start with a basic example of how to set-up a WebLogic cluster spanning multiple machines. Once we know the steps involved we cluster the Oracle Service Bus in the same manner.

The reason for a cluster set-up spanning multiple machines is scaling and improve the failover characteristics of the application. Note that by adding more machines we are scaling the application horizontally. We can also scale our application by adding more CPU’s and application server instances to a machine, which is referred to as vertical scaling. A combination of horizontal and vertical scaling can lead to better failover characteristics and better utilization of the server hardware.

In the case of a machine failure, pinned resources such as JMS servers and JTA transaction managers must be migrated from one server to another. For this purpose WebLogic provides whole server migration and service migration. An example of the latter can be found here, which shows how to set-up a distributed JMS environment in the example section. Another option is to use Veritas Cluster Servers that monitor and control the WebLogic server instances (or any other hardware and software resource). In the case of a machine failure the instances running on that machine are migrated to another machine. Note that using Veritas Cluster Servers comes at an additional cost in terms of money. In most cases the lower-cost migration options provided by WebLogic will typically meet our needs.

Basic example

First, we have to install the WebLogic software on the machine where the admin server will reside. The following software versions are used in the installation:

  • JRockit: jrrt-4.0.1-1.6.0-linux-x64.bin
  • WebLogic Server: wls1031_generic.jar

The steps to install the JRockit JVM are as follows:

  • Start the installation by using ./jrrt-4.0.1-1.6.0-linux-x64.bin.
  • Define /home/oracle/bea/jrrt-4.0.1-1.6.0 as the installation directory and click next.
  • Click done when installation is finished.

The steps to install the WebLogic Server are as follows:

  • Navigate to JRockit’s bin directory (/bea/jrrt-4.0.1-1.6.0/bin).
  • Enter the following command: ./java -d64 -Xms1024m -Xmx1024m -jar <location-weblogic-installer>/wls1031_generic.jar.
  • Define a middleware home, i.e., /home/oracle/beaand click next.
  • Decide if you want to stay informed on security updates.
  • Select custom installation type and click next.
  • Click next on the choose products screen.
  • Select the installed JRockit JVM and click next. (Click browse if the JVM is not shown in the list and navigate to the JRockit installation directory, i.e., /bea/jrrt-4.0.1-1.6.0).
  • Accept the defaults on the directories screen and click next.
  • Click next to start the installation.
  • Uncheck run quickstart and click done.

Next we need to set-up our domain. To this end we use a WSLT script. The script creates the base_domain with an admin server (running on the vm-wls-rh1 machine) and a cluster with two managed servers, Server1 and Server2, respectively running on the machines vm-wls-rh1 and vm-wls-rh3. The script looks as follows:

beahome = '/home/oracle/bea';
linux = true;

adminusername = 'weblogic';
adminpassword = 'transfer11g';
servername = 'AdminServer';
domainname = 'base_domain';

pathseparator = '/';
if not linux:
	pathseparator = '\\';

domaintemplate = beahome + pathseparator + 'wlserver_10.3' + pathseparator + 'common' + pathseparator + 'templates' + pathseparator + 'domains' + pathseparator + 'wls.jar';
domainlocation = beahome + pathseparator + 'user_projects' + pathseparator + 'domains' + pathseparator + domainname;
nodemanagerhomelocation = beahome + pathseparator + 'wlserver_10.3' + pathseparator + 'common' + pathseparator + 'nodemanager';
jvmlocation = beahome + pathseparator + 'jrrt-4.0.1-1.6.0';

print 'CREATE DOMAIN';
createDomain(domaintemplate, domainlocation, adminusername, adminpassword);

print 'START NODE MANAGER';
startNodeManager(verbose='true', NodeManagerHome=nodemanagerhomelocation, ListenPort='5556', ListenAddress='localhost');

print 'CONNECT TO NODE MANAGER';
nmConnect(adminusername, adminpassword, 'localhost', '5556', domainname, domainlocation, 'ssl');

print 'START ADMIN SERVER';
nmStart(servername);

print 'CONNECT TO ADMIN SERVER';
connect(adminusername, adminpassword);

print 'START EDIT MODE';
edit();
startEdit();

print 'DISABLE HOSTNAME VERIFICATION';
cd('/Servers/AdminServer/SSL/AdminServer');
cmo.setHostnameVerificationIgnored(true);
cmo.setHostnameVerifier(None);
cmo.setTwoWaySSLEnabled(false);
cmo.setClientCertificateEnforced(false);

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

print 'DISCONNECT ADMIN SERVER';
disconnect();

print 'STOPPING ADMIN SERVER';
nmKill(servername);

print 'START ADMIN SERVER';
nmStart(servername);

print 'CONNECT TO ADMIN SERVER';
connect(adminusername, adminpassword);

print 'START EDIT MODE';
edit();
startEdit();

cd('/');

print 'CREATE MACHINE: VM-WLS-RH1';
vmwlsrh1 = cmo.createUnixMachine('vm-wls-rh1');
vmwlsrh1.setPostBindUIDEnabled(true);
vmwlsrh1.setPostBindUID('oracle');
vmwlsrh1.getNodeManager().setNMType('ssl');

print 'CREATE MACHINE: VM-WLS-RH1';
vmwlsrh3 = cmo.createUnixMachine('vm-wls-rh3');
vmwlsrh3.setPostBindUIDEnabled(true);
vmwlsrh3.setPostBindUID('oracle');
vmwlsrh3.getNodeManager().setListenAddress('vm-wls-rh3');
vmwlsrh3.getNodeManager().setNMType('ssl');

cluster = cmo.createCluster('Cluster');
cluster.setClusterMessagingMode('unicast');

print 'CREATE MANAGED SERVER: server1';
server1 = cmo.createServer('Server1');
server1.setListenPort(7002);
server1.setAutoRestart(true);
server1.setAutoKillIfFailed(true);
server1.setRestartMax(2);
server1.setRestartDelaySeconds(10);
server1.getServerStart().setJavaHome(jvmlocation);
server1.getServerStart().setJavaVendor('Oracle');
server1.getServerStart().setArguments('-Xms512m -Xmx512m -Xns256m -Xgcprio:pausetime');

# Settings to prevent types of denial-of-service attacks
# Maximum number of bytes allowed in messages received over the HTTP protocol
# server1.setMaxHTTPMessageSize(1500000);
# The maximum post size this server allows for reading HTTP POST data in a servlet request
# server1.getWebServer().setMaxPostSize(1000000);
# The amount of time the server waits between receiving chunks of data in an HTTP POST data before it times out
# server1.getWebServer().setPostTimeoutSecs(30);
# The number of backlogged, new TCP connection requests that should be allowed for the server's ports
# server1.setAcceptBacklog(300);
# The login timeout for this server's default listen port
# server1.setLoginTimeoutMillis(2500);

print 'CREATE MANAGED SERVER: server2';
server2 = cmo.createServer('Server2');
server2.setListenPort(7003);
server2.setListenAddress('vm-wls-rh3');
server2.setAutoRestart(true);
server2.setAutoKillIfFailed(true);
server2.setRestartMax(2);
server2.setRestartDelaySeconds(10);
server2.getServerStart().setJavaHome(jvmlocation);
server2.getServerStart().setJavaVendor('Oracle');
server2.getServerStart().setArguments('-Xms512m -Xmx512m -Xns256m -Xgcprio:pausetime');

print 'ADD MANAGED SERVERS TO CLUSTER';
server1.setCluster(cluster);
server2.setCluster(cluster);

print 'ADD MANAGED SERVERS TO MACHINE';
server1.setMachine(vmwlsrh1);
server2.setMachine(vmwlsrh3);

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

Once the server environment setup is complete, it is time to install the WebLogic software on the machine where the managed server (Server2) resides. The same steps to install the JRockit JVM and the WebLogic Server can be followed as presented above.

WebLogic provides two command-line utilities, pack and unpack, that helps to create a new managed server directory on a remote machine, for example on the vm-wls-rh1 machine we do the following (first navigate to the ${WL_HOME}/common/bin directory):

[oracle@vm-wls-rh1 bin]$ ./pack.sh -managed=true -domain=/home/oracle/bea/user_projects/domains/base_domain -template=/home/oracle/temp/template.jar -template_name=base_domain

<< read domain from "/home/oracle/bea/user_projects/domains/base_domain"
>>  succeed: read domain from "/home/oracle/bea/user_projects/domains/base_domain"
<< set config option Managed to "true"
>>  succeed: set config option Managed to "true"
<< write template to "/home/oracle/temp/template.jar"
....................................................................................................
>>  succeed: write template to "/home/oracle/temp/template.jar"
<< close template
>>  succeed: close template

Next, we have to copy the created template.jar to the vm-wls-rh3 machine, for example by using

[oracle@vm-wls-rh1 ~]$ scp oracle@vm-wls-rh1:/home/oracle/temp/template.jar oracle@vm-wls-rh3:/home/oracle/temp/

The authenticity of host 'vm-wls-rh1' can't be established.
RSA key fingerprint is d9:ff:32:72:72:da:48:fb:0f:7e:f6:46:ad:00:d8:85.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'vm-wls-rh1' (RSA) to the list of known hosts.
oracle@vm-wls-rh1's password:
The authenticity of host 'vm-wls-rh3 (vm-wls-rh3)' can't be established.
RSA key fingerprint is d9:ff:32:72:72:da:48:fb:0f:7e:f6:46:ad:00:d8:85.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'vm-wls-rh3' (RSA) to the list of known hosts.
oracle@vm-wls-rh3's password:
template.jar                                  100%   43KB  43.3KB/s   00:00
Connection to vm-wls-rh1 closed.

To use unpack on the vm-wls-rh3 machine we can do the following (first navigate to the ${WL_HOME}/common/bin directory):

[oracle@vm-wls-rh3 bin]$ ./unpack.sh -domain=/home/oracle/bea/user_projects/domains/base_domain -template=/home/oracle/temp/template.jar

<< read template from "/home/oracle/temp/template.jar"
>>  succeed: read template from "/home/oracle/temp/template.jar"
<< set config option DomainName to "base_domain"
>>  succeed: set config option DomainName to "base_domain"
<< write Domain to "/home/oracle/bea/user_projects/domains/base_domain"
...............................................................................................
>>  succeed: write Domain to "/home/oracle/bea/user_projects/domains/base_domain"
<< close template
>>  succeed: close template

By default, WebLogic is installed and configured with the DemoIdentity.jks keystore. This certificate and key are created by utils.CertGen with the default options of containing only the host name in the common name field (cn), and not the fully-qualified DNS name. As a result, attempts to establish SSL connections may fail in some situations due to a host name verification exception. When using the demo certificates in a multi-server domain, managed server instances will fail to boot if they cannot establish an SSL connection with the Admin Server and the a BAD_CERTIFICATE error is generated. This error occurs because the host name verifier, which is enabled by default in all WebLogic domains and which is used during the SSL handshake, compares the value of the cn field in the certificate with the fully-qualified DNS name of the SSL server that accepts the SSL connection. If these names do not match, the SSL connection is dropped.

In the WLST script above we already turned off the hostname verification. We could also accomplish this by using the system property weblogic.security.SSL.ignoreHostnameVerification, i.e., edit the setDomainEnv file, find the JAVA_OPTIONS="${JAVA_OPTIONS}" entry and add -Dweblogic.security.SSL.ignoreHostnameVerification=true.

Before we can test our set-up make sure the node manager is running on the machines. Login to the WebLogic console and start the managed servers.

Oracle Service Bus example

The following software versions are used in the installation:

  • JRockit: jrrt-4.0.1-1.6.0-linux-x64.bin
  • WebLogic Server: wls1033_generic.jar
  • Oracle Service Bus: ofm_osb_generic_11.1.1.3.0_disk1_1of1.zip

The post Setting-up a High Available Tuned SOA Environment shows the steps involved to set-up the Oracle Service Bus. Once the set-up is completed and the environment started we can add new servers and machines to the environement. To this end open the WebLogic Console and perform, for example, the following steps:

  • Add a server to the osbcluster:
    • Click environment, servers, click new and enter the following parameters:
      • name: osb_server2
      • server listen address: vm-wls-rh3
      • server listen port: 8012
      • Should this server belong to a cluster? select yes and choose osbcluster
    • Click next and finish
  • Create a new machine and add the osb_server2 to the machine:
    • Click environment, machine, click new and enter the following parameters:
      • name: remoteosbmachine
      • machine os: Unix
    • Click ok
    • Configure the remoteosbmachine:
      • Click on remoteosbmachine and change post-bind UID and post-bind GID to appropriate values. In our case we set the post-bind UID to oracle and leave post-bind GID unchanged.
      • Click the nodemanager tab and enter the following parameters:
        • listen address: vm-wls-rh3
        • listen port: 5556
      • Add the osb_server2 to the remoteosbmachine

Just as we did in the basic example, on the vm-wls-rh1 we pack our domain, i.e.,

[oracle@vm-wls-rh1 bin]$ ./pack.sh -managed=true -domain=/home/oracle/aqualogic/user_projects/domains/base_domain -template=/home/oracle/temp/aqualogic.jar -template_name=base_domain

<< read domain from "/home/oracle/aqualogic/user_projects/domains/base_domain"
>>  succeed: read domain from "/home/oracle/aqualogic/user_projects/domains/base_domain"
<< set config option Managed to "true"
>>  succeed: set config option Managed to "true"
<< write template to "/home/oracle/temp/aqualogic.jar"
....................................................................................................
>>  succeed: write template to "/home/oracle/temp/aqualogic.jar"
<< close template
>>  succeed: close template

Copy the created template to the vm-wls-rh3 machine. Before we can use the unpack tool on the vm-wls-rh3 we have to install the Oracle Service Bus software (no configuration step is necessary – this will be our unpack action). Once the software has been installed we can use the following command to unpack the template:

[oracle@vm-wls-rh3 bin]$ ./unpack.sh -domain=/home/oracle/aqualogic/user_projects/domains/base_domain -template=/home/oracle/temp/aqualogic.jar 

<< read template from "/home/oracle/temp/aqualogic.jar"
>>  succeed: read template from "/home/oracle/temp/aqualogic.jar"
<< set config option DomainName to "base_domain"
>>  succeed: set config option DomainName to "base_domain"
<< write Domain to "/home/oracle/aqualogic/user_projects/domains/base_domain"
...............................................................................................
>>  succeed: write Domain to "/home/oracle/aqualogic/user_projects/domains/base_domain"
<< close template
>>  succeed: close template

Before we start the managed server on the vm-wls-rh3 machine, we adjust the hostname verification. In this case we will be using the WebLogic Console. In the WebLogic Console, click environment, server. Choose the admin server and click on the ssl configuration tab. Click on the advanced link and set the hostname verification attribute to none. To make the changes effective we have to restart the admin server. To this end we are going to use WLST, i.e.,

# SET THE WEBLOGIC ENVIRONMENT
[oracle@vm-wls-rh1 bin]$ . ./setWLSEnv.sh 

CLASSPATH=/home/oracle/aqualogic/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/aqualogic/patch_ocp353/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/aqualogic/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/aqualogic/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/aqualogic/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/aqualogic/modules/features/weblogic.server.modules_10.3.3.0.jar:/home/oracle/aqualogic/wlserver_10.3/server/lib/webservices.jar:/home/oracle/aqualogic/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/aqualogic/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:
PATH=/home/oracle/aqualogic/wlserver_10.3/server/bin:/home/oracle/aqualogic/modules/org.apache.ant_1.7.1/bin:/home/oracle/aqualogic/jrrt-4.0.1-1.6.0/jre/bin:/home/oracle/aqualogic/jrrt-4.0.1-1.6.0/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/oracle/bin
Your environment has been set.

# START THE WLST CONSOLE
[oracle@localhost bin]$ java weblogic.WLST

Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands

# FIRST CONNECT TO THE NODE MANAGER
wls:/offline> nmConnect('weblogic', 'transfer11g', 'localhost', '5556', 'base_domain', '/home/oracle/aqualogic/user_projects/domains/base_domain', 'ssl');

Connecting to Node Manager ...
Successfully Connected to Node Manager.

# STOP THE ADMIN SERVER
wls:/nm/base_domain> nmKill('AdminServer');

Killing server AdminServer ...
Successfully killed server AdminServer ...

# START THE ADMIN SERVER
wls:/nm/base_domain> nmStart('AdminServer');

Starting server AdminServer ...
Successfully started server AdminServer ...

Next, we have to make sure the node manager is running on the vm-wls-rh3 machine, for example, by using the following WLST command

startNodeManager(verbose='true', NodeManagerHome='/home/oracle/aqualogic/wlserver_10.3/common/nodemanager');

Once the node manager is running on the vm-wls-rh3 machine, we can start the osb_server2. Open the WebLogic Console, click environment, servers, control tab. Select the server to be started and click start.

To check which processes are running on the vm-wls-rh3 machine, we can use

[oracle@vm-wls-rh3 bin]$ ps -ef|grep java
oracle    9081     1  1 16:18 pts/1    00:00:02 /home/oracle/aqualogic/jrrt-4.0.1-1.6.0/jre/bin/java -classpath /home/oracle/aqualogic/jrrt-4.0.1-1.6.0/jre/lib/rt.jar:/home/oracle/aqualogic/jrrt-4.0.1-1.6.0/jre/lib/i18n.jar:/home/oracle/aqualogic/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/aqualogic/patch_ocp353/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/oracle/aqualogic/jrrt-4.0.1-1.6.0/lib/tools.jar:/home/oracle/aqualogic/wlserver_10.3/server/lib/weblogic_sp.jar:/home/oracle/aqualogic/wlserver_10.3/server/lib/weblogic.jar:/home/oracle/aqualogic/modules/features/weblogic.server.modules_10.3.3.0.jar:/home/oracle/aqualogic/wlserver_10.3/server/lib/webservices.jar:/home/oracle/aqualogic/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/oracle/aqualogic/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar -DNodeManagerHome=/home/oracle/aqualogic/wlserver_10.3/common/nodemanager -DQuitEnabled=true weblogic.NodeManager -v

oracle    9155  9110 36 16:19 ?        00:00:39 /home/oracle/aqualogic/jrrt-4.0.1-1.6.0/bin/java -jrockit -Xms256m -Xmx512m -Dweblogic.Name=osb_server2 -Djava.security.policy=/home/oracle/aqualogic/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.system.BootIdentityFile=/home/oracle/aqualogic/user_projects/domains/base_domain/servers/osb_server2/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=false -Dweblogic.ReverseDNSAllowed=false -Xverify:none -da -Dplatform.home=/home/oracle/aqualogic/wlserver_10.3 -Dwls.home=/home/oracle/aqualogic/wlserver_10.3/server -Dweblogic.home=/home/oracle/aqualogic/wlserver_10.3/server -Ddomain.home=/home/oracle/aqualogic/user_projects/domains/base_domain -Dcommon.components.home=/home/oracle/aqualogic/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/home/oracle/aqualogic/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/home/oracle/aqualogic/user_projects/domains/base_domain/config/fmwconfig -Doracle.server.config.dir=/home/oracle/aqualogic/user_projects/domains/base_domain/config/fmwconfig/servers/osb_server2 -Doracle.security.jps.config=/home/oracle/aqualogic/user_projects/domains/base_domain/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=/home/oracle/aqualogic/user_projects/domains/base_domain/config/fmwconfig/carml -Digf.arisidstack.home=/home/oracle/aqualogic/user_projects/domains/base_domain/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/home/oracle/aqualogic/oracle_common/modules/oracle.ossoiap_11.1.1,/home/oracle/aqualogic/oracle_common/modules/oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabled=false -Dweblogic.management.discover=false -Dweblogic.management.server=http://vm-wls-rh1:7001 -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/home/oracle/aqualogic/patch_wls1033/profiles/default/sysext_manifest_classpath:/home/oracle/aqualogic/patch_ocp353/profiles/default/sysext_manifest_classpath weblogic.Server

To test the environment we deploy a web service to the cluster and an osb configuration just as was done in the post Setting-up a High Available Tuned SOA Environment. When a web service is deployed on a different server it is very likely that the used endpoint is different. By using the OSB console we can edit imported OSB implementations in any way we like. To adjust the web service endpoint we use the following steps:

  • Click on project explorer and subsequently on the project name.
  • Click on BusinessServices and subsequently the project’s business service.
  • A configuration overview is shown.
  • Click create in the change center.
  • Click the edit icon of the transport configuration part.
  • Edit the endpoints, i.e, make sure the endpoints of the web service are correct.
  • Click last and click save.
  • Click submit.

To check if the load balancing actually works, perform multiple calls to the proxy service. Open the WebLogic Console click deployments, the deployed web service. Click on the work load monitoring tab to see how the work load is balanced between the osb_server1 on the vm-wls-rh1 machine and the osb_server2 on the vm-wls-rh3 machine.

One last remark is order. Through-out the examples we have used machine names, such as vm-wls-rh1 and vm-wls-rh3, instead of IP-addresses. In order to use machine names edit the /etc/hosts file, for example,

127.0.0.1 		vm-wls-rh1 localhost.localdomain localhost
::1				localhost6.localdomain6 localhost6
172.10.0.125  	vm-wls-rh3

and

127.0.0.1 		vm-wls-rh3 localhost.localdomain localhost
::1				localhost6.localdomain6 localhost6
172.10.0.122  	vm-wls-rh1

References

[1] Service Migration.
[2] Veritas Cluster Server.
[3] Veritas High Availability Agent for WebLogic Server.
[4] Configuring Identity and Trust.
[5] Creating Templates and Domains Using the Pack and Unpack Commands.