In general, it is a good idea to use log rotation to prevent having to stop a WebLogic server to remove a large log file that is filling up the file system. It is also a good idea to create a separate partition such that the root file system does not fill up. A partition is a contiguous set of blocks on a drive that are treated as if they were independent disks. The default installation of today’s Enterprise Linux distributions use flexible partitioning layouts by creating one or more logical volumes. Some of the partitions that we might want to consider separating out from root to provide more flexibility and better performance in the environment

• /home – A file server environment would benefit from separating out /home to its own partition. This is the home directory for all users on the system, if there are no disk quotas implemented, so separating this directory should isolate a user’s runaway consumption of disk space.
• /tmp – If we are running a high-performance computing environment, large amounts of temporary space are needed during compute time, then released upon completion.
• /usr – This is where the kernel source tree and Linux documentation (as well as most executable binaries) are located. The /usr/local directory stores the executables that must be accessed by all users on the system and is a good location to store custom scripts developed for our environment. If it is separated to its own partition, then files will not have to be reinstalled during an upgrade or reinstall by simply choosing not to have the partition reformatted
• /var – The /var partition is important in mail, Web, and print server environments because it contains the log files for these environments and the overall system log. Chronic messages can flood and fill this partition. If this occurs and the partition is not separate from the /, service interruptions are possible. Depending on the environment, further separation of this partition is possible by separating out /var/spool/mail for a mail server or /var/log for system logs.
• /opt – The installation of some third-party software products, such as WebLogic Server, default to this partition. If not separate, the installation will continue under / and, if there is not enough space allocated, could fail.

To learn how to create partitions on a disk drive and how to format them for use on a Linux system as swap or data space, we can use Create partitions and filesystems and Hard disk layout.

To configure the domain logging characteristics, we can use the domain’s logging configuration tab. Set the file name to, for example, base-dir/domain-name/domain-name.log. Here, base-dir could be /var/log/weblogic/domains. Set the rotation type attribute to by time, and the limit number of retained log files checkbox to checked. Note that it is also possible to move old log files to another directory, by using the log file rotation directy property. The default rotation are: rotation time 00:00 and rotation interval 24 hours (i.e., the log file is rotated daily at midnight). In practice the domain log file does not grow very quickly.

To configure server logging, we can use the servers’ general logging tabs. The file name can be set to base-dir/domain-name/server-name/server-name.log. When you expect a log file to fill up quickly it is useful to set the rotation interval to a shorter period. By using the HTTP logging tab, we can control the HTTP access log settings. In general, the HTTP access log files grow proportionally to the number of requests, so in this case depending on the amount of traffic to set the rotation to by size instead of by time, this guarantees that do not end up with a very large log file that could fill up the available disk space. Set the rotation file size to an appropriate value, for example 5000 kilobytes. Note that HTTP access logging is enabled by default. When using web server (such as Apache), it makes sense to disable HTTP access logging on WebLogic as the web server already creates an HTTP access log.

When we use the node manager, a nodemanager.log is created in the node manager’s home directory. This file contains the warning and errors related to the node manager life cycle, for example,

<Feb 26, 2013 11:00:56 AM> <INFO> <base_domain> <cluster-server2> <Startup configuration properties loaded from "/home/weblogic/weblogic12.1.1/configuration/domains/base_domain/servers/cluster-server2/data/nodemanager/startup.properties">
<Feb 26, 2013 11:00:56 AM> <INFO> <base_domain> <cluster-server1> <Startup configuration properties loaded from "/home/weblogic/weblogic12.1.1/configuration/domains/base_domain/servers/cluster-server1/data/nodemanager/startup.properties">
<Feb 26, 2013 11:00:56 AM> <INFO> <base_domain> <security-server> <Startup configuration properties loaded from "/home/weblogic/weblogic12.1.1/configuration/domains/base_domain/servers/security-server/data/nodemanager/startup.properties">
<Feb 26, 2013 11:00:56 AM> <INFO> <base_domain> <coherence-server1> <Startup configuration properties loaded from "/home/weblogic/weblogic12.1.1/configuration/domains/base_domain/servers_coherence/coherence-server1/data/nodemanager/startup.properties">
<Feb 26, 2013 11:00:57 AM> <INFO> <Secure socket listener started on port 5556, host middleware-magic.com/192.168.1.150>
<Feb 26, 2013 11:04:27 AM> <INFO> <base_domain> <AdminServer> <Rotated server output log to "/home/weblogic/weblogic12.1.1/configuration/domains/base_domain/servers/AdminServer/logs/AdminServer.out00001">
<Feb 26, 2013 11:04:27 AM> <INFO> <base_domain> <AdminServer> <Server error log also redirected to server log>
<Feb 26, 2013 11:04:27 AM> <INFO> <base_domain> <AdminServer> <Starting WebLogic server with command line: /home/weblogic/weblogic12.1.1/configuration/domains/base_domain/bin/startWebLogic.sh >
<Feb 26, 2013 11:04:27 AM> <INFO> <base_domain> <AdminServer> <Working directory is '/home/weblogic/weblogic12.1.1/configuration/domains/base_domain'>
<Feb 26, 2013 11:04:27 AM> <INFO> <base_domain> <AdminServer> <Server output log file is '/home/weblogic/weblogic12.1.1/configuration/domains/base_domain/servers/AdminServer/logs/AdminServer.out'>

As shown in the example above, when using the node manager, WebLogic captures the stdout and stderr output streams and writes them to disk. Note that these files are created in the server’s log directory and are called server-name.out. The server-name.out files allows us to determine why the server failed to start if the server fails before writing to its own log file. One thing to note is that there is no way to limit the size of the server-name.out files or automatically rotate it. The reduce the size of these files we can set the standard out logging destination’s severity level to, for example, critical using the advanced area of the server’s general logging tab. When on Linux when can use logrotate.

The following are the key files for logrotate to work properly

• /usr/bin/logrotate – the logrotate command
• /etc/cron.daily/logrotate – shell script that executes the logrotate command daily

  	#!/bin/sh
  
  	/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1
  	EXITVALUE=$?
  	if [ $EXITVALUE != 0 ]; then
  		/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
  	fi
  	exit 0
  	

• /etc/logrotate.conf – log rotation configuration

  	# see "man logrotate" for details
  	# rotate log files weekly
  	weekly
  	# keep 4 weeks worth of backlogs
  	rotate 4
  	# create new (empty) log files after rotating old ones
  	create
  	# use date as a suffix of the rotated file
  	dateext
  	# uncomment this if you want your log files compressed
  	#compress
  	# RPM packages drop log rotation information into this directory
  	include /etc/logrotate.d
  	# no packages own wtmp and btmp -- we'll rotate them here
  	/var/log/wtmp {
  		monthly
  		create 0664 root utmp
  		minsize 1M
  		rotate 1
  	}
  
  	/var/log/btmp {
  		missingok
  		monthly
  		create 0600 root utmp
  		rotate 1
  	}	
  
  	/home/weblogic/weblogic12.1.1/configuration/domains/base_domain/servers/AdminServer/logs/AdminServer.out {
  		missingok
  		nodateext
  		copytruncate
  		# create 0664 weblogic weblogic
  		rotate 10
  		size 10k
  	}
  	

• /etc/logrotate.d – RPM packages drop log rotation information into this directory, for example, the httpd log rotation configuration (/etc/logrotate.d/httpd) looks as follows

  	/var/log/httpd/*log {
  		missingok
  		notifempty
  		sharedscripts
  		delaycompress
  		postrotate
  			/sbin/service httpd reload > /dev/null 2>/dev/null || true
  		endscript
  	}
  	

By using the /etc/logrotate.d directory we can also add a WebLogic specific file that contains all the configuration related to server-name.out files. To test the example, presented in the /etc/logrotate.conf file, start the admin server using the node manager and at a certain point execute the logrotate -v /etc/logrotate.conf command. We can see that a file named AdminServer.out.1 is created that contains

...
<Feb 22, 2013 2:25:46 PM CET> <Notice> <Log Management> <BEA-170019> <The server log file /home/weblogic/weblogic12.1.1/configuration/domains/base_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>
<Feb 22, 2013 2:25:49 PM CET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Feb 22, 2013 2:25:53 PM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>
<Feb 22, 2013 2:25:53 PM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>

and a file AdminServer.out that contains

<Feb 22, 2013 2:25:57 PM CET> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on fe80:0:0:0:20c:29ff:fe9c:771e:7001 for protocols iiop, t3, ldap, snmp, http.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
<Feb 22, 2013 2:25:58 PM CET> <Warning> <Server> <BEA-002611> <The hostname "localhost", maps to multiple IP addresses: 127.0.0.1, 0:0:0:0:0:0:0:1.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7001 for protocols iiop, t3, ldap, snmp, http.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.1.150:7001 for protocols iiop, t3, ldap, snmp, http.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>
<Feb 22, 2013 2:25:58 PM CET> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>

The reason for this test is that certain processes write to a file at whichever offset it were last, i.e., logrotate truncates the file (the size is zero), the process writes to the file again, continuing at the offset it left off. Let us demostrate this behavior by using the following example

#!/bin/sh
# coherence options
COHERENCE_OPTIONS="-Dtangosol.coherence.mode=prod -Dtangosol.coherence.cacheconfig=security-cache-config.xml -Dtangosol.coherence.distributed.localstorage=false"
export COHERENCE_OPTIONS

MEASUREMENT_OPTIONS="-verbosegc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/home/weblogic/temp/gc.out"
export MEASUREMENT_OPTIONS

JAVA_HOME="/home/weblogic/jdk1.7.0_11"
export JAVA_HOME

MEM_ARGS="-server -Xms1024m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewRatio=2 -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=80 -XX:G1HeapRegionSize=2048k -XX:+UseTLAB -XX:LargePageSizeInBytes=2048k -XX:+UseLargePages"

CLASSPATH="/home/weblogic/weblogic12.1.1/installation/coherence_3.7/lib/coherence.jar:/home/weblogic/weblogic12.1.1/configuration/applications/base_domain/security/test.jar"
export CLASSPATH

# start the test
${JAVA_HOME}/bin/java ${MEM_ARGS} ${COHERENCE_OPTIONS} ${MEASUREMENT_OPTIONS} model.test.Test

In the example, we log the verbose garbage collection output of the G1 Collector. We add the following logrotate configuration to /etc/logrotate.conf

/home/weblogic/temp/gc.out {
    missingok
    nodateext
    copytruncate
    rotate 1
    size 100k
}

Let it run for a while and when the file reaches 100 kilobytes run the following command

[root@middleware-magic weblogic]# logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
...
reading config info for /home/weblogic/temp/gc.out
...
rotating pattern: /home/weblogic/temp/gc.out  102400 bytes (1 rotations)
empty log files are rotated, old logs are removed
considering log /home/weblogic/temp/gc.out
  log needs rotating
rotating log /home/weblogic/temp/gc.out, log->rotateCount is 1
dateext suffix '-20130226'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /home/weblogic/temp/gc.out.1 to /home/weblogic/temp/gc.out.2 (rotatecount 1, logstart 1, i 1),
old log /home/weblogic/temp/gc.out.1 does not exist
renaming /home/weblogic/temp/gc.out.0 to /home/weblogic/temp/gc.out.1 (rotatecount 1, logstart 1, i 0),
old log /home/weblogic/temp/gc.out.0 does not exist
log /home/weblogic/temp/gc.out.2 doesn't exist -- won't try to dispose of it
copying /home/weblogic/temp/gc.out to /home/weblogic/temp/gc.out.1
set default create context
truncating /home/weblogic/temp/gc.out

To see what happens to the gc.out we can use od -c gc.out | heap -n 10

[weblogic@middleware-magic temp]$ od -c gc.out | head -n 10
0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
0363540  \0       [   T   i   m   e   s   :       u   s   e   r   =   0
0363560   .   3   8       s   y   s   =   0   .   0   0   ,       r   e
0363600   a   l   =   0   .   3   4       s   e   c   s   ]      \n   T
0363620   o   t   a   l       t   i   m   e       f   o   r       w   h
0363640   i   c   h       a   p   p   l   i   c   a   t   i   o   n
0363660   t   h   r   e   a   d   s       w   e   r   e       s   t   o
0363700   p   p   e   d   :       0   .   3   4   0   0   1   1   0
0363720   s   e   c   o   n   d   s  \n   2   3   0   .   3   1   4   :

What this says is from offset 0 to 363540 the file consists of null bytes, and then some legible data. Getting to this state does not take the same amount of time it would take to actually write all those null-bytes. The ext{2,3,4} filesystems support so-called sparse files, so if we seek past a region of a file that does not contain anything, that region will be assumed to contain null-bytes and will not take up space on disk. Those null bytes will not actually be written, just assumed to be there. Note that when this happens the file is not useful anymore, so it is important to first test if the process can handle log rotation.

To configure the WebLogic logging by using WLST we can use something like the following

import socket;

base_dir = '/var/log/weblogic/domains';

admin_server_listen_address = socket.gethostname();
admin_server_url = 't3://' + admin_server_listen_address + ':' + admin_server_listen_port;

print 'CONNECT TO ADMIN SERVER';

connect(admin_username, admin_password, admin_server_url);

print 'CHANGE TO DOMAIN CONFIG';
domainConfig();

print 'START EDIT MODE';
edit();
startEdit();

print 'CHANGE DOMAIN AUDITING AND BACKUP SETTINGS';
cmo.setConfigurationAuditType('log');
cmo.setConfigBackupEnabled(java.lang.Boolean('true'));
cmo.setArchiveConfigurationCount(10);

print 'CHANGE DOMAIN LOG SETTINGS';
domain_name = cmo.getName();
domain_log = cmo.getLog();
domain_log.setFileName(base_dir + '/' + domain_name + '/' + domain_name + '.log');
domain_log.setRotationType('byTime');
domain_log.setFileTimeSpan(24);
domain_log.setRotationTime('00:00');
domain_log.setNumberOfFilesLimited(java.lang.Boolean('true'));
domain_log.setFileCount(10);

print 'CHANGE SERVER LOG SETTINGS';
servers = cmo.getServers();
for server in servers:
	server_name = server.getName();
	if server_name == 'AdminServer':
		server_log = server.getLog();
		server_log.setFileName(base_dir + '/' + domain_name + '/' + server_name + '/' + server_name + '.log');
		server_log.setRotationType('byTime');
		server_log.setFileTimeSpan(24);
		server_log.setRotationTime('00:00');
		server_log.setNumberOfFilesLimited(java.lang.Boolean('true'));
		server_log.setFileCount(10);
		server_log.setRedirectStderrToServerLogEnabled(java.lang.Boolean('true'));
		server_log.setRedirectStdoutToServerLogEnabled(java.lang.Boolean('true'));
		server_log.setLogFileSeverity('Warning');
		server_log.setStdoutSeverity('Warning');
		server_log.setDomainLogBroadcastSeverity('Warning');
		server_log.setMemoryBufferSeverity('Warning');
		web_server_log = server.getWebServer().getWebServerLog();
		web_server_log.setLoggingEnabled(java.lang.Boolean('false'));

print 'SAVE AND ACTIVATE CHANGES';
save();
activate(block='true');

One thing to note is that the .out files are still written to the domain-home/server/server-name/logs directory. Also by setting the -Dweblogic.Stdout and -Dweblogic.Stderr, for example,

if [ "${SERVER_NAME}" = "AdminServer" ] ; then

	if [ "${JAVA_VENDOR}" = "Sun" ] ; then
		# Memory arguments for JDK
		# USER_MEM_ARGS="-server -Xms512m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewRatio=2 -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -XX:+UseG1GC -XX:ParallelGCThreads=2 -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=80 -XX:G1HeapRegionSize=2048k -XX:+UseTLAB -XX:LargePageSizeInBytes=2048k -XX:+UseLargePages"
		USER_MEM_ARGS="-server -Xms512m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewRatio=2 -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -XX:+UseParallelGC -XX:MaxGCPauseMillis=200 -XX:+UseParallelOldGC"
		export USER_MEM_ARGS
	fi

	if [ "${JAVA_VENDOR}" = "Oracle" ] ; then
		# JMX_OPTIONS="-Djava.rmi.server.hostname=192.168.1.150 -Xmanagement:ssl=false,authenticate=false,port=7091,autodiscovery=true"
		# export JMX_OPTIONS

		LOGGING_OPTIONS="-Dweblogic.Stdout=/var/log/weblogic/domains/base_domain/AdminServer/AdminServer.out -Dweblogic.Stderr=/var/log/weblogic/domains/base_domain/AdminServer/AdminServer.out"
		export LOGGING_OPTIONS

		# Memory arguments for JRockit
		USER_MEM_ARGS="-jrockit -Xms512m -Xmx512m -Xns128m -XXkeepAreaRatio:25 -Xgc:pausetime -XpauseTarget:200ms ${LOGGING_OPTIONS}"
		export USER_MEM_ARGS
	fi
fi

an .out is created, specified by the -Dweblogic.Stdout option, but an .out file is also created (and written to) in the domain-home/server/server-name/logs directory. Note also the documentation states: “For each server instance that it controls, Node Manager maintains a log file that contains stdout and stderr messages generated by the server instance.” This means the node manager maintains the log file and not the server itself, so setting the -Dweblogic.Stdout on a server does not help and can be left out.

References

[1] Configuring Log Files and Filtering Log Messages.