Tag: Stuck Thread

Stuck Thread Alert using WebLogic SNMP Agent

Hi,
Jay SenSharma

Jay SenSharma

Based on a requirement of a  user named Shashi_sr on Oracle Forums we have created this post. Getting Alert is one of the most important  requirement in production environments. We can use many techinques to get alerts regarding the behaviour and scenario of WebLogic Server. WLST, JMX and WLDF are the most common techniques to monitor our WebLogic server resources and the health check of these resources.

Here we are going to see a very simple example of creating  a STUCK THREAD alert using SNMP Log Filters.  So that as soon as any thread will be declared by WebLogic  as STUCK we will immediately get an SNMP Alert.

You can also do the same thing using WLST script have a look at the post –  Sending Email Alert For Stuck Threads With Thread Dumps

Step1). Login to AdminConsole and then nevigate to “Diagnostics————> SNMP”

Step2). Now create an SNMP Agent like following:

Name: StuckThreadSNMPAgent
Enabled: true
Targeted Servers: AdminServer
UDP Port : 161
Community Prefix: public
Send Automatic Traps Enabled : true
Community Based Access Enabled : true
Inform Retry Interval : 10000
Maximum Inform Retry Count : 1
Credential Cache Invalidation Interval : 3600000
LogFilter_StuckThread_1

LogFilter_StuckThread_1

Step3). Now create a Log Filter  By clicking on

“Diagnostics—->SNMP—–>StuckThreadSNMPAgent—–>Configuration (Tab)—–>Log Filters”
like following:
Name: StuckThreadLogFilter
Enabled Servers: AdminServer
Severity Level: Error
Subsystem Names: WebLogicServer
Message IDs : 000337
Message Substring : [STUCK] ExecuteThread
Like following:
LogFilter_StuckThread_2

LogFilter_StuckThread_2

Step4). Now  Create a Trap Destination like following:

“Diagnostics—->SNMP—–>StuckThreadSNMPAgent—–>Configuration (Tab)—–>Trap Destinations”
Name: StuckThreadTrapDest
Community: public
Host: localhost
Port : 165
as following:
LogFilter_StuckThread_3

LogFilter_StuckThread_3

Step5). Now restart your Server  Just to make sure that every thing is OK.

Step6). Now open a command prompt and then run “setWLSEnv.cmd”/ setWLSEnv.sh  then start the

java weblogic.diagnostics.snmp.cmdline.Manager SnmpTrapMonitor -p 165

LogFilter_StuckThread_4

LogFilter_StuckThread_4

As soon as you will get any STUCK THREAD you will get an alert immediately as following:

LogFilter_StuckThread_5

LogFilter_StuckThread_5

.
.
Thank
Jay SenSharma

Sending Email Alert for Threads Pool Health Using WLST

Ravish Mody

This post is written considering to send an email alert message to Admin’s which would alert them about the Thread Pool Health State. All of us know if in Weblogic Health State is as WARNING state then it means something is not going correct in the server and we have to check it. Thus this WLST script would surely help all the Weblogic Administrators who are working in Production Environment.

In below  script has a properties file in which you can give all the details about the domain as well as time interval and the number of times the ratio has to be checked.

.

Steps to Create an Email Alert for Threads Pool Health

Step1) Create a Directory somewhere in your file system like : “C:\WLST”

Step2) Write a Properties file “domains.properties” inside “C:\WLST” like following:

admin.url=t3://localhost:8001
admin.username=weblogic
admin.password=weblogic

# Number of times the RATIO has to be checked
checkTimes_Number=25

# TIME INTERVAL between number of times the RATIO has to be checked (30000 milliseconds = 30 seconds)
checkInterval_in_Milliseconds=30000

############ Accouding to the above values the checker will run for total 25 times in an interval of 30 seconds each. #############

Step-3) Create a WLST Script somewhere in your file system with some name like “Alert_ThreadPoolHealth.py” inside “C:\WLST” contents will be something like following:

#############################################################################
#
# @author Copyright (c) 2010 - 2011 by Middleware Magic, All Rights Reserved.
#
#############################################################################

from java.io import FileInputStream
import java.lang
import os
import string

propInputStream = FileInputStream("domains.properties")
configProps = Properties()
configProps.load(propInputStream)

serverurl = configProps.get("server.url")
adminUser = configProps.get("admin.username")
adminPassword = configProps.get("admin.password")
checkTimes_Number = configProps.get("checkTimes_Number")
checkInterval_in_Milliseconds = configProps.get("checkInterval_in_Milliseconds")

i = 0
y = int(checkTimes_Number)

#############  This method would send the Alert Email  #################
def sendMailString():
	os.system('/bin/mailx -s  "ALERT: Thread Pool Health is in WARNING !!! " ABCD@COMPANY.com < rw_file')
	print '*********  ALERT MAIL HAS BEEN SENT  ***********'
	print ''

#############  This method is checking the Thread Pool Health   #################
def alertThreadPoolHealth(healthState):
	state ="Checking HealthState: " + healthState
	check = string.find(state,"HEALTH_WARN")
	if check != -1:
		print '!!!! ALERT !!!! Thread Pool Health is in WARNING State'
		print ''
		message =  'Please Check the Thread Pool is in WARNING State.'
		cmd = "echo " + message +" > rw_file"
		os.system(cmd)
		sendMailString()
	else:
		print 'Everything is working fine till now'

connect(adminUser,adminPassword,serverurl)
serverRuntime()
cd('ThreadPoolRuntime/ThreadPoolRuntime')

while (i < y):
	ls()
	healthState=cmo.getHealthState();
	alertThreadPoolHealth(str(healthState));
	Thread.sleep(int(checkInterval_in_Milliseconds))
	i = i + 1

Step-4) Open a command prompt and then run the “setWLSEnv.cmd” or “setWLSEnv.sh” to set the CLASSPATH and PATH variables. Better you do echo %CLASSPATH% or echo $CLASSPATH to see whether the CLASSPATH is set properly or not. If you see an Empty Classpath even after running the “setWLSEnv.sh” then please refer to the Note mentioned at Step3) in the Following post: http://middlewaremagic.com/weblogic/?page_id=1492

Step-5) Now run the WLST Script in the same command prompt like following:

java   weblogic.WLST  Alert_ThreadPoolHealth.py

You will see the following kind of results in the command prompt

java weblogic.WLST Alert_ThreadPoolHealth.py

Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands

Connecting to t3://localhost:8001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'Domain_8001'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Location changed to serverRuntime tree. This is a read-only tree with ServerRuntimeMBean as the root.
For more help, use help(serverRuntime)
-r--   CompletedRequestCount                        3714
-r--   ExecuteThreadIdleCount                       0
-r--   ExecuteThreadTotalCount                      6
-r--   ExecuteThreads                               weblogic.work.ExecuteThreadRuntime[weblogic.work.ExecuteThreadRuntime@39c4e135, weblogic.work.ExecuteThreadRuntime@2c9d7c34, weblogic.work.ExecuteThreadRuntime@174550ce, weblogic.work.ExecuteThreadRuntime@47bc1051, weblogic.work.ExecuteThreadRuntime@bd35aa2, weblogic.work.ExecuteThreadRuntime@60e347be]
-r--   HealthState                                  Component:threadpool,State:HEALTH_WARN,MBean:ThreadPoolRuntime,ReasonCode:[ThreadPool has stuck threads]
-r--   HoggingThreadCount                           2
-r--   MinThreadsConstraintsCompleted               94
-r--   MinThreadsConstraintsPending                 0
-r--   Name                                         ThreadPoolRuntime
-r--   PendingUserRequestCount                      0
-r--   QueueLength                                  0
-r--   SharedCapacityForWorkManagers                65536
-r--   StandbyThreadCount                           3
-r--   Suspended                                    false
-r--   Throughput                                   9.495252373813093
-r--   Type                                         ThreadPoolRuntime
-r-x   preDeregister                                Void :

!!!! ALERT !!!! Thread Pool Health is in WARNING State

*********  ALERT MAIL HAS BEEN SENT  ***********

NOTE: This script is using mailx (i.e. but Windows box does not have mailx utility) so please do check if your mailx is configured properly or else script would run properly but the mail would not be sent.

Alert Email

Alert Email

Regards,

Ravish Mody


Sending Email Alert for Hogger Threads Count Using WLST

Ravish Mody

This post is written considering to send an email alert message to Admin’s which would alert them about the hogger threads. Hogging threads can be called as a candidates for stuck threads in other words, those threads that “might” get stuck are called hogging threads. These threads will be declared as stuck threads after “StuckThreadMaxTimeout” seconds which by default value is 600secs.

Most of you guys might agree with me, that no one wants stuck threads in there production environment. Hence this script would surely help you guys to take an relevant actions once you get an alert about the hogging threads have exceeded the given ration.
In below  script we have used a properties file in which you can give all the details about the domain as well as hogger thread ration, time interval and the number of times the ratio has to be checked.

Steps to Create an Email Alert for Hogger Threads Count

Step1) Create a Directory somewhere in your file system like :  “C:\WLST”

Step2) Write a Properties file “domains.properties” inside “C:\WLST” like following:

admin.url=t3://localhost:7001
admin.username=weblogic
admin.password=weblogic

# This ExecuteThread_Vs_HoggerThreadRatio represtents the division of ExecuteThread/HoggerThreadRatio
ExecuteThread_Vs_HoggerThreadRatio=2

# Number of times the RATIO has to be checked
checkTimes_Number=25

# TIME INTERVAL between number of times the RATIO has to be checked (30000 milliseconds = 30 seconds)
checkInterval_in_Milliseconds=30000

############ Accouding to the above values the checker will run for total 25 times in an interval of 30 seconds each. #############

Step-3) Create a WLST Script somewhere in your file system with some name like “Alert_HoggerThreadCoung.py” inside “C:\WLST” contents will be something like following:

UPDATED [26-04-2011]: This script has been updated based on the comment — >  http://middlewaremagic.com/weblogic/?p=5423#comment-3760, thanks to sathya.

#############################################################################
#
# @author Copyright (c) 2010 - 2011 by Middleware Magic, All Rights Reserved.
#
#############################################################################

from java.io import FileInputStream
import java.lang
import os

propInputStream = FileInputStream("domains.properties")
configProps = Properties()
configProps.load(propInputStream)

adminUrl = configProps.get("admin.url")
adminUser = configProps.get("admin.username")
adminPassword = configProps.get("admin.password")
executeThread_Vs_HoggerThreadRatio = configProps.get("ExecuteThread_Vs_HoggerThreadRatio")
checkTimes_Number = configProps.get("checkTimes_Number")
checkInterval_in_Milliseconds = configProps.get("checkInterval_in_Milliseconds")

i = 0
y = int(checkTimes_Number)

#############  This method would send the Alert Email  #################
def sendMailString():
	os.system('/bin/mailx -s  "ALERT: Hogger Thread Count Exceeded the Limt !!! " abcd@company.com < rw_file')
	print '*********  ALERT MAIL HAS BEEN SENT  ***********'
	print ''

#############  This method is checking the Hogger Threads Ratio  #################
def alertHoggerThreads(executeTTC , hoggerTC):
	print 'Execute Threads : ', executeTTC
	print 'Hogger Thread Count : ', hoggerTC
	if hoggerTC != 0:
		ratio=(executeTTC/hoggerTC)
		print 'Ratio : ' , ratio
		print ''
		if (int(ratio) <= int(executeThread_Vs_HoggerThreadRatio)):
			print ' !!!! ALERT !!!! Stuck Threads are on its way.....'
			print ''
			message =  'ExecuteThreads Count= ' + str(executeTTC) + '   HoggingThreads= '+ str(hoggerTC) +'   ExecuteThreads/HoggingThreads Ratio= '+ str(ratio)
			cmd = "echo " + message +" > rw_file"
			os.system(cmd)
			sendMailString()
		else:
			print '++++++++++++++++++++++++++++++++++++'
			print 'Everything is working fine till now'
			print '++++++++++++++++++++++++++++++++++++'
	else:
			print '++++++++++++++++++++++++++++++++++++'
			print 'Everything is working fine till now'
			print '++++++++++++++++++++++++++++++++++++'

connect(adminUser,adminPassword,adminUrl)
serverRuntime()
cd('ThreadPoolRuntime/ThreadPoolRuntime')

while (i < y):
	ls()
	executeTTC=cmo.getExecuteThreadTotalCount();
	hoggerTC=cmo.getHoggingThreadCount();
	alertHoggerThreads(executeTTC , hoggerTC)
	print 'Sleeping for ', int(checkInterval_in_Milliseconds) , ' ...'
	print ''
	Thread.sleep(int(checkInterval_in_Milliseconds))
	i = i + 1

Step-4) Open a command prompt and then run the “setWLSEnv.cmd” or “setWLSEnv.sh” to set the CLASSPATH and PATH variables. Better you do echo %CLASSPATH% or echo $CLASSPATH to see whether the CLASSPATH is set properly or not. If you see an Empty Classpath even after running the “setWLSEnv.sh” then please refer to the Note mentioned at Step3) in the Following post: http://middlewaremagic.com/weblogic/?page_id=1492

Step-5) Now run the WLST Script in the same command prompt using the following command:

java   weblogic.WLST  Alert_HoggerThreadCoung.py

You will see the following kind of results in the command prompt

Initializing WebLogic Scripting Tool (WLST) ...
]$ java weblogic.WLST Alert_HoggerThreadCoung.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'Domain_7001'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Location changed to serverRuntime tree. This is a read-only tree with ServerRuntimeMBean as the root.
For more help, use help(serverRuntime)

-r--   CompletedRequestCount                        2606
-r--   ExecuteThreadIdleCount                       1
-r--   ExecuteThreadTotalCount                      6
-r--   ExecuteThreads                               weblogic.work.ExecuteThreadRuntime[weblogic.work.ExecuteThreadRuntime@78de59f8, weblogic.work.ExecuteThreadRuntime@4de4e6c6, weblogic.work.ExecuteThreadRuntime@6eeaf91d, weblogic.work.ExecuteThreadRuntime@48917cf, weblogic.work.ExecuteThreadRuntime@447a195c, weblogic.work.ExecuteThreadRuntime@2c170a23]
-r--   HealthState                                  Component:threadpool,State:HEALTH_OK,MBean:ThreadPoolRuntime,ReasonCode:[]
-r--   HoggingThreadCount                           0
-r--   MinThreadsConstraintsCompleted               104
-r--   MinThreadsConstraintsPending                 0
-r--   Name                                         ThreadPoolRuntime
-r--   PendingUserRequestCount                      0
-r--   QueueLength                                  0
-r--   SharedCapacityForWorkManagers                65536
-r--   StandbyThreadCount                           4
-r--   Suspended                                    false
-r--   Throughput                                   5.0
-r--   Type                                         ThreadPoolRuntime

-r-x   preDeregister                                Void :

Execute Threads :  6
Hogger Thread Count :  0
++++++++++++++++++++++++++++++++++++
Everything is working fine till now
++++++++++++++++++++++++++++++++++++
Sleeping for  30000  ...

NOTE: This script is using mailx (i.e. but Windows box does not have mailx utility) so please do check if your mailx is configured properly or else script would run properly but the mail would not be sent.

Alert Email

Alert Email

.

Regards,

Ravish Mody


  • Testimonials

  • RSS Middleware Magic – JBoss

  • Receive FREE Updates


    FREE Email updates of our new posts Enter your email address:



  • Magic Archives

  • Sitemeter Status

  • ClusterMap 7-Nov-2011 till Date

  • ClusterMap 6-Nov-2010 till 7-Nov-2011

  • Copyright © 2010-2012 Middleware Magic. All rights reserved. | Disclaimer