Tag: debugging

High CPU Utilization Finding Cause?

Hi All,

Jay SenSharma

Jay SenSharma

High CPU utilization is a very very common thing for an Application Server Administrator. Almost every week or quarter or may be every day we can see some High CPU utilization by the Application Server. It is very important for an Administrator or System Admin or a Application Server Support Person to find out what is causing the High CPU.

.
High CPU may cause System Crash or Server Crash or Slow Responsiveness of Server. So we need to find out which Thread is actually consuming more CPU cycles this will help us to find out whether the Thread is processing some application code ….there may be a code bug or it may be Application Framework bug as well…etc

.
So here we are going to see a very simple demonstration of generating High CPU and then we will see How to Analyze High CPU utilization. For this purpose we are going to use 2 very basic utilities

.
1). jps: to know How to use JPS utility please refer to : http://middlewaremagic.com/weblogic/?p=2291
2). top: This is an Operating System Utility .. Please refer to “man” (Manual of your OS like Linux/Solaris)
3). jstack: to know how to use jstack please refer to : http://middlewaremagic.com/weblogic/?p=2281

Generating High CPU

Step1). Create a Simple Web Application with the following JSP code:
“HighCpuGenerator.jsp”


<%
for (int i=0; i < 3; i++)
    {
       Thread x=new Thread(new Runnable(){

                  public void run()
                    {
                       System.out.println("Thread " +Thread.currentThread().getName() + " started");
                       double val=10;
                       for (;;)
                         {
                            Math.atan(Math.sqrt(Math.pow(val, 10)));
                         }
                     }
              });
 
        x.start();
    }
%>

<h3>High CPU generation ....Please collect the 4-5 times "Top" command Output and the 4-5 ThreadDumps.</h3>

Step2). Now Deploy your Application which is having this JSP.
Step3). Hit the Application using “http://localhost:7001/HighCpuWebApp/HighCpuGenerator.jsp

Analyzing High CPU:

Step4). Open a shell prompt and then get the Servers Process ID using “jps” utility which comes with Jdk1.5 and Higher inside the bin directory.

[jsenshar@jsenshar ~]$ export PATH=/home/jsenshar/MyJdks/jdk1.6.0_21/bin:$PATH
[jsenshar@jsenshar ~]$ jps -v

HighCpu_Collect5ing Process ID using JPS

HighCpu_Collect5ing Process ID using JPS

Step5). Now In the same command prompt run the “JStack” utility to collect some Thread Dumps in a File… Suppose if the Application Servers Process ID is “5398” which we got in previous step.

[jsenshar@jsenshar ~]$ jstack -l 5398 > ThreadDumps.log
[jsenshar@jsenshar ~]$ jstack -l 5398 >> ThreadDumps.log
[jsenshar@jsenshar ~]$ jstack -l 5398 >> ThreadDumps.log
[jsenshar@jsenshar ~]$ jstack -l 5398 >> ThreadDumps.log
[jsenshar@jsenshar ~]$ jstack -l 5398 >> ThreadDumps.log
HighCpu_ThreadDumps_JStack

HighCpu_ThreadDumps_JStack

Step6). Now In Parallel to collecting the Thread Dumps please collect the “top” commands output as well….like
(NOTE: Make sure that u collect the Thread Dumps and the Top commands results in Parallel whenever u observe any kind of High CPU utilization or Server Slow response)

[jsenshar@jsenshar HighCpu]$ top -H -b -p 5398

HighCPU_Top_Report

HighCPU_Top_Report

You will see some results like this :

top - 02:52:09 up 18:08,  4 users,  load average: 3.49, 3.25, 8.02
Tasks:  51 total,   6 running,  45 sleeping,   0 stopped,   0 zombie
Cpu(s): 96.6%us,  0.7%sy,  0.0%ni,  2.6%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   3848964k total,  3352224k used,   496740k free,    50940k buffers
Swap:  4194296k total,   400804k used,  3793492k free,  1116028k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5522 jsenshar  20   0 2884m 412m  19m R 77.8 11.0  40:00.46 java
 5521 jsenshar  20   0 2884m 412m  19m R 66.9 11.0  39:57.34 java
 6900 jsenshar  20   0 2884m 412m  19m R 65.2 11.0   0:07.82 java
 5520 jsenshar  20   0 2884m 412m  19m R 54.2 11.0  39:57.70 java

Step7). Now U will observe that there are some Thread IDs which are consuming more CPU Cycles like Child Thread PID=5522 is consuming 77.8% of CPU similarly Child Thread PID 5521 is consuming 66.9 % of CPU Cycles. These are very Hign CPU utilization Threads inside the Main Parent Process ID 5398.

Step8). Now convert those PIDs into Hexadecimal values…..Example     the Hex Decimal Value for     5522 will be      1592. So now please open the Thread Dump and then find out the word 1592

Decimal Value of the Thread ID provided by OS

Decimal Value of the Thread ID provided by OS

Converted the Decimal Thread ID Value to Hexa Decimal Code

Converted the Decimal Thread ID Value to Hexa Decimal Code

"Thread-17" daemon prio=10 tid=0x00007fe824439800 nid=0x1592 runnable [0x00007fe7fa9e2000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StrictMath.atan(Native Method)
        at java.lang.Math.atan(Math.java:187)
        at jsp_servlet.__highcpugenerator$1.run(__highcpugenerator.java:83)
        at java.lang.Thread.run(Thread.java:619)

Here you will see that The Thread with id=0x1592 is actually causing the High CPU …Now u can tell your Developers to check the Stack Trace of this Thread so that they can correct their Application Code or If you see any API Code or Framework code there then you know what u need to do and where the problem is.

Like in above case u can see that  My JSPs code is actually causing High CPU    “at jsp_servlet.__highcpugenerator$1.run(__highcpugenerator.java:83)

.

.

Thanks

Jay SenSharma


Multicast Errors in The Server Logs

Hi All,
Jay SenSharma

Jay SenSharma

Multicasting is the technology of delivering a message to a group of destination nodes (servers/jvms/any other network based software) Means broadcasting the piece of information to all it’s peers. Usually the IP Address range for multicast communication is any IP Address in the range of 224.0.0.0 to 239.255.255.255.
.
Application Server Clustering is one of the best implementation of Multicasting technology. In Middleware world Cluster is a logical entity in which many Member (Servers) will work together to provide LoadBalancing (Load Sharing), Failover (Reliablity) and Scalability to our applications.
.
Cluster Members usually communicates with each othr using following two ways:

1). IP Multicasting (One to Many):

In This technique every node of a cluster broadcast some piece of data/information to all the other members of the same Cluster. Using this techniqueue Servers achieves 2 main goals….
a). Each nodes sends the heartbeat messages to other nodes of the cluster. This makes other node of the cluster aware that the member whoever is sending messages is alive. The heartbeat message broadcasting helps the cluster master to maintain the “Dynamic Server List” (A List of servers who all are alive).
.
b). IP Multicasting techniqueue is also used for the JNDI objects replication among all the members of the cluster. The Object binded in the JNDI tree of a Clustered Node (Server) is broadcasted to rest of the members of the Cluster. It means the JNDI tree of a Clustered Server will be identicle to other members of the same cluster.

2). IP Socketing (One to One):

This technique is broadly used by the middleware cluster members for accessing the object from any other node of the cluster. This technique is actually used by the Cluster members to replicate the HttpSession Data or the EJB Session Objects.

Multicast Errors:

If we observe the multicast errors in the Server Logs …then it means our Cluster is not going to work as expected…one or more node of the clusters may be kicked out of the cluster….The errors will look something like this in the Server Logs:
java.io.OptionalDataException
<Error> <Cluster> <BEA-000110> <Multicast socket receive error: java.io.OptionalDataException
java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1285)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:322)
at weblogic.cluster.MulticastManager.execute(MulticastManager.java:411)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
java.lang.OutOfMemoryError: PermGen space
<Error> <Cluster> <testWeb> <MS1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <1264189488263> <BEA-000110> <Multicast socket receive error: java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:95)
at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:313)
at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1299)
at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:52)
java.io.StreamCorruptedException
<Error> <Cluster> <testDomain> <testServer> <ExecuteThread: '14' for queue: 'weblogic.kernel.Default'> <<WLS Kernel>> <BEA-000110> <Multicast socket receive error: java.io.StreamCorruptedException
java.io.StreamCorruptedException
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2347)
at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2380)
at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2452)
at java.io.DataInputStream.readInt(DataInputStream.java:443)
at java.io.ObjectInputStream$BlockDataInputStream.readInt(ObjectInputStream.java:2657)
at java.io.ObjectInputStream.readInt(ObjectInputStream.java:900)
at weblogic.cluster.MulticastManager.execute(MulticastManager.java:387)
java.io.EOFException
<Error> <Cluster> <BEA-000110> <Multicast socket receive error: java.io.EOFException
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:380)
at java.io.ObjectInputStream$BlockDataInputStream.readLong()J(Unknown Source)
at java.io.ObjectInputStream.readLong()J(Unknown Source)
at weblogic.cluster.HeartbeatMessage.readExternal(HeartbeatMessage.java:55)
at java.io.ObjectInputStream.readExternalData(Ljava.io.Externalizable;Ljava.io.ObjectStreamClass;)V(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Z)Ljava.lang.Object;(Unknown Source)
java.io.InterruptedIOException
<Error> <Cluster> <Multicast socket receive error : java.io.InterruptedIOException: Receive timed out
java.io.InterruptedIOException: Receive timed out
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:90)
at java.net.DatagramSocket.receive(DatagramSocket.java:404)
at weblogic.cluster.FragmentSocket.receive(FragmentSocket.java:145)
at weblogic.cluster.MulticastManager.execute(MulticastManager.java:298)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

Most Possible Causes of the Multicast Errors ?

Cause-1). Incorrect configuration of Multicast Addresses.
.
Cause-2). Less Number of File Descriptors Availability
.
Cause-3). Network Fluctuation/Interrupted Network Connectivity.
.
Cause-4). Multicast Blocking due to some Firewall restrictions. Try disabling the “iptables” as described at the bottom of the page.
.
Cause-5). Multicast Timeouts. Increase the MulticastTTL.
.
Cause-6). More than one Clusters present in the same Network using the same Multicast address.
.
Cause-7). In correct usages of Operating System Zoning or Multihoming Issues. MultiHoming means a Single Physical Box with multiple NIC Cards (Multiple IP Addresses)
.
Cause-8). Servers Listen Port was used as the Multicast Port.

What All Things Need to Debug ?

It is Always recommended to first of all go through the following link: http://download.oracle.com/docs/cd/E12840_01/wls/docs103/cluster/multicast_configuration.html

Point-1). Check & Make Sure that all the Clusters present in the same Network are using Uniqueue Multicast Address And Port
.
Point-2). Using “netstat” or “ping” commands we need to make sure that the Multicast Address and the Port are Ok for use.
.
Point-3). Opening a Socket or a File requires File Descriptors…So we Need to make sure that the required number of “File Descriptors” are available or not?
In Unix based OS we can use “lsof”  command (List Of Open Files)
Example :   “lsof -p <WLS_PID> | wc -l”
Here WLS_PID is the Process ID of WebLogic. To find WebLogic Process ID please refer to:  http://middlewaremagic.com/weblogic/?p=2291
Example: Suppose if the WebLogic Server’s Process ID is 4020 then run the following command:
[jaytest@jaytest bin]$ lsof -p 4020 | wc -l
666

[jaytest@jaytest bin]$ lsof -p 4020
COMMAND  PID     USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
java    4020 jaytest  cwd    DIR              253,4     4096 8657973 /NotBackedUp/WLS103/user_projects/domains/base_domain
java    4020 jaytest  rtd    DIR              253,1     4096       2 /
java    4020 jaytest  txt    REG              253,3    50810  133154 /home/jaytest/MyJdks/jdk1.6.0_21/bin/java
java    4020 jaytest  mem    REG              253,1   150672  542805 /lib64/ld-2.12.so
java    4020 jaytest  mem    REG              253,1  1838296  542806 /lib64/libc-2.12.so
java    4020 jaytest  mem    REG              253,1   145672  542818 /lib64/libpthread-2.12.so
java    4020 jaytest  mem    REG              253,1    22536  542808 /lib64/libdl-2.12.so
java    4020 jaytest  mem    REG              253,1   598816  542807 /lib64/libm-2.12.so
java    4020 jaytest  mem    REG              253,1    47072  542819 /lib64/librt-2.12.so
java    4020 jaytest  mem    REG              253,1   113904  542814 /lib64/libresolv-2.12.so
java    4020 jaytest  mem    REG              253,1   116136  542840 /lib64/libnsl-2.12.so
java    4020 jaytest  mem    REG              253,3     6676  133795 /home/jaytest/MyJdks/jdk1.6.0_21/jre/lib/amd64/librmi.so
java    4020 jaytest  mem    REG              253,3  1163700  133902 /home/jaytest/MyJdks/jdk1.6.0_21/jre/lib/resources.jar
java    4020 jaytest  mem    REG              253,3   842216  134434 /home/jaytest/MyJdks/jdk1.6.0_21/jre/lib/ext/localedata.jar
java    4020 jaytest  mem    REG              253,4   282279 8653766 /NotBackedUp/WLS103/wlserver_10.3/server/lib/consoleapp/consolehelp/WEB-INF/lib/console.jar
java    4020 jaytest  mem    REG              253,4   293750 8653774 /NotBackedUp/WLS103/wlserver_10.3/server/lib/consoleapp/consolehelp/WEB-INF/lib/standard.jar
java    4020 jaytest  mem    REG              253,4   779658 8653764 /NotBackedUp/WLS103/wlserver_10.3/server/lib/consoleapp/consolehelp/WEB-INF/lib/beehive-netui-core.jar
java    4020 jaytest  mem    REG              253,4    57299 8653775 /NotBackedUp/WLS103/wlserver_10.3/server/lib/consoleapp/consolehelp/WEB-INF/lib/struts-adapter.jar
java    4020 jaytest  mem    REG              253,4   531676 8653767 /NotBackedUp/WLS103/wlserver_10.3/server/lib/consoleapp/consolehelp/WEB-INF/lib/jh.jar
java    4020 jaytest  mem    REG              253,4  1490143 8653770 /NotBackedUp/WLS103/wlserver_10.3/server/lib/consoleapp/consolehelp/WEB-INF/lib/netuix_servlet.jar
java    4020 jaytest  mem    REG              253,4    54683 8653769 /NotBackedUp/WLS103/wlserver_10.3/server/lib/consoleapp/consolehelp/WEB-INF/lib/netuix_common_web.jar
java    4020 jaytest  mem    REG              253,4    46008 8653772 /NotBackedUp/WLS103/wlserver_10.3/server/lib/consoleapp/consolehelp/WEB-INF/lib/render_taglib.jar
.
Point-4). If we see any kind of “Multicast Receive timeout error” It means  we need to check the NIC card functioning properly or not.
.
Point-5). There May be Many a Multicast Storm going on in the Network (Storm means repeated transmission of the Multicast packets over the network). In this case we can try increasing the Multicast Buffer Size. Using “udp_max_buf” Parameter we can increase it. Please refer to :  http://docs.sun.com/app/docs/doc/816-0607/6m735r5gb?a=view for more details on it.
.
Point-6). In case of Multicast storm the network may be already flooded with the Multicast messages. If we find this then please disable the “igmp” snooping switch. This switch is part of the Internet Group Management Protocol (IGMP) and is used to prevent multicast flood problems on the managed switch.
Example:   igmp snooping=disable
For more details on this parameter Please refer to:  http://documentation.netgear.com/gs108t/enu/202-10337-01/GS108T_UM-06-21.html
.
Point-7). Set the Multicast Time-To-Live to the following:  MulticastTTL=32
NOTE: For WAN kind of larger network the Multicast Time To Live Parameters value must be kept High…sothat the Routers will not discard the Multicast Packets before they reach the Message destination.
.
Point-8). Perform the MulticastMonitor Test & MulticastTest  on the network…As described in the following link: http://middlewaremagic.com/weblogic/?p=980
.
Point-9). Try to enable the Cluster Debug to get more details:
java weblogic.Admin -url t3://localhost:7001 - username weblogic -password weblogic SET -type ServerDebug -property DebugCluster true
Point-10). If the Multicast still doesnt work….:) then disable the IPTables….And then check  http://kr.forums.oracle.com/forums/thread.jspa?threadID=767088
In RedHat Linux:        /etc/init.d/ipdables stop

.

.

Thanks
Jay SenSharma

Causes and First Aid of JVM Crash Issues?

Hi All,
Jay SenSharma

Jay SenSharma

Java Virtual Machine is a Native engine which allows our Java Applications to run. It performs the code optimization to improve the performance. In correct tuning, Low memory allocation, extensive code optimization, bad garbage collection strategy, API code leaking…etc are some of the reasons which may cause the JVM to crash.
.
Analyzing a JVM Crash is one of the very interesting and little time taking process sometimes it is even little complex to find out the root cause of the JVM Crash. Here in this article we will see some of the common mistakes, first aid solutions/debugging techniques to find out what kind of informations we can get by looking into the Core Dump.

What is Core Dump & Where to Find It?

Code dump is usually a Binary file which gets generated by the Operating System when JVM or any other process crashes. Sometimes it also happens that the JVM will not be able to generate the Crash dump. In Windows Operating Systems it will be generated in the Directore where the “Dr. Watson” tool is installed. In Windows it will be usually:  “C:Documents and SettingsAll UsersApplication DataMicrosoftDr Watson
.
By default in Unix based Operating Systems the Core Dump files are created in the directory where the Java Program/Server was started even sometimes it is generated in the “/tmp” directory of the Operating System. But using the following Java Option we can change it’s the Crash Dump/Heap   Dump generation locations:  -XX:HeapDumpPath=/opt/app/someLocaton/ and  -XX:+HeapDump JVM Options.
.
NOTE: These Flags does not gurantee that always the Heap/Crash dump will be generated at the time of JVM Crash. There are some more reasons behind the Core Dump not gets generated…like Process Limitations or the Less Disk Quota or unavailability of the Free File Descriptors.

Who Generates the Crash/Core Dump?

JVM does not generate the Coe dump. Rather it is the Operating System which generates the Core Dump. Core Dump is a Binary file which may be several hundred Mega Bytes or Giga Bytes in size. The Operating systems just log the exception/error messages and the details of the Threads along with the Native libraries loaded with that java process.
.
Many times a brief Textual Crash file is also generated by the JVM itself sometimes during Crash. Usually the file name is “hs_err_pid<WebLogicPID>.log” in case of Sun JDK. Similarly JRockit JVM also generates a Textual file with name “*.dump” in case of JVM Crash.

Use of -XX:+ShowMessageBoxOnError?

The Thread Dump is also very helpful to analyze the Server Crash. Thread dump tells us what was the status and activities performed by the Threads at the time of crash. It may be possible to get a thread dump before the process exits. HotSpot supports the JAVA_OPTIONS -XX:+ShowMessageBoxOnError
.
The corresponding JRockit JVM Option is -Djrockit.waitonerror.  When the JVM is crashing, it may prompt the user ” Do you want to debug the problem? ” This pauses the process, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.  However, this does not work in all cases (for eg., in case of stack overflow).
.
Along with above there are various options available to get the Thread Dumps as described in : http://middlewaremagic.com/weblogic/?p=823

What May Cause JVM Crash?

Reason-1). Usually Native Code causes the JVM Crash. Native code is a code written in Languages like C/C++, Java Native Interface APIs (JNI).
.
Reason-2). JDBC Drivers specially the Native Drivers.
.
Reason-3). JVM Code Optimization.
.
Reason-4). Less Memory availabity for Native Area of a Java Process.
.
Reason-5). Application Servers Native Performance Pack Libraries.
.
Reason-6). JVMs library itself can cause the Crash.
.
Reason-7). High CPU Utilization by the Threads. As described in : http://middlewaremagic.com/weblogic/?p=4348
.
Reason-8). Presence of Wrong Native Libraries in the PATH or in the “-Djava.library.path
.
Reason-9). Presence of A Different Version of Libraries in “java.library.path” or in “SHLIB” or in “LD_LIBRARY_PATH” variables. Like setting a 64-bit version of Library in a 32-bit version of JVM’s library path or vise-versa.
.
.

Tools To Analyze the Core Dumps?

Core/Crash dump is Operating System specific, So to analyze these Dumps we must use the Tools provided by the same Operating System vendors. Various kind of tools are provided by the Operating Systems to analyze these Core Dumps like:
Tool-1). Dr. Watson Tool  in Windows OS. Windows OS Start (Button)—>Run…—>drwtsn32
.
Tool-2). “pstack” and “pmap” in Solaris Operating System.
.
Tool-3). “procstack” and “procmap” in AIX Operating System.
.
Tool-4). “lsstack” and “pmap” in Linux Operating System.
.
Tool-5). “pflag” if available in HP-UX Operating System.
.
.

What May Help To Avoid JVM Crash?

It totally depends on What Caused the Crash or What Libraries caused the JVM crash to avoid the occurance of the JVM crash for next time. But following things should be taken in consideration while analyzing and avoiding the Crash.
.
Point-1). If the Native Jdbc Driver is causing issues, If our appliation is using the Native JDBC Drivers then Switching from Pure Native Jdbc Driver (Type-2 Drivers) to the Pure Java JDBC Driver (Type-4 Driver) may help.
.
Point-2). If the JVM Libraries are causing the Crash then Upgrading to a Later Version of the JDK. If that Application Server has that new JDK in it;s Supported Configuration list.
.
Point-3). If the JVM Code Optimization is causing the Crash, then Disabling the Code Optimization of JVM by applying the JVM Options.
For JRockit JVM Code Optimization can be disabled using JAVA_OPTION  -Xnoopt
For Sun JDK Code Optimization can be disabled using   JAVA_OPTION  -Xint
.
Point-4). Some times the “Just In Time Compiler” code generation also causes the JVM Crash. In these scenarios In Case of Sun JDKs disabling the JIT Compiler can help. We can disable the JIT Compiler by adding the JVM Option:  “-Djava.compiler=none
.
Point-5). Presence of a different bit version (32 bit or 64-bit libraries) of Library in the “-Djava.library.path
.
Point-6). Disabling the Application Servers Native Performance Packs. In WebLogic The Native IO Can be disabled using “-Dweblogic.NativeIOEnabled=false” JVM Option.
.
.
Thanks
Jay SenSharma

Copyright © 2010-2012 Middleware Magic. All rights reserved. |