Debug

It’s always best to enable the Garbage collection Logging in our production environment as well because it does not cause any resource overhead or any side effect on weblogic server or an other application server’s performance. GC log helps us in investigating man issues. Apart from issues it helps us to find out if some tuning is required based on the statistics of the Garbage collection.

.

Garbage collection logging can be enable and collected in a separate log file by using the following JAVA_OPTIONS:

As soon as you add these JAVA_OPTIONS which are JVM specific (above will work for Sun and Open JDKs fine) the JVM will start generating the garbage collection logging in the GCLog.log file. Now if you will open this file then you can

Point-1). [Full GC (System) [PSYoungGen: 38388K->0K(268800K)] It means a Full GC is happening on the complete Heap Area including all the Areas of the Java Heap Space.

.

Point-2). [GC [PSYoungGen: 230400K->19135K(268800K)] Indicates some small GCs which keep on happening in the young generation very frequently,This garbage collection cleans the Young Generation short living Objects.

.

Point-3). Meaning of the [GC [PSYoungGen: 230400K->19135K(268800K)] line is around 256MB (268800K) is the Young Generation Size, Before Garbage Collection in young generation the heap utilization in Young Generation area was around 255MB (230400K) and after garbage collection it reduced up to 18MB (19135K)

.

Point-4). Same thing we can see for Full Garbage collection as well….How effective the Garbage collection was…[Full GC (System) [PSYoungGen: 38388K->0K(268800K)] [PSOldGen: 13545K->51794K(1789952K)] Here it says that around

[(old)1789952K + young (268800K) ] memory space means OldGeneration is consuming 1.75GB space and Young Generation is consuming around 255 MB space So it means total Heap size is around 2GB.

.

But analyzing the Garbage collection log like above technique Line by Line is very bad…so here we have an alternative was to analyze the Garbage Collection log in few Seconds to see how much time the Full Garbage collection is taking as an average and other reports…etc.

High CPU utilization is a very very common thing for an Application Server Administrator. Almost every week or quarter or may be every day we can see some High CPU utilization by the Application Server. It is very important for an Administrator or System Admin or a Application Server Support Person to find out what is causing the High CPU.

.
High CPU may cause System Crash or Server Crash or Slow Responsiveness of Server. So we need to find out which Thread is actually consuming more CPU cycles this will help us to find out whether the Thread is processing some application code ….there may be a code bug or it may be Application Framework bug as well…etc

.
So here we are going to see a very simple demonstration of generating High CPU and then we will see How to Analyze High CPU utilization. For this purpose we are going to use 2 very basic utilities

Step5). Now In the same command prompt run the “JStack” utility to collect some Thread Dumps in a File… Suppose if the Application Servers Process ID is “5398” which we got in previous step.

[jsenshar@jsenshar ~]$ jstack -l 5398 > ThreadDumps.log

[jsenshar@jsenshar ~]$ jstack -l 5398 >> ThreadDumps.log

[jsenshar@jsenshar ~]$ jstack -l 5398 >> ThreadDumps.log

[jsenshar@jsenshar ~]$ jstack -l 5398 >> ThreadDumps.log

[jsenshar@jsenshar ~]$ jstack -l 5398 >> ThreadDumps.log

HighCpu_ThreadDumps_JStack

Step6). Now In Parallel to collecting the Thread Dumps please collect the “top” commands output as well….like
(NOTE: Make sure that u collect the Thread Dumps and the Top commands results in Parallel whenever u observe any kind of High CPU utilization or Server Slow response)

Step7). Now U will observe that there are some Thread IDs which are consuming more CPU Cycles like Child Thread PID=5522 is consuming 77.8% of CPU similarly Child Thread PID 5521 is consuming 66.9 % of CPU Cycles. These are very Hign CPU utilization Threads inside the Main Parent Process ID 5398.

Step8). Now convert those PIDs into Hexadecimal values…..Example the Hex Decimal Value for 5522 will be 1592. So now please open the Thread Dump and then find out the word 1592

Here you will see that The Thread with id=0x1592 is actually causing the High CPU …Now u can tell your Developers to check the Stack Trace of this Thread so that they can correct their Application Code or If you see any API Code or Framework code there then you know what u need to do and where the problem is.

Like in above case u can see that My JSPs code is actually causing High CPU “at jsp_servlet.__highcpugenerator$1.run(__highcpugenerator.java:83)”

Multicasting is the technology of delivering a message to a group of destination nodes (servers/jvms/any other network based software) Means broadcasting the piece of information to all it’s peers. Usually the IP Address range for multicast communication is any IP Address in the range of 224.0.0.0 to 239.255.255.255.

.

Application Server Clustering is one of the best implementation of Multicasting technology. In Middleware world Cluster is a logical entity in which many Member (Servers) will work together to provide LoadBalancing (Load Sharing), Failover (Reliablity) and Scalability to our applications.

.

Cluster Members usually communicates with each othr using following two ways:

1). IP Multicasting (One to Many):

In This technique every node of a cluster broadcast some piece of data/information to all the other members of the same Cluster. Using this techniqueue Servers achieves 2 main goals….

a). Each nodes sends the heartbeat messages to other nodes of the cluster. This makes other node of the cluster aware that the member whoever is sending messages is alive. The heartbeat message broadcasting helps the cluster master to maintain the “Dynamic Server List” (A List of servers who all are alive).

.

b). IP Multicasting techniqueue is also used for the JNDI objects replication among all the members of the cluster. The Object binded in the JNDI tree of a Clustered Node (Server) is broadcasted to rest of the members of the Cluster. It means the JNDI tree of a Clustered Server will be identicle to other members of the same cluster.

2). IP Socketing (One to One):

This technique is broadly used by the middleware cluster members for accessing the object from any other node of the cluster. This technique is actually used by the Cluster members to replicate the HttpSession Data or the EJB Session Objects.

Multicast Errors:

If we observe the multicast errors in the Server Logs …then it means our Cluster is not going to work as expected…one or more node of the clusters may be kicked out of the cluster….The errors will look something like this in the Server Logs:

Point-4). If we see any kind of “Multicast Receive timeout error” It means we need to check the NIC card functioning properly or not.

.

Point-5).There May be Many a Multicast Storm going on in the Network (Storm means repeated transmission of the Multicast packets over the network). In this case we can try increasing the Multicast Buffer Size. Using “udp_max_buf” Parameter we can increase it. Please refer to : http://docs.sun.com/app/docs/doc/816-0607/6m735r5gb?a=view for more details on it.

.

Point-6).In case of Multicast storm the network may be already flooded with the Multicast messages. If we find this then please disable the “igmp” snooping switch. This switch is part of the Internet Group Management Protocol (IGMP) and is used to prevent multicast flood problems on the managed switch.