Wednesday, March 27, 2013

Treating bad nutritional choices with drugs—choices that lead to morbidity in later life, after years and years of self-abuse—will never be an efficacious solution. We have to be responsible for our own health and rely on vigilant avoidance of the underlying causes of disease. We need to adopt scientifically supported superior nutrition and rid ourselves of the idea that doctors and pharmaceutical companies are our saviors, capable of enabling us to live long and productive lives.

Western Diets and Western Diseases

Today, the American diet takes over 60% of its calories from processed foods. Processed foods are generally mixed with additives, coloring agents, and preservatives to extend shelf life, and they're placed in plastic bags and cardboard boxes. Americans consume less than 10% of their calories from unrefined plant foods such as fruits, beans, seeds, and vegetables. Even worse, half the vegetable consumption is from white potato products, including fries and chips. Many phytochemicals in freshly harvested plant foods are lost or destroyed by modern processing techniques, including cooking. Since neither processed foods nor animal products contain a significant load of antioxidant nutrients or any phytochemicals, the western diet is dramatically disease-promoting[2].

As processed foods and fast foods expanded into the underdeveloped world, we saw rural areas starting to develop higher rates of cancer and obesity. The study by CDC in 2006 also shows that recent immigrants to the United States are far healthier than their US-born counterparts. The reason? The diets and lifestyles in the US are far less healthy than those in many other countries. The result today is a nation with exploding numbers of people with immune system disorders, allergies, autoimmune diseases, cardiovascular diseases, diabetes, and cancers.

The Issues

We have seen slow performance in one of our benchmarks (i.e., FOD). Here are the symptoms:

GC took about 50% of the application's CPU time

Full GC's were not triggered by full Perm Gen, but by full Old Gen

The old generation is completely full and it isn't clearing up anything but a few soft references during Full GC.

So, the symptoms all point to too many live objects kept in Old Gen and they have caused many Full GC's. There could be different causes for the above symptoms. To investigate, you need to create heap dumps and check out what objects HotSpot VM is holding onto. Below we will show you how to investigate this.

Generating Heap Dump

First, we have used a Java tool jmap to generate a summary of the heap contents:

Unfortunately, it didn't tell us much regarding the biggest java object that have taken ~800M bytes. But, we did notice that two other property maps also took up a lot of memory space. So, the next step is to generate a full dump as follows:

$./jmap -dump:live,file=/scratch/aime1/tmp/fod.hprof 16345

To analyze the full heap dump, you need to use Eclipse Memory Analyzer which we have chosen the standalone version.

Analyzing Heap Dump With Memory Analyzer (MAT)

I started the Memory Analyzer with the following command:

$ cd /scratch/sguan/mat/

$ ./MemoryAnalyzer &

When I opened the heap dump file (i.e., fod.hprof), MAT failed with a message related to Java Heap Space. So, I need to modified the following line in the MemoryAnalyzer.ini file:

-vmargs

-Xmx1024m

by changing -Xmx1024m to -Xmx6240m. Note that how big you should set for -Xmx option depends on:

Size of heap dump

Our original file size is 6.4g and was reduced to 3.6g after loading. Note that MAT will remove unreachable objects in the loading process.

Size of physical RAM in your system

The Culprit— WebLogic Session Objects

From the MAT, we have found out that the biggest live object kept in the heap was:

weblogic.servlet.internal.session.MemorySessionContext @ 0x705e0ff08

Also, for two previously-mentioned property map objects, they are also related to the session object. If you trace those objects back to the GC root, you will see that they are pointed to by the session state. They are the reason the session state is so big.

session-timeout Element in web.xml

Session object is used by WebLogic container to track a user's session that uses the StoreFrontModule web application in our FOD benchmark. Every time a new user comes in, it starts a session. This session object keeps user's data or states in it. If the user leaves, the session is considered idle and the server will retain it in memory for a period of time before reclaiming it.

Java web applications can be configured with a session timeout value which specifies the number of minutes a session can be idle before it is abandoned. This session timeout element is defined in the Web Application Deployment Descriptor web.xml[9]. For example, this was the session timeout value we had:

The problem with our benchmark's slowness is due to this session timeout value being too high. This has caused the sessions being piled up and taking up lots of memory. Setting it to be a lower value (see [10] on how to patch web.xml in an EAR), our benchmark then performs normally.

Saturday, March 23, 2013

WARNING: VERSION_DISPLAY_ENABLED_IN_PRODUCTION_STAGE The setting to enable version display will be ignored because the application is running in production mode.

In this article, we will show you:

How to eliminate those warnings from the server log file

How to patch web.xml in the StoreFrontModuleNew.ear

How to Eliminate "WARNING: VERSION_DISPLAY_ENABLED_IN_PRODUCTION_STAGE"?

To eliminate those warnings, you need to modify the following context parameter in the web.xml file:

<context-param><description>Whether the 'Generated by...' comment at the bottom of ADF Faces HTML pages should contain version number information.</description><param-name>oracle.adf.view.rich.versionString.HIDDEN</param-name><param-value>false</param-value></context-param>

by changing its value from false to true. Note that parameter

oracle.adf.view.rich.versionString.HIDDEN

determines whether the 'Generated by...' comment at the bottom of ADF Faces HTML pages should contain version number information or not.

How to Patch web.xml in Java EAR File?

StoreFrontModuleNew.ear is the application deployed in the WebLogic server for our ADF FOD benchmark. Here is the location of web.xml in the StoreFrontModuleNew.ear file.

Before you do that, you create a working directory tmp first. Then you copy EAR file to the tmp directory and un-jar it there. Similarly, to unjar StoreFrontWebAppNew.war, you create another working directory tmp2 at its level and copy it into tmp2 directory and un-jar it there. In the following, we show you the whole sequence of steps:

$cd /scratch/aime1/kjeyaram/

$mkdir tmp

$cd tmp

$cp ../StoreFrontModuleNew.ear .

$jar -xvf StoreFrontModuleNew.ear

$ls

$mkdir tmp2

$cd tmp2

$cp ../StoreFrontWebAppNew.war .

$jar xvf StoreFrontWebAppNew.war

$ls -lrt

$cd WEB-INF/

$vi web.xml

$cd ..

$zip -f ../StoreFrontWebAppNew.war WEB-INF/web.xml

$cd ..

$zip -f ../StoreFrontModuleNew.ear StoreFrontWebAppNew.war

$cd ..

$rm -rf tmp

We have used zip command[1] to replace existing web.xml in the zip archive with the following option:

-f

Replace (freshen) an existing entry in the zip archive only if it has been modified more recently than the version already in the zip archive; unlike the update option (-u) this will not add files that are not already in the zip archive.

There is strong evidence supporting the argument that the amount of time you sleep—even more than whether you smoke, exercise, or have high blood pressure or cholesterol levels—could be the most important predictor of how long you'll live.

On average, adults need 7-9 hours of sleep to stay healthy every night. Those who slept 5 hours or less a night had a 15% greater mortality risk compared with those sleeping 7 hours. While not getting enough sleep is clearly associated with increased health risks, so is getting too much sleep. Those who slept 9 hours had a 42% increase in mortality risk.

Too Little Sleep May Fuel Insulin Resistance

Sleep deficiency results in a higher than normal blood sugar level, which may increase your risk for diabetes.

After four nights of sleep deprivation (sleep time was only 4.5 hours per night), study participants' insulin sensitivity was 16 percent lower, while their fat cells' insulin sensitivity was 30 percent lower, and rivaled levels seen in those with diabetes or obesity.

Researchers at the University of Chicago found that losing just 3 to 4 hours of sleep over a period of several days is enough to trigger metabolic changes that are consistent with a prediabetic state.

We have two different users on a remote server and wish to set up public key authentication over SSH for both of them.

At beginning, we were able to connect to user bench, but not user aime1. This has stirred up our investigation into the issue. Yes, SSH does support multiple user authentications over SSH. It turns out the culprit was that we have forgotten to change the access permission of authorized_keys file for the user aime1:

$chmod 600 authorized_keys

In this article, we will show you:

How to investigate ssh connection issues?

What is going on underneath the ssh session?

The Solution

As described in [1], you need to copy the content of id_rsa.pub from the local server to the authorized_keysfile at the remote server (note that .ssh folders are located differently with different users) . Also, you need to change the authorized_keys' permission to 600. If you forgot to do latter, SSH won't automatically authenticate local server to remote server when local server offers its RSA public key. Note that we have chosen RSA, instead DSA, as the authentication keys over SSH.

As shown above, local server failed to automatically authenticated itself to the remote server (i.e., SSH still prompted user for the password). In the middle of output, the following three lines are interesting:

debug1: Server host key: RSA97:a2:85:f3:f1:28:ab:c2:70:df:58:2f:c3:61:65:9adebug1: Host 'remoteServer' is known and matches the RSA host key.debug1: Found key in /C=/Documents and Settings/aroot/.ssh/known_hosts:1
What happened is that the fingerprint of remote server's RSA public key was returned and it matched one entry stored in the known_hosts file of the local server. In public-key cryptography, a public key fingerprint is a short sequence of bytes used to authenticate or look up a longer public key[5]. Fingerprints are created by applying a cryptographic hash function to a public key. Since fingerprints are shorter than the keys they refer to, they can be used to simplify certain key management tasks.

The RSA public key from the remote server (i.e., a Linux box) is stored here:

SSH on the local server failed to authenticate itself to the remote server with its offered RSA key. So, it tried the following other ways of authentication in sequence:

id_dsa

id_ecdsa

password

Without much ado, we know what went wrong and have fixed the issue. Then when we tried it again, it was authenticated automatically. From the verbose output, we can tell the difference (i.e., it succeeded to authenticate with RSA key):

Monday, March 18, 2013

In our Fusion application deployed in WebLogic Server, it has generated too many messages. Therefore, we would like to disable most of them. However, before we do that, we also want to check out what current message levels of loggers are.

In this article, we will show you how to:

Redirect WLST print messages to an output file

List current message levels of all loggers associated with a managed server (i.e., "MS_1")

With the getLogLevel command

Redirect WLST print Statement

If you try to redirect WLST output to a log file using the following WLST command.

wls:/fod_domain/serverConfig> redirect('./logs/wlst.log', 'false')

It won't work. Instead , you should follow the instructions below[1] to redirect WLST's print statement to a file:

from java.io import File

from java.io import FileOutputStream

f = File("/scratch/aime1/tmp/wlst.log")

fos = FileOutputStream(f)

theInterpreter.setOut(fos)

print "start the script"

How to List Current Message Levels of All Loggers?

To get the current message level, you can use getLogLevel command. Note that you must be connected to WebLogic Server before you use the configuration commands.

From below, find the full list of commands that achieved the task:

$cd $MW_HOME/wlserver/common/bin
$./wlst.sh

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline> connect ('weblogic','weblogic1','t3://localhost:7001')Connecting to t3://localhost:7001 with userid weblogic ...Successfully connected to Admin Server "AdminServer" that belongs to domain "fod_domain".Warning: An insecure protocol was used to connect to theserver. To ensure on-the-wire security, the SSL port orAdmin port should be used instead.

wls:/fod_domain/serverConfig> from java.io import File

wls:/fod_domain/serverConfig> from java.io import FileOutputStream

wls:/fod_domain/serverConfig> f = File("/scratch/aime1/tmp/wlst.log")

wls:/fod_domain/serverConfig> fos = FileOutputStream(f)

wls:/fod_domain/serverConfig> theInterpreter.setOut(fos)

print "start the script"

listLoggers(target='MS_1')

Note that we typed CTRL-D to exit from WLST at the end.

The Output

Here is the output from the getLogLevel command (note that it lists messages levels of all loggers associated with a server named MS_1):

wls:/fod_domain/serverConfig> start the script

wls:/fod_domain/serverConfig> Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.

Saturday, March 16, 2013

Below are some highlights from that article. Cancer and FeverIn clinical practice, doctors observe that many cancer patients simply don't develop fevers, and cancer patients often report that they were never ill. Two of my friends told me exactly that and one of them is cancer victim and another cancer survivor.Fever and Heat TherapyHistorically, heat have been recognized for their beneficial effects. The following list shows how various cultures have used simple forms of heat as a way of both cleansing and healing.

Tuesday, March 12, 2013

There could be different causes that lead to out-of-memory error in HotSpot VM. For example, you can run out of memory in PermGen space:

java.lang.OutOfMemoryError: PermGen space

In this article, we will discuss:

Java Objects vs Java Classes

PermGen Collection[2]

Class unloading

How to find the classes allocated in PermGen?

How to enable class unloading for CMS?

Note that this article is mainly based on Jon's excellent article[1].

Java Objects vs Java Classes

Java objects are instantiations of Java classes. HotSpot VM has an internal representation of those Java objects and those internal representations are stored in the heap (in the young generation or the old generation[2]). HotSpot VM also has an internal representation of the Java classes and those are stored in the permanent generation.

PermGen Collector

The internal representation of a Java object and an internal representation of a Java class are very similar. From now on, we use Java objects and Java classes to refer to their internal representations. The Java objects and Java classes are similar to the extent that during a garbage collection both are viewed just as objects and are collected in exactly the same way.

Besides its basic fields, Java class also include the following:

Methods of a class (including the bytecodes)

Names of the classes (in the form of an object that points to a string also in the permanent generation)

Constant pool information (data read from the class file, see chapter 4 of the JVM specification for all the details).

Internal objects created by the JVM (java/lang/Object or java/lang/exception for instance)

Information used for optimization by the compilers (JITs)

There are a few other bits of information that end up in the permanent generation but nothing of consequence in terms of size. All these are allocated in the permanent generation and stay in the permanent generation.

Class Loading/Unloading

Back in old days, most classes were mostly static and custom class loaders were rarely used. Then class unloading may not be necessary. However, things have changed and sometimes you could run into the following message:

java.lang.OutOfMemoryError: PermGen space

In this case, there are at least two options:

Increasing the size of PermGen

Enabling class unloading

Increasing the Size of PermGen

Sometimes there is a legitimate need to increase PermGen size by setting the following options:

-XX:PermSize=384m -XX:MaxPermSize=384m

However, before you do that, you may want to find out what Java classes were allocated in PermGen by running

Enabling Class Unloading

If you use CMS collector and run into PermGen's out-of-memory error, you could consider enabling class unloading by setting:

-XX:+CMSClassUnloadingEnabled

-XX:+CMSPermGenSweepingEnabled

Depending on the release you may have, earlier versions (i.e., Java 6 Update 3 or earlier) require you to set both options. However, in later releases you only need to specify:

-XX:+CMSClassUnloadingEnabled

The following are the cases that you want to enable class unloading in CMS collectors:

If your application is using multiple class loaders and/or reflection, you may need to enable collecting of garbage in permanent space.

Objects in permanent space may have references to normal old space thus even if permanent space is not full itself, references from perm to old space may keep some dead objects unreachable for CMS if class unloading is not enabled.

Lots of redeployment may pressure PermGen space

A class and it's classloader have to both be unreachable in order for them to be unloaded. A class X with classloader A and the same class X with classloader B will result in two distinct objects (klassses) in the permanent generation.

In this article, we will discuss how to investigate and resolve this issue.

What to Check?

When using JRockit we have to make a hugetblfs file system available in the directory /mnt/hugepages. Since the message says that it cannot open temporary file /mnt/hugepages/jrock8SadIG, the first thing to check is if that directory was mounted (note that it was mounted before system maintenance).

$ mount -lnodev on /mnt/hugepages type hugetlbfs (rw)

As shown above, /mnt/hugepages directory was mounted. The next thing to check is why JRockit cannot open temporary file in that directory. Could it be privilege?

As described in [1], to make the /mnt/hugepages directory accessible for the oracle user, you need to do:

chmod -R 777 /mnt/hugepages

It turns out that it was the culprit. After issuing the above command, JRockit was able to create Java virtual machine.

Enabling Large Pages Support in Linux Kernel

We have followed the procedure described in [2,5] to enable huge-pages support in the Linux kernel. One of the requirements is to mount hugepages directory on the hugetlbfs type filesystem[6] (note that this step is required for JRockit, but not HotSpot). For example, we have hugepages directory mounted as follows:

So, the issue is related to Java code reservation in Huge Pages. After further investigation, we have found the real problem is that: when we mounted hugepages directory, we have chosen the following option:

noexec

Do not allow direct execution of any binaries on the mounted filesystem

After we have removed that constraint and rebooted the system, we have finally resolved the issue as shown below:

Disclaimer

The statements and opinions expressed here are my own and do not necessarily represent those of Oracle.For your computer health, follow me @xmlandmore. To improve your personal health, follow me @travel2health.

About Me

Healthy pursuits are like traveling. We know there is a wonderland called wellness. But, there are no fixed routes to reach there. The pursuits need effort and determination. We cannot act like tourists who don't know where they've been. We must take notes of warning signs sent by our bodies.

On the journey to wellness, there are many dangers to be avoided; there are many footsteps to be taken; unfortunately, there are no short-cuts.

Travelers walking in the night follow North Star. Healthy pursuits work similarly. We know our goal; we know the signposts; we know the dangers on the road; we can adjust pace if we are tired; we can change travel plans due to the weather. But, We walk steadily and persistently. With my companionship, hopefully, your journey will be made easier.