Tag Info

Hitting F1 or h will show you the key. But for reference, the default colors are:
CPU:
Blue = Low priority threads
Green = Normal priority threads
Red = Kernel threads
Memory:
Green = Used memory
Blue = Buffers
Yellow/Orange = Cache
There are a couple of different color-schemes available, you can see them through hitting F2.

I couldn't find this documented elsewhere. Looking into the code:
There are two modes for CPU metrics reporting: the default one, and a "detailed CPU time" which can be enabled from the Setup screen (Display Options / Detailed CPU time). All of them show the percentage of time spent in different processes:
Default mode
Blue: low priority processes (nice ...

While top is running, you can press c to toggle between showing the process name and the command line. To remember the toggle state for next time, press W to save the current configuration to ~/.toprc.

(1) I see that each of the running processes occupies a very small percentage of memory (%MEM no more than 0.2%, and most just 0.0%), but how the total memory is almost used as in the fourth line of output ("Mem: 130766620k total, 130161072k used, 605548k free, 919300k buffers")? The sum of used percentage of memory over all processes seems unlikely to ...

The first one is a 1 minute load average, second is 5 minutes, third is 15 minutes, just like the uptime command. They do not have any special correlation in htop other than to stand out from each other.
This is a good read on understanding what load is: http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages

You can use an I/O monitor like iotop, but it will show you only processes or threads actually doing I/O operations.
If you need to browse processus waiting for I/O, use watch to monitor processus with STAT flag 'D' like below:
watch -n 1 "(ps aux | awk '\$8 ~ /D/ { print \$0 }')"

In short:
Virtual size: is the amount of address space that a process is managing. The virtual address space contains everything that the process can access through pointers (memory address references). For example, if your program gets access to the framebuffer of your video card, that memory is mapped to the process virtual space and receives an address ...

TL;DR 1
Your server is within some kind of virtuozzo/openvz/virtualization-du-jour container. Trying to make sense of memory usage is tilting at windmills.
TL;DR 2
Linux ate your RAM! But that's okay, it does it to everyone.
The Long Story
Let's break it down!
In the Mem: section we have:
$n total: the amount of physical RAM in your machine
$n ...

Here are a few tools to find disk activity:
iotop
vmstat 1
iostat 1
lsof
strace -e trace=open <application>
strace -e trace=open -p <pid>
In ps auxf you'll also see which processes are are in uninterpretable disk sleep (D) because they are waiting for I/O.
Some days the load increase to reach 40 without increase of the number vistors.
...

Yes, each php process is using 30% of the total memory.
An yes again, shared memory is why you cannot simply add. See the column "SHR", this is the shared memory value, RES stands for resident. From "man top" :
n: %MEM -- Memory usage (RES)
A task's currently used share of available physical memory.
o: VIRT -- Virtual Image (kb)
The ...

Unix like systems, including linux, are designed to make the most efficient use of the available RAM possible. In very general terms, there are 3 states each MB of RAM can be in:
Free
Used by a Process
Used for Buffers
The 3rd state is only used as scratch space and is intended to be reassigned whenever necessary, i.e. your total available memory for ...

ps axu and look for processes which are in the "D" state. Based on the ps(1) manpage, processes that are in the D state are in uninterruptable sleep, which almost always means 'waiting for IO'. Unfortunately, killing these processes is usually not possible.

Zanchey's answer is the best I know to find out what is waiting for IO.
When you say your server is under high load, what do you mean by that? Something in particular is slow to respond?
If you are wondering if your Disk IO is the bottleneck, I would use the iostat command (part of the sysstat package) to see if the disk actually is under heavy load.
...

The Steal percentage (documented in the mpstat man-page) is indeed the hypervisor telling your VM that it can't have CPU resources the VM would otherwise use. This percentage is regulated in part by Amazon's CPU limiting, and VM load on that specific host. I/O load is monitored through the %io stat.

It's normal if you have a multi-threaded process on a machine with more than one core/thread/processor.
Abstract of man top:
In a true SMP environment, if a process is multi-threaded and top is not operating in Threads mode, amounts greater than 100% may be reported.

You can use this command to see the top 10 applications regarding RAM usage:
ps -A --sort -rss -o comm,pmem | head -n 11
Sometimes this command helps you if many sub processes have been generated:
ps auxf
This way you can see which processes belong together.

Running top in batch mode to report memory sizes periodically can be used to see who is using the memory when things go south. Runing sar in batch mode should give some good diagnostics on memory use, and related I/O. Running munin to monitor the system should give you a graph with good detail on what memory is being used for. This may help a lot.
You ...

The 57.6% means that mysqld is using .576 of one cpu. The discrepancy is likely to be a race condition between data collection for the system as a whole and collecting the per-process data.
EDIT:
Based on your update looks like you have 16 cores.
57.5% => .575/16 = .036 = 3.6%.
So that's where your 3% is coming from.
If you sum all the idle ...

A very good observation and we have run into this as well. Here's what I found -- http://www.metamul.com/blog/measuring-cpu-usage-from-within-ec2-it-doesnt-mean-what-you.html
When you run top within the EC2 instance, it is measuring the CPU usage of the physical core machine that is running your instance and others. This usage is incorrect if you want to be ...

There is top's secure mode, when it is invoked as
top -s
In that mode the user can't change the refresh delay of top, kill or renice processes.
If you want to make it system wide instead of on a per-invocation basis, you may use the /etc/toprc file with the following contents:
s
3.0
Only two lines: first one to set secure-mode, and the second to set ...

The data exposed by top is often insufficient or misleading in virtualized environments like Amazon EC2 and the reported percentage depends on your instance type and the under­ly­ing proces­sor core utilization (which usually doesn't match the virtualized hardware you are presented with from the hypervisor), amongst other things - what you are seeing is most ...

MongoDB uses memory mapped files - which can be much larger then the actual physical (or swap) available on your machine. The total virtual memory used will usually be the size of all the data on disk at the time as a result.
The resident memory will be the stat that represents the actual working set used, though depending on your resources on the box ...