Before we discuss about Huge pages, we should revise some of the basic concepts about Operating system memory management.

Operating systems basically deals with two types of memory for its operations one is Physical memory ( actual Physical RAM) and the second is Virtual memory ( combination of Physical and Swap). Virtual Memory is used to store entire program that is currently running and the physical memory is to store only specific part of memory that is important at this moment. And both physical memory and virtual memory will be segmented as pages, and the process of copying the pages of the program from to/from the virtual memory to/from the physical memory is called paging.

Operating system manages a page table to map virtual memory pages to physical memory pages. So when ever operating system needs to access memory for some information it will look at the paging table first and then look at the corresponding physical memory, if it doesn’t find information in physical memory then go back to virtual memory to get the required information.

Terminology That we use when referring to hugepages:

TLB (Transalation Lookaside Buffer) : It is a buffer ( or cache) in a CPU that contains parts of the page table. This is a fixed size buffer being used to do virtual address translation faster.

hugetlb: This is an entry in the TLB that points to a HugePage (a large/big page larger than regular 4K and predefined in size). HugePages are implemented via hugetlb entries, i.e. we can say that a HugePage is handled by a “hugetlb page entry”.

hugetlbfs: This is a new in-memory filesystem like tmpfs and is presented by 2.6 kernel. Pages allocated on hugetlbfs type filesystem are allocated in HugePages.

How operating system deals with process when there was no concept of Hugepages?

In unix environment every operation/task runs in the form of a process and every process has its own page table which is having entries that references to system wide page table. System wide page table will have complete information about memory pages that are required for the execution of system wide process.

During the process execution, each user process looks at its own local page table, and goes to system wide page table to find the location of the memory that it wants to access. And also sometimes it is possible that some related process will access same memory page in shared mode.

How Hugepages makes difference to regular Memory Management?

HugePages is a feature integrated into the Linux kernel with release 2.6. This feature basically provides the alternative to the 4K page size (16K for IA64) providing bigger pages. There are some basic differences between Huge pages and regular pages, and they are :

a. Huge pages can be allocated on-demand but they must be reserved during the system startup.Otherwise the allocation fails because the memory by default segmented into 4K pages.

b. HuHugePages are not swappable. Therefore there is no page-in/page-out mechanism overhead.HugePages are universally regarded as pinned.

c. Decreased Page Table Overhead: Each entry in page table occupies 64bytes memory , and is we assume we have 256GB of memory which is segmented into 4K memory pages, the pagetable alone takes around 800MB in size and managing such large table is an overhead to the operating system. In other case, if the huge pages are activated then we can reserve major part ( around 90%) of the memory to be accessed as 256MB hugepages so that the number of entries in the system page table reduces and the size of the page table also reduces to 40 to 50MB.

d. Overall memory performance will be improved : On virtual memory systems each memory operation is actually two abstract memory operations. Since there are less number of pages to work on, the possible bottleneck on page table access is clearly avoided

e. Size of the hugepages can vary from 2MB to 256MB depending on the kernel version.

Spread a word

I have started unixadminschool.com ( aka gurkulindia.com) in 2009 as my own personal reference blog, and later sometime i have realized that my leanings might be helpful for other unixadmins if I manage my knowledge-base in more user friendly format. And the result is today's' unixadminschool.com.
You can connect me at - https://www.linkedin.com/in/unixadminschool/

5 Responses

Hi Garvit, I am actually waiting for this question from the readers :). Thanks for raising the question.

The answer:

to set the number of pages we have to use the parameter ” vm.nr_hugepages = ” in /etc/sysctl.conf. And the number of hugepages has to be calculated as “Total-Huge-page-size-you-want-to-allocate / Size-of-each-huge-page” . Total size you want to allocate for huge pages is an input that we receive from the Database admins, and the size of each page is any thing from 2MB to 256MB. And most of the time we will leave the size of each page at 2MB.

For example:

Sybase server had 32 GB RAM, DBA wants 28GB to be allocated to sybase in the form of huge pages. Then the calculation follow as below:

sar -B output is showing memory paging rate.
Could you please provide more information regarding this output and how to find out wich process are consuming more paging rate ? How to make slow down the paging rate?

Paging is common memory Management feature and it is different than swapping. The applications which are bigger than physical memory do need the Paging to execute. All you can do here is find the application which is using highest memory during the startup and have it tuned to use the less memory and not a bad thing