Configuring HugePages for Oracle on Linux (x86-64)

Introduction

For large SGA sizes, HugePages can give substantial benefits in virtual memory management. Without HugePages, the memory of the SGA is divided into 4K pages, which have to be managed by the Linux kernel. Using HugePages, the page size is increased to 2MB (configurable to 1G if supported by the hardware), thereby reducing the total number of pages to be managed by the kernel and therefore reducing the amount of memory required to hold the page table in memory. In addition to these changes, the memory associated with HugePages can not be swapped out, which forces the SGA to stay memory resident. The savings in memory and the effort of page management make HugePages pretty much mandatory for Oracle 11g systems running on x86-64 architectures.

Just because you have a large SGA, it doesn't automatically mean you will have a problem if you don't use HugePages. It is typically the combination of a large SGA and lots database connections that leads to problems. To determine how much memory you are currently using to support the page table, run the following command at a time when the server is under normal/heavy load.

Thanks to Bjoern Rost for pointing out the issue when using the script against UEK3 and the suggested fix. I've subsequently added support for 3.10 and 4.1. There is a newer version of this script available from MOS (Doc ID 401749.1) which includes these kernel versions also.

Make the file executable.

$ chmod u+x hugepages_setting.sh

Make sure all the Oracle services are running as normal on the server, then run the script and make a note of the recommended "vm.nr_hugepages" value.

$ ./hugepages_setting.sh
Recommended setting: vm.nr_hugepages = 305
$

Edit the "/etc/sysctl.conf" file as the "root" user, adding the following entry, adjusted based on your output from the script. You should set the value greater than or equal to the value displayed by the script. You only need 1 or 2 spare pages.

vm.nr_hugepages=306

One person reported also needing the hugetlb_shm_group setting on Oracle Linux 6.5. I did not and it is listed as a requirement for SUSE only. If you want to set it, get the ID of the dba group.

# fgrep dba /etc/group
dba:x:54322:oracle
#

Use the resulting group ID in the "/etc/sysctl.conf" file.

vm.hugetlb_shm_group=54322

Run the following command as the "root" user.

# sysctl -p

Alternatively, edit the "/etc/grub.conf" file, adding "hugepages=306" to the end of the kernel line for the default kernel and reboot.

You can now see the HugePages have been created, but are currently not being used.

Add the following entries into the "/etc/security/limits.conf" script or "/etc/security/limits.d/99-grid-oracle-limits.conf" script, where the setting is at least the size of the HugePages allocation in KB (HugePages * Hugepagesize). In this case the value is 306*2048=626688.

* soft memlock 626688
* hard memlock 626688

If you prefer, you can set these parameters to a value just below the size of physical memory of the server. This way you can forget about it, unless you add more physical memory.

Check the MEMORY_TARGET parameters are not set for the database and SGA_TARGET and PGA_AGGREGATE_TARGET parameters are being used instead.

Remember, if you increase your memory allocation or add new instances, you need to retest the required number of HugePages, or risk Oracle running without them.

Force Oracle to use HugePages (USE_LARGE_PAGES)

Sizing the number of HugePages correctly is important because prior to 11.2.0.3, if the whole SGA doesn't fit into the available HugePages, the instance will start up without using any. From 11.2.0.3 onward, the SGA can run partly in HugePages and partly not, so the impact of this issue is not so great. Incorrect sizing may not be obvious to spot. Later releases of the database display a "Large Pages Information" section in the alert log during startup.

If you are running Oracle 11.2.0.2 or later, you can set the USE_LARGE_PAGES initialization parameter to "only" so the database fails to start if it is not backed by hugepages. You can read more about this here.

Disabling Transparent HugePages (RHEL6/OL6 and RHEL7/OL7)

Starting from RHEL6/OL6, Transparent HugePages are implemented and enabled by default. They are meant to improve memory management by allowing HugePages to be allocated dynamically by the "khugepaged" kernel thread, rather than at boot time like conventional HugePages. That sounds like a good idea, but unfortunately Transparent HugePages don't play well with Oracle databases and are associated with node reboots in RAC installations and performance problems on both single instance and RAC installations. As a result Oracle recommends disabling Transparent HugePages on all servers running Oracle databases, as described in this MOS note.

Thanks to Mor for pointing this out and directing me to the notes here and here.

With Transparent HugePages disabled, you should proceed to configure conventional HugePages, as described above.

Configuring 1G Hugepagesize

As mentioned by Eugene in the comments, Oracle currently don't recommend using 1G Hugepagesize. You can read more about this in MOS Doc ID 1607545.1. With that in mind, the rest of this section should probably be considered more of an academic exercise.

Check if your current hardware can support a Hugepagesize of 1G. If the following command produces any output, it can.

# cat /proc/cpuinfo | grep pdpe1gb

Thanks to Kevin Closson for pointing out the hardware support requirement.

Edit the "/etc/grub.conf" file, adding the following entries on to the kernel line of the default grub entry. Adjust the "hugepages" entry to the desired number of 1G pages. Notice this includes the disabling of Transparent HugePages, which is not mandatory, but a good idea.