Implement Huge Pages for Oracle Database

Advantages

Linux Huge Pages offer several advantages to an Oracle Database.  One the "real" page size increases from 4k to 2MB.  Second huge page memory cannot be swapped out therefore it is always resident.  The use of Huge Pages reduces virtual memory operations when dealing with large amounts of memory.   The fact that each page is larger, cannot be swapped out and reduces virtual memory I/O operations can improve performance of you Oracle database instances.  Oracle SGA sizes greater than 10GB have shown to have significant decreases in performance when huge pages are not utilized.  it is also considered best practice to implement Huge Pages for Oracle Database Instance SGAs.

Limitations

Linux Huge Pages cannot be used with Oracle 11g when using the Automatic Memory Management Feature (AMM).   When attempted to utilize huge pages and automatic memory management errors on instance startup will result. 

Calculating Huge Pages

Linux Huge Pages are calculated by understanding the amount of memory needed and utilizing a standard formula for setting the number of huge pages.   First you will need to determine the SGA Sizes (SGA)  for all databases on the node, it is a good idea to add some head room to huge pages so add some memory to total of SGA sizes as not to have to keep updating with every small increase in SGA sizes.  Next determine the operating system huge page size (hps) then can be done using grep Hugepagesize /proc/meminfo.  With this information you can calculate the number of huge pages required and set it. 

# grep Hugepagesize /proc/meminfo

Hugepagesize:     2048 kB

Apply formula:  # of huge pages = SGA / hugepagesize **

For example for a 10G SGA + 2 GB of head room for a total of 12 GB and knowing that the Hugepagesize is 2M.

12G/2M=6144, so 6144 is the number of huge pages required.

Check current hugepages

# grep Hugepage /proc/meminfo

Calculate the needed hugepages, You can do this manually or you can run a script that will calculate for you the value you need.  ** Script to Calculate Huge Pages (Oracle provides the Script in DOC ID 401749)

Setting up Huge Pages

There are several settings in Linux that need to be accounted for when setting up huge pages.  First you should set the memlock parameter for the oracle software account (oracle).  You will need to ensure that the memlock setting are sum greater than any single database instance memory use.  In the example a setting of 15G would be a good setting as it would allow the hugepages to be increased without further changes however to limit needed changes later it can be set to unlimited.

$ vi /etc/security/limits.conf

oracle   soft   memlock   unlimited

oracle   hard   memlock   unlimited

Next you will need to set the number of huge pages.

# vi /etc/sysctl.conf

vm.nr_hugepages = # of huge pages from applying formula

# sysctl -w vm.nr_hugepages=17500

# sysctl -p

net.ipv4.ip_forward = 0

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 1

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.shmmax = 68719476736

kernel.shmall = 4294967296

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 6815744

fs.aio-max-nr = 1048576

net.ipv4.neigh.bond0.locktime = 0

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 4194304

net.core.wmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_max = 2097152

vm.nr_hugepages = 17500

kernel.panic = 60

kernel.shmmax = 64547735961

kernel.shmall = 15758724

In 11gR2 there is bug (9251136) when running RAC instances where the ohasd does not abide by the ulimit setting for the memlock seting.  Therefore the best way to work around this is to set the ulimit in the $GRID_HOME/bin/ohasd file.  It is also recommended to be placed in the oracle OS user profile.

vi $HOME/.bash_profile

ulimit -l unlimited

ulimit -m unlimited

vi $GRID_HOME/bin/ohasd

ulimit -l unlimited

ulimit -m unlimited

Change the database to force the use of huge pages.   This is done by setting the use_large_pages database initialization parameter.  Setting this parameter to "only" will force the instance to use hugepages at startup or error.

Now lock the sga into memory to take full advantage of the Hug Pages by setting the lock_sga parameter=TRUE.  This parameter cannot be used with automatic memory management features.

Now everything is set, bounce the server and ensure that all settings are taking affect across node reboots.

Checking Huge Pages Utilization

Huge Pages utilization can be checked by looking at the /proc/meminfo file.  You will need to ensure you database(s) are started prior to checking huge pages utilization.

$ cat /proc/meminfo | grep HugePages

HugePages_Total:  6144

HugePages_Free:   2055     <-- are="" as="" free="" huge="" is="" lower="" notice="" pages="" span="" than="" that="" total="" used="">

oracle@dm01db01 ~> cat /proc/meminfo | grep Hugepage

Hugepagesize:     2048 kB

oracle@dmdrdb01 ~> ipcs -ma

------ Shared Memory Segments --------

key        shmid      owner      perms      bytes      nattch     status     

0x00000000 0          root      644        72         2                      

0x00000000 32769      root      644        16384      2                      

0x00000000 65538      root      644        280        2                      

0xa3c20e68 163844     oracle    660        4096       0                      

0x8755fd14 294917     oracle    660        26845642752 172                     

------ Semaphore Arrays --------

key        semid      owner      perms      nsems    

0x253c0f5c 131073     oracle    660        104      

0xeafd5270 524290     oracle    660        126      

0xeafd5271 557059     oracle    660        126       

0xeafd5272 589828     oracle    660        126      

0xeafd5273 622597     oracle    660        126      

0xeafd5274 655366     oracle    660        126      

------ Message Queues --------

key        msqid      owner      perms      used-bytes   messages