Advantages
Linux Huge Pages offer several advantages to an Oracle Database. One the "real" page size increases from 4k to 2MB. Second huge page memory cannot be swapped out therefore it is always resident. The use of Huge Pages reduces virtual memory operations when dealing with large amounts of memory. The fact that each page is larger, cannot be swapped out and reduces virtual memory I/O operations can improve performance of you Oracle database instances. Oracle SGA sizes greater than 10GB have shown to have significant decreases in performance when huge pages are not utilized. it is also considered best practice to implement Huge Pages for Oracle Database Instance SGAs.Limitations
Linux Huge Pages cannot be used with Oracle 11g when
using the Automatic Memory Management Feature (AMM). When attempted to utilize huge pages and
automatic memory management errors on instance startup will result.
Calculating Huge Pages
Linux Huge Pages are calculated by understanding the
amount of memory needed and utilizing a standard formula for setting the number
of huge pages. First you will need to
determine the SGA Sizes (SGA) for all
databases on the node, it is a good idea to add some head room to huge pages so
add some memory to total of SGA sizes as not to have to keep updating with
every small increase in SGA sizes. Next
determine the operating system huge page size (hps) then can be done using grep
Hugepagesize /proc/meminfo. With this
information you can calculate the number of huge pages required and set
it.
#
grep Hugepagesize /proc/meminfo
Hugepagesize: 2048 kB
Apply formula: #
of huge pages = SGA / hugepagesize **
For example for a 10G SGA + 2 GB of head room for a total
of 12 GB and knowing that the Hugepagesize is 2M.
12G/2M=6144, so 6144 is the number of huge pages
required.
Check current hugepages
#
grep Hugepage /proc/meminfo
Calculate the needed hugepages, You can do this manually or you can run a script that will calculate for you the value you need. ** Script to Calculate Huge Pages (Oracle provides the Script in DOC ID 401749)
Setting up Huge Pages
There are several settings in Linux that need to be
accounted for when setting up huge pages.
First you should set the memlock parameter for the oracle software
account (oracle). You will need to
ensure that the memlock setting are sum greater than any single database
instance memory use. In the example a
setting of 15G would be a good setting as it would allow the hugepages to be
increased without further changes however to limit needed changes later it can
be set to unlimited.
$
vi /etc/security/limits.conf
oracle soft
memlock unlimited
oracle hard memlock
unlimited
Next you will need to set the number of huge pages.
#
vi /etc/sysctl.conf
vm.nr_hugepages
= # of huge pages from applying formula
#
sysctl -w vm.nr_hugepages=17500
#
sysctl -p
net.ipv4.ip_forward
= 0
net.ipv4.conf.default.rp_filter
= 1
net.ipv4.conf.default.accept_source_route
= 0
kernel.sysrq
= 1
kernel.core_uses_pid
= 1
net.ipv4.tcp_syncookies
= 1
kernel.msgmnb
= 65536
kernel.msgmax
= 65536
kernel.shmmax
= 68719476736
kernel.shmall
= 4294967296
kernel.shmmni
= 4096
kernel.sem
= 250 32000 100 128
fs.file-max
= 6815744
fs.aio-max-nr
= 1048576
net.ipv4.neigh.bond0.locktime
= 0
net.ipv4.ip_local_port_range
= 9000 65500
net.core.rmem_default
= 4194304
net.core.wmem_default
= 262144
net.core.rmem_max
= 4194304
net.core.wmem_max
= 2097152
vm.nr_hugepages = 17500
kernel.panic
= 60
kernel.shmmax
= 64547735961
kernel.shmall
= 15758724
In 11gR2 there is bug (9251136) when running RAC
instances where the ohasd does not abide by the ulimit setting for the memlock
seting. Therefore the best way to work
around this is to set the ulimit in the $GRID_HOME/bin/ohasd file. It is also recommended to be placed in the
oracle OS user profile.
vi
$HOME/.bash_profile
ulimit
-l unlimited
ulimit
-m unlimited
vi
$GRID_HOME/bin/ohasd
ulimit
-l unlimited
ulimit
-m unlimited
Change the database to force the use of huge pages. This is done by setting the use_large_pages
database initialization parameter.
Setting this parameter to "only" will force the instance to
use hugepages at startup or error.
Now
lock the sga into memory to take full advantage of the Hug Pages by setting the
lock_sga parameter=TRUE. This parameter
cannot be used with automatic memory management features.
Now everything is set, bounce the server and ensure that
all settings are taking affect across node reboots.
Checking Huge Pages Utilization
Huge Pages utilization can be checked by looking at the
/proc/meminfo file. You will need to
ensure you database(s) are started prior to checking huge pages utilization.
$
cat /proc/meminfo | grep HugePages
HugePages_Total: 6144
HugePages_Free: 2055 <-- are="" as="" free="" huge="" is="" lower="" notice="" pages="" span="" than="" that="" total="" used="">
oracle@dm01db01 ~> cat
/proc/meminfo | grep Hugepage
Hugepagesize: 2048 kB
oracle@dmdrdb01
~> ipcs -ma
------
Shared Memory Segments --------
key shmid owner
perms bytes nattch
status
0x00000000
0 root 644
72 2
0x00000000
32769 root 644
16384 2
0x00000000
65538 root 644
280 2
0xa3c20e68
163844 oracle 660
4096 0
0x8755fd14
294917 oracle 660
26845642752 172
------
Semaphore Arrays --------
key semid owner
perms nsems
0x253c0f5c
131073 oracle 660
104
0xeafd5270
524290 oracle 660
126
0xeafd5271
557059 oracle 660
126
0xeafd5272
589828 oracle 660
126
0xeafd5273
622597 oracle 660
126
0xeafd5274
655366 oracle 660
126
------
Message Queues --------
key msqid owner
perms used-bytes messages
