Configure SQL Net using Infiniband Interface

Infiniband

Infiniband is a high speed network interface typically utilized in System/Storage Area Networks (SANs) which are networks connecting machines and storage in close proximity and require very low latency and very high bandwidth. Oracle has supported the use of infiniband for SQL Net traffic for quite some time.  Typically TCP over Infiniband has been utilized.

Sockets Direct Protocol Support

The Sockets Direct Protocol (SDP) is a standard protocol for Infiniband.  The SDP protocol is an interface the moves most of the messaging burden to the network interface card therefore reducing the amount of host CPU involved.  This frees the host CPU bandwidth for other processing.   This is a big advantage for mid-tier server connections to the database that require high performance communication between the database and middle tier application servers.  While Oracle provides support for SQL Net client activity via the SDP protocol for SQL Net traffic over an infiniband network it is recommended to stick with TCP over infiniband as I have never gotten the SDP protocol to work as of yet. 

Database Server Configuration

A database server with an infiniband network interface has the ability to communicate across that infiniband interface allowing database connections the increased bandwidth and lower latency of infiniband.  The SQL-Net interface for the infiniband can be configured using TCP protocol.  When using the TCP protocol it is recommended to configure the SDU sizes appropriately on the client and server.  On the database server side the Oracle TNS listener must configured to listen for the specific protocol on the infiniband network address so that will be used in the communication initiated to the database so that communication can be recognized.  To configure the server to accept connections to the database with the TCP it is recommended to configure 2 listeners, 1 listener for TCP communication over the normal network interface and a second listener for the TCP communication over the infiniband network.  If the infiniband connection is management network for activities such as backup, Storage communication, management, data guard, streams or golden gate connectivity a second dedicated listener for this communication is a must.  When the second listener is configured a second port is utilized and therefore the connection requests are completely isolated.

Additional Listener TCP

LISTENER_MGMT=

  (DESCRIPTION=

    (ADDRESS_LIST=

      (ADDRESS=(PROTOCOL=tcp)(HOST=infini-server)(PORT=1522)) ) )

Database Client Configuration

The database sqlnet client configuration will require a tns entry in the local tnsanames.ora as well as entries in the local sqlnet.ora.  When using SDP with infiniband rather than TCP there is no complicated listener or tns setup as with TCP when dealing with larger bandwidth data transfers.  Typically the hostname given to a database servers infiniband network interface is not in the DNS therefore you must add the database server hostname for that interface to the client and server hosts file (applies to both Windows and Unix/Linux).

hosts

# Connections (infiniband)

192.168.1.2      db02-priv.localdomain db02-priv

192.168.1.1      db01-priv.localdomain db01-priv

tnsnames.ora (non-rac example)

prd =

(DESCRIPTION=

   (ADDRESS_LIST=

     (ADDRESS=(PROTOCOL=tcp)(HOST=db01-priv)(PORT=1522))

   (CONNECT_DATA =

     (SERVICE_NAME = prd.localdomain)

   )

  )

tnsnames.ora (rac example)

prd_rac =

(DESCRIPTION=

   (ADDRESS_LIST=

     (ADDRESS=(PROTOCOL=tcp)(HOST=db01-priv)(PORT=1522))

     (ADDRESS=(PROTOCOL=tcp)(HOST=db02-priv)(PORT=1522))

   (LOAD_BALANCE = ON)

   (FAILOVER = ON)

   (CONNECT_DATA =

     (SERVICE_NAME = prd.localdomain)

     (FAILOVER_MODE=(TYPE=select)(METHOD=basic)(RETRIES=20)(DELAY=15))

   )

  )

Check Database Connection Configuration

After the configuration is complete a good way to test the connectivity is working is to test the connection across the infiniband from a client machine connected to the infiniband network.  Using SQL Plus or another client tool that will utilize the Oracle Client interface test the connection to the database through the listener listening on the infiniband network.

C:\> sqlplus messinam@prd

SQL*Plus: Release 11.2.0.1.0 Production on Wed Sep 1 18:00:10 2010

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Enter password:

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management,

OLAP,

Data Mining and Real Application Testing options

SQL>