Wednesday, August 21, 2013

Oracle Database 12c Important Facts


Feature Deprecation

There are several features that Oracle had indicated are deprecated which gives insight into features that Oracle intends to remove or de-support in future releases giving you time to plan and determine the alternatives that you will utilize going forward to replace the deprecated functionality.

Oracle Enterprise Manager Database Control

Oracle OEM Database Control is de-supported and is no longer going to be a component that can be utilized and is being replaced with Oracle Enterprise Manager Express.

Streams

Oracle Streams is being deprecated in Oracle 12c and is subject to full desupport and likely to be unavailable future releases. It is recommended to utilize GoldenGate to replace the capability Streams provided.

Advanced Replication

Oracle Advanced Replication is being deprecated in Oracle 12c. It is recommended that all Advanced Replication usage be moved to GoldenGate.  Golden Gate will replace Advanced Replication including multimaster replication, updatable materialized views, hierarchical materialized views, and deployment templates.

OCFS on Windows

In Oracle Database 12c, Oracle Cluster File System (OCFS) is desupported for Windows.  The support of OCFS (OCFS2) for Linux will continue and is not affected. http://docs.oracle.com/cd/E16655_01/server.121/e17642/deprecated.htm

Raw Devices

In Oracle Database 12c, raw devices will not be supported therefore you must move database data files on raw devices to ASM, a supported cluster file system, or NFS.    This also affects and applies to  OCR and VOTE for clusterware and must be moved before upgrade.

Change Data Capture

Oracle Change Data Capture (CDC) is not included in Oracle 12c therefore prior to upgrade all Change Data Capture usage must be removed and replaced.  It is recommended all CDC functionality be moved to GoldenGate.

Stored Outlines

Stored Outlines are deprecated, you should look at moved to SQL Profiles and Plan Baselines.


SQLNET.ALLOWED_LOGON_VERSION

This sqlnet.ora parameter is deprecated and is being replace by 2 new parameters:
SQLNET.ALLOWED_LOGON_VERSION_SERVER
SQLNET.ALLOWED_LOGON_VERSION_CLIENT

For environments that do not set this parameter means the environment was operating with the default setting of 8.  In upgrading to Oracle Database 12c you may need to set the SQLNET.ALLOWED_LOGIN_VERSION_SERVER to 8 to maintain compatibility of existing application clients.

CSSCAN and CSALTER

The csscan and csalter are deprecated in Oracle Database 12c and are replaced with the Database Migration Assistant for Unicode. http://docs.oracle.com/cd/E16655_01/doc.121/e36716/ch1overview.htm#DUMAG101\

Oracle Net Listener Password

The Oracle listener password is deprecated, Oracle ensures there is no loss of security as security is enforce through local OS authentication.

XML DB is Mandatory and Cannot Be Uninstalled

The XML DB is a part of the Oracle database that is no longer optional.  It can no longer be excluded from install on database creation and can no longer be removed from a database.

Deprecated Views

ALL_SCHEDULER_CREDENTIALS 
DBA_NETWORK_ACL_PRIVILEGES 
DBA_NETWORK_ACLS 
DBA_SCHEDULER_CREDENTIALS 
USER_NETWORK_ACL_PRIVILEGES 
 USER_SCHEDULER_CREDENTIALS 
V$OBJECT_USAGE
 

Initialization Parameters

With database 12c pluggable databases there are some database initialization parameters changes that you need to be aware of.

New Parameters

This is not intended to be a complete list of new parameter, but a mention of the ones seen as most notable.

To enable the use of pluggable databases the enable_pluggable_database initialization parameter is there to control that. 

enable_pluggable_database         TRUE

For a default file name convert used when creating a new pluggable database for file location.

pdb_file_name_convert

You will also notice that the v$parameter as a new column con_id that will indicate the database which the parameter is set for.  The parameters for each pluggable database are the same as a normal database.

Deprecated Parameters

active_instance_count
background_dump_dest
buffer_pool_keep
buffer_pool_recycle
commit_write
cursor_space_for_time
fast_start_io_target
global_context_pool_size
instance_groups
lock_name_space
log_archive_local_first
log_archive_start
max_enabled_roles
parallel_automatic_tuning
parallel_io_cap_enabled
parallel_server
parallel_server_instances
plsql_v2_compatibility
remote_os_authent
resource_manager_cpu_allocation
sec_case_sensitive_logon
serial_reuse
sql_trace
standby_archive_dest
user_dump_dest




Upgrade Path

Direct Upgrade Path


Source Database
Target Database
10.2.0.5 (or higher)
12.1.0.1.0
11.1.0.7 (or higher)
12.1.0.1.0
11.2.0.2 (or higher)
12.1.0.1.0

 

Indirect Upgrade Path


Source Database
Upgrade Path for Target  Database
Target Database
7.3.3.0.0 (or lower)
7.3.4.x --> 9.2.0.8
11.2.x
8.0.5.0.0 (or lower)
8.0.6.x --> 9.2.0.8
11.2.x
8.1.7.0.0 (or lower)
8.1.7.4 --> 9.2.0.8
11.2.x
9.0.1.3.0 (or lower)
9.0.1.4 --> 9.2.0.8
11.2.x

Implement Huge Pages for Oracle Database


Advantages

Linux Huge Pages offer several advantages to an Oracle Database.  One the "real" page size increases from 4k to 2MB.  Second huge page memory cannot be swapped out therefore it is always resident.  The use of Huge Pages reduces virtual memory operations when dealing with large amounts of memory.   The fact that each page is larger, cannot be swapped out and reduces virtual memory I/O operations can improve performance of you Oracle database instances.  Oracle SGA sizes greater than 10GB have shown to have significant decreases in performance when huge pages are not utilized.  it is also considered best practice to implement Huge Pages for Oracle Database Instance SGAs.

Limitations

Linux Huge Pages cannot be used with Oracle 11g when using the Automatic Memory Management Feature (AMM).   When attempted to utilize huge pages and automatic memory management errors on instance startup will result. 

Calculating Huge Pages

Linux Huge Pages are calculated by understanding the amount of memory needed and utilizing a standard formula for setting the number of huge pages.   First you will need to determine the SGA Sizes (SGA)  for all databases on the node, it is a good idea to add some head room to huge pages so add some memory to total of SGA sizes as not to have to keep updating with every small increase in SGA sizes.  Next determine the operating system huge page size (hps) then can be done using grep Hugepagesize /proc/meminfo.  With this information you can calculate the number of huge pages required and set it. 

# grep Hugepagesize /proc/meminfo
Hugepagesize:     2048 kB

Apply formula:  # of huge pages = SGA / hugepagesize **

For example for a 10G SGA + 2 GB of head room for a total of 12 GB and knowing that the Hugepagesize is 2M.

12G/2M=6144, so 6144 is the number of huge pages required.

Check current hugepages

# grep Hugepage /proc/meminfo

Calculate the needed hugepages, You can do this manually or you can run a script that will calculate for you the value you need.  ** Script to Calculate Huge Pages (Oracle provides the Script in DOC ID 401749)


Setting up Huge Pages

There are several settings in Linux that need to be accounted for when setting up huge pages.  First you should set the memlock parameter for the oracle software account (oracle).  You will need to ensure that the memlock setting are sum greater than any single database instance memory use.  In the example a setting of 15G would be a good setting as it would allow the hugepages to be increased without further changes however to limit needed changes later it can be set to unlimited.

$ vi /etc/security/limits.conf

oracle   soft   memlock   unlimited
oracle   hard   memlock   unlimited

Next you will need to set the number of huge pages.

# vi /etc/sysctl.conf
vm.nr_hugepages = # of huge pages from applying formula

# sysctl -w vm.nr_hugepages=17500
# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 1
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.neigh.bond0.locktime = 0
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 4194304
net.core.wmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_max = 2097152
vm.nr_hugepages = 17500
kernel.panic = 60
kernel.shmmax = 64547735961
kernel.shmall = 15758724

In 11gR2 there is bug (9251136) when running RAC instances where the ohasd does not abide by the ulimit setting for the memlock seting.  Therefore the best way to work around this is to set the ulimit in the $GRID_HOME/bin/ohasd file.  It is also recommended to be placed in the oracle OS user profile.

vi $HOME/.bash_profile
ulimit -l unlimited
ulimit -m unlimited

vi $GRID_HOME/bin/ohasd
ulimit -l unlimited
ulimit -m unlimited

Change the database to force the use of huge pages.   This is done by setting the use_large_pages database initialization parameter.  Setting this parameter to "only" will force the instance to use hugepages at startup or error.

Now lock the sga into memory to take full advantage of the Hug Pages by setting the lock_sga parameter=TRUE.  This parameter cannot be used with automatic memory management features.

Now everything is set, bounce the server and ensure that all settings are taking affect across node reboots.

Checking Huge Pages Utilization

Huge Pages utilization can be checked by looking at the /proc/meminfo file.  You will need to ensure you database(s) are started prior to checking huge pages utilization.

$ cat /proc/meminfo | grep HugePages
HugePages_Total:  6144
HugePages_Free:   2055     <-- are="" as="" free="" huge="" is="" lower="" notice="" pages="" span="" than="" that="" total="" used="">

oracle@dm01db01 ~> cat /proc/meminfo | grep Hugepage
Hugepagesize:     2048 kB

oracle@dmdrdb01 ~> ipcs -ma

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status     
0x00000000 0          root      644        72         2                      
0x00000000 32769      root      644        16384      2                      
0x00000000 65538      root      644        280        2                      
0xa3c20e68 163844     oracle    660        4096       0                      
0x8755fd14 294917     oracle    660        26845642752 172                     

------ Semaphore Arrays --------
key        semid      owner      perms      nsems    
0x253c0f5c 131073     oracle    660        104      
0xeafd5270 524290     oracle    660        126      
0xeafd5271 557059     oracle    660        126       
0xeafd5272 589828     oracle    660        126      
0xeafd5273 622597     oracle    660        126      
0xeafd5274 655366     oracle    660        126      

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    

Wednesday, February 20, 2013

OEM 12c Security Certificate Installation




Summary
The security certificate is just one component of the Oracle Enterprise Manager Security.  The certificate is utilized for https traffic therefore normal http traffic should be shut off for console and upload traffic there forcing all communication to the OMS to come from a secure channel that will utilize the certificate.  Though it is not required to shut off the normal http traffic it is recommended when using a certificate as what would be the point of the certificate if you can use unsecured http.

Prepare Security Certificate Request
1. login to the first OEM server as the owner of the OMS installation typically oracle.

2. Set the environment to the repository database home or a database home installed on the OMS Server, unfortunately if you do not have a database installation on the OMS server you will need to utilize this is where the orapki utility is


. oraenv
ORACLE_SID = [oemrep] ? oemrep
The Oracle base remains unchanged with value /u00/app/oracle


3. Create an auto login wallet, an auto login wallet is needed for this installation/setup.


# orapki wallet create -wallet /home/oracle/wallet -auto_login

4. Generate the security certificate request, this must be done by the oracle wallet manager or orapki utilizy, generating this any other way is not ok.  In this case we are calling the server oem.domain.com, this is not the actual server name, but the name utilized via the load balancer.  Ensure that you use the server name that you are generating the certificate for as it will be used in addressing the server which typically includes the domain.

# orapki wallet add -wallet /home/oracle/wallet -dn "CN=oem.domain.com, OU=EM, O=My Company, L=My City, ST=My State, C=US" -keysize "2048" -pwd password


# orapki wallet export -wallet /home/oracle/wallet -dn "CN=oem.domain.com, OU=EM, O=My Company, L=My City, ST=My State, C=US" -pwd password -request /home/oracle/wallet/csr_request.txt

5. Send the csr_request.txt to security to generate certificate(s)
you should be returned 3 filess a Root, intermediate and ssl user certificate.


6. Do not remove the generated Wallet files as you will need these once the security certificates are given to you

Install the Security Certificate
1. Import the Root certificate into the wallet


# orapki wallet add -wallet /home/oracle/wallet -trusted_cert -cert /home/oracle/wallet/Root.cer.txt -pwd password

2. Import the intermediate certificate into the wallet


# orapki wallet add -wallet /home/oracle/wallet -trusted_cert -cert /home/oracle/wallet/intermediate.cer.txt -pwd password

3. Import the SSL/User certificate into the wallet


# orapki wallet add -wallet /home/oracle/wallet -user_cert -cert /home/oracle/wallet/server.cer.txt -pwd password

4. Display the wallet


orapki wallet display -wallet /home/oracle/wallet

5. cat the immediate certificate and the root certificate to a trusted certificates file


# cd /home/oracle/wallet
# cat intermediate.cet.txt >> trusted.cer.txt
# cat root.cet.txt >> trusted.cer.txt


6. set the environemnt to the oms


# . oraenv
oms


7. Secure the console


# emctl secure console -wallet /home/oracle/wallet

8. import the wallet and certificate into OMS


# emctl secure oms -host oem.domain.com -wallet /home/oracle/wallet -trust_certs_loc /home/oracle/wallet/trusted.cer.txt

9. secure the agent on the local OMS host by securing the agent and uploading the agent. 


# . oraenv
agent
# emctl secure agent
# emctl upload agent


10. Lock the upload to only secure traffic

# emctl secure lock –upload

11. Lock the console access to only secure traffic (https)

# emctl secure lock -upload -console

12. Check the status of the OMS and agent in the console

http://oem.domain.com:7801/em


13. Login to the seconds OMS server and get to the oracle account and set the environment to the oms


# . oraenv
oms


14. Copy the certificates and wallets from the first OMS to same location /home/oracle/wallet

15. Secure the console


# emctl secure console -wallet /home/oracle/wallet


16. import the wallet and certificate into OMS


# emctl secure oms -host oem.domain.com -wallet /home/oracle/wallet -trust_certs_loc /home/oracle/wallet/trusted.cer.txt


17. secure the agent on the OMS server


# . oraenv
agent
# emctl secure agent 

# emctl upload agent

18. Check the status of the OMS and agent in the console

http://oem.domain.com:7801/em


Secure All Agents on All Hosts
1. Go to all targets with and agent and resecure agent
. oraenv
agent
emctl secure agent  or emctl secure agent -emdWalletSrcUrl https::/em
emctl upload agent




2. Check the status of the OMS and agent in the console for each target agent after secure

http://oem.domain.com:7801/em


** Special Note when blackout of target is set prior to certificate work
I had issues here as I blacked out all my targets before I started, after I stopped the blackout I had to go back to each agent and execute a clear state, reload and upload to clear out all blackouts

emctl clearstate agent
emctl reload agent
emctl upload agent

OEM 12c Manual Agent Installation



Get the agent software from OMS on the OMS server

** This only needs to be done 1 time per platform for agent installation this can be copied to other servers as required that are the same platform/OS this will stage the agent software installation on the OMS server.

1.       Login to the OMS using the emcli utility

# emcli login -username=sysman -password=passwordhere
Login successful

2.        Get List of supported platforms to see if your platform for an agent is there

# emcli get_supported_platforms

Getting list of platforms ...
Check the logs at /u00/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/setup/.emcli/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list  ...
-----------------------------------------------
Version = 12.1.0.2.0
 Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.


3.       Get the agent for your platform, you will use the platform value from the previous step.

# emcli get_agentimage -destination=/home/oracle -platform="Linux x86-64" -version=12.1.0.2.0

Platform:Linux x86-64
Destination:/home/oracle
 === Partition Detail ===
Space free : 2 GB
Space required : 1 GB
Check the logs at /u00/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/setup/.emcli/get_agentimage_2012-12-05_21-04-34-PM.log
Setting property ORACLE_HOME to:/u00/app/oracle/Middleware/oms
calling pulloneoffs with arguments:/u00/app/oracle/Middleware/oms/u00/app/oracle/Middleware/oms/sysman/agent/12.1.0.2.0_AgentCore_226.zip12.1.0.2.0linux_x64
Check this logs for more information: /u00/app/oracle/Middleware/oms/sysman/prov/agentpush/logs

Install Agent on Each Server
** For every server that needs an agent installation do the following steps
1.        Copy the extracted agent software to the node where agent is to be installed
               
# cd /u01/app/oracle
# mkdir software
# cd software
# scp :/home/oracle/12.1.0.2.0_AgentCore_226.zip ./

2.       unzip the copied agent installation

# unzip 12.1.0.2.0_AgentCore_226.zip

3.        Update the agent.rsp file, note that the ORACLE_HOSTNAME has to be update with each install, the example below is for guidance and will vary in your environment.

# vi agent.rsp

Example:
OMS_HOST=oem.domain.com
EM_UPLOAD_PORT=4901
AGENT_REGISTRATION_PASSWORD=weblogic031oemrep
AGENT_INSTANCE_HOME=/u01/app/oracle/product/agent/agent_inst
AGENT_PORT=3872
b_startAgent=true
ORACLE_HOSTNAME=server.domain.com
s_agentHomeName="agent12cR2"


4.        If a load balancer is utilized you will need to ensure the load balancer name is in the /etc/host or DNS otherwise the OMS server name you are using will need to be in /etc/hosts or in DNS

# vi /etc/hosts

# Example
192.168.1.112    oem.domain.com                   oem


5.       Execute the agent silent installation

# /u01/app/oracle/software/agentDeploy.sh AGENT_BASE_DIR=/u01/app/oracle/product/agent RESPONSE_FILE=/u01/app/oracle/software/agent.rsp

Example Output:
Validating the OMS_HOST & EM_UPLOAD_PORT
Executing command : /u01/app/oracle/product/agent/core/12.1.0.2.0/jdk/bin/java -classpath /u01/app/oracle/product/agent/core/12.1.0.2.0/jlib/agentInstaller.jar:/u01/app/oracle/product/agent/core/12.1.0.2.0/oui/jlib/OraInstaller.jar oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/product/agent/core/12.1.0.2.0 /u01/app/oracle/software /u01/app/oracle/product/agent -prereq

Validating oms host & port with url: http://oem.domain.com:4901/empbs/genwallet
Validating oms host & port with url: https://oem.domain.com:4901/empbs/genwallet
Return status:3
Unzipping the agentcoreimage.zip to /u01/app/oracle/product/agent ....
12.1.0.2.0_PluginsOneoffs_226.zip
Executing command : /u01/app/oracle/software/unzip -o /u01/app/oracle/software/12.1.0.2.0_PluginsOneoffs_226.zip -d /u01/app/oracle/product/agent
Executing command : /u01/app/oracle/product/agent/core/12.1.0.2.0/jdk/bin/java -classpath /u01/app/oracle/product/agent/core/12.1.0.2.0/oui/jlib/OraInstaller.jar:/u01/app/oracle/product/agent/core/12.1.0.2.0/oui/jlib/xmlparserv2.jar:/u01/app/oracle/product/agent/core/12.1.0.2.0/oui/jlib/srvm.jar:/u01/app/oracle/product/agent/core/12.1.0.2.0/oui/jlib/emCfg.jar:/u01/app/oracle/product/agent/core/12.1.0.2.0/jlib/agentInstaller.jar:/u01/app/oracle/product/agent/core/12.1.0.2.0/oui/jlib/share.jar oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/product/agent/core/12.1.0.2.0 /u01/app/oracle/software /u01/app/oracle/product/agent /u01/app/oracle/product/agent/agent_inst AGENT_BASE_DIR=/u01/app/oracle/product/agent AGENT_BASE_DIR=/u01/app/oracle/product/agent RESPONSE_FILE=/u01/app/oracle/software/agent.rsp


Executing agent install prereqs...
Executing command: /u01/app/oracle/product/agent/core/12.1.0.2.0/oui/bin/runInstaller -ignoreSysPrereqs -prereqchecker -silent -ignoreSysPrereqs -waitForCompletion  -prereqlogloc /u01/app/oracle/product/agent/core/12.1.0.2.0/cfgtoollogs/agentDeploy -entryPoint oracle.sysman.top.agent_Complete -detailedExitCodes PREREQ_CONFIG_LOCATION=/u01/app/oracle/product/agent/core/12.1.0.2.0/prereqs  -J-DORACLE_HOSTNAME=server.domain.com -J-DAGENT_PORT=3872 -J-DAGENT_BASE_DIR=/u01/app/oracle/product/agent
Prereq Logs Location:/u01/app/oracle/product/agent/core/12.1.0.2.0/cfgtoollogs/agentDeploy/prereq.log
Agent install prereqs completed successfully

Cloning the agent home...
Executing command: /u01/app/oracle/product/agent/core/12.1.0.2.0/oui/bin/runInstaller -ignoreSysPrereqs -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/u01/app/oracle/product/agent/core/12.1.0.2.0 -responseFile /u01/app/oracle/software/agent.rsp  AGENT_BASE_DIR=/u01/app/oracle/product/agent AGENT_BASE_DIR=/u01/app/oracle/product/agent RESPONSE_FILE=/u01/app/oracle/software/agent.rsp -noconfig  ORACLE_HOME_NAME=agent12cR2 -force b_noUpgrade=true
Clone Action Logs Location:/u01/app/oraInventory/logs/cloneActions.log
Cloning of agent home completed successfully

Attaching sbin home...
Executing command: /u01/app/oracle/product/agent/core/12.1.0.2.0/oui/bin/runInstaller -ignoreSysPrereqs -attachHome -waitForCompletion -nowait ORACLE_HOME=/u01/app/oracle/product/agent/sbin ORACLE_HOME_NAME=sbin12c1 -force
Attach Home Logs Location:/u01/app/oracle/product/agent/core/12.1.0.2.0/cfgtoollogs/agentDeploy/AttachHome.log
Attach home for sbin home completed successfully.

Updating home dependencies...
Executing command: /u01/app/oracle/product/agent/core/12.1.0.2.0/oui/bin/runInstaller -ignoreSysPrereqs -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/u01/app/oracle/product/agent/sbin:/u01/app/oracle/product/agent/core/12.1.0.2.0,} -invPtrLoc /u01/app/oracle/product/agent/core/12.1.0.2.0/oraInst.loc -force
Update Home Dependencies Location:/u01/app/oracle/product/agent/core/12.1.0.2.0/cfgtoollogs/agentDeploy/UpdateHomeDeps.log
Update home dependency completed successfully.

Performing the agent configuration...
Executing command: /u01/app/oracle/product/agent/core/12.1.0.2.0/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/product/agent/core/12.1.0.2.0 RESPONSE_FILE=/u01/app/oracle/product/agent/core/12.1.0.2.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true
Configuration Log Location:/u01/app/oracle/product/agent/core/12.1.0.2.0/cfgtoollogs/cfgfw/CfmLogger.log
Agent Configuration completed successfully

The following configuration scripts need to be executed as the "root" user.
#!/bin/sh
#Root script to run
 /u01/app/oracle/product/agent/core/12.1.0.2.0/root.sh
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts
Agent Deployment Successful.
Agent deployment log location:
/u01/app/oracle/product/agent/core/12.1.0.2.0/cfgtoollogs/agentDeploy/agentDeploy_2012-12-05_22-12-33-PM.log
Agent deployment completed successfully.


7. Run the root.sh as indicated

# su -
# /u01/app/oracle/product/agent/core/12.1.0.2.0/root.sh

Finished product-specific root actions.
/etc exist

Creating /etc/oragchomelist file...
Finished product-specific root actions.


****** Note: All nodes in a cluster must have agent installed and discovered in OEM prior to being able to add any cluster items such as databases, ASM, etc.