10 Managing Oracle Software and Applying Patches

Checking the ORACLE_HOME Environment Variable

OPatch verifies if the Oracle home is present. You must ensure that the ORACLE_HOME environment variable is set to the Oracle home of the product you are trying to patch.

Check the respective vendor documentation for the details to set the environment variable.

To check the current setting of the ORACLE_HOME variable on Linux:

1. In a command window, log in to the operating system as the oracle user.

2. Use the echo command to display the current setting of the ORACLE_HOME environment variable.

   echo $ORACLE_HOME
   

Performing a Backup Before Patching Software

It is highly recommended to back up the software directory you are patching before performing any patch operation. This applies to Oracle Database or Oracle Grid Infrastructure software installation directories.

To back up the software installation:

1. Back up the software installed in the specified Oracle home using:

   1. An operating system utility, such as zip, cp -r, tar, or cpio, to back up the software in the Oracle home directory to disk.

   2. The Oracle Secure Backup utility to back up the software in the Oracle home directory to tape.

Staging the Patch on Each Node

After you down the patch from My Oracle Support, you must copy the patch to each node that runs the software being patched.

Updating the PATH Environment Variable

The opatch binary file is located in the Oracle_home/OPatch directory. You can either specify this path when executing OPatch, or you can update the PATH environment variable to include the OPatch directory.

To update the PATH environment variable on Oracle Linux systems:

1. In a command window, log in to the operating system.

2. Use a shell command similar to the following to update the value of the PATH environment variable, where /u01/app/oracle/product/11.2.0/dbhome_1 is the location of your Oracle home directory:

   $ export PATH=$PATH:/u01/app/oracle/product/11.2.0/dbhome_1/OPatch
   

   You could also modify the shell profile script for the current user to have this variable configured every time you log in.

Configuring SSH User Equivalency

Before you patch a system, make sure the user equivalency is working. User equivalency exists when the operating system user on each node is considered to be identical. For information on configuring user equivalency, see "Configuring Operating System Users and Groups".

To test SSH user equivalency:

1. On the system where you want to run OPatch, log in as the oracle user.

2. Use the following command to test user equivalency:

   [oracle@racnode1] $ ssh racnode2 date
   

   If the date is returned, then user equivalency between the source and destination node has been configured.

3. If you see output similar to the following, then SSH user equivalency is not enabled:

   Enter passphrase for key '/home/oracle/.ssh/id_rsa': 
   

   Enable SSH user equivalency before continuing with the patching operation.

To enable SSH user equivalency:

1. On the system where you want to run OPatch, open a command window and log in as the oracle user.

2. Start the SSH agent and load the SSH keys into memory using the following commands:

   $ /usr/bin/ssh-agent $SHELL
   $ /usr/bin/ssh-add
   

   These commands start the ssh-agent on the local node, and load the RSA and DSA keys into the current session's memory so that you are not prompted to use pass phrases when issuing SSH commands.

3. At the prompt, enter the pass phrase for each key that you generated when configuring Secure Shell, for example:

   [oracle@racnode1 .ssh]$ exec /usr/bin/ssh-agent $SHELL
   [oracle@racnode1 .ssh]$ /usr/bin/ssh-add
   Enter passphrase for /home/oracle/.ssh/id_rsa
   Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)
   Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)
   

4. To test if you have configured SSH correctly, run the following command. If you have configured SSH correctly, then you are not prompted for a password or a pass phrase.

   [oracle@racnode1] $ ssh racnode2 date
   

Patching the Grid Home

There are two parts to patching the Oracle Grid Infrastructure for a cluster software:

• Unlock the Grid home

• Apply patches to the Grid home

Before you can apply a patch or make other modifications to the software files in the Grid home, you must first unlock the Grid home.

To unlock the Grid home, perform the following steps:

1. Change directory to the path Grid_home/crs/install, where Grid_home is the path to the Grid home, and at the root user unlock the Grid home using commands similar to the following:

   cd /u01/app/11.2.0/grid/crs/install 
   perl rootcrs.pl -unlock -crshome /u01/app/11.2.0/grid
   

   When you run the rootcrs.pl script with the -unlock flag, it stops the Oracle Clusterware stack and unlocks the files in the Grid home so they can be modified.

2. Change user to the Oracle Grid Infrastructure for a cluster software owner and apply the patch to the Grid home, using one of the patching methods described in this section.

3. After you have finished modifying the Grid home, lock it again as the root user using commands similar to the following:

   cd /u01/app/11.2.0/grid/crs/install 
   perl rootcrs.pl -patch
   

   The rootcrs.pl script with the -patch flag locks the Grid home again and restarts the Oracle Clusterware stack.

All Node Patching

In all node patching, all the nodes in the cluster are initially shut down and the patch is applied on all the nodes. After all the nodes have been patched, then the Oracle Clusterware stack and all its registered resources are restarted on each node. This method is typically used for very critical patches and it leads to maximum downtime. OPatch uses this method if the patch cannot be applied in a rolling fashion, and you did not specify the minimize_downtime option.

To implement all node patching:

1. Stop all user applications that use the Oracle home directory being patched.

2. If you are patching only the Oracle RAC home directory, then shut down all Oracle RAC instances on all nodes in the cluster. To shut down all Oracle RAC instances for a cluster database, enter the following command in a command window, where CRS_home is the location of the Grid home directory and sales is the name of the database:

   $ CRS_home/bin/srvctl stop database -d sales
   

3. If you are patching the Grid home directory, then you must stop all the Oracle RAC databases for the cluster and all single-instance databases in the cluster that use the ASM installation being patched.

   When patching the Grid home directory you can use a single command to stop the Oracle Clusterware stack on all servers in the cluster. This command attempts to gracefully stop resources managed by Oracle Clusterware while attempting to stop the Oracle Clusterware stack. If any resources that Oracle Clusterware manages are still running after you run the crsctl stop cluster command, then the command fails. Use the -f option to unconditionally stop all resources and stop the Oracle Clusterware stack.Use a command similar to the following, where CRS_home is the home directory of your Oracle Grid Infrastructure for a cluster installation:

   $ CRS_home/crs/bin/crsctl stop cluster -all
   

4. After you have stopped the database instances and Oracle Clusterware stack on each node in the cluster, use the crsctl utility to verify that all cluster resources have been stopped on each node.

   $ Grid_home/bin/crsctl status resource -t
   

5. Set your current directory to the directory where the patch is located, for example:

   $ cd Oracle_home/OPatch/4519934/4519934
   

6. Make sure the ORACLE_HOME environment variable points to the software directory you want to patch, for example:

   $ echo $ORACLE_HOME
   /u01/app/oracle/product/11.2.0/dbhome_1
   

7. Run OPatch by entering the following command:

   opatch apply
    
   

8. If you applied the patch to the Grid home directory, then restart the Oracle Clusterware stack on all nodes by issuing the following command as the root user on any node, where Grid_home is the home directory of your Oracle Grid Infrastructure for a cluster installation:

   # Grid_home/bin/crsctl start cluster -all
   

9. After you have restarted the Oracle Clusterware stack on all nodes, use the crsctl utility to verify that the cluster resources were restarted on each node.

   $ Grid_home/bin/crsctl status resource -t
   

   If any of the cluster resources did not restart, then use either the CRSCTL or SRVCTL utility to restart them. For example, you can use commands similar the following to restart various cluster resources, where Grid_home is the home directory of your Oracle Grid Infrastructure for a cluster installation and Oracle_home is the home directory of your Oracle RAC database:

   $ Oracle_home/bin/srvctl start instance -d sales -i "sales1"
   # Grid_home/bin/crsctl start resource myResource -n docrac2
   

10. Run any post-patch scripts that are mentioned in the patch instructions, for example:

    $ sqlplus /nolog 
    SQL> connect sys/password@sales1 AS SYSDBA
    SQL> @Oracle_home/rdbms/admin/catbundle.sql cpu apply
    SQL> exit
    

Rolling Patching

In rolling patching, one group of nodes is shut down, the patch is applied to those nodes, and the nodes are brought back up. This is performed group by group, separately, until all the nodes in the cluster are patched. This is the most efficient means of applying an interim patch to an Oracle RAC or Oracle Grid Infrastructure for a cluster installation. By patching groups of nodes individually, there is zero downtime for the cluster database because at least one instance is available at all times on a different node.

While most patches can be applied in a rolling fashion, some patches cannot be applied in this fashion. The README file for the patch indicates whether or not you can apply the patch using the rolling patch method. If the patch cannot be applied using the rolling patch method, then you must use either "Minimum Downtime Patching" or "All Node Patching" to apply the patch.

To apply a patch using the rolling patch method:

1. In a command window, change to the directory where the unzipped patch is staged on disk, for example:

   $ cd Oracle_home/OPatch/12419331/12419331
   

2. Verify the patch can be applied in a rolling fashion, using the following command:

   $ opatch query -is_rolling_patch [unzipped patch location]
   

   To be applied in a rolling fashion, the patch must be designated as a "rolling updatable patch" or simply "rolling patch". Refer to patch README file to find out whether the patch is rolling patch or not.

3. Stop all user applications that use the Oracle home directory for the group of nodes being patched. For example, to stop Enterprise Manager Database Control on the local node, use the following command, where Oracle_home is the home directory for your Oracle RAC installation:

   $ Oracle_home/bin/emctl stop dbconsole
   

4. If you are patching only the Oracle RAC home directory, then shut down all Oracle RAC instances in the group of nodes being patched. To shut down an instance for an Oracle RAC database, enter a command similar to the following example, where Oracle_home is the location of the Oracle RAC home directory, sales is the name of the database, and sales1 is the name of the instance:

   $ Oracle_home/bin/srvctl stop instance -d sales -i "sales1" -f
   

5. If you are patching the Grid home directory, then you must stop all Oracle RAC database instances that are running on the group of nodes being patched. Additionally, stop all single-instance databases and user applications that use the ASM installation in the Grid home on the nodes being patched.

   When patching the Grid home directory you can use a single command to stop the Oracle Clusterware stack on specific servers in the cluster. This command attempts to gracefully stop resources managed by Oracle Clusterware while attempting to stop the Oracle Clusterware stack. If any resources that Oracle Clusterware manages are still running after you run the crsctl stop cluster command, then the command fails. Use the -f option to unconditionally stop all resources and stop the Oracle Clusterware stack on the specified servers.Use a command similar to the following, where Grid_home is the home directory of your Oracle Grid Infrastructure for a cluster installation and docrac1 is one of the nodes in the group of nodes being patched:

   $ Grid_home/crs/bin/crsctl stop cluster -n docrac1
   

6. After you have stopped the database instances and Oracle Clusterware stack on specific nodes in the cluster, use the crsctl utility to verify that all cluster resources have been stopped on those nodes.

   $ Grid_home/bin/crsctl status resource -t
   

7. Make sure the ORACLE_HOME environment variable points to the software directory you want to patch, for example:

   $ echo $ORACLE_HOME
   /u01/app/oracle/product/11.2.0/dbhome_1
   

8. If you are patching nodes individually, then use the following command to instruct OPatch to apply the patch to only the local node. If you run this command from the directory where the patch is located, then you do not need to specify the patch ID.

   $ opatch apply -local
   

   If you are patching nodes individually, then you can use a command similar to the following to instruct OPatch to apply the patch to only the next node to be patched. If you run this command from the directory where the patch is located, then you do not need to specify the patch ID.

   $ opatch apply -remote_nodes docrac2
   

   If you are patching a group of nodes, then use a command similar to the following to instruct OPatch to apply the patch to the group of nodes being patched:

   $ opatch apply [-local_node docrac1] -remote_nodes docrac2,docrac3
   

9. After you have applied the patch to a node or group of nodes, then you can restart Oracle Clusterware and Oracle RAC on those nodes.

   1. If you applied the patch to only the Oracle home, then restart the instances and database applications on the patched nodes. For example, you would use commands similar to the following:

      $ Oracle_home/bin/srvctl start instance -d sales -i "sales1"
      $ Oracle_home/bin/emctl start dbconsole
      

   2. If you applied the patch to the Grid home directory, then restart the Oracle Clusterware stack on the patched nodes by issuing the following command as the root user one of those nodes, where Grid_home is the home directory of your Oracle Grid Infrastructure for a cluster installation:

      # Grid_home/bin/crsctl start cluster -n docrac1
      

      After you have restarted Oracle Clusterware and Oracle ASM, you can then restart any Oracle Database instances and associated applications on the patched nodes.

10. After you have restarted the Oracle Clusterware stack on all nodes, use the crsctl utility to verify that the cluster resources were restarted on each node.

    $ Grid_home/bin/crsctl status resource -t
    

    If any of the cluster resources did not restart, then use either the CRSCTL or SRVCTL utility to restart them. For example, you can use commands similar the following to restart various cluster resources, where Grid_home is the home directory of your Oracle Grid Infrastructure for a cluster installation and Oracle_home is the home directory of your Oracle RAC database:

    $ Oracle_home/bin/srvctl start instance -d sales -i "sales1"
    # Grid_home/bin/crsctl start resource myResource -n docrac1
    

11. Repeat Step 3 through Step 10 for the next group of nodes.

    If you have more than two groups of nodes to be patched, then repeat Step 3 through Step 10 for each group of nodes until all the nodes in the cluster have been patched.

12. Run any post-patch scripts that are mentioned in the patch instructions, for example:

    $ sqlplus /nolog 
    SQL> connect sys/password@sales1 AS SYSDBA
    SQL> @Oracle_home/rdbms/admin/catbundle.sql cpu apply
    SQL> exit
    

Minimum Downtime Patching

In minimum downtime patching, one set of nodes is shut down and the patch is applied to those nodes. After the first set of nodes has been patched, the second set of nodes is shut down. The first set of nodes is then restarted and the patch is applied to the second set of nodes. After the patch has been applied to the second set of nodes, those nodes are restarted. This method leads to less downtime for Oracle RAC, compared to having all the nodes shut down at the same time.

When you use the minimum downtime patching method, the following actions occur:

• The local node is always patched first.

• The local node is used as a base to patch the other nodes.

• The user is prompted for the set of nodes to patch first from the remaining nodes.

• For each node in this first set, the user is asked to stop the instance and then the patch is propagated to that node before continuing to the next node. When the initial set of nodes has been patched, the user is asked to shut down the remaining nodes.

• After the local node is patched, the patch is propagated to the last set of nodes and the inventory is updated. The last instances are stopped on the remote nodes. You can then start up the patched nodes (the first set of nodes) before patching the remaining nodes.

To apply a patch to your cluster database using the minimum downtime method:

1. Change to the directory where the unzipped patch is staged on disk, for example:

   $ cd Oracle_home/OPatch/12419331/12419331
   

2. Stop all user applications that use the Oracle RAC home directory for the group of nodes being patched. For example, to stop Enterprise Manager Database Control on the local node, use the following command, where Oracle_home is the home directory for your Oracle RAC installation:

   $ Oracle_home/bin/emctl stop dbconsole
   

3. Shut down all Oracle RAC instances on the local node. To shut down an instance for an Oracle RAC database, enter a command similar to the following example, where Oracle_home is the home directory for your Oracle RAC database installation, sales is the name of the database, and sales1 is the name of the instance:

   $ Oracle_home/bin/srvctl stop instance -d sales -i "sales1" -f
   

4. Make sure the ORACLE_HOME environment variable points to the software directory you want to patch, for example:

   $ echo $ORACLE_HOME
   /u01/app/oracle/product/11.2.0/dbhome_1
   

5. Use the following command from within the patch directory:

   $ opatch apply -minimize_downtime
   

   If you run the OPatch command from the directory where the patch is staged on disk, then you do not need to specify the patch ID.

   OPatch asks if you are ready to patch the local node. After you confirm that the Oracle RAC instances on the local node have been shut down, OPatch applies the patch to the Oracle home directory on the local node. You are then asked to select the next nodes to be patched.

6. After you shut down the Oracle RAC instances on the other nodes in the cluster, you can restart the Oracle RAC instance on the local node. Then, instruct OPatch that you are ready to patch the remaining nodes.

7. After all the nodes have been patched, restart the Oracle RAC instances on the other nodes in the cluster. The following command shows how to start the orcl2 instance for the Oracle RAC database named orcl:

   $ Oracle_home/bin/srvctl start instance -d orcl -i "orcl2"
    
   

8. Verify that all the Oracle Clusterware resources were restarted on all the nodes in the cluster.

   $ crsctl check cluster
   

   If any of the cluster resources did not restart, then use either the CRSCTL or SRVCTL utility to restart them. For example, you can use commands similar the following to restart various cluster resources, where Grid_home is the home directory of your Oracle Grid Infrastructure for a cluster installation and Oracle_home is the home directory of your Oracle RAC database:

   $ Oracle_home/bin/srvctl start instance -d sales -i "sales1"
   # Grid_home/bin/crsctl start resource myResource -n docrac1
   

9. Run any post-patch scripts that are mentioned in the patch instructions, for example:

   $ sqlplus /nolog 
   SQL> connect sys/password@sales1 AS SYSDBA
   SQL> @Oracle_home/rdbms/admin/catbundle.sql cpu apply
   SQL> exit
   

Updating the Node List for OPatch

If OPatch does not automatically detect Oracle RAC or its nodes, then investigate the contents of the inventory and ensure they are complete.

To update the node list for OPatch:

If the list of nodes for your cluster is not complete, then you can update it by using Oracle Universal Installer and the -updateNodeList flag, as demonstrated in the following example:

Oracle_home/oui/bin/runInstaller -updateNodeList 
ORACLE_ HOME=/u01/app/oracle/product/11.2.0/dbhome_1 
CLUSTER_NODES=racnode1,racnode2,racnode3 -noClusterEnabled

About OPatch Log and Trace Files

Logging and tracing is a common aid for debugging. OPatch maintains logs for all apply, rollback, and lsinventory operations. The log files are located in the Oracle_home/cfgtoollogs/opatch directory. Each log file is tagged with the time stamp of the operation. Log files are named as opatch_mm-dd-yyyy_hh-mm-ss.log, where mm-dd-yyyy is the current date and hh-mm-ss is the current time. Each time you run OPatch, a new log file is created.

For example, if a log file is created on May 17, 2007 at 11:55 PM, then it is named as follows:

opatch_05-17-2007_23-55-00.log

OPatch also maintains an index of the commands processed by OPatch and the log files associated with it in the opatch_history.txt file located in the Oracle_home/cfgtoollogs/opatch directory. A sample of the opatch_history.txt file is as follows:

Date & Time : Tue Apr 26 23:00:55 PDT 2007
Oracle Home : /u01/app/oracle/product/11.2.0/dbhome_1/
OPatch Ver. : 11.2.0.0.0
Current Dir : /scratch/oui/OPatch
Command     : lsinventory
Log File    : 
/u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch-2007_Apr_26_23-00-55-PDT_Tue.log

Resolving the "Not a valid patch area" Error

You might get this error if the directory that the OPatch utility is using to do the patch does not match the template for what it is checking, or if the OPatch utility is run from an invalid directory.

The Patch_Shiphome directory should have the following structure:

• An etc directory that has the metadata files

• A files directory that has the patch files

• The etc/config/inventory.xml file and the actions.xml file under the same directory

To resolve the "Not a valid patch area" error:

• Perform one of the following actions:

  1. Remove the patch shiphome directory and re-create it with the proper structure (by extracting the files again).

  2. Start the OPatch utility from the directory where the patch to be installed has been unzipped and staged on disk.

  3. Use the following command when starting OPatch:

     opatch apply /Patch_Shiphome
     

     where Patch_Shiphome is the location where the patch has been staged on disk.

Resolving the "Unable to remove a partially installed interim patch" Error

If the patching process is interrupted, then you might get the error "Unable to remove a partially installed interim patch" when you try to install the patch a second time.

To resolve the partially installed patch error:

1. Ensure that the environment variable ORACLE_HOME is set to the Oracle home directory you are attempting to patch.

2. Go to the Oracle_home/.patch_storage/patch-id_timestamp directory and run the restore.sh script (or restore.bat on Windows platforms) as follows:

   Oracle_home/.patch_storage/patch-id_timestamp/restore.sh
   

3. On Linux and UNIX systems, use the Oracle_home/.patch_storage/patch-id_timestamp/make.txt file (if available) to modify your operating system environment, as follows:

   /bin/sh make.txt
   

4. Attempt to apply the patch again.