This chapter contains the following topics:
The best practices discussed in this section apply to Oracle Database 11g with Oracle Real Application Clusters (Oracle RAC). These best practices build on the Oracle Database 11g configuration best practices described in Chapter 5, "Configuring Oracle Database" and Chapter 6, "Configuring Oracle Database with Oracle Clusterware." These best practices are identical for the primary and standby databases if they are used with Data Guard in Oracle Database 11g with Oracle RAC and Data Guard—MAA. Some best practices may use your system resources more aggressively to reduce or eliminate downtime. This can, in turn, affect performance service levels, so be sure to assess the impact in a test environment before implementing these practices in a production environment.
Instance recovery is the process of recovering the redo thread from the failed instance. Instance recovery is different from crash recovery, which occurs when all instances accessing a database have failed. Crash recovery is the only type of recovery when an instance fails using a single-instance Oracle Database.
When using Oracle RAC, the SMON process in one surviving instance performs instance recovery of the failed instance.
In both Oracle RAC and single-instance environments, checkpointing is the internal mechanism used to bound Mean Time To Recover (MTTR). Checkpointing is the process of writing dirty buffers from the buffer cache to disk. With more aggressive checkpointing, less redo is required for recovery after a failure. Although the objective is the same, the parameters and metrics used to tune MTTR are different in a single-instance environment versus an Oracle RAC environment.
In a single-instance environment, you can set the FAST_START_MTTR_TARGET initialization parameter to the number of seconds the crash recovery should take. Note that crash recovery time includes the time to startup, mount, recover, and open the database.
Oracle provides several ways to help you understand the MTTR target your system is currently achieving and what your potential MTTR target could be, given the I/O capacity.
|
See Also: The MAA white paper "Best Practices for Optimizing Availability During Unplanned Outages Using Oracle Clusterware and Oracle Real Application Clusters" for more information from the MAA Best Practices area for Oracle Database at |
The FAST_START_PARALLEL_ROLLBACK parameter determines how many processes are used for transaction recovery, which is done after redo application. Optimizing transaction recovery is important to ensure an efficient workload after an unplanned failure. If the system is not CPU bound, setting this parameter to HIGH is a best practice. This causes Oracle to use four times the CPU_COUNT (4 X CPU_COUNT) parallel processes for transaction recovery. The default setting for this parameter is LOW, or two times the CPU_COUNT (2 X CPU_COUNT). Set the parameter as follows:
ALTER SYSTEM SET FAST_START_PARALLEL_ROLLBACK=HIGH SCOPE=BOTH;
|
See Also: Oracle Database VLDB and Partitioning Guide for information about Parameters Affecting Resource Consumption for Parallel DML and Parallel DDL |
Using asynchronous I/O is a best practice that is recommended for all Oracle Databases. For more information, see Section 5.1.7, "Set DISK_ASYNCH_IO Initialization Parameter".
Use redundant dedicated connections and sufficient bandwidth for public traffic, Oracle RAC interconnects, and I/O.
In Oracle terms, an extended cluster is a two or more node configuration where the nodes are separated in two physical locations. For an extended cluster and for other Oracle RAC configurations, separate dedicated channels on one fibre may be needed, or you can optionally configure Dense Wavelength Division Multiplexing (DWDM) to allow communication between the sites without using repeaters and to allow greater distances, greater than 10 km, between the sites. However, the disadvantage is that DWDM can be prohibitively expensive.
|
See Also: Oracle Database 2 Day + Real Application Clusters Guide for more information About Network Hardware Requirements |
Oracle RAC One Node is a single instance of an Oracle Real Application Clusters (Oracle RAC) database that runs on one node in a cluster with an option to failover or migrate to other nodes in the same cluster. This option adds to the flexibility that Oracle offers for database consolidation. You can consolidate many databases into one cluster with minimal overhead while also providing the high availability benefits of failover protection, online rolling patch application, and rolling upgrades for the operating system and Oracle Clusterware.
|
See Also: Oracle Real Application Clusters Administration and Deployment Guide for more information about Administering Oracle RAC One Node |
An Oracle RAC extended cluster is an architecture that provides extremely fast recovery from a site failure and allows for all nodes, at all sites, to actively process transactions as part of single database cluster. An extended cluster provides greater high availability than a local Oracle RAC cluster, but because the sites are typically in the same metropolitan area, this architecture may not fulfill all disaster recovery requirements for your organization.
The best practices discussed in this section apply to Oracle Database 11g with Oracle RAC on extended clusters, and build on the best practices described in Section 7.1, "Configuring Oracle Database with Oracle RAC."
Use the following best practices when configuring an Oracle RAC database for an extended cluster environment:
Spread the Workload Evenly Across the Sites in the Extended Cluster
Configure the Nodes to Be Within the Proximity of a Metropolitan Area
Use Host-Based Storage Mirroring with Oracle ASM Normal or High Redundancy
|
See Also:
|
A typical Oracle RAC architecture is designed primarily as a scalability and availability solution that resides in a single data center. To build and deploy an Oracle RAC extended cluster, the nodes in the cluster are separated by greater distances. When configuring an Oracle RAC database for an extended cluster environment, you must:
Configure one set of nodes at Site A and another set of nodes at Site B.
Spread the cluster workload evenly across both sites to avoid introducing additional contention and latency into the design. For example, avoid client/server application workloads that run across sites, such that the client component is in site A and the server component is in site B.
Most extended clusters have only two storage systems (one at each site). During normal processing each node writes and reads a disk heartbeat at regular intervals, but if the heartbeat cannot complete, all affected nodes are evicted from the cluster forcing them to restart their processes and retry to acquire access to the shared resources safely as a member. Thus, the site that houses the majority of the voting disks is a potential single point of failure for the entire cluster. For availability reasons, you should add a third site that can act as the arbitrator in case either: one site fails, or a communication failure occurs between the sites.
In some cases, you can also use standard NFS to support a third voting disk on an extended cluster. You can configure the quorum disk on inexpensive, low end, standard NFS mounted device somewhere on the network. Oracle recommends putting the NFS voting disk on a dedicated server which belongs to a production environment.
If you have an extended cluster and do not configure a third site, you must find out which of the two sites is the primary site. Then, if the primary site fails, you must manually restart the secondary site.
|
Note: Oracle Clusterware supports NFS, iSCSI, Direct Attached Storage (DAS), Storage Area Network (SAN) storage, and Network Attached Storage (NAS). If your system does not support NFS, use an alternative. For example, on Windows systems you can use iSCSI. |
|
See Also: For more information, see the Technical Article "Using standard NFS to support a third voting file for extended cluster configurations" at
|
Extended clusters provide the highest level of availability for server and site failures when data centers are in close enough proximity to reduce latency and complexity. The preferred distance between sites in an extended cluster is within a metropolitan area. High internode and interstorage latency can have a major effect on performance and throughput. Performance testing is mandatory to assess the impact of latency. In general, distances of 50 km or less are recommended.
Testing has shown the distance (greatest cable stretch) between Oracle RAC cluster nodes generally affects the configuration, as follows:
Distances less than 10 km can be deployed using normal network cables.
Distances equal to or more than 10 km require Dense Wavelength Division Multiplexing (DWDM) links.
Distances from 10 to 50 km require storage area network (SAN) buffer credits to minimize the performance impact due to the distance. Otherwise, the performance degradation due to the distance can be significant.
For distances greater than 50 km, there are not yet enough proof points to indicate the effect of deployments. More testing is needed to identify what types of workloads could be supported and what the effect of the chosen distance would have on performance.
Use host-based mirroring with Oracle ASM normal or high redundancy configured disk groups so that a storage array failure does not affect the application and database availability.
Oracle recommends host-based mirroring using Oracle ASM to internally mirror across the two storage arrays. Implementing mirroring with Oracle ASM provides an active/active storage environment in which system write I/Os are propagated to both sets of disks, making the disks appear as a single set of disks that is independent of location. Do not use array-based mirroring because only one storage site is active, which makes the architecture vulnerable to this single point of failure and longer recovery times.
The Oracle ASM volume manager provides flexible host-based mirroring redundancy options. You can choose to use external redundancy to defer the mirroring protection function to the hardware RAID storage subsystem. The Oracle ASM normal and high-redundancy options allow two-way and three-way mirroring, respectively.
|
Note: Array based mirroring can be used in an Oracle RAC extended cluster. Using this approach has the result that the two mirror sites will be in an active-passive configuration and this will result in a complete outage if one site fails. Service becomes available if the remaining mirror site is brought up. For this reason array based mirroring is not recommended from an HA perspective. To work with two active sites, host based mirroring is recommended. |
Beginning with Oracle Database Release 11g, Oracle ASM includes a preferred read capability that ensures that a read I/O accesses the local storage instead of unnecessarily reading from a remote failure group. When you configure Oracle ASM failure groups in extended clusters, you can specify that a particular node reads from a failure group extent that is closest to the node, even if it is a secondary extent. This is especially useful in extended clusters where remote nodes have asymmetric access for performance, thus leading to better usage and lower network loading. Using preferred read failure groups is most useful in extended clusters.
The ASM_PREFERRED_READ_FAILURE_GROUPS initialization parameter value is a comma-delimited list of strings that specifies the failure groups that should be preferentially read by the given instance. This parameter is instance specific, and it is generally used only for clustered Oracle ASM instances. It's value can be different on different nodes. For example:
diskgroup_name1.failure_group_name1, ...
|
See Also:
|
Consider the following additional factors when implementing an extended cluster architecture:
Network, storage, and management costs increase.
Write performance incurs the overhead of network latency. Test the workload performance to assess impact of the overhead.
Because this is a single database without Oracle Data Guard, there is no protection from data corruption or data failures.
The Oracle release, the operating system, and the clusterware used for an extended cluster all factor into the viability of extended clusters.
When choosing to mirror data between sites:
Host-based mirroring requires a clustered logical volume manager to allow active/active mirrors and thus a primary/primary site configuration. Oracle recommends using Oracle ASM as the clustered logical volume manager.
Array-based mirroring allows active/passive mirrors and thus a primary/secondary configuration.
Extended clusters need additional destructive testing, covering
Site failure
Communication failure
For full disaster recovery, complement the extended cluster with a remote Data Guard standby database, because this architecture:
Maintains an independent physical replica of the primary database
Protects against regional disasters
Protects against data corruption and other potential failures
Provides options for performing rolling database upgrades and patch set upgrades
A data protection plan is not complete without a sound backup and recovery strategy to protect against system and storage failures. Oracle delivers a comprehensive data protection suite for backup and recovery of Oracle database and unstructured, application files.
The primary focus of this chapter is best practice configuration for backup and recovery for the Oracle database. File system data protection offerings are introduced along with pointers on where to find more information.
This chapter contains the following topics:
Backup and Recovery Configuration and Administration Best Practices
Backup and Recovery Operations and Maintenance Best Practices
Table 8-1provides a quick reference summary of the Oracle backup and recovery suite.
Table 8-1 Backup and Recovery Summary
| Technology | Recommended use with Oracle Database | Recommended use with File System Data | Comments |
|---|---|---|---|
|
Recovery Manager (RMAN) |
Yes |
No |
Native backup utility for the Oracle database |
|
Oracle Secure Backup |
Yes |
Yes |
Tape backup management software |
|
Oracle Secure Backup Cloud Module |
Yes |
No |
Backup to Amazon S3 storage |
|
Flashback Technologies |
Yes |
No |
Logical error correction leveraging undo data of the Oracle database |
|
Flashback Database |
Yes |
No |
Continuous Data Protection (CDP) leveraging flashback logs |
|
Automatic Clustered File System (ACFS) Snapshots (for file system clones and disaster recovery) |
No |
Yes |
Read-only or read/write copy-on-write version of the file system. Replication available. |
|
ZFS Snapshots (for database clones such as dev/test) |
Yes |
Yes |
Read-only or read/write copy-on-write version of the database for testing and development |
|
ZFS Snapshots for backup/restore |
No |
Yes |
Read-only or read/write copy-on-write version of the file system for testing, development and backup |
This section discusses the motivation and tools for maintaining good database backups, for using Oracle database recovery features, and for using backup options and strategies made possible with Oracle database features.
Using backups to resolve an unscheduled outage of a production database may not allow you to meet your Recovery Time Objective (RTO) and Recovery Point Objective (RPO) or service-level requirements. For example, some outages are handled best by using Flashback Database or a standby database. However, some situations require using database backups, including the sample situations shown in Table 8-2.
Table 8-2 Sample Situations that Require Database Backup
| Situations that require Database Backup | Description |
|---|---|
|
Setting Up the Initial Data Guard Environment |
During initial setup of a standby database, you can either use a backup of the primary database that is made accessible to the secondary site to create the initial standby database, or use RMAN network-enabled database duplication to create the standby database without the need for a pre-existing backup. To perform an over-the-network duplication you must include the RMAN |
|
Recovering from Data Failures Using File or Block Media Recovery |
When a block corruption, media failure, or other physical data failure occurs in an environment that does not include Data Guard, the only method of recovery is to restore from existing backups. |
|
Resolving a Double Failure |
A double failure scenario is a situation that affects the availability of both the production and the standby databases. An example of a double failure scenario is a site outage at the secondary site, which eliminates fault tolerance, followed by a media failure on the production database. Whether the standby must be re-created depends on the type of outage at the secondary site. If the secondary site outage was temporary and did not involve the physical destruction of files, then after the secondary site is brought back online it can continue to receive redo data from the production database. Otherwise, the resolution of this situation is to re-create the production database from an available backup and then re-create the standby database. Some multiple failures, or more appropriately disasters, such as a primary site outage followed by a secondary site outage, might require the use of backups that exist only in an offsite location. Developing and following a process to deliver and maintain backup tapes at an offsite location is necessary to restore service in this worst case scenario. |
|
See Also:
|
Recovery Manager (RMAN) is Oracle's utility to backup and recover the Oracle Database. Because of its tight integration with the database, RMAN determines automatically what files must be backed up. More importantly, RMAN knows what files must be restored for media-recovery operations. RMAN uses server sessions to perform backup and recovery operations and stores metadata about backups in a repository. RMAN offers many advantages over typical user-managed backup methods, including:
Online database backups without placing tablespaces in backup mode
Efficient block-level incremental backups
Data block integrity checks during backup and restore operations
Test backups and restores without actually performing the operation
Synchronize a physical standby database with the primary database
RMAN automates backup and recovery. While user-managed methods require you to:
Locate backups for each data file
Copy backups to the correct place using operating system commands
Choose which logs to apply
RMAN fully automates these backup and recovery tasks.
There are also capabilities of Oracle backup and recovery that are only available when using RMAN, such as automated tablespace point-in-time recovery and block media recovery.
|
See Also: For more information, see the following chapters:
|
Oracle Secure Backup delivers unified data protection for heterogeneous environments with a common management interface across the spectrum of servers. Protecting both Oracle databases and unstructured data, Oracle Secure Backup provides centralized tape backup management for your entire IT environment, including:
Oracle database through the Oracle Secure Backup built-in integration with Recovery Manager (RMAN)
File system data protection: For UNIX, Windows, and Linux servers
Network Attached Storage (NAS) data protection leveraging the Network Data Management Protocol (NDMP)
Oracle Secure Backup is integrated with RMAN providing the media management layer (MML) for Oracle database tape backup and restore operations. The tight integration between these two products delivers high-performance Oracle database tape backup.
Specific performance optimizations between RMAN and Oracle Secure Backup that reduce tape consumption and improve backup performance are:
Unused block compression: Eliminates the time and space usage needed to backup unused blocks
Backup undo optimization: Eliminates the time and space usage needed to backup undo that is not required to recover the current backup.
You can manage the Oracle Secure Backup environment using the command line, the Oracle Secure Backup Web tool, and Oracle Enterprise Manager.
Using the combination of RMAN and Oracle Secure Backup provides an end-to-end tape backup solution, eliminating the need for third-party backup software.
|
See Also:
|
Users can take advantage of the Internet-based data storage services offered by Amazon Web Services (AWS) Simple Storage Service (S3) for their database backup needs. The OSB Cloud Module enables RMAN to use S3 as a repository for Oracle Database backups. This provides an easy-to-manage, cost-efficient, and scalable alternative to maintaining in-house data storage and a local, fully configured backup infrastructure. RMAN with the OSB Cloud module is also the recommended means of backing up an Oracle Database that is running on AWS's Elastic Compute Cloud (EC2).
Users must establish an account with AWS to pay for their storage and network transfer costs as appropriate. Additionally, users must consider their security requirements and network resources, and configure their backups appropriately. Since backups to S3 may travel to AWS on the public Internet, and will be stored on a public cloud facility, it may be necessary to encrypt them to protect the data in transit and at rest at S3. This requirement may not apply for certain databases or in certain configurations, for example when the database runs on EC2 or inside AWS's Virtual Private Cloud (VPC).
Users must also configure the degree of network parallelism (using RMAN channels) to match the network capabilities between the database and S3 (which may include a portion of the public Internet). Multiple simultaneous channels may achieve the highest throughput overall, as RMAN takes full advantage of such parallelism.
Oracle provides restore points and guaranteed restore points:
Restore points protect against logical failures at risky points during database maintenance. Creating a normal restore point assigns a restore point name to a specific point in time or SCN, that is a snapshot of the data as of that time. Normal restore points are available with Flashback Table, Flashback Database, and all RMAN recovery-related operations.
Guaranteed restore points are recommended for database-wide maintenance such as database or application upgrades, or running batch processes. Guaranteed restore points are integrated with Flashback Database and enforce the retention of all flashback logs required for flashing back to the guaranteed restore point. After maintenance activities complete and the results are verified, you should delete guaranteed restore points that are no longer needed to reclaim flashback log space.
|
See Also: Oracle Database Backup and Recovery User's Guide for more information about using restore points and guaranteed restore points with a Flashback Database |
This section describes best practices for determining backup frequency, using the RMAN recovery catalog, and for using Oracle database backup options such as Block Change Tracking.
It is important to determine a backup frequency policy and to perform regular backups. A backup retention policy helps ensure that needed data is not destroyed.
Factors Determining Backup Frequency Frequent backups are essential for any recovery scheme. You should base the frequency and content of backups on the following criteria:
Criticality of the data: The Recovery Point Objective (RPO) determines how much data your business can acceptably lose if a failure occurs. The more critical the data, the lower the RPO and the more frequently data should be backed up. If you are going to back up certain tablespaces more often than others, with the goal of getting better RPO for those tablespaces, then you also must plan for doing TSPITR as part of your recovery strategy. This requires considerably more planning and practice than DBPITR, because you must ensure that the tablespaces you plan to TSPITR are self-contained.
Estimated repair time: The Recovery Time Objective (RTO) determines the acceptable amount of time needed for recovery. Repair time is dictated by restore time plus recovery time. The lower the RTO, the higher the frequency of backups, that is, backups are more current, thereby reducing recovery time.
Volume of changed data: The rate of database change effects how often data is backed up:
For read-only data, perform backups frequently enough to adhere to retention policies.
For frequently changing data, perform backups more often to reduce the RTO.
To simplify database backup and recovery, the Oracle Suggested Backup Strategy uses the fast recovery area, incremental backups, and incrementally updated backup features. After the initial image copy backup to the FRA, only the changed blocks are captured in the incremental backups thereafter and subsequently applied to the image copy, thereby updating the copy to the most current incremental backup time (that is, incrementally updating the backup).
|
See Also:
|
Establishing a Backup Retention Policy A backup retention policy is a rule set regarding which backups must be retained, on disk or other backup media, to meet recovery and other requirements. It may be safe to delete a specific backup because it has been superseded by more recent backups or because it has been stored on tape. You may also have to retain a specific backup on disk for other reasons such as archival or regulatory requirements. A backup that is no longer needed to satisfy the backup retention policy is said to be obsolete.
Base your backup retention policy on redundancy or on a recovery window:
In a redundancy-based retention policy, specify a number n such that you always keep at least n distinct backups of each file in your database.
In a recovery window-based retention policy, specify an earlier time interval, for example, one week or one month, and keep all backups required to let you perform point-in-time recovery to any point during that window.
Keeping Archival Backups Some businesses must retain some backups for much longer than their day-to-day backup retention policy. RMAN allows for this with the Long-term Archival Backup feature. Rather than becoming obsolete according to the database's backup retention policy, archival backups either never become obsolete or become obsolete when their time limit expires.
You can use the RMAN BACKUP command with the KEEP option to retain backups for longer than your ordinary retention policy. This option specifies the backup as an archival backup, which is a self-contained backup that is exempt from the configured retention policy. This allows you to retain certain backups for much longer than usual, when needed for such reasons as satisfying statutory retention requirements. Using the KEEP FOREVER option, a recovery catalog is required because the backup records eventually age out of the control file (otherwise, without a recovery catalog, loss may occur when you retain backups for much longer than usual using the database control file). Only the archived redo log files required to make an archival backup consistent are retained. For more information about the RMAN recovery catalog, see Section 8.2.2, "Use an RMAN Recovery Catalog".
|
See Also: Oracle Database Backup and Recovery User's Guide for information about Archival Backups for Long-Term Storage |
To protect and keep backup metadata for longer retention times than can be accommodated by the control file, you can create a recovery catalog. You should create the recovery catalog schema in a dedicated standalone database. Do not locate the recovery catalog with other production data. If you use Oracle Enterprise Manager, you can create the recovery catalog schema in the Oracle Enterprise Manager repository database.
The advantages of using a recovery catalog include:
Storing backup information for a longer retention period than what can be feasibly stored in the control file. If the control file is too small to hold additional backup metadata, then existing backup information is overwritten, making it difficult to restore and recover using those backups.
Stores metadata for multiple databases.
Offloading backups to a physical standby database and using those backups to restore and recover the primary database. Similarly, you can back up a tablespace on a primary database and restore and recover it on a physical standby database. Note that backups of logical standby databases are not usable at the primary database.
|
See Also: Oracle Database Backup and Recovery User's Guide for more information about RMAN repository and the recovery catalog |
When creating backups to disk or tape, use the target database control file as the RMAN repository so that the success of the backup does not depend on the availability of the database connection to the recovery catalog. To use the target database control file as the RMAN repository, run RMAN with the NOCATALOG option. Immediately after the backup is complete, the new backup information stored in the target database control file should be synchronized to the recovery catalog using the RESYNC CATALOG command.
|
See Also: Oracle Database Backup and Recovery Reference for more information about theRESYNC CATALOG command |
Oracle database includes the BLOCK CHANGE TRACKING feature for incremental backups which improves incremental backup performance by keeping track of which database bloc ks have changed since the previous backup. If BLOCK CHANGE TRACKING is enabled then RMAN uses the block change tracking file to identify which blocks to include in an incremental backup. This avoids the need to scan every block in the data file, reducing the number of disk reads during backup.
Starting with Oracle Database 11g, you can enable BLOCK CHANGE TRACKING on both the primary and physical standby databases. You should enable change tracking for any database where incremental backups are being performed. For example, if backups have been completely offloaded to a physical standby database, then Block Change Tracking should be enabled for that database (this requires Active Data Guard). If backups are being performed on both the primary and physical standby databases, then enable Block Change Tracking for both databases.
|
See Also:
|
You should configure RMAN to automatically back up the control file and the server parameter file (SPFILE) whenever the database structure metadata in the control file changes or when a backup record is added.
The control file autobackup option enables RMAN to recover the database even if the current control file, catalog, and SPFILE are lost. Enable the RMAN autobackup feature with the CONFIGURE CONTROLFILE AUTOBACKUP ON statement.
You should enable autobackup for both the primary and standby databases. For example, after connecting to the primary database, as the target database, and the recovery catalog, issue the following command:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
|
See Also:
|
In an Oracle Data Guard configuration you can offload the process of backing up control files, data files, and archived redo log files to a physical standby database system, thereby minimizing the effect of performing backups on the primary system. You can use these backups to recover the primary or standby database.
|
Note: Backups of logical standby databases are not usable on the primary database. |
|
See Also: Oracle Data Guard Concepts and Administration for information about using RMAN to back up and restore files |
To ensure a database is enabled to use Flashback Query, Flashback Versions Query, and Flashback Transaction Query, implement the following:
Set the UNDO_MANAGEMENT initialization parameter to AUTO. This ensures the database is using an undo tablespace.
Set the UNDO_RETENTION initialization parameter to a value that allows UNDO to be kept for a length of time that allows success of your longest query back in time or to recover from human errors.
Set the RETENTION GUARANTEE clause for the undo tablespace to guarantee that unexpired undo will not be overwritten.
The Flashback Table also relies on the undo data to recover the tables. Enabling Automatic Undo Management is recommended and the UNDO_RETENTION parameter must be set to a period for which the Flashback Table is needed. If a given table does not contain the required data after a Flashback Table, it can be flashed back further, flashed forward, or back to its original state, if there is sufficient UNDO data.
|
See Also:
|
Review the following priorities to determine your disk backup strategy:
Overall backup time
Impact to resource consumption
Space used by the backup
Recovery time
Table 8-3 compares different backup alternatives against the different priorities you might have. Using Table 8-3 as a guide, you can choose the best backup approach for your specific business requirements. You might want to minimize backup space while sacrificing recovery time. Alternatively, you might choose to place a higher priority on recovery and backup times while space is not an issue.
Table 8-3 Comparing Backup to Disk Options
Backup to Disk: Best Practices for Optimizing Recovery Times If restore time is your primary concern then perform either a database copy or an incremental backup with immediate apply of the incremental to the copy. These are the only options that provide an immediate usable backup of the database, which you then must recover only to the time of the failure using archived redo log files created since the last incremental backup was performed.
Backup to Disk: Best Practices for Minimizing Space Usage If space usage is your primary concern then perform an incremental backup with a deferred apply of the incremental to the copy. If you perform a cumulative level 1 incremental backup, then it stores only those blocks that have been changed since the last level 0 backup:
With a cumulative incremental backup apply only the last level 1 backup to the level 0 backup.
With a differential incremental backup apply all level 1 backups to the level 0 backup.
A cumulative incremental backup usually consumes more space in the fast recovery area than a differential incremental backup.
Backup to Disk: Best Practices for Minimizing System Resource Consumption (I/O and CPU) If system resource consumption is your primary concern then an incremental backup with a Block Change Tracking enabled consumes the least amount of resources on the database.
Example
For many applications, only a small percentage of the entire database is changed each day even if the transaction rate is very high. In many cases, applications repeatedly modify the same set of blocks; so, the total unique, changed block set is small.
For example, a database contains about 600 GB of user data, not including temp files and redo logs. Every 24 hours, approximately 2.5% of the database is changed, which is approximately 15 GB of data. In this example, MAA testing recorded the following results:
Level 0 backup takes 180 minutes, including READs from the data area and WRITEs to the fast recovery area
Level 1 backup takes 20 minutes, including READs from the data area and WRITEs to the fast recovery area
Rolling forward and merging an existing image copy in the fast recovery area with a newly created incremental backup takes only 45 minutes, including READs and WRITEs from the fast recovery area.
In this example, the level 0 backup (image copy) takes 180 minutes. This is the same amount of time it takes to perform a full backup set.
Subsequent backups are level 1 (incremental), which take 20 minutes, so the potential impact on the data area is reduced. That backup is then applied to the existing level 0 backup, which takes 45 minutes. This process does not perform I/O to the data area, so there is no impact (assuming the fast recovery area and data area use separate storage). The total time to create the incremental backup and apply it to the existing level 0 backup is 65 minutes (20+45).
The result is the same using incrementally updated backups or full backup sets, a full backup of the database is created. The incremental approach takes 115 minutes less time (64% less) than simply creating a full backup set. In addition, the I/O impact is less, particularly against the data area which should have less detrimental effect on production database performance.
Thus, for this example when you compare a full backup set strategy versus starting with an image copy, performing only incremental backups, and then rolling forward the copy, the net savings are:
Thus, for this example when you compare always taking full backups versus starting with a level 0 backup, performing only incremental backups, and then rolling forward the level 0 backup, the net savings are:
115 minutes or 64% time savings to create a complete backup
Reduced I/O on the database during backups
|
See Also: Oracle Database Backup and Recovery User's Guide for more information about backing up the database |
Recovery Manager (RMAN) provides automated disk backup for the Oracle database and is integrated with media management products such as Oracle Secure Backup for backup to tape. Whether your Oracle database backup strategy uses disk, tape, or both, the combination of RMAN and Oracle Secure Backup delivers a comprehensive solution to meet your specific requirements.
When installing Oracle Secure Backup, the System Backup Tape (SBT) libraries for RMAN tape backups are automatically linked. Using Oracle Enterprise Manager Database Control you can manage the Oracle Secure Backup backup domain from tape vaulting to backup and restore operations. The tight integration between RMAN, Oracle Enterprise Manager Database Control, and Oracle Secure Backup makes initial configuration a simple process.
Perform the following four steps to perform initial configuration and prepare to backup a database to tape:
Define your Oracle Secure Backup Administrative Server in Oracle Enterprise Manager Database Control enabling the Oracle Secure Backup domain to be managed through Oracle Enterprise Manager.
Pre-authorize an Oracle Secure Backup user for use with RMAN allowing the RMAN backup/restore be performed without having to explicitly login to Oracle Secure Backup.
Set-up media policies in Oracle Secure Backup to be used for RMAN backups.
Establish RMAN backup settings such as parallelism and compression.
|
Note: If you use Oracle Secure Backup or tape-side compression, do not also use RMAN compression. |
|
See Also: Oracle Secure Backup Administrator's Guide for more information about using Recovery Manager with Oracle Secure Backup |
Once backup data stored on tape is no longer needed, its lifecycle is complete and the tape media can be reused. Management requirements during a tape's lifecycle (retention period) may include duplication and vaulting across multiple storage locations. Oracle Secure Backup provides effective media lifecycle management through user-defined media policies, including:
Retention
Tape duplication
Vaulting: rotation of tapes between multiple locations
Media lifecycle management may be as simple as defining appropriate retention settings or more complex to include tape duplication with the original and duplicate(s) having different retention periods and vaulting requirements. Oracle Secure Backup media families, often referred to as tape pools, provide the media lifecycle management foundation.
The best practice recommendation is to leverage content-managed media families which use defined RMAN retention parameters associated with the database to determine when the tape may be reused (effectively an expired tape). A specific expiration date is not associated with content-managed tapes as is done with time-managed. The expiration or recycling of these tapes is based on the attribute associated with the backup images on the tape. All backup images written to content-managed tapes automatically have an associated "content-manages reuse" attribute. Since the recycling of content-managed tapes adheres to user-defined RMAN retention settings, RMAN instructs Oracle Secure Backup when to change the backup image attribute to "deleted".
The RMAN DELETE OBSOLETE command communicates which backup pieces (images) are no longer required to meet the user-defined RMAN retention periods. Once Oracle Secure Backup receives this communication, the backup image attribute is changed to "deleted". The actual backup image is not deleted but the attribute is updated within the Oracle Secure Backup catalog. Once all backup images on tape have a deleted attribute, Oracle Secure Backup considers the tape eligible for reuse, similar to that of an expired time-managed tape.
Oracle Secure Backup provides policy-based media management for RMAN backup operations through user-defined Database Backup Storage Selectors. One Database Backup Storage Selector (SSEL) may apply to multiple databases or multiple SSELs may be associated with a single database. For example, you would create two SSEL for a database when using RMAN duplexing and each copy should be written to a different media family. The SSEL contains the following information:
Database name / ID or applicable to all databases
Hostname or applicable to all hosts
Content: archive logs, full, incremental, autobackup or applicable to all
RMAN copy number (applicable when RMAN duplexing is configured)
Media family name
Name(s) of devices to which operations are restricted (if no device restrictions are configured, Oracle Secure Backup uses any available device)
Wait time (duration) for available tape resources
Encryption setting
Oracle Secure Backup automatically uses the storage selections defined within a SSEL without further user intervention. To override the storage selections for one time backup operations or other exceptions, define alternate media management parameters in the RMAN backup script. For more information, see:
http://www.oracle.com/technetwork/database/secure-backup/documentation/index.html
You can easily backup the Fast Recovery Area (FRA) to tape with the RMAN command: BACKUP RECOVERY AREA. Using this disk to tape backup method instead of performing a separate backup of the production database to tape provides a few distinct advantages:
Saves tape consumption by creating an optimized backup of the Fast Recovery Area (FRA) thereby eliminating unnecessary backup of files already protected on tape
Enables RMAN to use better restore intelligence from disk then tape as necessary, otherwise, RMAN would restore from the most recent backup regardless of media type
Reduces I/O on the production database since the FRA uses a separate disk group
Upon restoration, RMAN automatically selects the most appropriate backup to restore from disk or tape. If the required backup is on tape, RMAN would restore or recovery the database directly from tape media through integration with Oracle Secure Backup. As RMAN has intimate knowledge of what files are necessary for recovery, restoration from disk or tape is an automated process.
While it is possible to backup the FRA or other RMAN disk backup to tape outside of RMAN by performing a file system backup of the disk area using the media management software, it is not recommended. If RMAN is not aware of the tape backup then restoration is an error-prone, manual process:
DBA must determine what files are needed for the restoration.
Media manager administrator would then restore designated files from tape backups to a disk location.
Once files on disk, DBA would initiate an RMAN restore or recovery from the disk location.
The combination of RMAN and Oracle Secure Backup provides an integrated Oracle database tape backup solution.
Backup tapes are highly portable and often stored at offsite locations for disaster recovery purposes. These tapes are first created from within a tape device but often are removed from the hardware device. Once backup tapes are removed from a hardware device the tapes may be stored in an on-site or offsite location. You can effectively manage tape movement between multiple locations using Oracle Secure Backup rotation policies.
For Oracle database restoration, a restore request is submitted from RMAN to Oracle Secure Backup. If the tapes are within the library, the restore begins immediately assuming device availability. However, if the tapes needed for restore could be offsite; you may want to confirm the location of tapes before you issue the restore command. With RMAN and Oracle Secure Backup you can easily do so by issuing the following RMAN command(s):
RESTORE DATABASE PREVIEW command provides a list of tapes needed for restoration which are offsite.
RESTORE DATABASE PREVIEW RECALL command initiates a recall operation through Oracle Secure Backup to return the tapes from offsite to the tape device for restoration. Once the tapes are on-site, you can begin the RMAN restore operation.
|
See Also:
|
This section outlines procedures to regularly check for corruption of data files using the Data Recovery Advisor, for testing recovery procedures, and for backing up the recovery catalog database.
Data Recovery Advisor is an Oracle Database tool that automatically diagnoses data failures, determines and presents appropriate repair options, and executes repairs at the user's request. In this context, a data failure is a corruption or loss of persistent data on disk. By providing a centralized tool for automated data repair, Data Recovery Advisor improves the manageability and reliability of an Oracle database and thus helps reduce the mean time to recover (MTTR).
|
See Also: Oracle Database Backup and Recovery User's Guide for information about using Data Recovery Advisor |
Use the RMAN VALIDATE command to regularly check database files for block corruption that has not yet been reported by a user session or by normal backup operations. RMAN scans the specified files and checks for physical and logical errors, but does not actually perform the backup or recovery operation. Oracle database records the address of the corrupt block and the type of corruption in the control file. Access these records through the V$DATABASE_BLOCK_CORRUPTION view, which can be used by RMAN block media recovery.
To detect all types of corruption that are possible to detect, specify the CHECK LOGICAL option.
|
See Also: Oracle Database Backup and Recovery User's Guide for information about Validating Database Files and Backups |
Comple-xte, successful, and tested backups are fundamental to the success of any recovery. Create test plans for different outage types. Start with the most common outage types and progress to the least probable. Using the RMAN DUPLICATE command is a good way to perform recovery testing, because it requires restoring from backups and performing media recovery.
Monitor the backup procedure for errors, and validate backups by testing your recovery procedures periodically. Also, validate the ability to restore the database using the RMAN command RESTORE...VALIDATE.
|
See Also:
|
Include the recovery catalog database in your backup and recovery strategy. If you do not back up the recovery catalog and a disk failure occurs that destroys the recovery catalog database, then you may lose the metadata in the catalog. Without the recovery catalog contents, recovery of your other databases is likely to be more difficult.
The Oracle Secure Backup catalog maintains backup metadata, scheduling and configuration details for the backup domain. Just as it's important to protect the RMAN catalog or control file, the Oracle Secure Backup catalog should be backed up on a regular basis.
|
See Also:
|
Oracle Secure Backup provides tape backup for non-Oracle files. Oracle Secure Backup does not have pro-active checking as the database does. Use the features available to backup files outside the database. For more information, see Section 8.6, "Backup Files Outside the Database".
Oracle IT environments include both database and application files that must be protected for short and long-term retention requirements. Differences exist between backing up database and unstructured files. In addition, managing backup and recovery often crosses organizational areas such as DBA for the database and system administration for file system data could cross organizational areas. The Oracle data protection suite offers a cohesive solution meeting your complete needs for Oracle database and non Oracle database storage.
An Oracle ACFS snapshot is an online, read-only, point in time copy of an Oracle ACFS file system. The snapshot copy is space-efficient and uses Copy-On-Write functionality. Before an Oracle ACFS file extent is modified or deleted, its current value is copied to the snapshot to maintain the point-in-time view of the file system.
Oracle ACFS snapshot can support the online recovery of files inadvertently modified or deleted from a file system. With up to 63 snapshot views supported for each file system, flexible online file recovery solutions spanning multiple views can be employed. An Oracle ACFS snapshot can also be used as the source of a file system backup, as it can be created on demand to deliver a current, consistent, online view of an active file system.
|
See Also:
|
Oracle's Sun ZFS Storage Appliance provides an integrated high performance backup solution and is also a cost effective platform for disaster recovery for non-Database files. Ever-growing amounts of data present system and database administrators with many challenges—the thorniest of which are associated with the complex process of backup and recovery. Without reliable data protection and processes, mission-critical data is at risk.
Oracle's Sun ZFS Storage Appliance is an easy-to-deploy Unified Storage System that ensures that backup window and recovery time objectives (RTO) are met by providing timely recovery in the event of a disaster.
Oracle's Sun ZFS Storage Appliance supports unlimited snapshot capability. A snapshot similar to Oracle ACFS is a read-only, point-in-time copy of a file system (for information about Oracle ACFS, see Section 8.6.1, "ACFS Snapshots"). It is instantaneously created and no space is allocated initially. Blocks are allocated as changes are made to the base file system (copy-on-write). The snapshots are either initiated manually or can be automated by scheduling at specific intervals. The snapshot data can be directly accessed for any backup purposes. Any reads to the snapshot blocks are served by the base file system's block. When changes happen to the base file system, the older block is now referenced by the snapshot and the new changed block is referenced by the file system.
Oracle's Sun ZFS Storage Appliance is also recommended for development and test systems that are snapshots taken from the standby database in a Data Guard environment.
Snapshot rollback is the process to bring the base file system to the point in time when the snapshot is taken. The rollback process discards all the changes that happened to the base file system from the time of the snapshot to the time of rollback. This removes the need for a data restore process.
|
See Also:
|
Oracle Secure Backup provides centralized tape backup management for heterogeneous file system data and the Oracle database. Oracle Secure Backup offers multiple backups levels with full, cumulative and differential incrementals. For more information, see Section 8.4, "Backup to Tape Best Practices". In addition, a full offsite backup level may be scheduled without interfering with the regular full/incremental schedule. File system backups can be performed at the file, directory, file system, or raw partition level meeting even the most stringent requirements within user-defined backup windows.
For file system backup operations, you define Oracle Secure Backup "datasets" which describes what to backup. A dataset is a textual description employing a lightweight language to communicate how to build and organize files to be protected. Being Oracle Database aware, Oracle Secure Backup can skip database files during file system backups by using the "exclude oracle database files" directive within the dataset.
|
See Also: The Oracle Secure Backup Administrator's Guide for more information about file system backup operations |
High Availability Best Practices
11g Release 2 (11.2)
E10803-02
May 2012
Oracle Database High Availability Best Practices 11g Release 2 (11.2)
E10803-02
Copyright © 2005, 2012, Oracle and/or its affiliates. All rights reserved.
Primary Authors: Lawrence To, Viv Schupmann, Thomas Van Raalte, Virginia Beecher
Contributing Author: Janet Stern
Contributors: Andrew Babb, Janet Blowney, Larry Carpenter, Timothy Chien, Jay Davison, Senad Dizdar, Ray Dutcher, Mahesh Girkar, Stephan Haisley, Wei Ming Hu, Holger Kalinowski, Nitin Karkhanis, Frank Kobylanski, Rene Kundersma, Joydip Kundu, Barb Lundhild, Roderick Manalac, Pat McElroy, Robert McGuirk, Joe Meeks, Markus Michalewicz, Valarie Moore, Michael Nowak, David Parker, Darryl Presley, Hector Pujol, Michael T. Smith, Vinay Srihari, Mark Townsend, Douglas Utzig, Thomas Van Raalte, James Viscusi, Vern Wagman, Steve Wertheimer, Shari Yamaguchi
This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.
If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065.
This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.
This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.
Use operational best practices to provide a successful MAA implementation.
This chapter contains the following topics:
Provide a Plan to Test and Apply Recommended Patches and Software
Configure Monitoring and Service Request Infrastructure for High Availability
Understand and document your high availability and performance service-level agreements (SLAs) and create an outage and solution matrix:
Document the business's cost of downtime, Recovery Time Objectives (RTO or recovery time) and Recovery Point Objectives (RPO or data loss tolerance) for the outages described in Oracle Database High Availability Overview.
Build an outage and solution matrix similar those shown in Table 13-1, "Recovery Times and Steps for Unscheduled Outages on the Primary Site" and Table 14-1, "Solutions for Scheduled Outages on the Primary Site".
Implement a high availability environment to achieve the optimal high availability architecture:
Install or update your software with the latest certified patch sets
Configure your software using best practices
Document your choices and configuration
Validate and automate repair operations to ensure that you meet your target HA service-level agreements (SLAs). You should validate the backup, restore, and recovery operations and periodically evaluate all repair operations for various types of possible outages (see Table 13-1 for more information).
If you use Oracle Data Guard for disaster recovery and data protection, Oracle recommends that you perform periodic switchover operations or conduct full application and database failover tests. Also, periodically execute Application and Data Guard switchovers to fully validate all role transition procedures. For more information see Section 2.8, "Execute Data Guard Role Transitions".
Corporate data can be at grave risk if placed on a system or database that does not have proper security measures in place. A well-defined security policy can help protect your systems from unwanted access and protect sensitive corporate information from sabotage. Proper data protection reduces the chance of outages due to security breaches. For more information, see the Oracle Database Security Guide.
Institute procedures that manage and control changes as a way to maintain the stability of the system and to ensure that no changes are incorporated in the primary database unless they have been rigorously evaluated on your test systems.
Review the changes and get feedback and approval from your change management team, which should include representatives for any component that affects the business requirements, functionality, performance, and availability of your system. For example, the team can include representatives for end-users, applications, databases, networks, and systems.
By periodically testing and applying the latest recommended patches and software versions, you ensure that your system has the latest security and software fixes required to maintain stability and avoid many known issues. Remember to validate all updates and changes on a test system before performing the upgrade on the production system. For more information, see "Oracle Recommended Patches -- Oracle Database" in My Oracle Support Note 756671.1 at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=756671.1
Proper testing and patching are important prerequisites for preventing instability. You must validate any change in your test systems thoroughly before applying it to your production environment. These practices involve the following:
The test system should be a close replica of the production and standby environment and workload to execute functional tests, performance tests, and availability tests (for more information, see Table 13-1). Any changes should be validated in the test environment first, including evaluating the effect of changes and the fallback procedures before introducing the changes in the production environment.
With a properly configured test system, many problems can be avoided because changes are validated with an equivalent production and standby database configuration containing a full data set and using a workload framework to mimic production (for example, using Oracle Real Application Testing).
Do not try to reduce costs by eliminating the test system because that decision ultimately affects the stability and the availability of your production applications. Using only a subset of system resources for testing and QA has the tradeoffs shown in Table 2-1.
Table 2-1 Tradeoffs for Different Test and QA Environments
| Test Environment | Benefits and Tradeoffs |
|---|---|
|
Full Replica of Production and Standby Systems |
Validate all patches and software changes. Validate all functional tests. Full performance validation at production scale. Full HA validation. |
|
Full Replica of Production Systems |
Validate all patches and software changes. Validate all functional tests. Full performance validation at production scale. Full HA validation minus the standby system. No functional, performance, HA and disaster recovery validation with standby database. |
|
Standby System |
Validate most patches and software changes. Validate all functional tests. Full performance validation if using Data Guard Snapshot Standby but this can extend recovery time if a failover is required. Role transition validation. Resource management and scheduling is required if standby and test databases exist on the same system. |
|
Shared System Resource |
Validate most patches and software changes. Validate all functional tests. This environment may be suitable for performance testing if enough system resources can be allocated to mimic production. Typically, however, the environment includes a subset of production system resources, compromising performance testing/validation. Resource management and scheduling is required. |
|
Smaller or Subset of the system resources |
Validate all patches and software changes. Validate all functional tests. No performance testing at production scale. Limited full-scale high availability evaluations. |
|
Different hardware or platform system resources but same operating system |
Validate most patches and software changes. Limited firmware patching test. Validate all functional tests unless limited by new hardware features. Limited production scale performance tests. Limited full-scale high availability evaluations. |
Pre-production validation and testing of software patches or any change is an important way to maintain stability. The high-level pre-production validation steps are:
Review the patch or upgrade documentation or any document relevant to that change. Evaluate the possibility of performing a rolling upgrade if your SLAs require zero or minimal downtime. Evaluate any rolling upgrade opportunities to minimize or eliminate planned downtime. Evaluate whether the patch qualifies for Standby-First Patching.
|
Note: Standby-First Patch enables you to apply a patch initially to a physical standby database while the primary database remains at the previous software release (this applies to certain types of patches and does not apply to Oracle patch sets and major release upgrades; use the Data Guard transient logical standby method for patch sets and major releases). Once you are satisfied with the change, then perform a switchover to the standby database. The fallback is to switchback if required. Alternatively, you can proceed to the following step and apply the change to your production environment. For more information, see "Oracle Patch Assurance - Data Guard Standby-First Patch Apply" in My Oracle Support Note 1265700.1 at
|
Validate the application in a test environment and ensure the change meets or exceeds your functionality, performance, and availability requirements. Automate the procedure and be sure to also document and test a fallback procedure. This requires comparing metrics captured before and after patch application on the test and against metrics captured on the production system. Real Application Testing may be used to capture the workload on the production system and replay it on the test system. AWR and SQL Performance Analyzer may be used to assess performance improvement or regression resulting from the patch.
Validate the new software on a test system that mimics your production environment, and ensure the change meets or exceeds your functionality, performance, and availability requirements. Automate the patch or upgrade procedure and ensure fallback. Being thorough during this step eliminates most critical issues during and after the patch or upgrade.
See Section 2.7.1, "Configuring the Test System and QA Environments" for more information about configuring your test system.
Optionally, use the Oracle Real Application Testing option that enables you to perform real-world testing of Oracle Database. Oracle Real Application Testing captures production workloads and assesses the impact of system changes before production deployment; thus, Oracle Real Application Testing minimizes the risk of instabilities associated with changes. Oracle GoldenGate can also be used as another logical replica to apply changes.
See Section 2.7.1, "Configuring the Test System and QA Environments" for more information about configuring your test system.
If applicable, perform final pre-production validation of all changes on a Data Guard standby database before applying them to production. Apply the change in an Oracle Data Guard environment, if applicable. For more information about Data Guard transient logical standby method, see Section 14.2.6, "Database Upgrades".
Apply the change in your production environment.
|
See Also:
|
When you have a standby database(s), it is important to ensure that the operations and DBA teams are well prepared to use the standby database(s) at anytime when the primary database is down or underperforming, according to a predetermined threshold. By reacting and executing efficiently, which includes detection and making the decision to failover, overall downtime can be reduced from hours to minutes.
If you use Oracle Data Guard for disaster recovery and data protection, Oracle recommends that you perform periodic switchover operations every quarter or conduct full application and database failover tests. For more information about configuring Oracle Data Guard and role transition best practices, see Chapter 9, "Configuring Oracle Data Guard" and Section 9.4.1, "Oracle Data Guard Switchovers Best Practices."
|
See: My Oracle Support provides notes for Data Guard switchovers:
|
Establish escalation management procedures so repair is not hindered. Most repair solutions, when conducted properly are automatic and transparent with the MAA solution. The challenges occur when the primary database or system is not meeting availability or performance SLAs and failover procedures are not automatic as in the case with some Data Guard failover scenarios. Downtime can be prolonged if proper escalation policies are not followed and decisions are not made quickly.
Availability is the top priority, and a contingency plan should be created to gather sufficient data for future Root Cause Analysis (RCA).
For more information about MAA outage and repair, check the MAA web page on the Oracle Technology Network (OTN) at
To maintain your High Availability environment, you should configure the monitoring infrastructure that can detect and react to performance and high availability related thresholds. Also, where available, Oracle can detect failures, dispatch field engineers, and replace failing hardware without customer involvement.
You should configure and use Enterprise Manager and the monitoring infrastructure that detects and reacts to performance and high availability related thresholds to avoid potential downtime. The monitoring infrastructure assists you with monitoring for High Availability and enables you to do the following:
Monitor system, network, application, database and storage statistics
Monitor performance and service statistics
Create performance and high availability thresholds as early warning indicators of system or application problems
|
See Also:
|
In addition to monitoring infrastructure with Enterprise Manager in the Oracle HA environment where available, Oracle can detect failures, dispatch field engineers, and replace failing hardware without customer involvement. For example, Oracle Auto Service Request (ASR) is a secure, scalable, customer-installable software solution available as a feature. The software resolves problems faster by using auto-case generation for Oracle's Sun server and storage systems when specific hardware faults occur.
|
See Also: See "Oracle Auto Service Request" in My Oracle Support Note 1185493.1 at
|
Using Oracle Enterprise Manager 11g is the MAA best practice recommendation for configuring your entire high availability environment. Oracle Enterprise Manager is Oracle's single, integrated solution for managing all aspects of the Oracle Grid and the applications running on it. Oracle Enterprise Manager Grid Control couples top-down monitoring for applications with automated configuration management, provisioning, and administration. This powerful combination provides unequaled management for any size Oracle data center.
Using Oracle Enterprise Manager you can perform most configuration tasks. For example, you can:
Migrate to Oracle Automatic Storage Management (Oracle ASM)
Migrate a single-instance Oracle Database to Oracle Clusterware and Oracle Real Application Clusters (Oracle RAC)
Create Oracle Data Guard standby databases
Configure backup and recovery
Implement Oracle Active Data Guard
Use the MAA Advisor to implement Oracle's best practices and achieve a high availability architecture
For information about the configuration Best Practices for Oracle Database, see the following chapters:
|
See Also: Oracle Enterprise Manager online help system, and the documentation set available at
|
By implementing and using Oracle Maximum Availability Architecture (MAA) best practices, you can provide high availability for the Oracle database and related technology.
This chapter contains the following topics:
Designing and implementing a high availability architecture can be a daunting task given the broad range of Oracle technologies and hardware, software, and deployment options. A successful effort begins with clearly defined and thoroughly understood business requirements. Thorough analysis of the business requirements enables you to make intelligent design decisions and develop an architecture that addresses your business needs in the most cost effective manner. The architecture you choose must achieve the required levels of availability, performance, scalability, and security. Moreover, the architecture you choose should have a clearly defined plan for deployment and ongoing management that minimizes complexity and business risk.
Once your business requirements are understood, you should begin designing your high availability architecture by reading the Oracle Database High Availability Overview to get a high-level view of the various Oracle solutions that comprise the Oracle Maximum Availability Architecture (MAA). This should result in a design for an architecture that can be fully examined and validated using the best practices documented in this book.
Oracle High Availability (HA) best practices help you deploy a highly available architecture throughout your enterprise. Having a set of configuration and operational best practices helps you achieve high availability and reduces the cost associated with the implementation and ongoing maintenance of your enterprise. Also, employing best practices can optimize usage of system resources.
By implementing the HA best practices you can:
Reduce the cost of creating an Oracle Database high availability system by following detailed guidelines on configuring your database, storage, application failover, backup and recovery. See Chapter 3, "Overview of Configuration Best Practices" for more information.
Use operational best practices to maintain your system. See Chapter 2, "Operational Prerequisites to Maximizing Availability" for more information.
Detect and quickly recover from unscheduled outages caused by computer failure, storage failure, human error, or data corruption. For more information, see Section 5.1.6, "Protect Against Data Corruption" and Chapter 13, "Recovering from Unscheduled Outages".
Eliminate or reduce downtime due to scheduled maintenance such as database patches or application upgrades as described in Chapter 14, "Reducing Downtime for Planned Maintenance".
Oracle Maximum Availability Architecture (MAA) is Oracle's best practices blueprint based on Oracle High Availability (HA) technologies, extensive validation performed by the Oracle MAA development team, and the accumulated production experience of customers who have successfully deployed business critical applications on Oracle.
MAA covers Oracle products within the following technologies:
Oracle Database as described in this book
Oracle Exadata Database Machine and Oracle Exalogic Elastic Cloud
Oracle Fusion Middleware and Oracle WebLogic Server
Oracle Applications (Siebel, Peoplesoft, E-Business Suite)
Oracle Collaboration Suite
Oracle Enterprise Manager
This book, Oracle Database High Availability Best Practices primarily focuses on high availability best practices for the Oracle Database. There are also other components for which you might want to consider Oracle Maximum Availability Architecture (MAA) best practices. For more information go to:
http://www.oracle.com/goto/MAA
|
See:
|
The goal of MAA is to achieve the optimal HA architecture at the lowest cost and complexity. MAA provides:
Best practices that span the Exadata Database Machine, Oracle Database, Oracle Fusion Middleware, Oracle Applications, Oracle Enterprise Manager, and solutions provided by Oracle Partners.
Accommodates a range of business requirements to make these best practices as widely applicable as possible.
Leverages lower-cost servers and storage.
Uses hardware and operating system independent features and evolves with new Oracle versions and features. The only exception is Exadata MAA which has specific and customized configuration and operating practices for Exadata Database Machine.
Makes high availability best practices as widely applicable as possible considering the various business service level agreements (SLA).
Uses the Oracle Grid Infrastructure with Database Server Grid and Database Storage Grid to provide highly resilient, scalable, and lower cost infrastructures.
Provides the ability to control the length of time to recover from an outage and the amount of acceptable data loss from any outage.
For more information about MAA and documentation about best practices for all MAA components, visit the MAA website at
This chapter provides best practices for monitoring your system using Enterprise Manager and to monitor and maintain a highly available environment across all tiers of the application stack.
This chapter contains the following topics:
Continuous monitoring of the system, network, database operations, application, and other system components ensures early detection of problems. Early detection improves the user's system experience because problems can be avoided or resolved faster. In addition, monitoring captures system metrics to indicate trends in system performance, growth, and recurring problems. This information can facilitate prevention, enforce security policies, and manage job processing. For the database server, a sound monitoring system must measure availability and detect events that can cause the database server to become unavailable, and provide immediate notification about critical failures to responsible parties.
The monitoring system itself must be highly available and adhere to the same operational best practices and availability practices as the resources it monitors. Failure of the monitoring system leaves all monitored systems unable to capture diagnostic data or alert the administrator about problems.
Enterprise Manager provides management and monitoring capabilities with many different notification options. Recommendations are available for methods of monitoring the environment's availability and performance, and for using the tools in response to changes in the environment.
A major benefit of Enterprise Manager is its ability to manage components across the entire application stack, from the host operating system to a user or packaged application. Enterprise Manager treats each of the layers in the application as a target. Targets—such as databases, application servers, and hardware—can then be viewed along with other targets of the same type, or can be grouped by application type. You can also review all targets in a single view from the High Availability Console (for more information, Section 12.3.3, "Manage Database Availability with the High Availability Console"). Each target type has a default generated home page that displays a summary of relevant details for a specific target. You can group different types of targets by function; that is, as resources that support the same application.
Every target is monitored by an Oracle Management Agent. Every Management Agent runs on a system and is responsible for a set of targets. The targets can be on a system that is different from the one that the Management Agent is on. For example, a Management Agent can monitor a storage array that cannot host an agent natively. When a Management Agent is installed on a host, the host is automatically discovered along with other targets that are on the machine.
Moreover, to help you implement the Maximum Availability Architecture (MAA) best practices, Enterprise Manager provides the MAA Advisor (for more information, see Section 12.3.4, "Configure High Availability Solutions with MAA Advisor"). The MAA Advisor page recommends Oracle solutions for most outage types and describes the benefits of each solution.
In addition to monitoring infrastructure with Enterprise Manager in the Oracle HA environment, Oracle Auto Service Request (ASR) can be used to resolve problems faster by using auto-case generation for Oracle's Sun server and storage systems when specific hardware faults occur. For more information, see "Oracle Auto Service Request" in My Oracle Support Note 1185493.1 at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1185493.1
|
See Also: Oracle Enterprise Manager Concepts for information about Enterprise Manager Architecture and the Oracle Management Agent |
The Enterprise Manager home page in Figure 12-1 shows the availability of all discovered targets.
The Enterprise Manager home page includes the following information:
A snapshot of the current availability of all targets. The All Targets Status pie chart gives the administrator an immediate indication of any target that is Available (Up), unavailable (Down), or has lost communication with the console (Unknown).
An overview of how many alerts and problems (for jobs) are known in the entire monitored system. You can display detailed information by clicking the links, or by navigating to the Alerts tab from any Enterprise Manager page.
A view of the severity and total number of policy violations for all managed targets. Drill down to determine the source and type of violation.
All Targets Jobs lists the number of scheduled, running, suspended, and problem (stopped/failed) executions for all Enterprise Manager jobs. Click the number next to the status group to view a list of those jobs.
An overview of what is actually discovered in the system. This list can be shown at the hardware level and the Oracle level.
Alerts are generated by a combination of factors and are defined on specific metrics. A metric is a data point sampled by a Management Agent and sent to the Oracle Management Repository. An alert could be the availability of a component through a simple heartbeat test, or an evaluation of a specific performance measurement such as "disk busy" or percentage of processes waiting for a specific wait event.
There are four states that can be checked for any metric: error, warning, critical, and clear. The administrator must make policy decisions such as:
What objects should be monitored (databases, nodes, listeners, or other services)?
What instrumentation should be sampled (such as availability, CPU percent busy)?
How frequently should the metric be sampled?
What should be done when the metric exceeds a predefined threshold?
All of these decisions are predicated on the business needs of the system. For example, all components might be monitored for availability, but some systems might be monitored only during business hours. Systems with specific performance problems can have additional performance tracing enabled to debug a problem.
|
See Also: Oracle Enterprise Manager Cloud Control Introduction for more information about monitoring and using metrics in Enterprise Manager |
Notification Rules are defined sets of alerts on metrics that are automatically applied to a target when it is discovered by Enterprise Manager. For example, an administrator can create a rule that monitors the availability of database targets and generates an e-mail message if a database fails. After that rule is generated, it is applied to all existing databases and any database created in the future. Access these rules by navigating to Preferences and then choosing Rules.
The rules monitor problems that require immediate attention, such as those that can affect service availability, and Oracle or application errors. Service availability can be affected by an outage in any layer of the application stack: node, database, listener, and critical application data. A service availability failure, such as the inability to connect to the database, or the inability to access data critical to the functionality of the application, must be identified, reported, and reacted to quickly. Potential service outages such as a full archive log directory also must be addressed correctly to avoid a system outage.
Enterprise Manager provides a series of default rules that provide a strong framework for monitoring availability. A default rule is provided for each of the preinstalled target types that come with Enterprise Manager. You can modify these rules to conform to the policies of each individual site, and you can create rules for site-specific targets or applications. You can also set the rules to notify users during specific time periods to create an automated coverage policy.
Use the following best practices:
Modify each rule for high-value components in the target architecture to suit your availability requirements by using the rules modification wizard. For the database rule, set the metrics in Table 12-1, Table 12-2, and Table 12-3 for each target. The frequency of the monitoring is determined by the service-level agreement (SLA) for each component.
Use Beacon functionality to track the performance of individual applications. A Beacon can be set to perform a user transaction representative of normal application work. Enterprise Manager can then break down the response time of that transaction into its component pieces for analysis. In addition, an alert can be triggered if the execution time of that transaction exceeds a predefined limit.
Add Notification Methods and use them in each Notification Rule. By default, the easiest method for alerting an administrator to a potential problem is to send e-mail. Supplement this notification method by adding a callout to an SNMP trap or operating system script that sends an alert by some method other than e-mail. This avoids problems that might occur if a component of the e-mail system fails. Set additional Notification Methods by using the Setup link at the top of any Enterprise Manager page.
Modify Notification Rules to notify the administrator when there are errors in computing target availability. This might generate a false positive reading on the availability of the component, but it ensures the highest level of notification to system administrators.
|
See Also:
|
Figure 12-2 shows the Edit Notification Rule property page for choosing availability states, with the Down option chosen.
Figure 12-2 Setting Notification Rules for Availability
In addition, ensure that the metrics listed in Table 12-1, Table 12-2, and Table 12-3 are added to the database notification rule. Configure those metrics using the Metrics and Policy Settings page, which can be accessed from the Related Links section of the Database Homepage.
Use the metrics shown in Table 12-1 to monitor space management conditions that have the potential to cause a service outage.
Table 12-1 Recommendations for Monitoring Space
| Metric | Recommendation |
|---|---|
|
Tablespace Space Used (%) |
Set this database-level metric to check the Available Space Used (%) for each tablespace. For cluster databases, this metric is monitored at the cluster database target level and not by member instances. This metric enables the administrator to choose the threshold percentages that Enterprise Manager tests against, and the number of samples that must occur in error before a message is generated and sent to the administrator. If the percentage of used space is greater than the values specified in the threshold arguments, then a warning or critical alert is generated. The recommended default settings are 85% for a warning and 97% for a critical space usage threshold, but you should adjust these values appropriately, depending on system usage. Also, you can customize this metric to monitor specific tablespaces. Note: there is an Enterprise Manager Job in the Job Library named:
Use this Job to disable alerts for all |
|
Dump Area Used (%) |
Set this metric to monitor the dump directory destinations. Dump space must be available so that the maximum amount of diagnostic information is saved the first time an error occurs. The recommended default settings are 70% for a warning and 90% for an error, but these should be adjusted depending on system usage. Set this metric in the Dump Area metric group. |
|
Recovery Area Free Space (%) |
This is a database-level metric that is evaluated by the server every 15 minutes or during a file creation, whichever occurs first. The metric is also printed in the alert log. For cluster databases, this metric is monitored at the cluster database target level and not by member instances. The Critical Threshold is set for < 3% and the Warning Threshold is set for < 15%. You cannot customize these thresholds. An alert is returned the first time the alert occurs, and the alert is not cleared until the available space rises above 15%. |
|
File System Available(%) |
By default, this metric monitors the root file system per host. The default warning level is 20% and the critical warning is 5%. |
|
Archive Area Used (%) |
Set this metric to return the percentage of space used on the archive area destination. If the space used is more than the threshold value given in the threshold arguments, then a warning or critical alert is generated. If the database is not running in |
In Enterprise Manager 11g the mechanism for monitoring the Database Alert Log is tightly integrated with the Support Workbench, with the benefits of being able to generate packages for each problem or incident reported and quickly upload them to support.
As part of integrating with the Support Workbench, errors are categorized into different classes and groups, each served by a separate metric. At the highest level of categorization there are two different classes of errors: incidents and operational errors.
Incidents are errors that are recorded in the database alert log file, which signify that the database being monitored has detected a critical error condition. For example a critical error condition could be a generic internal error or an access violation.
Operational Errors are errors that are recorded in the database alert log file, which signify that the database being monitored has detected an error that may affect the operation of the database. For example, an operational error could be an indication that the archiver is hung or a media failure.
Configure the metrics that raise alerts for errors reported in the Alert Log as shown in Table 12-2.
|
Note: For more information about the Alert Log metrics in Table 12-2 and Alert Log Monitoring for 11g database targets in Enterprise Manager, see "Monitoring 11g Database Alert Log Errors in Enterprise Manager" in My Oracle Support Note 949858.1 at
|
Table 12-2 Recommendations for Monitoring the Alert Log
| Metric | Recommendation |
|---|---|
|
Generic Internal Error Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Access Violation Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Session Terminated Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Out of Memory Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Redo Log Corruption Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Inconsistent DB State Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Deadlock Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more Note: This metric does not raise alerts when application level deadlocks ( |
|
Internal SQL Error Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Cluster Error Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Data Block Corruption Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Media Failure Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more |
|
Generic Incident Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more of the errors: |
|
Generic Operational Error Status |
Set the Critical threshold to 0 to ensure an alert is raised each time one or more generic operational errors have been reported in the Alert Log since the last time the metric was collected. |
Monitor the system to ensure that the processing capacity is not exceeded. The warning and critical thresholds for these metrics should be modified based on the usage pattern of the system, following the recommendations in Table 12-3.
Table 12-3 Recommendations for Monitoring Processing Capacity
| Metric | Recommendation |
|---|---|
|
Process limit |
Set thresholds for this metric to warn if the number of current processes approaches the value of the |
|
Session limit |
Set thresholds for this metric to warn if the instance is approaching the maximum number of concurrent connections allowed by the database. |
Figure 12-3 shows the Metric and Policy settings page for setting and editing metrics. The online help contains complete reference information for every metric. To access reference information for a specific metric, use the online help search feature.
Figure 12-3 Setting Notification Rules for Metrics
|
See Also:
|
The Database target Home page in Figure 12-4 shows system performance, space usage, and the configuration of important availability components such as the percentage space used in the Fast Recovery Area, and Flashback Database Logging status (the Fast Recovery Area is labeled Flash Recovery Area (%) in Enterprise Manager 11g).
You can see the most recent alerts for the target under the Alerts table, as shown in Figure 12-4. You can access further information about alerts by clicking the links in the Message column.
Performance Analysis and Performance Baseline
Many of the metrics for Database targets in Enterprise Manager pertain to performance. A system that is not meeting performance service-level agreements is not meeting High Availability system requirements. While performance problems seldom cause a major system outage, they can still cause an outage to a subset of customers. Outages of this type are commonly referred to as application service brownouts. The primary cause of brownouts is the intermittent or partial failure of one or more infrastructure components. IT managers must be aware of how the infrastructure components are performing (their response time, latency, and availability), and how they are affecting the quality of application service delivered to the end user.
A performance baseline, derived from normal operations that meet the service-level agreement should determine what constitutes a performance metric alert. Baseline data should be collected from the first day that an application is in production and should include the following:
Application statistics (transaction volumes, response time, web service times)
Database statistics (transaction rate, redo rate, hit ratios, top 5 wait events, top 5 SQL transactions)
Operating system statistics (CPU, memory, I/O, network)
You can use Enterprise Manager to capture a baseline snapshot of database performance and create an Automatic Workload Repository (AWR) baseline. Enterprise Manager compares these values against system performance and displays the result on the database Target page. Enterprise Manager can also send alerts if the values deviate too far from the established baseline. See "Use Automatic Performance Tuning Features" for more information about Automatic Workload Repository.
Set the database notification rule to capture the metrics listed in Table 12-4 for all database targets.
Table 12-4 Recommendations for Performance Related Metrics
| Metric | Level | Recommendation |
|---|---|---|
|
I/O Requests (per second) |
Instance |
This metric represents the total rate of I/O read and write requests for the database. It sends an alert when the number of operations exceeds a user-defined threshold. Use this metric with operating system-level metrics that are also available with Enterprise Manager. Set this metric based on the total I/O throughput available to the system, the number of I/O channels available, network bandwidth (in a SAN environment), the effects of the disk cache if you are using a storage array device, and the maximum I/O rate and number of spindles available to the database. |
|
Database CPU Time (%) |
Instance |
This metric represents the percentage of database call time that is spent on the CPU. It can be used to detect a change in the operation of a system, for example, a drop in Database CPU time from 50% to 25%. The |
|
Wait Time (%) |
Instance |
Excessive idle time indicates that a bottleneck for one or more resources is occurring. Set this instance-level metric based on the system wait time when the application is performing as expected. |
|
Network Bytes per Second |
Instance |
This metric reports network traffic that Oracle generates. This metric can indicate a potential network bottleneck. Set this metric based on actual usage during peak periods. |
|
Pages Paged-in (per second) |
Host |
For UNIX-based systems, represents the number of pages paged in (read from disk to resolve fault memory references) per second. This metric checks the number of pages paged in for the CPU(s) specified by the Host CPU(s) parameter, such as For Microsoft Windows, this metric is the rate at which pages are read from disk to resolve hard page faults. Hard page faults occur when a process refers to a page in virtual memory that is not in its working set or elsewhere in physical memory, and must be retrieved from disk. When a page is faulted, the system tries to read multiple contiguous pages into memory to maximize the benefit of the read operation. |
|
Run Queue Length |
Host |
For UNIX-based systems, the Run Queue Length metrics represent the average number of processes in memory and subject to be run in the last interval (1 minute average, 5 minute average, and 15 minute average). It is recommended to alert when Run Queue Length = # of CPUs. (An alternative way to do this is to monitor the Load Average metric and compare it to Maximum CPU.) This metric is not available on Microsoft Windows. |
|
See Also:
|
Set Enterprise Manager metrics to monitor the availability of Data Guard configurations. Table 12-5 shows the metrics that are available for monitoring Data Guard databases.
Table 12-5 Recommendations for Setting Data Guard Metrics
| Metric | Recommendation |
|---|---|
|
Notifies you about system problems in a Data Guard configuration. | |
|
Displays (in seconds) how far the standby is behind the primary database. This metric generates an alert on the standby database if it falls behind more than the user-specified threshold (if any). | |
|
Displays the approximate number of seconds required to failover to this standby database. | |
|
Displays the Redo Apply rate in KB/second on this standby database. | |
|
Displays the approximate number of seconds of redo that is not yet available on this standby database. The lag may be because the redo data has not yet been transported or there may be a gap. This metric generates an alert on the standby database if it falls behind more than the user-specified threshold (if any). |
Use Enterprise Manager as a proactive part of administering any system and for problem notification and analysis, with the following recommendations:
Use Enterprise Manager to Manage Oracle Patches and Maintain System Baselines
Manage Database Availability with the High Availability Console
Enterprise Manager comes with a pre-installed set of policies and recommendations of best practices for all databases. These policies are checked by default and the number of violations is displayed on the Targets page in the Policy Violations area, as shown in Figure 12-5.
Figure 12-5 Database Home Page with Targets Showing Policy Violations
To see more details on violations, select a link in the Policy Violations area. Figure 12-6 shows the Policy Tend Overview page.
Figure 12-6 Database Targets Policy Trend Overview Page
To see Policy Violations, select Violations from the Compliance tab, as shown in Figure 12-7.
Figure 12-7 Shows Compliance Tab with Policy Violations
For any monitored system in the application environment, you can use Enterprise Manager to download and manage patches from My Oracle Support at
You can set up a job to routinely check for patches that are relevant to the user environment. Those patches can be downloaded and stored directly in the Management Repository. Patches can be staged from the Management Repository to multiple systems and applied during maintenance windows.
You can examine patch levels for one system and compare them between systems in either a one-to-one or one-to-many relationship. In this case, a system can be identified as a baseline and used to demonstrate maintenance requirements in other systems. This can be done for operating system patches and database patches.
|
See Also:
|
The High Availability (HA) Console is a one stop, dashboard-style page for monitoring the availability of each database. You can use it on any database and if a database is part of a Data Guard configuration, the HA Console allows you to switch your view from the primary database to any of the standby databases.
Use the HA Console to:
Display high availability events including events from related targets such as standby databases
View the high availability summary that includes the status of the database
View the last backup status
View the Fast Recovery Area Usage, if configured
If Oracle Data Guard is configured: View the Data Guard summary, set up Data Guard standby databases for any database target, manage switchover and failover of database targets other than the database that contains the Management Repository, and monitor the health of a Data Guard configuration at a glance
If Oracle RAC is configured: View the Oracle RAC Services summary including Top Services
|
Note: Oracle Enterprise Manager Database Control uses the name Fast Recovery Area for the renamed Flash Recovery Area. In places, the HA Console and Enterprise Manager use the name Flash Recovery Area. For more information about the Fast Recovery Area, see Section 5.1.3, "Use a Fast Recovery Area". |
Figure 12-8 shows the HA Console. This figure shows summary information, details, and historical statistics for the primary database and shows the standby databases for the primary target, various Data Guard standby performance metrics and settings, and the data protection mode.
Figure 12-8 Monitoring a Primary Database in the High Availability Console
In Figure 12-8, the Availability Summary shows that the primary database is up and its availability is currently 100%. The Availability Summary also shows Oracle ASM instances status. The Availability Events table shows specific high availability events (alerts). You can click the message to obtain more details (or to suppress the event). To set up, manage, and configure a specific solution area for this database, under Availability Summary, next to MAA Advisor, click Details to go to the Maximum Availability Architecture (MAA) Advisor page (described in more detail in Section 12.3.4, "Configure High Availability Solutions with MAA Advisor").
The Backup/Recovery Summary area displays the Last Backup and Next Backup information. The Fast Recovery Area Usage chart indicates about 83% of the fast recovery area is currently used. The Used (Non-reclaimable) Fast Recovery Area (%) chart shows the usage over the last 2 hours. You can click the chart to display the page with the metric details.
The Data Guard Summary area shows the primary database is running in Maximum Availability mode and has Fast-Start Failover enabled. You can click the link next to Protection Mode to modify the data protection mode. In the Standby Databases table, the physical standby database (north) is caught up with the primary database (Apply/Transport Lag) metrics are showing 0 seconds, and the Used Fast Recovery Area (FRA) is 16.02%. The Primary Database Redo Rate chart shows the redo trend over the past 2 hours. Note that if Data Guard is not configured, the "Switch To" box in the corner of the console is not displayed.
Figure 12-9 shows information similar to figure Figure 12-8, but for the standby database (north), which is a physical standby database running real-time query. In the Standby Databases table, the Apply/Transport Lag metrics indicate that the physical standby database is caught up with the primary database, and the Used Fast Recovery Area (FRA) is 16%. Note that if Data Guard is not configured, the "Switch To" box in the corner of the console is not displayed.
Figure 12-9 Monitoring the Standby Database in the High Availability Console
Figure 12-10 shows sample values for Services Summary and Services Details areas. These areas show summary and detail information about Oracle RAC Services, including Top Services and problem services.
Figure 12-10 Monitoring the Cluster in the High Availability Console Showing Services
|
See Also: Oracle Enterprise Manager Cloud Control Introduction for information about Database Management |
The goal of the MAA Advisor is to help you implement Oracle's best practices to achieve the optimal high availability architecture.
From the Availability Summary section on the High Availability Console, you can link to the MAA Advisor to:
View recommended Oracle solutions for each outage type (site failures, computer failures, storage failures, human errors, and data corruptions)
View the configuration status and use the links in the Oracle Solution column to go to the Enterprise Manager page where the solution can be configured.
Understand the benefits of each solution
Link to the MAA website for white papers, documentation, and other information
The MAA Advisor page contains a table that lists the outage type, Oracle solutions for each outage, configuration status, and benefits. The MAA Advisor allows you to view High Availability solutions in the following ways:
Primary Database Recommendations Only—This condensed view shows only the recommended solutions (the default view) for the primary database.
All Solutions —This expanded view shows all configuration recommendations and status for all primary and standby databases in this configuration. It includes an extra column Target Name:Role that provides the database name and shows the role (Primary, Physical Standby, or Logical Standby) of the database.
Figure 12-11 shows an example of the MAA Advisor page with the Show All Solutions view selected.
Figure 12-11 Maximum Availability Architecture (MAA) Advisor Page in Enterprise Manager
You can click the link in the Oracle Solution column to go to a page where you can set up, manage, and configure the specific solution area. Once a solution has been configured, click Refresh to update the configuration status. Once the page is refreshed, click Advisor Details on the Console page to see the updated values.
The Cluster Health Monitor (CHM) gathers operating system metrics in real time and stores them in its repository for later analysis to determine the root cause of many Oracle Clusterware and Oracle RAC issues with the assistance of Oracle Support. It also works with Oracle Database Quality of Service Management (Oracle Database QoS Management) by providing metrics to detect memory over-commitment on a node. With this information, Oracle Database QoS Management can take action to relieve the stress and preserve existing workloads.
|
See: Oracle Clusterware Administration and Deployment Guide for an Overview of Managing Oracle Clusterware Environments and for more information about Cluster Health Monitor (CHM) |
This chapter describes best practices for configuring a fault-tolerant storage subsystem that protects data while providing manageability and performance. These practices apply to all Oracle Database high availability architectures described in Oracle Database High Availability Overview.
This chapter contains the following topics:
Evaluate Database Performance and Storage Capacity Requirements
Use Automatic Storage Management (Oracle ASM) to Manage Database Files
Characterize your database performance requirements using different application workloads. Extract statistics during your target workloads by gathering the beginning and ending statistical snapshots. Some examples of target workloads include:
Average load
Peak load
Application workloads such as batch processing, Online Transaction Processing (OLTP), decision support systems (DSS) and reporting, Extraction, Transformation, and Loading (ETL)
Evaluating Database Performance Requirements
You can gather the necessary statistics by using Automatic Workload Repository (AWR) reports or by querying the GV$SYSSTAT view. Along with understanding the database performance requirements, you must evaluate the performance capabilities of a storage array.
Choosing Storage
When you understand the performance and capacity requirements, choose a storage platform to meet those requirements.
|
See Also: Oracle Database Performance Tuning Guide for Overview of the Automatic Workload Repository (AWR) and on Generating Automatic Workload Repository Reports |
Oracle ASM is a vertical integration of both the file system and the volume manager built specifically for Oracle database files. Oracle ASM extends the concept of stripe and mirror everything (SAME) to optimize performance, while removing the need for manual I/O tuning (distributing the data file layout to avoid hot spots). Oracle ASM helps manage a dynamic database environment by letting you grow the database size without shutting down the database to adjust the storage allocation. Oracle ASM also enables low-cost modular storage to deliver higher performance and greater availability by supporting mirroring and striping.
Oracle ASM provides data protection against drive and SAN failures, the best possible performance, and extremely flexible configuration and reconfiguration options. Oracle ASM automatically distributes the data across all available drivers, transparently and dynamically redistributes data when storage is added or removed from the database.
Oracle ASM manages all of your database files. You can phase Oracle ASM into your environment by initially supporting only the fast recovery area.
|
Note: Oracle recommends host-based mirroring using Oracle ASM. |
|
See:
|
The Grid Infrastructure is the software that provides the infrastructure for an enterprise grid architecture. In a cluster, this software includes Oracle Clusterware and Oracle ASM.
You can use clustered Oracle ASM with both Oracle single-instance databases and Oracle Real Application Clusters (Oracle RAC). In an Oracle RAC environment, there is one Oracle ASM instance for each node, and the Oracle ASM instances communicate with each other on a peer-to-peer basis. Only one Oracle ASM instance is required and supported for each node regardless of the number of database instances on the node. Clustering Oracle ASM instances provides fault tolerance, flexibility, and scalability to your storage pool.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for more information about clustered Oracle ASM |
Oracle Restart improves the availability of your Oracle database. When you install the Oracle Grid Infrastructure for a standalone server, it includes both Oracle ASM and Oracle Restart. Oracle Restart runs out of the Oracle Grid Infrastructure home, which you install separately from Oracle Database homes.
Oracle Restart provides managed startup and restart of a single-instance (non-clustered) Oracle Database, Oracle ASM instance, service, listener, and any other process running on the server. If an interruption of a service occurs after a hardware or software failure, Oracle Restart automatically takes the necessary steps to restart the component.
With Server Control Utility (SRVCTL) you can add a component, such as an Oracle ASM instance to Oracle Restart. You then enable Oracle Restart protection for the Oracle ASM instance. With SRVCTL, you also remove or disable Oracle Restart protection.
|
See Also:
|
Use the following Oracle ASM strategic best practices:
Oracle recommends using Oracle high redundancy disk groups (3 way mirroring) or an external redundancy disk group with equivalent mirroring resiliency for mission critical applications. This higher level of mirroring provides greater protection and better tolerance of different storage failures. This is especially true during planned maintenance windows when a subset of the storage is offline for patching or upgrading. For more information about redundancy, see Chapter 4, "Use Redundancy to Protect from Disk Failure."
When you use Oracle ASM for database storage, create two disk groups: one disk group for the data area and another disk group for the fast recovery area:
data area: contains the active database files and other files depending on the level of Oracle ASM redundancy. If Oracle ASM with high redundancy is used, then the data area can also contain OCR, Voting, spfiles, control files, online redo log files, standby redo log files, broker metadata files, and change tracking files used for RMAN incremental backup.
For example (high redundancy):
CREATE DISKGROUP data HIGH REDUNDANCY
FAILGROUP controller1 DISK
'/devices/c1data01' NAME c1data01,\
'/devices/c1data02' NAME c1data02
FAILGROUP controller2 DISK
'/devices/c2data01' NAME c2data01,
'/devices/c2data02' NAME c2data02
FAILGROUP controller3 DISK
'/devices/c3data01' NAME c3data01,
'/devices/c3data02' NAME c3data02
ATTRIBUTE 'au_size'='4M',
'compatible.asm' = '11.2',
'compatible.rdbms'= '11.2',
'compatible.advm' = '11.2';
fast recovery area: contains recovery-related files, such as a copy of the current control file, a member of each online redo log file group, archived redo log files, RMAN backups, and flashback log files.
For example (normal redundancy):
CREATE DISKGROUP reco NORMAL REDUNDANCY
FAILGROUP controller1 DISK
'/devices/c1reco01' NAME c1reco01,
'/devices/c1reco02' NAME c1reco02
FAILGROUP controller2 DISK
'/devices/c2reco01' NAME c2reco01,
'/devices/c2reco02' NAME c2reco02
ATTRIBUTE 'au_size'='4M',
'compatible.asm' = '11.2',
'compatible.rdbms'= '11.2',
'compatible.advm' = '11.2';
|
Note 1: If you are using ASMLib in a Linux environment, then create the disks using theORACLEASM CREATEDISK command. ASMLib is a support library for Oracle ASM and is not supported on all platforms. For more information about ASMLib, see Section 4.4.6, "Use ASMLib On Supported Platforms".
For example: /etc/init.d/oracleasm createdisk lun1 /devices/lun01 Then, create the disk groups. For example: CREATE DISKGROUP DATA DISK 'ORCL:lun01','ORCL:lun02','ORCL:lun03','ORCL:lun04'; |
|
Note 2: Oracle recommends using four (4) or more disks in each disk group. Having multiple disks in each disk group spreads kernel contention accessing and queuing for the same disk. |
To simplify file management, use Oracle Managed Files to control file naming. Enable Oracle Managed Files by setting the following initialization parameters: DB_CREATE_FILE_DEST and DB_CREATE_ONLINE_LOG_DEST_n.
For example:
DB_CREATE_FILE_DEST=+DATA DB_CREATE_ONLINE_LOG_DEST_1=+RECO
You have two options when partitioning disks for Oracle ASM:
Allocate entire disks to the data area and fast recovery area disk groups. Figure 4-1 illustrates allocating entire disks.
Partition each disk into two partitions, one for the data area and another for the fast recovery area. Figure 4-2 illustrates partitioning each disk into two partitions.
The advantages of the option shown in Figure 4-1 are:
Easier management of the disk partitions at the operating system level because each disk is partitioned as just one large partition.
Quicker completion of Oracle ASM rebalance operations following a disk failure because there is only one disk group to rebalance.
Fault isolation, where storage failures only cause the affected disk group to go offline.
Patching isolation, where you can patch disks or firmware for individual disks without impacting every disk.
The disadvantage of the option shown in Figure 4-1 is:
Less I/O bandwidth, because each disk group is spread over only a subset of the available disks.
Figure 4-2 illustrates the partitioning option where each disk has two partitions. This option requires partitioning each disk into two partitions: a smaller partition on the faster outer portion of each drive for the data area, and a larger partition on the slower inner portion of each drive for the fast recovery area. The ratio for the size of the inner and outer partitions depends on the estimated size of the data area and the fast recovery area.
The advantages of the option shown in Figure 4-2 for partitioning are:
More flexibility and easier to manage from a performance and scalability perspective.
Higher I/O bandwidth is available, because both disk groups are spread over all available spindles. This advantage is considerable for the data area disk group for I/O intensive applications.
There is no need to create a separate disk group with special, isolated storage for online redo logs or standby redo logs if you have sufficient I/O capacity.
You can use the slower regions of the disk for the fast recovery area and the faster regions of the disk for data.
The disadvantages of the option shown in Figure 4-2 for partitioning are:
A double partner disk failure will result in loss of both disk groups, requiring the use of a standby database or tape backups for recovery. This problem is eliminated when using high redundancy ASM disk groups.
An Oracle ASM rebalance operation following a disk failure is longer, because both disk groups are affected.
|
See Also:
|
When setting up redundancy to protect from hardware failures, there are two options to consider:
|
See Also:
|
If you are using a high-end storage array that offers robust built-in RAID solutions, then Oracle recommends that you configure redundancy in the storage array by enabling RAID protection, such as RAID1 (mirroring) or RAID5 (striping plus parity). For example, to create an Oracle ASM disk group where redundancy is provided by the storage array, first create the RAID-protected logical unit numbers (LUNs) in the storage array, and then create the Oracle ASM disk group using the EXTERNAL REDUNDANCY clause:
CREATE DISKGROUP DATA EXTERNAL REDUNDANCY DISK
'/devices/lun1','/devices/lun2','/devices/lun3','/devices/lun4';
|
See Also:
|
Oracle ASM provides redundancy with the use of failure groups, which are defined during disk group creation. The disk group type determines how Oracle ASM mirrors files. When you create a disk group, you indicate whether the disk group is a normal redundancy disk group (2-way mirroring for most files by default), a high redundancy disk group (3-way mirroring), or an external redundancy disk group (no mirroring by Oracle ASM). You use an external redundancy disk group if your storage system does mirroring at the hardware level, or if you have no need for redundant data. The default disk group type is normal redundancy. After a disk group is created the redundancy level cannot be changed.
Failure group definition is specific to each storage setup, but you should follow these guidelines:
If every disk is available through every I/O path, as would be the case if using disk multipathing software, then keep each disk in its own failure group. This is the default Oracle ASM behavior if creating a disk group without explicitly defining failure groups.
CREATE DISKGROUP DATA NORMAL REDUNDANCY DISK '/devices/diska1','/devices/diska2','/devices/diska3','/devices/diska4', '/devices/diskb1','/devices/diskb2','/devices/diskb3','/devices/diskb4';
For an array with two controllers where every disk is seen through both controllers, create a disk group with each disk in its own failure group:
CREATE DISKGROUP DATA NORMAL REDUNDANCY DISK '/devices/diska1','/devices/diska2','/devices/diska3','/devices/diska4', '/devices/diskb1','/devices/diskb2','/devices/diskb3','/devices/diskb4';
If every disk is not available through every I/O path, then define failure groups to protect against the piece of hardware that you are concerned about failing. Here are some examples:
For an array with two controllers where each controller sees only half the drives, create a disk group with two failure groups, one for each controller, to protect against controller failure:
CREATE DISKGROUP DATA NORMAL REDUNDANCY FAILGROUP controller1 DISK '/devices/diska1','/devices/diska2','/devices/diska3','/devices/diska4' FAILGROUP controller2 DISK '/devices/diskb1','/devices/diskb2','/devices/diskb3','/devices/diskb4';
For a storage network with multiple storage arrays, you want to mirror across storage arrays, then create a disk group with two failure groups, one for each array, to protect against array failure:
CREATE DISKGROUP DATA NORMAL REDUNDANCY FAILGROUP array1 DISK '/devices/diska1','/devices/diska2','/devices/diska3','/devices/diska4' FAILGROUP array2 DISK '/devices/diskb1','/devices/diskb2','/devices/diskb3','/devices/diskb4';
When determining the proper size of a disk group that is protected with Oracle ASM redundancy, enough free space must exist in the disk group so that when a disk fails Oracle ASM can automatically reconstruct the contents of the failed drive to other drives in the disk group while the database remains online. The amount of space required to ensure Oracle ASM can restore redundancy following disk failure is in the column REQUIRED_MIRROR_FREE_MB in the V$ASM_DISKGROUP view. The amount of free space that you can use safely in a disk group, taking mirroring into account, and still be able to restore redundancy after a disk failure is in the USABLE_FILE_MB column in the V$ASM_DISKGROUP view. The value of the USABLE_FILE_MB column should always be greater than zero. If USABLE_FILE_MB falls below zero, then add more disks to the disk group.
Oracle Automatic Storage Management (Oracle ASM) and Oracle Clusterware are installed into a single home directory, which is called the Grid home. Oracle Grid Infrastructure for a cluster software refers to the installation of the combined products. The Grid home is separate from the home directories of other Oracle software products installed on the same server.
|
See Also: Oracle Database 2 Day + Real Application Clusters Guide for information about Oracle ASM and the Grid home. |
Although ensuring that all disks in the same disk group have the same size and performance characteristics is not required, doing so provides more predictable overall performance and space utilization. When possible, present physical disks (spindles) to Oracle ASM as opposed to Logical Unit Numbers (LUNs) that create a layer of abstraction between the disks and Oracle ASM.
If the disks are the same size, then Oracle ASM spreads the files evenly across all of the disks in the disk group. This allocation pattern maintains every disk at the same capacity level and ensures that all of the disks in a disk group have the same I/O load. Because Oracle ASM load balances workload among all of the disks in a disk group, different Oracle ASM disks should not share the same physical drive.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for complete information about administering Oracle ASM disk groups |
Using failure groups to define a common failure compo nent ensures continuous access to data when that component fails. For maximum protection, use at least three failure groups for normal redundancy and at least five failure groups for high redundancy. Doing so enables Oracle ASM to tolerate multiple failure group failures and avoids the confusing state of having Oracle ASM running without full redundancy.
Intelligent Data Placement enables you to specify disk regions on Oracle ASM disks for best performance. Using the disk region settings you can ensure that frequently accessed data is placed on the outermost (hot) tracks which have greater speed and higher bandwidth. In addition, files with similar access patterns are located physically close, reducing latency. Intelligent Data Placement also enables the placement of primary and mirror extents into different hot or cold regions.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for more information about Intelligent Data Placement |
Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is a multi-platform, scalable file system, and storage management technology that extends Oracle Automatic Storage Management (Oracle ASM) functionality to support customer files maintained outside of Oracle Database. Oracle ACFS includes a volume management service and comes with fine grained security policies, encryption, snapshotting and replication.
Oracle ACFS supports many database and application files, including executables, database trace files, database alert logs, application reports, BFILEs, and configuration files. Other supported files are video, audio, text, images, engineering drawings, and other general-purpose application file data.
|
Note: Oracle database binaries can be put on Oracle ACFS but not binaries in the Grid Infrastructure home. |
|
See Also: Oracle Automatic Storage Management Administrator's Guide for more information about Oracle ACFS |
Use the following Oracle ASM configuration best practices:
Disk multipathing software aggregates multiple independent I/O paths into a single logical path. The path abstraction provides I/O load balancing across host bus adapters (HBA) and nondisruptive failovers when there is a failure in the I/O path. You should use disk multipathing software with Oracle ASM.
When specifying disk names during disk group creation in Oracle ASM, use the logical device representing the single logical path. For example, when using Device Mapper on Linux 2.6, a logical device path of /dev/dm-0 may be the aggregation of physical disks /dev/sdc and /dev/sdh. Within Oracle ASM, the ASM_DISKSTRING parameter should contain /dev/dm-* to discover the logical device /dev/dm-0, and that logical device is necessary during disk group creation:
asm_diskstring='/dev/dm-*' CREATE DISKGROUP DATA DISK '/dev/dm-0','/dev/dm-1','/dev/dm-2','/dev/dm-3';
|
Note: For more information about using the combination of ASMLib and Multipath Disks, see "Configuring Oracle ASMLib on Multipath Disks" in My Oracle Support Note 309815.1 at
|
|
See Also:
|
Use the SGA_TARGET and PGA_AGGREGATE_TARGET initialization parameters in the Oracle ASM instance to manage ASM process memory using the automatic shared memory management (ASSM) functionality.
To use automatic shared memory management the values for MEMORY_TARGET and MEMORY_MAX_TARGET should be set to 0.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for information about memory-related initialization parameters for Oracle ASM |
The PROCESSES initialization parameter affects Oracle ASM, but the default value is usually suitable. However, if multiple database instances are connected to an Oracle ASM instance, you can use the following formulas:
| For < 10 instances per node... | For > 10 instances per node... |
|---|---|
PROCESSES = 50 * (n + 1) |
PROCESSES = 50 * MIN (n + 1, 11) + 10 * MAX (n - 10, 0) |
where n is the number database instances connecting to the Oracle ASM instance.
|
See Also:
|
Disk labels ensure consistent access to disks across restarts. ASMLib is the preferred tool for disk labeling. For more information about ASMLib, see Section 4.4.6, "Use ASMLib On Supported Platforms".
The DISK_REPAIR_TIME disk group attribute specifies how long a disk remains offline before Oracle ASM drops the disk. If a disk is made available before the DISK_REPAIR_TIME parameter has expired, the storage administrator can issue the ONLINE DISK command and Oracle ASM resynchronizes the stale data from the mirror side. In Oracle Database 11g, the online disk operation does not restart if there is a failure of the instance on which the disk is running. You must reissue the command manually to bring the disk online.
You can set a disk repair time attribute on your disk group to specify how long disks remain offline before being dropped. The appropriate setting for your environment depends on how long you expect a typical transient type of failure to persist.
Set the DISK_REPAIR_TIME disk group attribute to the maximum amount of time before a disk is definitely considered to be out of service.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for information about restoring the redundancy of an Oracle ASM disk group after a transient disk path failure |
To improve manageability use ASMLib on platforms where it is available. ASMLib is a support library for Oracle ASM.
Although ASMLib is not required to run Oracle ASM, using ASMLib is beneficial because ASMLib:
Eliminates the need for every Oracle process to open a file descriptor for each Oracle ASM disk, thus improving system resource usage.
Simplifies the management of disk device names, makes the discovery process simpler, and removes the challenge of having disks added to one node and not be known to other nodes in the cluster.
Eliminates the impact when the mappings of disk device names change upon system restart.
|
Note: ASMLib is not supported on all platforms. |
|
See Also:
|
A general rule of thumb is to disable variable sized extents if the amount of space managed by a single Oracle ASM cluster is less than or equal to 330TB Raw.
|
Note: This rule of thumb assumes that read-only tablespaces are not being shared across multiple databases. |
Use the following Oracle ASM operational best practices:
The Oracle ASM instance is managed by a privileged role called SYSASM, which grants full access to Oracle ASM disk groups. Using SYSASM enables the separation of authentication for the storage administrator and the database administrator. By configuring a separate operating system group for Oracle ASM authentication, you can have users that have SYSASM access to the Oracle ASM instances and do not have SYSDBA access to the database instances.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for information about authentication to access Oracle ASM instances |
Higher Oracle ASM rebalance power limits make a rebalance operation run faster but can also affect application service levels. Rebalancing takes longer with lower power values, but consumes fewer processing and I/O resources that are shared by other applications, such as the database.
After performing planned maintenance, for example adding or removing storage, it is necessary to subsequently perform a rebalance to spread data across all of the disks. There is a power limit associated with the rebalance. You can set a power limit to specify how many processes perform the rebalance. If you do not want the rebalance to impact applications, then set the power limit lower. However, if you want the rebalance to finish quickly, then set the power limit higher. To determine the default power limit for rebalances, check the value of the ASM_POWER_LIMIT initialization parameter in the Oracle ASM instance.
If the POWER clause is not specified in an ALTER DISKGROUP statement, or when rebalance is run implicitly when you add or drop a disk, then the rebalance power defaults to the value of the ASM_POWER_LIMIT initialization parameter. You can adjust the value of this parameter dynamically.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for more information about rebalancing Oracle ASM disk groups |
Mounting multiple disk groups in the same command ensures that disk discovery runs only one time, thereby increasing performance. Disk groups that are specified in the ASM_DISKGROUPS initialization parameter are mounted automatically at Oracle ASM instance startup.
To mount disk groups manually, use the ALTER DISKGROUP...MOUNT statement and specify the ALL keyword:
ALTER DISKGROUP ALL MOUNT;
|
Note: TheALTER DISKGROUP...MOUNT command only works on one node. For cluster installations use the following command:
srvctl start diskgroup -g |
|
See Also: Oracle Automatic Storage Management Administrator's Guide for information about mounting and dismounting disk groups |
Oracle ASM permits you to add or remove disks from your disk storage system while the database is operating. When you add a disk to a disk group, Oracle ASM automatically redistributes the data so that it is evenly spread across all disks in the disk group, including the new disk. The process of redistributing data so that it is also spread across the newly added disks is known as rebalancing. By executing storage maintenance commands in the same command, you ensure that only one rebalance is required to incur minimal impact to database performance.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for information about Altering Disk Groups |
You should periodically check disk groups for imbalance. Occasionally, disk groups can become unbalanced if certain operations fail, such as a failed rebalance operation. Periodically checking the balance of disk groups and running a manual rebalance, if needed, ensures optimal Oracle ASM space utilization and performance.
Use the following methods to check for disk group imbalance:
To check for an imbalance on all mounted disk groups, see "Script to Report the Percentage of Imbalance in all Mounted Diskgroups" in My Oracle Support Note 367445.1 at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=367445.1
To check for an imbalance from an I/O perspective, query the statistics in the V$ASM_DISK_IOSTAT view before and after running a large SQL*Plus statement. For example, if you run a large query that performs only read I/O, the READS and BYTES_READ columns should be approximately the same for all disks in the disk group.
You should proactively mine vendor logs for disk errors and have Oracle ASM move data off the bad disk spots. Disk vendors usually provide disk-scrubbing utilities that notify you if any part of the disk is experiencing problems, such as a media sense error. When a problem is found, use the ASMCMD utility REMAP command to move Oracle ASM extents from the bad spot to a good spot.
Note that this is only applicable for data that is not accessed by the database or Oracle ASM instances, because in that case Oracle ASM automatically moves the extent experiencing the media sense error to a different location on the same disk. In other words, use the ASMCMD utility REMAP command to proactively move data from a bad disk spot to a good disk spot before that data is accessed by the application.
|
See Also: Oracle Automatic Storage Management Administrator's Guide for information about the ASMCMD utility |
Use the ASMCMD utility to ease the manageability of day-to-day storage administration. Use the ASMCMD utility to view and manipulate files and directories in Oracle ASM disk groups and to list the contents of disk groups, perform searches, create and remove directories and aliases, display space usage. Also, use the ASMCMD utility to backup and restore the metadata of the disk groups (using the md_backup and md_restore commands).
|
Note: As a best practice to create and drop Oracle ASM disk groups, use SQL*Plus, ASMCA, or Oracle Enterprise Manager. |
|
See Also: Oracle Automatic Storage Management Administrator's Guide for more information about ASMCMD Disk Group Management Commands |
Oracle ASM Configuration Assistant (ASMCA) supports installing and configuring Oracle ASM instances, disk groups, volumes, and Oracle Automatic Storage Management Cluster File System (Oracle ACFS). In addition, you can use the ASMCA command-line interface as a silent mode utility.
The Oracle Storage Grid consists of either:
Oracle ASM and third-party storage using external redundancy.
Oracle ASM and Oracle Exadata or third-party storage using Oracle ASM redundancy. The Oracle Storage Grid with Exadata seamlessly supports MAA-related technology, improves performance, provides unlimited I/O scalability, is easy to use and manage, and delivers mission-critical availability and reliability to your enterprise.
To protect storage against unplanned outages:
Set the DB_BLOCK_CHECKSUM initialization parameter to TYPICAL (default) or FULL. For more information, see Section 9.3.8.3, "Set DB_BLOCK_CHECKSUM=FULL and DB_BLOCK_CHECKING=MEDIUM or FULL".
|
Note: Oracle Exadata Database Machine also prevents corruptions from being written to disk by incorporating the hardware assisted resilient data (HARD) technology in its software. HARD uses block checking, in which the storage subsystem validates the Oracle block contents, preventing corrupted data from being written to disk. HARD checks in Oracle Exadata operate completely transparently and no parameters must be set for this purpose at the database or storage tier. For more information see the White Paper "Optimizing Storage and Protecting Data with Oracle Database 11g" at
|
Choose Oracle ASM redundancy type (NORMAL or HIGH) based on your desired protection level and capacity requirements
The NORMAL setting stores two copies of Oracle ASM extents, while the HIGH setting stores three copies of Oracle ASM extents. Normal redundancy provides more usable capacity and high redundancy provides more protection.
If a storage component is to be offlined when one or more databases are running, then verify that taking the storage component offline does not impact Oracle ASM disk group and database availability. Before dropping a failure group or offlining a storage component perform the appropriate checks.
Ensure I/O performance can be sustained after an outage
Ensure that you have enough I/O bandwidth to support your service-level agreement if a failure occurs. For example, a typical case for a Storage Grid with n storage components would be to ensure that n-1 storage components could support the application service levels (for example, to handle a storage component failure).
Use the following list of best practices for planned maintenance:
Size I/O for performance first, and then set it for capacity:
When building your Oracle Storage Grid, make sure you have enough drives to support I/O's per second and MBs per second to meet your service-level requirements. Then, make sure you also have enough capacity. The order is important because you do not want to buy enough drives to support capacity but then find the system cannot meet your performance requirements.
When you are sizing, you must consider what happens to performance when you offline a subset of storage for planned maintenance. For Example, when a subset of the overall storage is offlined you still must make sure you get the required number of IOPS if that is important to meet your SLAs. Also, if offlining storage means the system cannot add more databases, then one has to consider that upfront.
Set Oracle ASM power limit for faster rebalancing For more information, see Section 4.5.2, "Set Rebalance to the Maximum Limit that Does Not Affect Service Levels".
|
See Also:
|
This chapter describes the Oracle operational best practices that can tolerate or manage each unscheduled outage type and minimize downtime.
This chapter contains the following topics:
|
See Also: Chapter 14, "Reducing Downtime for Planned Maintenance" for information about scheduled outages. |
This section describes unscheduled outages that affect the primary or secondary site components, and describes the recommended methods to repair or minimize the downtime associated with each outage.
Unscheduled outages are unanticipated failures in any part of the technology infrastructure that supports the application, including the following components:
Hardware
Software
Network infrastructure
Naming services infrastructure
Database
Your monitoring and high availability infrastructure should provide rapid detection and recovery from downtime. Chapter 12, "Monitoring for High Availability" describes detection, while this chapter focuses on reducing downtime.
Solutions for unscheduled outages are critical for maximum availability of the system. Table 13-1 compares the most common Oracle high availability architectures and summarizes the recovery steps for unscheduled outages on the primary site. For outages that require multiple recovery steps, the table includes links to the detailed descriptions in Section 13.2, "Recovering from Unscheduled Outages".
Table 13-1 Recovery Times and Steps for Unscheduled Outages on the Primary Site
| Outage Scope | Oracle Database 11g | Oracle Database 11g with Oracle RAC and Oracle Clusterware | Oracle Database 11g with Data GuardFoot 1 | Oracle Database 11g MAA |
|---|---|---|---|---|
|
|
Hours to days
|
Hours to days
|
Seconds to 5 minutesFoot 2 |
Seconds to 5 minutesFootref 2 |
|
|
Not applicable |
Hours to days
|
Not applicable |
Seconds to 5 minutes |
|
computer failure (node) |
Minutes to hoursFoot 3
|
No downtimeFoot 4 Managed automatically by Section 13.2.3, "Oracle RAC Recovery for Unscheduled Outages (for Node or Instance Failures)" |
Seconds to 5 minutesFootref 3 |
No downtimeFootref 4 Managed automatically by Section 13.2.3, "Oracle RAC Recovery for Unscheduled Outages (for Node or Instance Failures)" |
|
computer failure (instance) |
MinutesFootref 3
|
No downtimeFootref 4 Managed automatically by Section 13.2.3, "Oracle RAC Recovery for Unscheduled Outages (for Node or Instance Failures)" |
MinutesFootref 3
or Seconds to 5 minutesFootref 2 |
No downtimeFootref 4 Managed automatically by Section 13.2.3, "Oracle RAC Recovery for Unscheduled Outages (for Node or Instance Failures)" |
|
|
No downtimeFoot 5 Section 13.2.5, "Oracle ASM Recovery After Disk and Storage Failures" |
No downtimeFootref 5 Section 13.2.5, "Oracle ASM Recovery After Disk and Storage Failures" |
No downtimeFootref 5 Section 13.2.5, "Oracle ASM Recovery After Disk and Storage Failures" |
No downtimeFootref 5 Section 13.2.5, "Oracle ASM Recovery After Disk and Storage Failures" |
|
|
Minutes to hours |
Minutes to hours |
Possible no downtime with Active Data Guard: Section 13.2.6.2, "Use Active Data Guard" Seconds to 5 minutes |
Possible no downtime with Active Data Guard: Section 13.2.6.2, "Use Active Data Guard" Seconds to 5 minutes |
|
|
< 30 minutesFoot 6 Section 13.2.7, "Recovering from Human Error (Recovery with Flashback)" |
< 30 minutesFootref 6 Section 13.2.7, "Recovering from Human Error (Recovery with Flashback)" |
<30 minutesFootref 6 Section 13.2.7, "Recovering from Human Error (Recovery with Flashback)" |
< 30 minutesFootref 6 Section 13.2.7, "Recovering from Human Error (Recovery with Flashback)" |
|
|
customized and configurable Foot 7 |
customized and configurable Footref 7 |
customized and configurable Foot 8 |
customized and configurableFootref 7 andFootref 8 |
Footnote 1 While Data Guard physical replication is the most common data protection and availability solution used for Oracle Database, there are cases where active-active logical replication may be preferred, especially when control over the application makes it possible to implement. You may use Oracle GoldenGate in place of Data Guard for these requirements. See the topic, "Oracle Active Data Guard and Oracle GoldenGate" for additional discussion of the trade-offs between physical and logical replication at http://www.oracle.com/technetwork/database/features/availability/dataguardgoldengate-096557.html
Footnote 2 Recovery time indicated applies to database and existing connection failover. Network connection changes and other site-specific failover activities may lengthen overall recovery time.
Footnote 3 Recovery time consists largely of the time it takes to restart the failed system.
Footnote 4 Database is still available, but portion of application connected to failed system is temporarily affected.
Footnote 5 Storage failures are prevented by using Oracle ASM with mirroring and its automatic rebalance capability.
Footnote 6 Recovery times from human errors depend primarily on detection time. If it takes seconds to detect a malicious DML or DLL transaction, then it typically only requires seconds to flash back the appropriate transactions, if properly rehearsed. Referential or integrity constraints must be considered.
Footnote 7 Oracle Enterprise Manager or a customized application heartbeat can be configured to detect application or response time slowdown and react to these SLA breaches. For example, you can configure the Enterprise Manager Beacon to monitor and detect application response times. Then, after a certain threshold expires, Enterprise Manager can alert and possibly restart the database.
Footnote 8 Oracle Enterprise Manager or a customized application heartbeat can be configured to detect application or response time slowdown and react to these SLA breaches. For example, you can configure the Enterprise Manager Beacon to monitor and detect application response times. Then, after a certain threshold expires, Enterprise Manager can call the Oracle Data Guard DBMS_DG.INITIATE_FS_FAILOVER PL/SQL procedure to initiate a failover.
|
See:
|
Outages on the standby site do not impact the availability of the primary database when using Data Guard Maximum Availability (synchronous communication with net_timeout) or Maximum Performance (asynchronous communication).
|
Note: Outages to a system that uses the Active Data Guard option with the standby database can affect applications that are using the standby database for read activity, but such outages do not impact the availability of the primary database (the availability is based on the mode you specify). |
Data Guard Maximum Protection, however, has an impact on availability if the primary database does not receive acknowledgment from a standby database running in SYNC transport mode (net_timeout does not apply to Maximum Protection). For this reason, if you are using Maximum Protection you should follow the MAA best practice of deploying two SYNC standby databases, each at its own site. With two standby databases a single standby outage does not impact primary availability or zero data loss protection.
If limited system resources make it impractical to deploy two standby databases, then the availability of the primary database can be restored simply by downgrading the data protection mode to Maximum Availability and restarting the primary database.
Table 13-2 summarizes the recovery steps for unscheduled outages of the standby database on the secondary site. For outages that require multiple recovery steps, the table includes links to the detailed descriptions in Section 13.2, "Recovering from Unscheduled Outages".
Table 13-2 Recovery Steps for Unscheduled Outages on the Secondary Site
| Outage Type | Recovery Steps for Single-Instance or Oracle RAC Standby Database |
|---|---|
|
Computer failure (instance) |
The broker automatically restarts the log apply services. Note 1: If there is only one standby database and if Maximum Protection is configured, then the primary database shuts down to ensure that there is no data divergence with the standby database (no unprotected data). Note 2: If this is an Oracle RAC standby database, then there is no affect on primary database availability if you configured the primary database Oracle Net descriptor to use connect-time failover to an available standby instance. If you are using the broker, connect-time failover is configured automatically. |
|
Data corruption |
Section 13.3.5, "Restoring Fault Tolerance After a Standby Database Data Failure" |
|
Primary database opens with |
Section 13.3.6, "Restoring Fault Tolerance After the Primary Database Was Opened Resetlogs" |
|
See Also:
|
This section describes best practices for recovering from various types of unscheduled outages.
With complete site failover, the database, the middle-tier application server, and all user connections fail over to a secondary site that is prepared to handle the production load.
If the standby site meets the prerequisites, then complete site failover is recommended for the following scenarios:
Primary site disaster, such as natural disasters or malicious attacks
Primary network-connectivity failures
Primary site power failures
To expedite site failover in minutes:
Use the Data Guard configuration best practices in Section 9.3, "General Data Guard Configuration Best Practices"
Use Data Guard fast-start failover to automatically fail over to the standby database, with a recovery time objective (RTO) of less than 30 seconds (described in Section 9.4.2.3, "Fast-Start Failover Best Practices")
Maintain a running middle-tier application server on the secondary site to avoid the startup time, or redirect existing applications to the new primary database using the Fast Connection Failover best practices described in:
The MAA white paper: "Client Failover Best Practices for Data Guard 11g Release 2" from the MAA Best Practices area for Oracle Database at
Configure Automatic Domain Name Server (DNS) failover procedure. Automatic DNS failover occurs after a primary site is inaccessible and the wide-area traffic manager at the secondary site returns the virtual IP address of a load balancer at the secondary site and clients are directed automatically on the subsequent reconnect.
The potential for data loss is dependent on the Data Guard protection mode used: Maximum Protection, Maximum Availability, or Maximum Performance.
A wide-area traffic manager on the primary and secondary sites provides the site failover function. The wide-area traffic manager can redirect traffic automatically if the primary site, or a specific application on the primary site, is not accessible. It can also be triggered manually to switch to the secondary site for switchovers. Traffic is directed to the secondary site only when the primary site cannot provide service due to an outage or after a switchover. If the primary site fails, then user traffic is directed to the secondary site automatically.
Figure 13-1 illustrates the possible network routes before site failover:
Client requests enter the client tier of the primary site and travel by the WAN traffic manager.
Client requests are sent through the firewall into the demilitarized zone (DMZ) to the application server tier.
Requests are forwarded through the active load balancer to the application servers.
Requests are sent through another firewall and into the database server tier.
The application requests, if required, are routed to an Oracle RAC instance.
Responses are sent back to the application and clients by a similar path.
Figure 13-1 Network Routes Before Site Failover
Figure 13-2 illustrates the network routes after site failover. Client or application requests enter the secondary site at the client tier and follow the same path on the secondary site that they followed on the primary site.
Figure 13-2 Network Routes After Site Failover
The following steps describe the effect of a failover or switchover on network traffic:
The administrator has failed over or switched over the primary database to the secondary site. This is automatic if you are using Data Guard fast-start failover.
The administrator starts the middle-tier application servers on the secondary site, if they are not running.
The wide-area traffic manager selection of the secondary site can be automatic for an entire site failure. The wide-area traffic manager at the secondary site returns the virtual IP address of a load balancer at the secondary site and clients are directed automatically on the subsequent reconnect. In this scenario, the site failover is accomplished by an automatic domain name system (DNS) failover.
Alternatively, a DNS administrator can manually change the wide-area traffic manager selection to the secondary site for the entire site or for specific applications. The following is an example of a manual DNS failover:
Change the DNS to point to the secondary site load balancer:
The master (primary) DNS server is updated with the zone information, and the change is announced with the DNS NOTIFY announcement.
The slave DNS servers are notified of the zone update with a DNS NOTIFY announcement, and the slave DNS servers pull the zone information.
|
Note: The master and slave servers are authoritative name servers. Therefore, they contain trusted DNS information. |
Clear affected records from caching DNS servers.
A caching DNS server is used primarily for performance and fast response. The caching server obtains information from an authoritative DNS server in response to a host query and then saves (caches) the data locally. On a second or subsequent request for the same data, the caching DNS server responds with its locally stored data (the cache) until the time-to-live (TTL) value of the response expires. At this time, the server refreshes the data from the zone master. If the DNS record is changed on the primary DNS server, then the caching DNS server does not pick up the change for cached records until TTL expires. Flushing the cache forces the caching DNS server to go to an authoritative DNS server again for the updated DNS information.
Flush the cache if the DNS server being used supports such a capability. The following is the flush capability of common DNS BIND versions:
BIND 9.3.0: The command rndc flushname name flushes individual entries from the cache.
BIND 9.2.0 and 9.2.1: The entire cache can be flushed with the command rndc flush.
BIND 8 and BIND 9 up to 9.1.3: Restarting the named server clears the cache.
Refresh local DNS service caching.
Some operating systems might cache DNS information locally in the local name service cache. If so, this cache must also be cleared so that DNS updates are recognized quickly.
Solaris: nscd
Linux: /etc/init.d/nscd restart
Microsoft Windows: ipconfig /flushdns
Apple Mac OS X: lookupd -flushcache
The secondary site load balancer directs traffic to the secondary site middle-tier application server.
The secondary site is ready to take client requests.
Failover also depends on the client's web browser. Most browser applications cache the DNS entry for a period. Consequently, sessions in progress during an outage might not fail over until the cache timeout expires. To resume service to such clients, close the browser and restart it.
Failover is the operation of transitioning one standby database to the role of primary database. A failover operation is invoked when an unplanned failure occurs on the primary database and there is no possibility of recovering the primary database in a timely fashion.
With Oracle Data Guard, you can automate the failover process using the broker and fast-start failover, or you can perform the failover manually:
Fast-start failover eliminates the uncertainty of a process that requires manual intervention and automatically executes a zero loss or minimum-loss failover (that you configure using the FastStartFailoverLagLimit property) within seconds of an outage being detected. See Section 9.4.2.3, "Fast-Start Failover Best Practices" for configuration best practices.
Manual failover allows for a failover process where decisions are user driven using any of the following methods:
Oracle Enterprise Manager
The broker command-line interface (DGMGRL)
SQL*Plus statements
See Section 13.2.2.3, "Best Practices for Performing Manual Failover".
A database failover is accompanied by an application failover and, in some cases, preceded by a site failover. After the Data Guard failover, the secondary site hosts the primary database. You must reinstate the original primary database as a new standby database to restore fault tolerance of the configuration. See Section 13.3.2, "Restoring a Standby Database After a Failover."
A failover operation typically occurs in under a minute, and with little or no data loss.
|
See Also:
|
When a primary database failure cannot be repaired in time to meet your Recovery Time Objective (RTO) using local backups or Flashback technology, you should perform a failover using Oracle Data Guard.
You should perform a failover manually due to an unplanned outage such as:
A site disaster, which results in the primary database becoming unavailable
Damage resulting from user errors that cannot be repaired in a timely fashion
Data failures, which impact the production application
A failover requires that you reinstate the initial primary database as a standby database to restore fault tolerance to your environment. You can quickly reinstate the standby database using Flashback Database provided the original primary database has not been damaged. See Section 13.3.2, "Restoring a Standby Database After a Failover."
A fast-start failover is completely automated and requires no user intervention.
There are no procedural best practices to consider when performing a fast-start failover. However, it is important to address all of the configuration best practices described in Section 9.4.2.3, "Fast-Start Failover Best Practices".
|
See Also: The MAA white paper "Data Guard Switchover and Failover Best Practices" from the MAA Best Practices area for Oracle Database at |
When performing a manual failover:
Follow the configuration best practices outlined in Section 9.4.2.4, "Manual Failover Best Practices."
Choose from the following methods:
Oracle Enterprise Manager
See Oracle Data Guard Broker for complete information about how to perform a manual failover using Oracle Enterprise Manager. The procedure is the same for both physical and logical standby databases.
Oracle Data Guard broker command-line interface (DGMGRL)
See Oracle Data Guard Broker for complete information about how to perform a manual failover using Oracle Enterprise Manager. The procedure is the same for both physical and logical standby databases.
SQL*Plus statements:
Oracle Data Guard Concepts and Administration for information about Physical standby database steps for "Performing a Failover to a Physical Standby Database"
Oracle Data Guard Concepts and Administration for information about Logical standby database steps for "Performing a Failover to a Logical Standby Database"
Oracle RAC Recovery is performed automatically when there is a node or instance failure. In regular multi instance Oracle RAC environments, surviving instances automatically recover the failed instances and potentially aid in the automatic client failover. Recover times can be bounded by adopting the database and Oracle RAC configuration best practices and can usually lead to instance recovery times of seconds to minutes in very large busy systems, with no data loss. For Oracle RAC One Node configurations recover times are expected to take longer than full Oracle RAC; with Oracle RAC One Node a replacement instance must be started first before it can do the instance recovery.
For instance or node failures with Oracle RAC and Oracle RAC One Node, use the following recovery methods:
Instance failure occurs when software or hardware problems cause an instance to shutdown or abort. After instance failure, Oracle automatically uses the online redo log file to perform database recovery.
Instance recovery in Oracle RAC does not include restarting the failed instance or the recovery of applications that were running on the failed instance. Applications will run continuously using service relocation and fast application notification (as described in Section 13.2.3.2, "Automatic Service Relocation").
When one instance performs recovery for another instance, the recovering instance:
Reads redo log entries generated by the failed instance and uses that information to ensure that committed transactions are recorded in the database. Thus, data from committed transactions is not lost
Rolls back uncommitted transactions that were active at the time of the failure and releases resources used by those transactions
When multiple instances fail, if one instance survives Oracle RAC performs instance recovery for any other instances that fail. If all instances of an Oracle RAC database fail, then on subsequent restart of any instance a crash recovery occurs and all committed transactions are recovered. Data Guard is the recommended solution to survive outages when all instances of a cluster fail.
Service reliability is achieved by configuring and failing over among the surviving instances. A service will be made available by multiple database instances to provide a service that is needed. If a hardware failure occurs and the failure adversely affects an Oracle RAC database instance, then depending on the configuration, Oracle Clusterware does one the following:
Oracle Clusterware automatically moves any services on the failed database instance to another available instance, as configured with DBCA or Enterprise Manager. Oracle Clusterware recognizes when a failure affects a service and automatically fails over the service across the surviving instances supporting the service.
|
Note: With Oracle RAC One Node the relocation occurs when another instance on a different node is started and enabled for the appropriate services. Thus, Oracle RAC One Node starts a new instance when an instance fails but the new instance is not a "surviving instance." |
A service can be made available on multiple instances by default. In this case, when one of those multiple instances is lost the clients continue to use the available services across the surviving instances, but there are less resources to do the work.
In parallel, Oracle Clusterware attempts to restart and integrate the failed instances and dependent resources back into the system and Cluster Ready Services (CRS) will try to restart the database instance three times. Clients can "subscribe" to node failure events, in this way clients can be notified of instance problems quickly and new connections can be setup (Oracle Clusterware does not setup the new connections, the clients setup the new connections). Notification of failures using fast application notification (FAN) events occur at various levels within the Oracle Server architecture. The response can include notifying external parties through Oracle Notification Service (ONS), advanced queuing, or FAN callouts, recording the fault for tracking, event logging, and interrupting applications. Notification occurs from a surviving node when the failed node is out of service. The location and number of nodes serving a service is transparent to applications. Restart and recovery after a node shutdown or clusterware restart are done automatically.
Loss of the Oracle Cluster Registry (OCR) file affects the availability of Oracle RAC and Oracle Clusterware. The OCR file can be restored from a backup that is automatically created or from an export file that is manually created by using the ocrconfig tool (also use ocrconfig to restore the backup). Additionally, Oracle can optionally mirror the OCR so that a single OCR device failure can be tolerated. Ensure the OCR mirror is on a physically separate device and preferably on a separate controller. For more information, see Section 6.2.7, "Mirror Oracle Cluster Registry (OCR) and Configure Multiple Voting Disks with Oracle ASM".
If all of the voting disks are corrupted, then you must restore them. To do this you use the crsctl command. The steps you use depend on where you store your voting files. If the voting disks are stored in Oracle ASM, then run the commands to migrate the voting disks to the Oracle ASM disk group you specify, with: crsctl replace votedisk. If you did not store voting disks in Oracle ASM, then you run the commands to delete and add the voting disks: crsctl delete css votedisk and crsctl add css votedisk.
|
See Also:
|
With a minimal configuration, applications can receive fast and efficient notification when instances providing services become unavailable. When notified, application reconnects occur transparently to the surviving instances of an Oracle RAC database or to a standby database that has assumed the primary role following a failover.
In an Oracle RAC configuration, services are essential to achieving fast and transparent application failover. Clients are notified of a service relocation through Fast Application Notification (FAN).
In an Oracle Data Guard configuration, you can configure services for client failover across sites. After a site failure in a Data Guard configuration, the new primary database can automatically publish the production service while notifying affected clients, through FAN events, that the services are no longer available on the failed primary database.
For hangs or situations in which the response time is unacceptable, you can configure Oracle Enterprise Manager or a custom application heartbeat to detect application or response time slowdown and react to these situations. For example, you can configure the Enterprise Manager Beacon to monitor and detect application response times. Then, after a certain time threshold expires, Enterprise Manager can call the Oracle Data Guard DBMS_DG.INITIATE_FS_FAILOVER PL/SQL procedure to initiate a database failover immediately followed by an application failover using FAN notifications and service relocation.
FAN notifications and service relocation enable automatic and fast redirection of clients if any failure or planned maintenance results in an Oracle RAC or Oracle Data Guard fail over.
|
See Also:
|
Table 13-3 summarizes the impacts and recommended repairs for various Oracle ASM failure types.
Table 13-3 Types of Oracle ASM Failures and Recommended Repair
| Failure | Description | Impact | Recommended Repair |
|---|---|---|---|
|
Oracle ASM instance failure |
Oracle ASM instance fails |
All database instances accessing Oracle ASM storage from the same node shut down |
Automatic Section 13.2.3, "Oracle RAC Recovery for Unscheduled Outages (for Node or Instance Failures)" If Oracle RAC is not used, use Data Guard failover (see Section 13.2.2.2, "Best Practices for Implementing Fast-Start Failover") If Oracle RAC and Data Guard are not used, fix the underlying problem and then restart Oracle ASM and the database instances |
|
Oracle ASM disk failure |
One or more Oracle ASM disks fail, but all disk groups remain online |
All data remains accessible. This is possible only with normal or high redundancy disk groups |
Oracle ASM automatically rebalances to the remaining disk drives and reestablishes redundancy. There must be enough free disk space in the remaining disk drives to restore the redundancy or the rebalance may fail with an |
|
Data area disk-group failure |
One or more Oracle ASM disks fail, and data area disk group goes offline |
Databases accessing the data area disk group shut down |
Perform Data Guard failover or local recovery as described in Section 13.2.5.3, "Data Area Disk Group Failure" |
|
Fast recovery area disk-group failure |
One or more Oracle ASM disks fail, and the fast recovery area disk group goes offline |
Databases accessing the fast recovery area disk group shut down |
Perform local recovery or Data Guard failover as described in Section 13.2.5.4, "Fast Recovery Area Disk Group Failure" |
If the Oracle ASM instance fails, then database instances accessing Oracle ASM storage from the same node shut down. The following list describes failover processing:
If the primary database is an Oracle RAC database, then application failover occurs automatically and clients connected to the database instance reconnect to remaining instances. Thus, the service is provided by other instances in the cluster and processing continues. The recovery time typically occurs in seconds.
If the primary database is not an Oracle RAC database, then an Oracle ASM instance failure shuts down the entire database.
If the configuration uses Oracle Data Guard and fast-start failover is enabled, a database failover is triggered automatically and clients automatically reconnect to the new primary database after the failover completes. The recovery time is the amount of time it takes to complete an automatic Data Guard fast-start failover operation. If fast-start failover is not configured, then you must recover from this outage by either restarting the Oracle ASM and database instances manually, or by performing a manual Data Guard failover.
If the configuration includes neither Oracle RAC nor Data Guard, then you must manually restart the Oracle ASM instance and database instances. The recovery time depends on how long it takes to perform these tasks.
If the Oracle ASM disk fails, then failover processing is as follows:
If an Oracle ASM disk group is configured as an external redundancy type, then a failure of a single disk is handled by the storage array and should not be seen by the Oracle ASM instance. All Oracle ASM and database operations using the disk group continue normally.
However, if the failure of an external redundancy disk group is seen by the Oracle ASM instance, then the Oracle ASM instance takes the disk group offline immediately, causing Oracle instances accessing the disk group to crash. If the disk failure is temporary, then you can restart Oracle ASM and the database instances and crash recovery occurs after the disk group is brought back online.
If an Oracle ASM disk group is configured as a normal or a high-redundancy type, then disk failure is handled transparently by Oracle ASM and the databases accessing the disk group are not affected.
An Oracle ASM instance automatically starts an Oracle ASM rebalance operation to distribute the data of one or more failed disks to the remaining, intact disks of the Oracle ASM disk group. While the rebalance operation is in progress, subsequent disk failures may affect disk group availability if the disk contains data that has yet to be remirrored. When the rebalance operation completes successfully, the Oracle ASM disk group is no longer at risk in the event of a subsequent failure. Multiple disk failures are handled similarly, provided the failures affect only one failure group in an Oracle ASM disk group with normal redundancy.
The failure of multiple disks in multiple failure groups where a primary extent and all of its mirrors have been lost causes the disk group to go offline.
When Oracle ASM disks fail, use the following recovery methods:
Figure 13-3 shows Enterprise Manager reporting disk failures. Five of 14 alerts are shown. The five alerts shown are Offline messages for Disk RECO2.
Figure 13-3 Enterprise Manager Reports Disk Failures
Figure 13-4 shows Enterprise Manager reporting the status of data area disk group DATA, database Data Guard disk group DBFS_DG, and recovery area disk group RECO.
Figure 13-4 Enterprise Manager Reports Oracle ASM Disk Groups Status
Figure 13-5 shows Enterprise Manager reporting a pending REBAL operation on the DATA disk group. The operation is almost done, as shown in % Complete, and the Remaining Time is estimated to be 0 minutes.
Figure 13-5 Enterprise Manager Reports Pending REBAL Operation
Perform these steps after one or more failed disks of one specific failure group have been dropped and must be replaced with new disks:
Add the one or more replacement disks to the failed disk group with the following SQL command:
ALTER DISKGROUP disk_group ADD FAILGROUP failure_group DISK 'disk1','disk2',...;
Check the progress of the operation:
SELECT * FROM V$ASM_OPERATION;
A data area disk group failure should occur only when there have been multiple failures. For example, if the data area disk group is defined as external redundancy, a single-disk failure should not be exposed to Oracle ASM. However, multiple disk failures in a storage array may be seen by Oracle ASM causing the disk group to go offline. Similarly, multiple disk failures in different failure groups in a normal or high-redundancy disk group may cause the disk group to go offline.
When one or more disks fail in a normal or high redundancy disk group, and the Oracle ASM disk group is accessible, there is no loss of data and no immediate loss of accessibility. An Oracle ASM instance automatically starts an Oracle ASM rebalance operation to distribute the data on the one or more failed disks to the disks that remain intact in the Oracle ASM disk group. When the rebalance operation completes successfully, the Oracle ASM disk group is no longer at risk if a second failure occurs. There must be enough free disk space on the remaining disks in the disk group for the rebalance to complete successfully.
Table 13-4 summarizes the possible solutions for recovering from a data area disk group failure.
Table 13-4 Recovery Options for Data Area Disk Group Failure
| Recovery Option | Recovery Time Objective (RTO) | Recovery Point Objective (RPO) |
|---|---|---|
|
Data Guard failover (see Section 13.2.5.4, "Fast Recovery Area Disk Group Failure") |
Five minutes or less |
Varies depending on the data protection level chosen |
|
Local Recovery (see "Local Recovery Steps") |
Database restore and recovery time |
Zero |
If Data Guard is being used and fast-start failover is configured, then an automatic failover occurs when the database shuts down due to the data area disk group going offline. If fast-start failover is not configured, then perform a manual failover.
If you decide to perform a Data Guard failover then the recovery time objective (RTO) is expressed in terms of minutes or seconds, depending on the presence of the Data Guard observer process and fast-start failover. However, if a manual failover occurs and not all data is available on the standby site, then data loss might result.
After Data Guard failover has completed and the application is available, you must resolve the data area disk group failure. Continue with the following "Local Recovery Steps" procedure to resolve the Oracle ASM disk group failure.
The RTO for local recovery only is based on the time required to:
Repair and replace the failed storage components
Restore and recover the database
Because the loss affects only the data-area disk group, there is no loss of data. All transactions are recorded in the Oracle redo log members that reside in the fast recovery area, so complete media recovery is possible.
If you are not using Data Guard, then perform the following local recovery steps. The time required to perform local recovery depends on how long it takes to restore and recover the database. There is no data loss when performing local recovery.
Perform these steps after one or more failed disks have been replaced and access to the storage has been restored:
|
Note: If you have performed an Oracle Data Guard failover to a new primary database, then you can now use the following procedure to restore and sync the Data Guard setup. Also, see Section 13.3.2, "Restoring a Standby Database After a Failover". |
Rebuild the Oracle ASM disk group using the new storage location by issuing the following SQL*Plus statement on the Oracle ASM instance:
SQL> CREATE DISKGROUP DATA NORMAL REDUNDANCY DISK 'path1','path2',...force;
Start the database instance NOMOUNT by issuing the following RMAN command:
RMAN> STARTUP FORCE NOMOUNT;
Restore the control file from the surviving copy located in the recovery area:
RMAN> RESTORE CONTROLFILE FROM 'recovery_area_controlfile';
Start the database instance MOUNT:
RMAN> STARTUP FORCE MOUNT;
Restore the database:
RMAN> RESTORE DATABASE
Recover the database:
RMAN> RECOVER DATABASE;
If you use block change tracking, then disable and re-enable the block change tracking file using SQL*Plus statements:
SQL> ALTER DATABASE DISABLE BLOCK CHANGE TRACKING; SQL> ALTER DATABASE ENABLE BLOCK CHANGE TRACKING;
Open the database:
SQL> ALTER DATABASE OPEN;
Re-create the log file members on the failed Oracle ASM disk group:
SQL> ALTER DATABASE DROP LOGFILE MEMBER 'filename'; SQL> ALTER DATABASE ADD LOGFILE MEMBER 'disk_group' TO GROUP group_no;
Perform an incremental level 0 backup using the following RMAN command:
RMAN> BACKUP INCREMENTAL LEVEL 0 DATABASE;
When the fast recovery-area disk group fails, the database crashes because the control file member usually resides in the fast recovery area and Oracle requires that all control file members are accessible. The fast recovery area can also contain the flashback logs, redo log members, and all backup files.
Because the failure affects only the fast recovery-area disk group, there is no loss of data. No database media recovery is required, because the data files and the online redo log files are still present and available in the data area.
A fast recovery area disk group failure typically occurs only when there have been multiple failures. For example, if the fast recovery-area disk group is defined as external redundancy, a single-disk failure should not be exposed to Oracle ASM. However, multiple disk failures in a storage array may affect Oracle ASM and cause the disk group to go offline. Similarly, multiple disk failures in different failure groups in a normal or high-redundancy disk group may cause the disk group to go offline.
Table 13-5 summarizes possible solutions when the fast recovery-area disk group fails.
Table 13-5 Recovery Options for Fast Recovery Area Disk Group Failure
| Recovery Option | Recovery Time Objective (RTO) | Recovery Point Objective (RPO) |
|---|---|---|
|
Local recovery (see Section 13.2.5.4.1, "Local Recovery for Fast Recovery Area Disk Group Failure") |
Five minutes or less |
Zero |
|
Data Guard failover or switchover (see Section 13.2.5.4.2, "Data Guard Role Transition for Fast Recovery Area Disk Group Failure") |
Five minutes or less |
Zero |
If you decide to perform local recovery then you must perform a fast local restart to start the primary database after removing the controlfile member that is located in the fast recovery area from the init.ora and allocate another disk group as the fast recovery area for archiving.
For a fast local restart, perform the following steps on the primary database:
Change the CONTROL_FILES initialization parameter to specify only the members in the Data Area:
ALTER SYSTEM SET CONTROL_FILES='+DATA/sales/control1.dbf' SCOPE=spfile;
Change local archive destinations and the fast recovery area to the local redundant, scalable destination:
ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='+DATA' SCOPE=spfile;
Start the database with the new settings:
STARTUP MOUNT:
Drop the redo log members that were in the lost disk group. For example, issue the following command:
ALTER DATABASE DROP LOGFILE MEMBER '+RECO/dbm/onlinelog/group_2.258.750768395';
If the flashback logs were damaged or lost, it may be necessary to disable and reenable Flashback Database:
ALTER DATABASE FLASHBACK OFF; ALTER DATABASE FLASHBACK ON; ALTER DATABASE OPEN;
However, this is a temporary fix until you create a fast recovery area to replace the failed storage components. Oracle recommends using the Local Recovery Steps. For more information, see "Data Guard Role Transition for Fast Recovery Area Disk Group Failure Local Recovery Steps".
If you decide to perform a Data Guard role transition then the recovery time objective (RTO) can be expressed in terms of seconds or minutes, depending on the presence of the Data Guard observer process and fast-start failover.
If the protection level is maximum performance or the standby database is unsynchronized with the primary database, then:
Temporarily start the primary database by removing the controlfile member and pointing to a temporary fast recovery area (file system) in the SPFILE.
Perform a Data Guard switchover to ensure no data loss.
After the switchover has completed and the application is available, resolve the fast recovery area disk group failure.
Shut down the affected database and continue by using the instructions in the Local Recovery Steps to resolve the Oracle ASM disk group failure. For more information, see "Data Guard Role Transition for Fast Recovery Area Disk Group Failure Local Recovery Steps".
|
Note: If you performed an Oracle Data Guard failover to a new primary database, then you cannot use this procedure to reintroduce the original primary database as a standby database. This is because Flashback Database log files that are required as part of reintroducing the database have been lost. You must perform a full reinstatement of the standby database. |
Replace or get access to storage to use for a fast recovery area
Rebuild the Oracle ASM disk group using the storage location by issuing the following SQL*Plus statement:
SQL> CREATE DISKGROUP RECO NORMAL REDUNDANCY DISK 'path1','path2',...force;
Start the database instance NOMOUNT using the following RMAN command:
RMAN> STARTUP FORCE NOMOUNT;
Restore the control file from the surviving copy located in the data area:
RMAN> RESTORE CONTROLFILE FROM 'data_area_controlfile';
Start the database instance MOUNT:
RMAN> STARTUP FORCE MOUNT;
If you use Flashback Database, then disable it with the following SQL*Plus statement:
SQL> ALTER DATABASE FLASHBACK OFF;
Open the database and allow instance recovery to complete:
SQL> ALTER DATABASE OPEN;
Issue the following statements only if Flashback Database is required:
SQL> SHUTDOWN IMMEDIATE; SQL> STARTUP MOUNT; SQL> ALTER DATABASE FLASHBACK ON; SQL> ALTER DATABASE OPEN;
Re-create the log file members on the failed Oracle ASM disk group:
SQL> ALTER DATABASE DROP LOGFILE MEMBER 'filename'; SQL> ALTER DATABASE ADD LOGFILE MEMBER 'disk_group' TO GROUP group_no;
Synchronize the control file and the fast recovery area using the following RMAN commands:
RMAN> CATALOG RECOVERY AREA; RMAN> CROSSCHECK ARCHIVELOG ALL; RMAN> CROSSCHECK BACKUPSET; RMAN> CROSSCHECK DATAFILECOPY ALL; RMAN> LIST EXPIRED type; RMAN> DELETE EXPIRED type;
In the example, the type variable is a placeholder for both LIST EXPIRED BACKUP and LIST EXPIRED COPY commands, and also for the DELETE EXPIRED BACKUP and DELETE EXPIRED COPY commands. You should run all of these commands now.
Assuming that data has been lost, perform a backup:
RMAN> BACKUP INCREMENTAL LEVEL 0 DATABASE;
A data block is corrupted when it is not in a recognized Oracle Database format, or its contents are not internally consistent. Data block corruption can damage internal Oracle control information or application and user data, leading to crippling loss of critical data and services. The Oracle Database corruption prevention, detection, and repair capabilities are built on internal knowledge of the data and transactions it protects, and on the intelligent integration of its comprehensive high availability solutions. For more information, see Section 5.1.6, "Protect Against Data Corruption".
The recovery process begins when you either suspect or discover a block corruption (for example: ORA-1578, ORA-752, and ORA-600 [3020]). Once a corrupt block is found, Oracle provides various techniques for recovering from most block corruptions.
There are various techniques for recovering data blocks, including:
Data Recovery Advisor is the easiest way to diagnose and repair most problems. For more information, see Section 13.2.6.1, "Use Data Recovery Advisor".
Active Data Guard can automatically repair corrupt data blocks in a primary or standby database. For more information, see Section 13.2.6.2, "Use Active Data Guard".
RMAN block media recovery can repair individual corrupted blocks by retrieving the blocks from good backups. For more information, see Section 13.2.6.3, "Use RMAN and Block Media Recovery".
Data Guard switchover or failover to a standby database. For more information, see Section 13.2.6.2.2, "Extracting Data from a Physical Standby Databases".
Data File Media Recovery with RMAN. For more information, see Section 13.2.6.3, "Use RMAN and Block Media Recovery"
When encountering lost write corruptions that result in ORA-752 or ORA-600 [3020], follow the guidelines in "Resolving ORA-752 or ORA-600 [3020] During Standby Recovery" in My Oracle Support Note 1265884.1 at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1265884.1
Whatever method you use to recover corrupted blocks, you first must analyze the type and degree of corruption to perform the recovery. Implementing the optimal techniques to prevent and prepare for data corruptions can save time, effort, and stress when dealing with the possible consequences-lost data and downtime.
MAA best practices provide a step-by-step process for resolving most corruptions and stray or lost writes, including the following:
|
See Also:
|
Data Recovery Advisor enables you to perform restore operations and recovery procedures or use Flashback Database as follows:
Perform block media recovery of data files that have corrupted blocks
Perform point-in-time recovery of the database or selected tablespaces
Rewind the entire database with Flashback Database
Completely restore and recover the database from a backup
Data Recovery Advisor has both a command-line and GUI interface. The GUI interface is available when you click Perform Recovery with Oracle Enterprise Manager Database Control (Support Workbench); this allows you use Data Recovery Advisor.
Using the RMAN command-line interface, the Data Recovery Advisor commands include: LIST FAILURE, ADVISE FAILURE, REPAIR FAILURE, and CHANGE FAILURE.
If the Data Recover Advisor fixes the problem, then there is no need to continue with any further recovery methods. However, continue to periodically check the alert log for any ORA- errors and any corruption warnings on the primary and standby databases. While the database is operational and corruption is detected, corruption errors are recorded as ORA-600 or ORA-01578 in the alert log.
|
Note: In the current release, Data Recovery Advisor only supports single-instance databases. Oracle RAC databases are not supported. See Oracle Database Backup and Recovery User's Guide for more information about Data Recovery Advisor supported database configurations. |
|
See Also:
|
There are two options for using a standby database to repair block corruption on the primary database:
Alternatively, if the corruption is widespread, you may choose to failover or switchover to the standby database while you make repairs to the primary database. For more information, see Section 13.2.6.4, "Perform a Data Guard Role Transition".
Starting in Oracle Database 11g Release 2 (11.2), the primary database automatically attempts to repair a corrupted block in real time by fetching a good version of the same block from a physical standby database. This capability is referred to as Automatic Block Repair; Automatic Block Repair allows corrupt data blocks to be automatically repaired as soon as the corruption is detected. Automatic Block Repair reduces the amount of time that data is inaccessible due to block corruption and reduces block recovery time by using up-to-date good blocks in real-time, as opposed to retrieving blocks from disk or tape backups, or from Flashback logs.
Thus, with Automatic Block Repair you use an Oracle Active Data Guard standby database for automatic repair of data corruptions detected by the primary database. Additionally if the corruption is discovered on an Active Data Guard physical standby database the corruption is automatically repaired with a good block from the Primary. Both of these operations are transparent to the applications.
|
Note: Automatic Block Repair requires the use of the Oracle Active Data Guard option. |
|
See Also: Oracle Data Guard Concepts and Administration for more information about Oracle Active Data Guard option and the Automatic Block Repair feature |
You can use a Data Guard physical standby database to repair data file wide block corruption on the primary database by replacing the corrupted data files with good copies from the standby database. Once the files are restored on the primary database, data file or tablespace recovery makes the data files consistent with the rest of the database.
|
See Also: Oracle Data Guard Concepts and Administration for information about Recovery from Loss of Data Files on the Primary Database Using Files On a Standby Database |
Block media recovery recovers one block or a set of data blocks marked "media corrupt" in a data file by using the RMAN RECOVER BLOCK command. When a small number of data blocks are marked media corrupt and require media recovery, you can selectively restore and recover damaged blocks rather than whole data files. Block media recovery minimizes redo application time and avoids I/O overhead during recovery. Block media recovery also enables affected data files to remain online during recovery of the corrupt blocks. The corrupt blocks, however, remain unavailable until they are completely recovered.
Use block media recovery when:
A small number of blocks require media recovery and you know which blocks need recovery.
Blocks are marked corrupt (you can verify this with the RMAN VALIDATE CHECK LOGICAL command).
The backup file for the corrupted data file is available locally or can be retrieved from a remote location.
|
Note: Do not use block media recovery to recover from user errors or software bugs that cause logical corruption where the data blocks are intact. |
If a significant portion of the data file is corrupt or if the amount of corruption is unknown, then use either RMAN to restore the file from a backup or switch to an on disk image copy, or switchover to your Data Guard standby database.
When corruption is detected, recover the block through the Oracle Enterprise Manager Restore and Recovery Wizard or directly with RMAN. For example, to recover a specific corrupt block using RMAN block media recovery:
RMAN> RECOVER BLOCK DATAFILE 7 BLOCK 3;
After a corrupt block is repaired, the row identifying this corrupted block is deleted from the V$DATABASE_BLOCK_CORRUPTION view.
|
See Also: Oracle Database Backup and Recovery User's Guide for information about RMAN's block media recovery |
If the primary database corruption is widespread due to a bad controller or other hardware or software problem, then you may want to failover or switchover to the standby database while repairs to the primary database server are made. Use Data Guard switchover or failover for data corruption or data failure when:
The database is down or when the database is up but the application is unavailable because of data corruption or failure, and the time to restore and recover locally is long or unknown.
Recovering locally takes longer than the business service-level agreement or RTO.
|
See Also: Oracle Data Guard Concepts and Administration for more information about Data Guard failovers and switchovers |
When you cannot use any of the following methods to resolve corruptions and stray or lost writes, then use RMAN traditional media recovery:
|
Note: If you do not have a Data Guard Physical standby, then you must use traditional media recovery. Using traditional media recovery, a backup copy of one or more files is restored and then data file, tablespace, or database recovery brings the database back to a consistent state. |
Data file media recovery affects an entire data file or set of data files for a database by using the RMAN RECOVER command. When a large or unknown number of data blocks are marked "media corrupt" and require media recovery, or when an entire file is lost, you must restore and recover the applicable data files.
|
See Also: Oracle Database Backup and Recovery User's Guide for information about Advanced User-Managed Recovery Scenarios |
Oracle Flashback technology revolutionizes data recovery. Before Flashback technology, it took seconds to damage a database but from hours to days to recover it. With Flashback technology, the time to correct errors can be as short as the time it took to make the error. Fixing human errors that require rewinding the database, table, transaction, or row level changes to a previous point in time is easy and does not require any database or object restoration. Flashback technology provides fine-grained analysis and repair for localized damage such as erroneous row deletion. Flashback technology also enables correction of more widespread damage such as accidentally running the wrong application batch job. Furthermore, Flashback technology is exponentially faster than a database restoration.
Flashback technologies are applicable only to repairing the following human errors:
Erroneous or malicious update, delete, or insert transactions
Erroneous or malicious DROP TABLE statements
Erroneous or malicious batch job or wide-spread application errors
Flashback technologies cannot be used for media or data corruption such as block corruption, bad disks, or file deletions. See Section 13.2.2, "Database Failover with a Standby Database" to repair these outages.
|
Note: For information about Flashback Database configuration best practices, see Section 5.1.4, "Enable Flashback Database" |
Table 13-6 summarizes the Flashback solutions for outage varying in scope from destroying a row, such as through a bad update, to destroying a whole database (such as by deleting all the underlying files at the operating system level).
Table 13-6 Flashback Solutions for Different Outages
| Outage Scope | Examples of Human Errors | Flashback Solutions | See Also |
|---|---|---|---|
|
Row or transaction |
Accidental deletion of row Erroneous transaction |
|
See Also: Section 13.2.7.2, "Resolving Row and Transaction Inconsistencies" |
|
Table |
Dropped table Erroneous transactions affecting one table or a set of tables |
|
See Also: Section 13.2.7.1, "Resolving Table Inconsistencies" |
|
Tablespace or database |
Erroneous batch job affecting many tables or an unknown set of tables Series of database-wide malicious transactions |
Enable Flashback Database or use multiple Flashback Table commands |
See Also: Section 13.2.7.3, "Resolving Database-Wide Inconsistencies" |
|
Single tablespace or a subset of tablespaces |
Erroneous transactions affecting a small number of tablespaces |
RMAN Tablespace Point-in-Time Recovery (TSPITR) |
See Also: Section 13.2.7.4, "Resolving One or More Tablespace Inconsistencies" |
Table 13-7 summarizes each Flashback feature.
Table 13-7 Summary of Flashback Features
Flashback Database uses the Oracle Database flashback logs, while all other features of flashback technology use the Oracle Database unique undo and multiversion read consistency capabilities. For more information, see the configuration best practices for the database, as documented in Section 5.1, "Database Configuration High Availability and Fast Recoverability Best Practices" to configure Flashback technologies to ensure that the resources from these solutions are available at a time of failure.
|
See Also:
|
In general, the recovery time when using Flashback technologies is equivalent to the time it takes to cause the human error plus the time it takes to detect the human error.
Flashback technologies allow recovery up to the point that the human error occurred.
Use the following recovery methods:
Dropping or deleting database objects by accident is a common mistake. Users soon realize their mistake, but by then it is too late and there has been no way to easily recover the dropped tables and its indexes, constraints, and triggers. Objects once dropped were dropped forever. Loss of very important tables or other objects (like indexes, partitions or clusters) required DBAs to perform a point-in-time recovery, which can be time-consuming and lead to loss of recent transactions.
Oracle provides the following statements to help resolve table inconsistencies:
Flashback Table statement to restore a table to a previous point in the database
Flashback Drop statement to recover from an accidental DROP TABLE statement
Flashback Transaction statement to roll back one or more transactions and their dependent transactions, while the database remains online
Flashback Table
Flashback Table provides the ability to quickly recover a table or a set of tables to a specified point in time. In many cases, Flashback Table alleviates the more complicated point-in-time recovery operations. For example:
FLASHBACK TABLE orders, order_items
TO TIMESTAMP
TO_DATE('28-Jun-11 14.00.00','dd-Mon-yy hh24:mi:ss');
This statement rewinds any updates to the ORDERS and ORDER_ITEMS tables that have been done between the current time and a specified timestamp in the past. Flashback Table performs this operation online and in place, and it maintains referential integrity constraints between the tables.
Flashback Drop
Flashback Drop provides a safety net when dropping objects. When a user drops a table, Oracle places it in a recycle bin. Objects in the recycle bin remain there until the user decides to permanently remove them or until space limitations begin to occur on the tablespace containing the table. The recycle bin is a virtual container where all dropped objects reside. Users view the recycle bin and undrop the dropped table and its dependent objects. For example, the employees table and all its dependent objects would be undropped by the following statement:
FLASHBACK TABLE employees TO BEFORE DROP;
Flashback Transaction
Oracle Flashback Transaction increases availability during logical recovery by easily and quickly backing out a specific transaction or set of transactions and their dependent transactions, while the database remains online.
Use the DBMS_FLASHBACK.TRANSACTION_BACKOUT() PL/SQL procedure to roll back a transaction and its dependent transactions. This procedure uses undo data to create and execute the compensating transactions that return the affected data to its pre-transaction state.
|
See Also:
|
Resolving row and transaction inconsistencies might require a combination of Flashback Query, Flashback Version Query, Flashback Transaction Query, and the compensating SQL statements constructed from undo statements to rectify the problem. This section describes a general approach using a human resources example to resolve row and transaction inconsistencies caused by erroneous or malicious user errors.
Flashback Query
Flashback Query enables an administrator or user to query any data from some earlier point in time. Use this feature to view and reconstruct data that might have been deleted or changed by accident.
Developers can use Flashback Query to build self-service error correction into their applications, empowering end users to undo and correct their errors without delay, and freeing database administrators from having to perform this task. Flashback Query is easy to manage because the database automatically keeps the necessary information to reconstruct data for a configurable time into the past.
The following partial statement displays rows from the EMPLOYEES table starting from 2:00 p.m. on June 28, 2011.
SELECT * FROM EMPLOYEES
AS OF TIMESTAMP
TO_DATE('28-Jun-11 14:00','DD-Mon-YY HH24:MI')
WHERE ...
Flashback Version Query
Flashback Version Query provides a way to view changes made to the database at the row level. Flashback Version Query is an extension to SQL and enables the retrieval of all the different versions of a row across a specified time interval. For example:
SELECT * FROM EMPLOYEES
VERSIONS BETWEEN TIMESTAMP
TO_DATE('28-Jun-11 14:00','dd-Mon-YY hh24:mi') AND
TO_DATE('28-Jun-11 15:00','dd-Mon-YY hh24:mi')
WHERE ...
This statement displays each version of the row, each entry changed by a different transaction, between 2 and 3 p.m. on June 28, 2011. A database administrator can use this to pinpoint when and how data is changed and trace it back to the user, application, or transaction. Flashback Version Query enables the database administrator to track down the source of a logical corruption in the database and correct it. It also enables application developers to debug their code.
Flashback Transaction Query
Flashback Transaction Query provides a way to view changes made to the database at the transaction level. Flashback Transaction Query is an extension to SQL that enables you to see all changes made by a transaction. For example:
SELECT UNDO_SQL FROM FLASHBACK_TRANSACTION_QUERY WHERE XID = '000200030000002D';
This statement shows all of the changes that resulted from this transaction. In addition, compensating SQL statements are returned and can be used to undo changes made to all rows by this transaction. Using a precision tool like Flashback Transaction Query, the database administrator and application developer can precisely diagnose and correct logical problems in the database or application.
Consider a human resources (HR) example involving the SCOTT schema. The HR manager reports to the database administrator that there is a potential discrepancy in Ward's salary. Sometime before 9:00 a.m., Ward's salary was increased to $1875. The HR manager is uncertain how this occurred and wishes to know when the employee's salary was increased. In addition, he instructed his staff to reset the salary to the previous level of $1250. This was completed around 9:15 a.m.
The following steps show how to approach the problem.
Assess the problem.
Fortunately, the HR manager has provided information about the time when the change occurred. You can query the information as it was at 9:00 a.m. using Flashback Query.
SELECT EMPNO, ENAME, SAL
FROM EMP
AS OF TIMESTAMP TO_DATE('28-JUN-11 09:00','dd-Mon-yy hh24:mi')
WHERE ENAME = 'WARD';
EMPNO ENAME SAL
---------- ---------- ----------
7521 WARD 1875
You can confirm that you have the correct employee by the fact that Ward's salary was $1875 at 09:00 a.m. Rather than using Ward's name, you can now use the employee number for subsequent investigation.
Query previous rows or versions of the data to acquire transaction information.
Although it is possible to restrict the row version information to a specific date or SCN range, you might want to query all the row information that is available for the employee WARD using Flashback Version Query.
SELECT EMPNO, ENAME, SAL, VERSIONS_STARTTIME, VERSIONS_ENDTIME, VERSIONS_XID
FROM EMP
VERSIONS BETWEEN TIMESTAMP MINVALUE AND MAXVALUE
WHERE EMPNO = 7521
ORDER BY NVL(VERSIONS_STARTSCN,1);
EMPNO ENAME SAL VERSIONS_STARTTIME VERSIONS_ENDTIME VERSIONS_XID
----- ------ --- ---------------------- -------------------- ---------------
7521 WARD 1250 28-JUN-11 08.48.43 AM 28-JUN-11 08.54.49 AM 0006000800000086
7521 WARD 1875 28-JUN-11 08.54.49 AM 28-JUN-11 09.10.09 AM 0009000500000089
7521 WARD 1250 28-JUN-11 09.10.09 AM 000800050000008B
You can see that WARD's salary was increased from $1250 to $1875 at 08:54:49 the same morning and was subsequently reset to $1250 at approximately 09:10:09.
Also, you can see that the ID of the erroneous transaction that increased WARD's salary to $1875 was "0009000500000089".
Query the erroneous transaction and the scope of its effect.
With the transaction information (VERSIONS_XID pseudocolumn), you can now query the database to determine the scope of the transaction, using Flashback Transaction Query.
SELECT UNDO_SQL
FROM FLASHBACK_TRANSACTION_QUERY
WHERE XID = HEXTORAW('0009000500000089');
UNDO_SQL
----------------------------------------------------------------------------
update "SCOTT"."EMP" set "SAL" = '950' where ROWID = 'AAACV4AAFAAAAKtAAL';
update "SCOTT"."EMP" set "SAL" = '1500' where ROWID = 'AAACV4AAFAAAAKtAAJ';
update "SCOTT"."EMP" set "SAL" = '2850' where ROWID = 'AAACV4AAFAAAAKtAAF';
update "SCOTT"."EMP" set "SAL" = '1250' where ROWID = 'AAACV4AAFAAAAKtAAE';
update "SCOTT"."EMP" set "SAL" = '1600' where ROWID = 'AAACV4AAFAAAAKtAAB';
6 rows selected.
You can see that WARD's salary was not the only change that occurred in the transaction. Now you can send the information that was changed for the other four employees at the same time as employee WARD, back to the HR manager for review.
Determine if the corrective statements should be executed.
If the HR manager decides that the corrective changes suggested by the UNDO_SQL column are correct, then the database administrator can execute the statements individually.
Query the FLASHBACK_TRANSACTION_QUERY view for additional transaction information. For example, to determine the user that performed the erroneous update, issue the following query:
SELECT LOGON_USER FROM FLASHBACK_TRANSACTION_QUERY
WHERE XID = HEXTORAW('0009000500000089');
LOGON_USER
----------------------------
MSMITH
In this example, the query shows that the user MSMITH was responsible for the erroneous transaction.
To bring an Oracle database to a previous point in time, the traditional method is point-in-time recovery. However, point-in-time recovery can take hours or even days, because it requires the whole database to be restored from backup and recovered to the point in time just before the error was introduced into the database. With the size of databases constantly growing, it takes hours or even days just to restore the whole database.
Flashback Database is a strategy for doing point-in-time recovery. Flashback Database quickly rewinds an Oracle database to a previous time to correct any problems caused by logical data corruption or user error. Flashback logs are used to capture old versions of changed blocks. When recovery must be performed the flashback logs are quickly replayed to restore the database to a point in time before the error and just the changed blocks are restored. Flashback Database is extremely fast and reduces recovery time from hours to minutes. In addition, it is easy to use. A database can be recovered to 2:05 p.m. by issuing a single statement. Before the database can be recovered, all instances of the database must be shut down and one instance subsequently mounted. The following is an example of a FLASHBACK DATABASE statement.
FLASHBACK DATABASE TO TIMESTAMP SYSDATE-1;
No restoration from tape, no lengthy downtime, and no complicated recovery procedures are required to use it. You can also use Flashback Database and then open the database in read-only mode and examine its contents. If you determine that you flashed back too far or not far enough, then you can reissue the FLASHBACK DATABASE statement or continue recovery to a later time to find the proper point in time before the database was damaged. Flashback Database works with a primary database, a physical standby database, or a logical standby database.
These steps are recommended for using Flashback Database:
Determine the time or the SCN to which to flash back the database.
Verify that there is sufficient flashback log information.
SELECT OLDEST_FLASHBACK_SCN,
TO_CHAR(OLDEST_FLASHBACK_TIME, 'mon-dd-yyyy HH:MI:SS')
FROM V$FLASHBACK_DATABASE_LOG;
Flash back the database to a specific time or SCN. (The database must be mounted to perform a Flashback Database.)
FLASHBACK DATABASE TO SCN scn;
or
FLASHBACK DATABASE TO TIMESTAMP TO_DATE date;
Open the database in read-only mode to verify that it is in the correct state.
ALTER DATABASE OPEN READ ONLY;
If more flashback data is required, then issue another FLASHBACK DATABASE statement. (The database must be mounted to perform a Flashback Database.)
If you want to move forward in time, then issue a statement similar to the following:
RECOVER DATABASE UNTIL [TIME date | CHANGE scn];
Open the database:
ALTER DATABASE OPEN RESETLOGS;
Other considerations when using Flashback Database are as follows:
If there are not sufficient flashback logs to flash back to the target time, then use an alternative:
Use Data Guard to recover to the target time if the standby database lags behind the primary database or flash back to the target time if there's sufficient flashback logs on the standby.
Restore from backups.
After flashing back a database, any dependent database such as a standby database must be flashed back. See Section 13.3, "Restoring Fault Tolerance".
Flashback Database does not automatically fix a dropped tablespace, you can use Flashback Database to significantly reduce the downtime. You can flash back the primary database to a point before the tablespace was dropped and then restore a backup of the corresponding data files using SET NEWNAME from the affected tablespace and recover to a time before the tablespace was dropped.
Recovery Manager (RMAN) automatic tablespace point-in-time recovery (TSPITR) enables you to quickly recover one or more tablespaces in a database to an earlier time without affecting the rest of the tablespaces and objects in the database. You can only use TSPITR on tablespaces whose data is completely segregated from the rest of the database. This usually means that TSPITR is something for which you must plan in advance.
RMAN TSPITR is most useful for the following situations:
To recover a logical database to a point different from the rest of the physical database, when multiple logical databases exist in separate tablespaces of one physical database. For example, you maintain logical databases in the Orders and Personnel tablespaces. An incorrect batch job or DML statement corrupts the data in only one tablespace.
To recover data lost after DDL operations that change the structure of tables. You cannot use Flashback Table to rewind a table to before the point of a structural change such as a truncate table operation.
To recover a table after it has been dropped with the PURGE option.
To recover from the logical corruption of a table.
You perform TSPITR by using the RMAN RECOVER TABLESPACE command.
|
See Also: Oracle Database Backup and Recovery User's Guide for detailed information about performing RMAN TSPITR |
Some applications may update multiple databases and participate in distributed transactions. Global consistency between the participating databases may be expected and crucial to the application.
If one database in a distributed database environment requires recovery to an earlier time, it is often necessary to recover all other databases in the configuration to the same point in time when global data consistency is required by the application.
To achieve coordinated, time-based, distributed database recovery:
Recover the database that requires the recovery operation using time-based recovery.
For example, if a database must be recovered because of a media failure, then recover this database first using time-based recovery. Do not recover the other databases at this point.
After you have recovered the database and opened it with the RESETLOGS option, search the alert_SID.log of the database for the RESETLOGS message. Your next step depends on the message that you find in the log file, as described in following table:
| If the message returned is ... | Then ... |
|---|---|
"RESETLOGS after complete recovery through change nnn" | Recovery is complete. You have applied all the changes in the database and performed complete recovery. Do not recover any of the other databases in the distributed system because this unnecessarily removes database changes. |
"RESETLOGS after incomplete recovery UNTIL CHANGE nnn" | You have successfully performed an incomplete recovery. Record the change number from the message and proceed to the next step. |
Recover or flash back all other databases in the distributed database system using change-based recovery, specifying the change number (SCN) that you recorded in Step 2.
|
Note: If a database that is participating in distributed transactions fails, in-doubt distributed transactions may exist in the participating databases. If the failed database recovers completely and communications resume between the databases, then the in-doubt transactions is automatically resolved by the Oracle recoverer process (RECO) process. If you cannot wait until the failed database becomes available, you can also manually commit or rollback in-doubt transactions. |
|
See Also:
|
Whenever a component in a high availability architecture fails, the full protection, or fault tolerance, of the architecture is compromised and possible single points of failure exist until the component is repaired. Restoring the high availability architecture to full fault tolerance to reestablish full Oracle RAC, Data Guard, or MAA protection requires repairing the failed component. While full fault tolerance might be sacrificed during planned downtime, the method of repair is well understood because it is planned, the risk is controlled, and it ideally occurs at times best suited for continued application availability. However, for unplanned downtime the risk of exposure to a single point of failure must be clearly understood.
This section provides the following topics that describe the steps needed to restore database fault tolerance:
For Oracle Database 11g with Oracle RAC or Oracle RAC One Node
For Oracle Database 11g with Data Guard and Oracle Database 11g with Oracle RAC and Data Guard - MAA
Ensuring that application services fail over quickly and automatically in an Oracle RAC cluster—or between primary and secondary sites—is important when planning for both scheduled and unscheduled outages. Similarly, using Oracle RAC One Node you must ensure that applications failover to the new instance that starts if an Oracle RAC One Node instance fails. To ensure that the environment is restored to full fault tolerance after any errors or issues are corrected, it is also important to understand the steps and processes for restoring failed instances or nodes within an Oracle RAC cluster or databases between sites.
Adding a failed node back into the cluster or restarting a failed Oracle RAC instance or Oracle RAC One Node instance is easily done after the core problem that caused the specific component to originally fail has been corrected. However, you should also consider:
When to perform these tasks to incur minimal or no effect on the current running environment
Failing back or rebalancing existing connections
After the problem that caused the initial node or instance failure has been corrected, a node or instance can be restarted and added back into the Oracle RAC environment at any time. For an Oracle RAC One Node, you can also restart a failed instance and go back to running the instance on the original node. Processing to complete the reconfiguration of a node may require additional system resources.
Table 13-8 summarizes additional processing that may be required when adding a node.
Table 13-8 Additional Processing When Restarting or Rejoining a Node or Instance
| Action | Additional Resources |
|---|---|
|
Restarting a node or rejoining a node into a cluster |
When using only Oracle Clusterware, there is no impact when a node joins the cluster. When using vendor clusterware, there may be performance degradation while reconfiguration occurs to add a node back into the cluster. The impact on current applications should be evaluated with a full test workload. |
|
Restarting or rejoining of an Oracle RAC instance |
When you restart an Oracle RAC instance, there might be some potential performance impact while lock reconfiguration takes place. The impact on current applications is usually minimal, but it should be evaluated with a full test workload. |
Use the following recovery methods:
|
See Also:
|
|
Note: These descriptions for recovering service availability are for Oracle RAC and do not apply for Oracle RAC One Node systems. |
After a failed node has been brought back into the cluster and its instance has been started, Cluster Ready Services (CRS) automatically manages the virtual IP address used for the node and the services supported by that instance automatically. A particular service might or might not be started for the restored instance. The decision by CRS to start a service on the restored instance depends on how the service is configured and whether the proper number of instances are currently providing access for the service. A service is not relocated back to a preferred instance if the service is still being provided by an available instance to which it was moved by CRS when the initial failure occurred.
CRS restarts services on the restored instance if the number of instances that are providing access to a service across the cluster is less than the number of preferred instances defined for the service. After CRS restarts a service on a restored instance, CRS notifies registered applications of the service change.
For example, suppose the HR service is defined with instances A and B as preferred and instances C and D as available in case of a failure. If instance B fails and CRS starts the HR service on C automatically, then when instance B is restarted, the HR service remains at instance C. CRS does not automatically relocate a service back to a preferred instance.
Suppose a different scenario in which the HR service is defined with instances A, B, C, and D as preferred and no instances defined as available, spreading the service across all nodes in the cluster. If instance B fails, then the HR service remains available on the remaining three nodes. CRS automatically starts the HR service on instance B when it rejoins the cluster because it is running on fewer instances than configured. CRS notifies the applications that the HR service is again available on instance B.
Oracle RAC One Node databases are administered slightly differently from Oracle RAC or single-instance databases. For administrator-managed Oracle RAC One Node databases, you must monitor the candidate node list and make sure a server is always available for failover, if possible. Candidate servers reside in the Generic server pool and the database and its services will fail over to one of those servers.
For policy-managed Oracle RAC One Node databases, you must ensure that the server pools are configured such that a server will be available for the database to fail over to in case its current node becomes unavailable. Also, for policy-managed Oracle RAC One Node databases, the destination node for online database relocation must be located in the database's server pool.
|
See Also: Oracle Real Application Clusters Administration and Deployment Guide for information about Administering Oracle RAC One Node |
After an Oracle RAC instance has been restored, additional steps might be required, depending on the current resource usage and system performance, the application configuration, and the network load balancing that has been implemented.
Existing connections, that might have failed over or started as a new session, on the surviving Oracle RAC instances are not automatically redistributed or failed back to an instance that has been restarted. Failing back or redistributing users might or might not be necessary, depending on the current resource utilization and the capability of the surviving instances to adequately handle and provide acceptable response times for the workload. If the surviving Oracle RAC instances do not have adequate resources to run a full workload or to provide acceptable response times, then it might be necessary to move (disconnect and reconnect) some existing user connections to the restarted instance.
|
Note: In Oracle RAC One Node there is only one instance for a database (unless you are migrating). Thus an Oracle RAC One Node configuration does not require you to rethink the strategy for 'rebalancing' the connections as there is only one. Clients using Oracle RAC One Node must be able to work with FAN and other client and service facilities to be informed about the status of services. |
Connections are started as they are needed, on the least-used node, assuming connection load balancing has been configured. Therefore, the connections are automatically load-balanced over time.
An application service can be:
Managed with services running on a subset of Oracle RAC instances
Nonpartitioned so that all services run equally across all nodes
This is valuable for modularizing application and database form and function while still maintaining a consolidated data set. For cases where an application is partitioned or has a combination of partitioning and nonpartitioning, you should consider the response time and availability aspects for each service.
If redistribution or failback of connections for a particular service is required, then you can rebalance workloads automatically using Oracle Universal Connection Pool (UCP). If you are using UCP, then connections are automatically redistributed to the new node.
|
Note: Oracle Universal Connection Pool (UCP) provides fast and automatic detection of connection failures and removes terminated connections for any Java application using, Fast Connection Failover, and FAN events |
For load-balancing application services across multiple Oracle RAC instances, Oracle Net connect-time failover and connection load balancing are recommended. This feature does not require changes or modifications for failover or restoration. It is also possible to use hardware-based load balancers. However, there might be limitations in distinguishing separate application services (which is understood by Oracle Net Services) and restoring an instance or a node. For example, when a node or instance is restored and available to start receiving connections, a manual step might be required to include the restored node or instance in the hardware-based load balancer logic, whereas Oracle Net Services does not require manual reconfiguration.
Table 13-9 summarizes the considerations for new and existing connections after an instance has been restored. The considerations differ depending on whether the application services are partitioned, nonpartitioned, or are a combination of both. The actual redistribution of existing connections might or might not be required depending on the resource utilization and response times.
Table 13-9 Restoration and Connection Failback
| Application Services | Failback or Restore Existing Connections | Failback or Restore New Connections |
|---|---|---|
|
Partitioned |
Existing sessions are not automatically relocated back to the restored instance. Use the |
Automatically routes to the restored instance by using the Oracle Net Services configuration. |
|
Nonpartitioned |
No action is necessary unless the load must be rebalanced, because restoring the instance means that the load there is low. If the load must be rebalanced, then the same problems are encountered as if application services were partitioned. |
Automatically routes to the restored instance (because its load should be lowest) by using the Oracle Net Services configuration |
Figure 13-6 shows a two-node partitioned Oracle RAC database. Each instance services a different portion of the application (HR and Sales). Client processes connect to the appropriate instance based on the service they require.
Figure 13-6 Partitioned Two-Node Oracle RAC Database
Figure 13-7 shows what happens when one Oracle RAC instance fails.
Figure 13-7 Oracle RAC Instance Failover in a Partitioned Database
If one Oracle RAC instance fails, then the service and existing client connections can be automatically failed over to another Oracle RAC instance. In this example, the HR and Sales services are both supported by the remaining Oracle RAC instance. In addition, you can route new client connections for the Sales service to the instance now supporting this service.
After the failed instance has been repaired and restored to the state shown in Figure 13-6 and the Sales service is relocated to the restored instance, then you might need to identify and failback any failed-over clients and any new clients that had connected to the Sales service on the failed-over instance. Client connections that started after the instance has been restored should automatically connect back to the original instance. Therefore, over time, as older connections disconnect, and new sessions connect to the Sales service, the client load migrates back to the restored instance. Rebalancing the load immediately after restoration depends on the resource utilization and application response times.
Figure 13-8 shows a nonpartitioned application. Services are evenly distributed across both active instances. Each instance has a mix of client connections for both HR and Sales.
Figure 13-8 Nonpartitioned Oracle RAC Instances
If one Oracle RAC instance fails, then Oracle Clusterware moves the services that were running on the failed instance. If one Oracle RAC instance fails, new client connections are only accepted on the remaining instances that offers that service.
After the failed instance has been repaired and restored to the state shown in Figure 13-8, some clients might have to be moved back to the restored instance. For nonpartitioned applications, identifying appropriate services is not required for rebalancing the client load among all available instances. Also, this is necessary only if a single-instance database is not able to adequately service the requests.
Client connections that started after the instance has been restored should automatically connect back to the restored instance because it has a smaller load. Therefore, over time, as older connections disconnect and new sessions connect to the restored instance, the client load evenly balances again across all available Oracle RAC instances. Rebalancing the load immediately after restoration depends on the resource usage and application response times.
Following unplanned downtime on a primary database that requires a failover, full fault tolerance is compromised until the standby database is reestablished. Full database protection should be restored as soon as possible. The steps for restoring fault tolerance differ slightly between physical and logical standby databases.
Reinstating databases is automated if you are using Data Guard fast-start failover. After a fast-start failover completes, the observer automatically attempts to reinstate the original primary database as a standby database. Reinstatement restores high availability to the broker configuration so that, if the new primary database fails, another fast-start failover can occur. The reinstated database can act as the fast-start failover target for the primary database, making a subsequent fast-start failover possible. The standby database is a viable target of a failover when it begins applying redo data received from the new primary database. If you want to prevent automatic reinstatement (for example, to perform diagnostic or repair work after failover has completed), set the FastStartFailoverAutoReinstate configuration property to FALSE.
The FastStartFailoverAutoReinstate configuration property controls whether the observer should automatically reinstate the original primary after a fast-start failover occurred because a fast-start failover was initiated due to the primary database being isolated for longer than the number of seconds specified by the FastStartFailoverThreshold property. In some cases, an automatic reinstatement might not be wanted until further diagnostic or recovery work is done.
To reinstate the original primary database, the database must be started and mounted, but it cannot be opened. The broker reinstates the database as a standby database of the same type (physical or logical) as the original standby database.
If the original primary database cannot be reinstated automatically, you can manually reinstate it using either the DGMGRL REINSTATE command or Enterprise Manager. Step-by-step instructions for manual reinstatement are described in Oracle Data Guard Broker.
Standby databases do not have to be re-created if you use the Oracle Flashback Database feature. Flashback Database has the following advantages:
Saves hours of database restoration time
Reduces overall complexity in restoring fault tolerance
Reduces the time that the system is vulnerable because the standby database is re-created more quickly
|
See Also:
|
This section includes the following topics:
Reinstating the Original Primary Database After a Fast-Start Failover
Reinstating a Standby Database Using Enterprise Manager After a Failover
Following a fast-start failover, the observer periodically attempts to reconnect to the original primary database. When the observer regains network access to the original primary database, it initiates a request for the broker to automatically reinstate it as a standby database to the new primary. This quickly restores disaster protection and high availability for the configuration.
You can enable fast-start failover from any site, including the observer site, in Enterprise Manager while connected to any database in the broker configuration. The broker simplifies switchovers and failovers by allowing you to invoke them using a single key click in Oracle Enterprise Manager, as shown in Figure 13-9.
Figure 13-9 Fast-Start Failover and the Observer Are Successfully Enabled
Furthermore, you can use Enterprise Manager to reinstate the original primary as the new standby. Figure 13-10 shows an example of the warning message that shows in Enterprise Manager when a reinstatement is needed.
Figure 13-10 Reinstating the Original Primary Database After a Fast-Start Failover
Follow the steps in Section 13.2.5.3, "Data Area Disk Group Failure" or Section 13.2.5.4, "Fast Recovery Area Disk Group Failure".
After performing the planned maintenance on the secondary site, the standby database and log apply services must be restarted, and then the Data Guard redo transport services automatically catch up the standby database with the primary database. You can use Enterprise Manager and the broker to monitor the Data Guard state.
The following steps are required to restore full fault tolerance after planned downtime on a secondary site or clusterwide outage:
|
Note: The following steps can be accomplished manually (as described below) or automatically using Enterprise Manager. |
Start the standby database
You might have to restore the standby database from local backups, local tape backups, or from the primary site backups if the data in the secondary site has been damaged. Re-create the standby database from the new primary database by following the steps for creating a standby database in Oracle Data Guard Concepts and Administration.
After the standby database has been reestablished, start the standby database.
Start Redo Apply (physical standby) or SQL Apply (logical standby):
Verify redo transport services on the primary database
You might have to reenable the primary database remote archive destination. Query the V$ARCHIVE_DEST_STATUS view first to see the current state of the archive destinations:
SELECT DEST_ID, DEST_NAME, STATUS, PROTECTION_MODE, DESTINATION, ERROR, SRL
FROM V$ARCHIVE_DEST_STATUS;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_n=ENABLE;
ALTER SYSTEM ARCHIVE LOG CURRENT;
Verify redo transport services between the primary and standby databases by checking for errors. Query the V$ARCHIVE_DEST and V$ARCHIVE_DEST_STATUS views:
SELECT STATUS, TARGET, LOG_SEQUENCE, TYPE, PROCESS, REGISTER, ERROR
FROM V$ARCHIVE_DEST;
SELECT * FROM V$ARCHIVE_DEST_STATUS WHERE STATUS!='INACTIVE';
Verify that recovery is progressing on standby database
For a physical standby database, verify that there are no errors from the managed recovery process and that the recovery has applied the redo from the archived redo log files:
SELECT MAX(SEQUENCE#), THREAD# FROM V$LOG_HISTORY GROUP BY THREAD;
SELECT PROCESS, STATUS, THREAD#, SEQUENCE#, CLIENT_PROCESS
FROM V$MANAGED_STANDBY;
For a logical standby database, verify that there are no errors from the logical standby process and that the recovery has applied the redo from the archived redo logs:
SELECT THREAD#, SEQUENCE# SEQ#
FROM DBA_LOGSTDBY_LOG LOG, DBA_LOGSTDBY_PROGRESS PROG
WHERE PROG.APPLIED_SCN BETWEEN LOG.FIRST_CHANGE# AND LOG.NEXT_CHANGE#
ORDER BY NEXT_CHANGE#;
Restore primary database protection mode
If you had to change the protection mode of the primary database from maximum protection to either maximum availability or maximum performance because of the standby database outage, then change the primary database protection mode back to maximum protection depending on your business requirements.
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE [PROTECTION | AVAILABILITY];
Following unplanned downtime on the standby database that requires a full or partial data file restoration (such as data or media failure), full fault tolerance is compromised until the standby database is brought back into service. Full database protection should be restored as soon as possible.
To repair data corruption and data failures on a logical standby database, you require a backup of the logical standby file and not a backup from the primary database. Otherwise, you must reinstate or re-create the relevant objects affected by the corruption.To repair data corruption or data failures on the standby database, you can use the following repair solutions:
Use RMAN and Block Media Recovery (described in Section 13.2.6.3, "Use RMAN and Block Media Recovery")
Use RMAN and Data File Media Recovery (described in Section 13.2.6.5, "Use RMAN and Data File Media Recovery")
Re-Create Objects Manually for logical standby databases only (described in Section 14.2.6.7, "Re-Create Objects Manually")
If you had to change the protection mode of the primary database from maximum protection to either maximum availability or maximum performance because of the standby database outage, then change the primary database protection mode back to maximum protection (depending on your business requirements).
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE [PROTECTION | AVAILABILITY];
If the primary database is activated because it was flashed back to correct a logical error or because it was restored and recovered to a point in time, then the corresponding standby database might require additional maintenance. No additional work is required if the primary database completed recovery with no resetlogs.
After opening the primary database with the RESETLOGS option, execute the queries shown in Table 13-12.
Table 13-12 Queries to Determine RESETLOGS SCN and Current SCN OPEN RESETLOGS
| Database | Query |
|---|---|
|
Primary |
|
|
Physical standby |
|
|
Logical standby |
|
Table 13-13 shows the actions you take to restore fault tolerance if the standby database is behind the primary database's resetlogs SCN.
Table 13-13 SCN on Standby Database is Behind RESETLOGS SCN on the Primary Database
| Database | Action |
|---|---|
|
Physical standby |
|
|
Logical standby |
Ensure that the standby database has received an archived redo log file from the primary database. See Also: "Verify redo transport services on the primary database" |
Table 13-14 shows the actions you take to restore fault tolerance if the standby database is ahead of the primary database's resetlogs SCN.
Table 13-14 SCN on the Standby is Ahead of Resetlogs SCN on the Primary Database
| Database | Action |
|---|---|
|
Physical standby |
|
|
Logical standby |
|
If a dual failure affecting both the standby and primary databases occurs, then you must re-create the primary database first. Because the sites are identical, the primary database can be created wherever the most recent backup resides.
Table 13-15 summarizes the recovery strategy depending on the type of backups that are available.
Table 13-15 Re-Creating the Primary and Standby Databases
| Available Backups | Re-Creating the Primary Database |
|---|---|
|
Local backup on primary and standby databases |
Restore backup from the primary database. Recover and activate the database as the new primary database. |
|
Local backup only on standby database. Tape backups on standby database. |
Restore the local standby backup to the standby database. Recover and activate the database as the new primary database. |
|
Tape backups only |
Restore tape backups locally. Recover the database and activate it as the new primary database. |
|
See Also: Oracle Data Guard Concepts and Administration for the steps for creating a standby database after the primary database is re-created |
Oracle Active Data Guard option
A physical standby database can be open for read-only access while Redo Apply is active if a license for the Oracle Active Data Guard option has been purchased. This capability, known as real-time query, also provides the ability to have block-change tracking on the standby database, thus allowing incremental backups to be performed on the standby.
clusterwide failure
The whole cluster hosting the Oracle RAC database is unavailable or fails. This includes failures of nodes in the cluster, and any other components that result in the cluster being unavailable and the Oracle database and instances on the site being unavailable.
computer failure
An outage that occurs when the system running the database becomes unavailable because it has crashed or is no longer accessible.
data corruption
A corrupt block is a block that has been changed so that it differs from what Oracle Database expects to find. Block corruptions fall under two categories: physical and logical block corruptions.
hang or slow down
Hang or slow down occurs when the database or the application cannot process transactions because of a resource or lock contention. Perceived hang can be caused by lack of system resources.
human error
An outage that occurs when unintentional or malicious actions are committed that cause data in the database to become logically corrupt or unusable. The service level impact of a human error outage can vary significantly depending on the amount and critical nature of the affected data.
Three-bit identifiers used on a SCSI bus to distinguish between up to eight devices (logical units) with the same SCSI ID.
lost write
A lost write is another form of data corruption that can occur when an I/O subsystem acknowledges the completion of the block write, while in fact the write did not occur in the persistent storage. No error is reported by the I/O subsystem back to Oracle.
The Maximum Availability architecture provides the most comprehensive set of solutions for both unplanned and because it inherits the capabilities and advantages of both Oracle Database 11g with Oracle RAC and Oracle Database 11g with Data Guard.
MAA involves high availability best practices for all Oracle products across the entire technology stack—Oracle Database, Oracle WebLogic Server, Oracle Applications, Oracle Collaboration Suite, and Enterprise Manager.
The Data Guard network server processes, also referred to as LNSn processes, on the primary database perform a network send to the RFS process on the standby database. There is one network server process for each destination.
real-time query
If a license for the Oracle Active Data Guard option has been purchased, you can open a physical standby database while Redo Apply continues to apply redo data received from the primary database.
recovery point objective (RPO)
The maximum amount of data an IT-based business process may lose before causing harm to the organization. RPO indicates the data-loss tolerance of a business process or an organization in general. This data loss is often measured in terms of time, for example, five hours or two days worth of data loss.
The maximum amount of time that an IT-based business process can be down before the organization suffers significant material losses. RTO indicates the downtime tolerance of a business process or an organization in general.
site failure
An outage that occurs when an event causes all or a significant portion of an application to stop processing or slow to an unusable service level. A site failure may affect all processing at a data center, or a subset of applications supported by a data center.
snapshot standby database
An updatable standby database that you create from a physical standby database. A snapshot standby database receives and archives redo data received from the primary database, but the snapshot standby database does not apply redo data from the primary database while the standby database is open for read/write I/O. Thus, the snapshot standby typically diverges from the primary database over time. Moreover, local updates to the snapshot standby database cause additional divergence. However, a snapshot standby protects the primary database because the snapshot standby can be converted back into a physical standby database.
storage failure
An outage that occurs when the storage holding some or all of the database contents becomes unavailable because it has shut down or is no longer accessible.
The proper configuration of Oracle Data Guard is essential to ensuring that all standby databases work properly and perform their roles within the necessary service levels after switchovers and failovers.
The best practices for Oracle Data Guard build on the best practices described in Chapter 5, "Configuring Oracle Database."
This chapter contains the following topics:
Data Guard is the Oracle optimized solution for Data availability and protection. It excels at simple, fast, and reliable one-way replication of a complete Oracle Database to provide High Availability and Disaster Recovery. Data Guard offers various deployment options that address unplanned outages, pre-production testing, and planned maintenance. Active Data Guard, an extension of basic Data Guard capabilities, further enables production offload of read-only workload to a synchronized physical standby database, automatic repair of corrupt blocks, and offload of fast incremental backups.
The focus of Data Guard is High Availability and Data Recovery. Data Guard design principles are simplicity, high performance, and application transparency.
Data Guard is not intended to be a full-featured replication solution. Oracle GoldenGate is the solution recommended for advanced replication requirements, such as multi-master replication, granular replication of a subset of a database, many to one replication topologies, and data integration. Oracle GoldenGate also provides additional options for reducing downtime for planned maintenance and for heterogeneous platform migrations.
Depending upon your requirements, the most efficient solution to use may be using Data Guard alone, using Data Guard with Oracle GoldenGate in a complementary manner, or just using Oracle GoldenGate.
For more information about Data Guard and Oracle GoldenGate see the Product Technical Brief on Oracle Active Data Guard and Oracle GoldenGate at
http://www.oracle.com/technetwork/middleware/goldengate/overview/index.html
Table 9-1 provides a summary of the Data Guard deployment options that are appropriate, depending on your requirements. Two or more options may be used in combination to address multiple requirements. This chapter also presents the Best practices for implementing each option.
Table 9-1 Requirements and Data Guard Deployment Options
| Requirement | Data Guard Deployment Options |
|---|---|
|
Zero data loss protection and availability for Oracle Database |
Data Guard Maximum Protection or Maximum Availability (SYNC transport) and Redo Apply (physical standby) |
|
Near-zero data loss (single-digit seconds) and availability for Oracle Database |
Data Guard Maximum Performance (ASYNC transport) and Redo Apply |
|
Multi-site protection, including topology with local zero data loss standby for HA and remote asynchronous standby for geographic disaster recovery for Oracle Database |
Multi-standby Data Guard configuration and Redo Apply |
|
Fastest possible database failover |
Data Guard Fast-Start Failover with Oracle Data Guard broker for automatic failure detection and database failover. Automatic failover of accompanying client applications to the new production database is implemented using Oracle Fast Application Notification (FAN) and Oracle Client Failover Best Practices. For more information, see the MAA white paper "Client Failover Best Practices for Data Guard 11g Release 2" from the MAA Best Practices area for Oracle Database at |
|
Offload read-only queries and fast incremental backups to a synchronized standby database. Use the standby database to automatically repair corrupt blocks, transparent to the application and user |
Active Data Guard. Active Data Guard can be purchased in either of the following ways: (1) standalone as an option license for Oracle Database Enterprise Edition, or (2) included with an Oracle GoldenGate license. |
|
Pre-production testing |
Snapshot Standby. A snapshot standby is a physical standby database that is temporarily open read/write for test and other read/write activity independent of primary database transactions. A snapshot standby is easily converted back into a synchronized standby database when testing is complete. Snapshot Standby is an included feature of Data Guard Redo Apply and is an ideal complement for Oracle Real Application Testing. |
|
Planned maintenance: certain platform migrations such as Windows to Linux, data center moves, patching and upgrading system software or Oracle Database |
Data Guard switchover, planned role transition, using Redo Apply. Redo Apply and Standby-First Patch Apply for qualifying patches from 11.2.0.1 onward. SQL Apply and Data Guard Database Rolling Upgrades (10.1 onward). Data Guard Transient Logical Standby (Upgrades Made Easy) from 11.1.0.7 onward. For more information, see the MAA white paper, "Database Rolling Upgrades Made Easy by Using a Data Guard Physical Standby Database", from the MAA Best Practices area for Oracle Database at |
|
Data Protection for data residing outside of the Oracle Database |
When practical, move operating system file system data into Oracle Database using Oracle Database File System (DBFS). Data Guard protects DBFS data in the same manner as any other Oracle data. Data that must remain in operating system files can be protected using Oracle ASM Cluster File System (Oracle ACFS) or storage mirroring, and Data Guard. |
|
Note: Standby-First Patch allows you to apply a patch initially to a physical standby database while the primary database remains at the previous software release (this applies for certain types of patches and does not apply for Oracle patch sets and major release upgrades; use the Data Guard transient logical standby method for patch sets and major releases). Once you are satisfied with the change, then you perform a switchover to the standby database. The fallback is to switchback if required. For more information, see "Oracle Patch Assurance - Data Guard Standby-First Patch Apply" in My Oracle Support Note 1265700.1 at
|
|
See Also:
|
Oracle Data Guard Zero Data Loss protection provides both a guarantee of that data is protected and the simplest recovery. For these reasons a Zero Data Loss protection mode, either Oracle Data Guard Maximum Protection or Maximum Availability, is recommended. While both modes use Oracle Data Guard synchronous redo transport by default, there are differences in the rule-sets used to govern behavior at failover time that must be considered, as described below. Oracle Data Guard synchronous redo transport, however, can impact primary database performance if round-trip network latency between primary and standby databases is too great (latency is a function of distance and how 'clean' the network is). If this is the case (testing is easy to do, a DBA may change protection modes and transport methods dynamically), then use Oracle Data Guard Maximum Performance. Maximum Performance uses Oracle Data Guard asynchronous transport services and does not have any impact on primary database performance regardless of network latency. In an environment with sufficient bandwidth to accommodate redo volume, data loss potential is measured in single-digit seconds when using Maximum Performance.
To determine the appropriate data protection mode for your application, consult Oracle Data Guard Concepts and Administration.
Best practices for the protection mode:
Maximum Protection mode guarantees that no data loss will occur if the primary database fails, even in the case of multiple failures (for example, the network between the primary and standby fails, and then at a later time, the primary fails). This is enforced by never signaling commit success for a primary database transaction until at least one synchronous Data Guard standby has acknowledged that redo has been hardened to disk. Without such an acknowledgment the primary database will stall and eventually shut down rather than allow unprotected transactions to commit. To maintain availability in cases where the primary database is operational but the standby database is not, the best practice is to always have a minimum of two synchronous standby databases in a Maximum Protection configuration. Primary database availability is not impacted if it receives acknowledgment from at least one synchronous standby database.
Maximum Availability mode guarantees that no data loss will occur in cases where the primary database experiences the first failure to impact the configuration. Unlike the previous protection mode, Maximum Availability will wait a maximum of NET_TIMEOUT seconds for an acknowledgment from a standby database, after which it will signal commit success to the application and move to the next transaction. Primary database availability (thus the name of the protection mode) is not impacted by an inability to communicate with the standby (for example, due to standby or network outages). Oracle Data Guard will continue to ping the standby and automatically re-establish connection and resynchronize the standby database when possible, but during the period when primary and standby have diverged there will be data loss should a second failure impact the primary database. For this reason, it is a best practice to monitor protection level (simple to do using Enterprise Manager Grid Control) and quickly resolve any disruption in communication between primary and standby before a second failure can occur.
Maximum Performance mode (the default mode) provides the highest level of data protection that is possible without affecting the performance or the availability of the primary database. This is accomplished by allowing a transaction to commit as soon as the redo data needed to recover that transaction is written to the local online redo log at the primary database (the same behavior as if there were no standby database). Oracle Data Guard transmits redo to the standby database directly from the primary log buffer asynchronous to the local online redo log write. There is never any wait for standby acknowledgment. Similar to Maximum Availability, it is a best practice to monitor protection level (simple to do using Enterprise Manager Grid Control) and quickly resolve any disruption in communication between primary and standby before a second failure can occur.
|
See Also: Oracle Data Guard Concepts and Administration for information about Data Guard Protection Modes |
At a high level, the Redo Transport best practices for planning and implementing redo transport services for Oracle Data Guard are as follows:
Use the SYNC redo transport mode for a high degree of synchronization between the primary and standby databases. Use SYNC redo transport for zero data loss protection where performance service levels can tolerate the impact caused by network latency.
Use the ASYNC redo transport mode for minimal impact on the primary database, but with a lower degree of synchronization. Use ASYNC redo transport when zero data loss protection is not required or when the performance impact caused by network latency makes it impractical to use SYNC.
Optimize network throughput following the best practices described in Section 9.2.2, "Assess Performance with Proposed Network Configuration".
Oracle recommends that you conduct a performance assessment with your proposed network configuration and current, or anticipated, peak redo rate. The network effect between the primary and standby databases, and the effect on the primary database throughput must be understood. Because the network between the primary and standby databases is essential for the two databases to remain synchronized, the infrastructure must have the following characteristics:
Sufficient bandwidth to accommodate the maximum redo generation rate
If using the SYNC transport, then minimal latency is necessary to reduce the performance impact on the primary database
Multiple network paths for network redundancy
In configurations that use a dedicated network connection the required bandwidth is determined by the maximum redo rate of the primary database and the efficiency of the network. Depending on the data protection mode, there are other recommended practices and performance considerations. Maximum protection mode and maximum availability mode require SYNC transport.
The maximum performance protection mode use ASYNC redo transport. Use ASYNC redo transport when data loss can be tolerated or when the performance impact caused by network latency makes it impractical to use SYNC (use SYNC redo transport for zero data loss protection).
Unlike the ASYNC transport mode, the SYNC transport mode can affect the primary database performance due to the incurred network latency. Distance and network configuration directly influence latency, while high latency can slow the potential transaction throughput and quicken response time. The network configuration, number of repeaters, the overhead of protocol conversions, and the number of routers also affect the overall network latency and transaction response time.
Use the following configuration best practices for Data Guard:
Use Oracle Data Guard broker to create, manage, and monitor a Data Guard configuration. You can perform all Data Guard management operations locally or remotely through the Oracle Data Guard broker's easy-to-use interfaces: the Data Guard management pages in Oracle Enterprise Manager, which is the broker's graphical user interface (GUI), and the Data Guard command-line interface called DGMGRL.
The broker's interfaces improve usability and centralize management and monitoring of the Data Guard configuration. Available as a feature of the Enterprise Edition and Personal Edition of the Oracle database, the broker is also integrated with the Oracle database and Oracle Enterprise Manager.
The benefits of using Oracle Data Guard broker include:
Enhanced disaster protection.
Higher availability and scalability with Oracle Real Application Clusters (Oracle RAC) Databases.
Automated creation of a Data Guard configuration.
Easy configuration of additional standby databases.
Simplified, centralized, and extended management.
Simplified switchover and failover operations.
Built-in monitoring and alert and control mechanisms.
Transparent to application.
|
See Also: Oracle Data Guard Broker for more information about the benefits of using Data Guard Broker |
Oracle recommends that you use the Recovery Manager (RMAN) utility to simplify the process of creating a physical standby database.
You can either create a standby database from backups of your primary database, or create a standby database over the network:
Use the RMAN DUPLICATE TARGET DATABASE FOR STANDBY command to create a standby database from backups of your primary database.
You can use any backup copy of the primary database to create the physical standby database if the necessary archived redo log files to completely recover the database are accessible by the server session on the standby host. RMAN restores the most recent data files unless you execute the SET UNTIL command.
Use the RMAN FROM ACTIVE DATABASE option to create the standby database over the network if a preexisting database backup is not accessible to the standby system.
RMAN copies the data files directly from the primary database to the standby database. The primary database must be mounted or open.
You must choose between active and backup-based duplication. If you do not specify the FROM ACTIVE DATABASE option, then RMAN performs backup-based duplication. Creating a standby database over the network is advantageous because:
You can transfer redo data directly to the remote host over the network without first having to go through the steps of performing a backup on the primary database. (Restoration requires multiple steps including storing the backup locally on the primary database, transferring the backup over the network, storing the backup locally on the standby database, and then restoring the backup on the standby database.)
With active duplication you can backup a database (as it is running) from Oracle ASM, and restore the backup to a host over the network and place the files directly into Oracle ASM.
Before this feature, restoration required you to backup the primary and copy the backup files on the primary host file system, transfer the backup files over the network, place the backup files on the standby host file system, and then restore the files into Oracle ASM.
|
See Also:
|
Enable Flashback Database on both the primary and standby database so that, in case the original primary database has not been damaged, you can reinstate the original primary database as a new standby database following a failover. If there is a failure during the switchover process, then it can easily be reversed when Flashback Database is enabled. For more information, see Section 5.1.4, "Enable Flashback Database".
When the primary database is in FORCE LOGGING mode, all database data changes are logged. FORCE LOGGING mode ensures that the standby database remains consistent with the primary database. If this is not possible because you require the load performance with NOLOGGING operations, then you must ensure that the corresponding physical standby data files are subsequently synchronized. To synchronize the physical standby data files, either apply an incremental backup created from the primary database or replace the affected standby data files with a backup of the primary data files taken after the nologging operation. Before the file transfer, you must stop Redo Apply on the physical standby database.
You can enable force logging immediately by issuing an ALTER DATABASE FORCE LOGGING statement. If you specify FORCE LOGGING, then Oracle waits for all ongoing unlogged operations to finish.
|
See Also:
|
This archiving strategy is based on the following assumptions:
Each database uses a fast recovery area.
The primary database instances archive remotely to only one apply instance.
Table 9-2 describes the recommendations for a robust archiving strategy when managing a Data Guard configuration through SQL*Plus. All of the following items are handled automatically when Oracle Data Guard broker is managing a configuration.
Table 9-2 Archiving Recommendations
The following example illustrates the recommended initialization parameters for a primary database communicating to a physical standby database. There are two instances, SALES1 and SALES2, running in maximum protection mode.
*.DB_RECOVERY_FILE_DEST=+RECO
*.LOG_ARCHIVE_DEST_1='SERVICE=SALES_stby SYNC AFFIRM NET_TIMEOUT=30
REOPEN=300 VALID_FOR=(ONLINE_LOGFILES, ALL_ROLES) DB_UNIQUE_NAME=SALES_stby'
*.LOG_ARCHIVE_DEST_STATE_1=ENABLE
The fast recovery area must be accessible to any node within the cluster and use a shared file system technology such as automatic storage management (Oracle ASM), a cluster file system, a global file system, or high availability network file system (HA NFS). You can also mount the file system manually to any node within the cluster very quickly. This is necessary for recovery because all archived redo log files must be accessible on all nodes.
On the standby database nodes, recovery from a different node is required when a failure occurs on the node applying redo and the apply service cannot be restarted. In that case, any of the existing standby instances residing on a different node can initiate managed recovery. In the worst case, when the standby archived redo log files are inaccessible, the managed recovery process (MRP) on the different node fetches the archived redo log files using the FAL server to retrieve from the primary node directly.
When configuring hardware vendor shared file system technology, verify the performance and availability implications. Investigate the following issues before adopting this strategy:
Is the shared file system accessible by any node regardless of the number of node failures?
What is the performance impact when implementing a shared file system?
Is there any effect on the interconnect traffic?
You should configure standby redo logs on both sites for improved availability and performance. To determine the recommended number of standby redo logs, use the following formula:
(maximum # of logfile groups +1) * maximum # of threads
For example, if a primary database has two instances (threads) and each thread has three online log groups, then there should be eight standby redo logs ((3 + 1) * 2 = 8), this reduces the likelihood that the logwriter process for the primary instance is blocked because a standby redo log cannot be allocated on the standby database.
The statements in Example 9-1 create two standby log members for each group and each member is 1 GB. One member is created in the directory specified by the DB_CREATE_FILE_DEST initialization parameter, and the other member is created in the directory specified by DB_RECOVERY_FILE_DEST initialization parameter. Because this example assumes that there are three online redo log groups in two threads, the next group is group seven.
Example 9-1 Create Standby Log Members
SQL> ALTER DATABASE ADD STANDBY LOGFILE THREAD 1
GROUP 7 ('+RECO') SIZE 1G,
GROUP 8 ('+RECO') SIZE 1G,
GROUP 9 ('+RECO') SIZE 1G,
Group 10 ('+RECO') SIZE 1G;
SQL> ALTER DATABASE ADD STANDBY LOGFILE THREAD 2
GROUP 11 ('+RECO') SIZE 1G,
GROUP 12 ('+RECO') SIZE 1G,
GROUP 13 ('+RECO') SIZE 1G,
GROUP 14 ('+RECO') SIZE 1G;
Consider the following additional guidelines when creating standby redo logs:
Create the same number of standby redo logs on both the primary and standby databases.
Create all online redo logs and standby redo logs for both primary and standby databases so that they are the same size.
Create standby redo logs in the first available ASM high redundancy disk group, or ensure that the logs are protected using external storage redundancy.
In an Oracle RAC environment, create standby redo logs on a shared disk.
In an Oracle RAC environment, assign a thread when the standby redo log is created as described in Example 9-1.
Do not multiplex the standby redo logs.
To check the number and group numbers of the redo logs, query the V$LOG view:
SQL> SELECT * FROM V$LOG;
To check the results of the ALTER DATABASE ADD STANDBY LOGFILE THREAD statements, query the V$STANDBY_LOG view:
SQL> SELECT * FROM V$STANDBY_LOG;
You can also see the members created by querying the V$LOGFILE view:
SQL> SELECT * FROM V$LOGFILE;
|
See Also: Oracle Data Guard Concepts and Administration for information about Configuring an Oracle Database to Receive Redo Data |
The best practices for Data Guard transport and network configuration include:
In most cases the default for LOG_ARCHIVE_MAX_PROCESSES is sufficient. However, in a Data Guard configurations that have multiple standby databases it may be necessary to increase the number of archive processes. The value of the LOG_ARCHIVE_MAX_PROCESSES initialization parameter must be at least one greater than the total number of all remote destinations. Use the following equation when setting the LOG_ARCHIVE_MAX_PROCESSES parameter for highly available environments:
LOG_ARCHIVE_MAX_PROCESSES = sum(remote_destinations) + count(threads)
You can adjust these parameter settings after evaluating and testing the initial settings in your production environment.
|
See Also: Oracle Database Administrator's Guide for more information about Adjusting the Number of Archiver Processes |
To set the network configuration and highest network redo rates:
Properly Configure TCP Send / Receive Buffer Sizes
To achieve high network throughput, especially for a high-latency, high-bandwidth network, the minimum recommended setting for the sizes of the TCP send and receive socket buffers is the bandwidth-delay product (BDP) of the network link between the primary and standby systems. Settings higher than the BDP may show incremental improvement. For example, in the MAA Linux test lab, simulated high-latency, high-bandwidth networks realized small, incremental increases in throughput when using TCP send and receive socket buffer settings up to three times the BDP.
BDP is product of the network bandwidth and latency. Socket buffer sizes are set using the Oracle Net parameters RECV_BUF_SIZE and SEND_BUF_SIZE, so that the socket buffer size setting affects only Oracle TCP connections. The operating system may impose limits on the socket buffer size that must be adjusted so Oracle can use larger values. For example, on Linux, the parameters net.core.rmem_max and net.core.wmem_max limit the socket buffer size and must be set larger than RECV_BUF_SIZE and SEND_BUF_SIZE.
Set the send and receive buffer sizes at either the value you calculated or 10 MB (10,485,760 bytes), whichever is larger. For example, if bandwidth is 622 Mbits and latency is 30 ms, then you would calculate the minimum size for the RECV_BUF_SIZE and SEND_BUF_SIZE parameters as follows: 622,000,000 / 8 x 0.030 = 2,332,500 bytes. Then, multiply the BDP 2,332,500 x 3 for a total of 6,997,500.
In this example, you would set the initialization parameters as follows:
RECV_BUF_SIZE=10485760
SEND_BUF_SIZE=10485760
Increase SDU Size
With Oracle Net Services it is possible to control data transfer by adjusting the size of the Oracle Net setting for the session data unit (SDU). Oracle internal testing has shown that setting the SDU to its maximum value of 65535 can improve performance for the SYNC transport. You can set SDU on a per connection basis using the SDU parameter in the local naming configuration file (TNSNAMES.ORA) and the listener configuration file (LISTENER.ORA), or you can set the SDU for all Oracle Net connections with the profile parameter DEFAULT_SDU_SIZE in the SQLNET.ORA file.
Note that the ASYNC transport uses the new streaming protocol and increasing the SDU size from the default has no performance benefit.
|
See Also: Oracle Database Net Services Reference for more information about theSDU and DEFAULT_SDU_SIZE parameters |
Set TCP.NODELAY to YES
To preempt delays in buffer flushing in the TCP protocol stack, disable the TCP Nagle algorithm by setting TCP.NODELAY to YES in the SQLNET.ORA file on both the primary and standby systems.
|
See Also: Oracle Database Net Services Reference for more information about theTCP.NODELAY parameter |
Determine When to Use Redo Transport Compression
In Oracle Database 11g Release 2 (11.2.0.2) redo transport compression is no longer limited to compressing redo data only when a redo gap is being resolved. When compression is enabled for a destination, all redo data sent to that destination is compressed.
In general, compression is most beneficial when used over low bandwidth networks. As the network bandwidth increases, the benefit is reduced. Compressing redo in a Data Guard environment is beneficial if:
Sufficient CPU resources are available for the compression processing.
The database redo rate is being throttled by a low bandwidth network.
Before enabling compression, assess the available CPU resources and decide if enabling compression is feasible. For complete information about enabling compression, see "Redo Transport Compression in a Data Guard Environment" in My Oracle Support Note 729551.1 at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=729551.1
To improve the Redo Apply rate of a physical standby database (and media recovery):
Maximize I/O Rates on Standby Redo Logs and Archived Redo Logs
Set DB_BLOCK_CHECKSUM=FULL and DB_BLOCK_CHECKING=MEDIUM or FULL
Set DB_CACHE_SIZE to a Value Greater than on the Primary Database
|
See Also: The MAA white paper "Active Data Guard 11g Best Practices (includes best practices for Redo Apply)" from the MAA Best Practices area for Oracle Database at |
Measure read I/O rates on the standby redo logs and archived redo log directories. Concurrent writing of shipped redo on a standby database might reduce the redo read rate due to I/O saturation. The overall recovery rate is always bounded by the rate at which redo can be read; so ensure that the redo read rate surpasses your required recovery rate.
To obtain the history of recovery rates, use the following query to get a history of recovery progress:
SELECT * FROM V$RECOVERY_PROGRESS;
If your ACTIVE APPLY RATE is greater than the maximum redo generation rate at the primary database or twice the average generation rate at the primary database, then no tuning is required; otherwise follow the tuning tips below. The redo generation rate for the primary database can be monitored from Enterprise Manager or extracted from AWR reports under statistic REDO SIZE. If CHECKPOINT TIME PER LOG is greater than ten seconds, then investigate tuning I/O and checkpoints.
DB_BLOCK_CHECKSUM=FULL and DB_BLOCK_CHECKING=MEDIUM or FULLRedo apply performance should be fast enough to keep up with most applications' redo generation rates but you can temporarily disable DB_BLOCK_CHECKING to speed up recovery. If you disable DB_BLOCK_CHECKING, you will disable in-memory block semantic checks as described in My Oracle Support note 1302539.1.
|
Note: To check for block corruption that was not preventable through theDB_BLOCK_CHECKING parameter, use:
|
Set the DB_LOST_WRITE_PROTECT parameter to FULL on the standby database to enable Oracle to detect writes that are lost in the I/O subsystem. The impact on redo apply is very small for OLTP applications and generally less than 5 percent.
Set DB_CACHE_SIZE to a value greater than that for the primary database. Set DB_KEEP_CACHE_SIZE and DB_RECYCLE_CACHE_SIZE to 0.
Having a large database cache size can improve media recovery performance by reducing the amount of physical data block reads. Because media recovery does not require DB_KEEP_CACHE_SIZE and DB_RECYCLE_CACHE_SIZE or require a large SHARED_POOL_SIZE, the memory can be reallocated to the DB_CACHE_SIZE.
Before converting the standby database into a primary database, reset these parameters to the primary database settings.
With the Active Data Guard option and real-time query, you can use Statspack from the primary database to collect data from a standby database that is opened read-only and performing recovery. Any tuning or troubleshooting exercise should start with collecting Standby Statspack reports. For complete details about installing and using Standby Statspack, see "Installing and Using Standby Statspack in 11g" in My Oracle Support Note 454848.1 at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=454848.1
If you do not have a license for the Active Data Guard option, you can determine the top system and session wait events by querying the standby database's V$SYSTEM_EVENT, V$SESSION_WAIT, and V$EVENT_HISTOGRAM and looking for the largest TIME_WAITED value. You may have to capture multiple snapshots of the query results and manually extract the difference to accurately assess a certain time period.
If recovery is applying a lot of redo data efficiently, the system is I/O bound and the I/O wait should be reasonable for your system. The vast majority of wait events related to parallel recovery coordinators and slaves apply to the coordinator. Slaves are either applying changes (clocking on CPU) or waiting for changes to be passed from the coordinator.
Typically, in a properly tuned system, the top wait event is db file parallel write followed by checkpoint completed. Consult the table below for tuning advice in cases where db file parallel write is not the top wait event. The database wait events are shown in Table 9-3 and Table 9-4.
Table 9-3 Parallel Recovery Coordinator Wait Events
| Wait Name | Description | Tuning |
|---|---|---|
|
|
The parallel recovery coordinator is waiting on I/O from the online redo log or the archived redo log. |
Tune or increase the I/O bandwidth for the ASM diskgroup where the archive logs or online redo logs reside. |
|
|
This event indicates that all read buffers are being used by slaves, and usually indicates that the recovery slaves lag behind the coordinator. |
Tune or increase the I/O bandwidth for the ASM diskgroup where data files reside. |
|
|
The parallel recovery coordinator is waiting for a buffer to be released by a recovery slave. Again, this is a sign the recovery slaves are behind the coordinator. |
Tune or increase the I/O bandwidth for the ASM diskgroup where data files reside. |
|
|
The parallel recovery coordinator is waiting for a file resize to finish, as would occur with file auto extend. |
Tune or increase the I/O bandwidth for the ASM diskgroup where data files reside. |
|
|
The coordinator has sent a synchronous control messages to all slaves, and is waiting for all slaves to reply. |
This is a non-tunable event. |
When dealing with recovery slave events, it is important to know how many slaves were started. Divide the wait time for any recovery slave event by the number of slaves. Table 9-4 describes the parallel recovery slave wait events.
Table 9-4 Parallel Recovery Slave Wait Events
| Wait Name | Description | Tuning |
|---|---|---|
|
|
The parallel recovery slave is waiting for a change to be shipped from the coordinator. This is in essence an idle event for the recovery slave. To determine the amount of CPU a recovery slave is using, divide the time spent in this event by the number of slaves started and subtract that value from the total elapsed time. This may be close, because there are some waits involved. |
Tune or increase the I/O bandwidth for the ASM diskgroup where the archive logs or online redo logs reside. |
|
|
A parallel recovery slave (or serial recovery process) is waiting for a batch of synchronous data block reads to complete. |
Tune or increase the I/O bandwidth for the ASM diskgroup where data files reside. |
|
|
Recovery is waiting for checkpointing to complete, and Redo Apply is not applying any changes currently. |
Tune or increase the I/O bandwidth for the ASM diskgroup where data files reside. Also, increase the number of |
|
|
A parallel recovery slave is waiting for a batched data block I/O. |
Tune or increase the I/O bandwidth for the ASM diskgroup where data files reside. |
DBWR must write out modified blocks from the buffer cache to the data files. Always use native asynchronous I/O by setting DISK_ASYNCH_IO to TRUE (default). In the rare case that asynchronous I/O is not available, use DBWR_IO_SLAVES to improve the effective data block write rate with synchronous I/O.
Ensure that you have sufficient I/O bandwidth and that I/O response time is reasonable for your system either by doing some base I/O tests, comparing the I/O statistics with those for the primary database, or by looking at some historical I/O metrics. Be aware that I/O response time may vary when many applications share the same storage infrastructure such as with a Storage Area Network (SAN) or Network Attached Storage (NAS).
Use system commands such as UNIX sar and vmstat commands, or use system monitoring tools to assess the system resources. Alternatively, you can monitor using Oracle Enterprise Manager, AWR reports, or performance views such as V$SYSTEM_EVENT, V$ASM_DISK and V$OSSTAT.
If there are I/O bottlenecks or excessive wait I/O operations, then investigate operational or application changes that increased the I/O volume. If the high waits are due to insufficient I/O bandwidth, then add more disks to the relevant Oracle ASM disk group. Verify that this is not a bus or controller bottleneck or any other I/O bottleneck. The read I/O rate from the standby redo log should be greater than the expected recovery rate.
Check for excessive swapping or memory paging.
Check to ensure the recovery coordinator or MRP is not CPU bound during recovery.
You should deploy multiple standby databases for any of the following purposes. When desired, use standby databases for these purposes while reserving at least one standby database to serve as the primary failover target:
To provide continuous protection following failover
The standby databases in a multiple standby configuration that are not the target of the role transition (these databases are referred to as bystander standby databases) automatically apply redo data received from the new primary database.
To achieve zero data loss protection while also guarding against widespread geographic disasters that extend beyond the limits of synchronous communication
For example, one standby database that receives redo data synchronously is located 200 miles away, and a second standby database that receives redo data asynchronously is located 1,500 miles away from the primary.
To perform rolling database upgrades while maintaining disaster protection throughout the rolling upgrade process
To perform testing and other ad-hoc tasks while maintaining disaster-recovery protection
Use Multiple Standby Databases Best Practices
The Oracle Database High Availability Overview describes how a multiple standby database architecture is virtually identical to that of single standby database architectures. Therefore, the configuration guidelines for implementing multiple standby databases described in this section complement the existing best practices for physical and logical standby databases.
When deploying multiple standby databases, use the following best practices:
Use Oracle Data Guard broker (described in Chapter 12, "Monitoring for High Availability") to manage your configuration and perform role transitions. However, if you choose to use SQL*Plus statements, see the MAA white paper "Multiple Standby Databases Best Practices" for best practices from the MAA Best Practices area for Oracle Database at
If you are using Flashback Database for the sole purpose of reinstating databases following a failover, a DB_FLASHBACK_RETENTION_TARGET of 120 minutes is the minimum recommended value. When you use Flashback Database to quickly reinstate the original primary as the standby after a failover, instead of re-creating the entire standby database from backups or from the primary database, when using Fast-start Failover, ensure the UNDO_RETENTION and DB_FLASHBACK_RETENTION_TARGET initialization parameters are set to a minimum of 120 so that reinstatement is still possible after a prolonged outage. On a standby the flashback barrier cannot be guaranteed to be published every 30 minutes as it is on a primary. Thus, when enabling flashback database on a standby, the DB_FLASHBACK_RETENTION_TARGET should be a minimum of 120. Since the primary and standby should match, this implies the same for the primary.
Enable supplemental logging in configurations containing logical standby databases. When creating a configuration with both physical and logical standby databases, issue the ALTER DATABASE ADD SUPPLEMENTAL LOG DATA statement to enable supplemental logging in the following situations:
When adding a logical standby database to an existing configuration consisting of all physical standby databases, you must enable supplemental logging on all existing physical standby databases in the configuration.
When adding a physical standby database to an existing configuration that contains a logical standby database, you must enable supplemental logging on the physical standby database when you create it.
As part of the logical standby database creation supplemental logging is automatically enabled on the primary. Enabling supplemental logging is a control file change and therefore the change is not propagated to each physical standby database. Supplemental logging is enabled automatically on a logical standby database when it is first converted from a physical standby database to a logical standby database as part of the dictionary build process.To enable supplemental logging, issue the following SQL*Plus statement when connected to a physical standby database:
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY, UNIQUE INDEX) COLUMNS;
If logical standby databases are not configured to perform real-time queries, then consider configuring SQL Apply to delay applying redo data to the logical standby database. By delaying the application of redo, you can minimize the need to manually reinstate the logical standby database after failing over to a physical standby database.
To set a time delay, use the DELAY=minutes attribute of the LOG_ARCHIVE_DEST_n initialization parameter.
|
See Also:
|
With proper planning and execution, Data Guard role transitions can effectively minimize downtime and ensure that the database environment is restored with minimal impact on the business. Using a physical standby database, MAA testing has determined that switchover and failover times with Oracle Data Guard 11g have been reduced to seconds. This section describes best practices for both switchover and failover.
A database switchover performed by Oracle Data Guard is a planned transition that includes a series of steps to switch roles between a standby database and a primary database. Following a successful switchover operation, the standby database assumes the primary role and the primary database becomes a standby database. Switchovers are typically completed in only seconds to minutes. At times the term switchback is also used within the scope of database role management. A switchback operation is a subsequent switchover operation to return the roles to their original state.
Data Guard enables you to change these roles dynamically by:
Using Oracle Enterprise Manager
Using the Oracle Data Guard broker's DGMGRL command-line interface
Issuing SQL statements, as described in Section 14.2.1.3, "How to Perform Data Guard Switchover"
|
See Also: Oracle Data Guard Broker for information about using Oracle Enterprise Manager or Oracle Data Guard broker's DGMGRL command-line interface to perform database switchover |
To optimize switchover processing, perform the following steps before performing a switchover:
Disconnect all sessions possible using the ALTER SYSTEM KILL SESSION SQL*Plus command.
Stop job processing by setting the AQ_TM_PROCESSES parameter to 0.
Cancel any specified apply delay by using the NODELAY keyword to stop and restart log apply services on the standby database.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE NODELAY;
You can view the current delay setting on the primary database by querying the DELAY_MINS column of the V$ARCHIVE_DEST view.
For physical standby databases in an Oracle RAC environment, ensure there is only one instance active for each primary and standby database.
Configure the standby database to use real-time apply and, if possible, ensure the databases are synchronized before the switchover operation to optimize switchover processing.
For the fastest switchover, use real-time apply so that redo data is applied to the standby database as soon as it is received, and the standby database is synchronized with the primary database before the switchover operation to minimize switchover time. To enable real-time apply use the following SQL*Plus statement:
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT USING CURRENT LOGFILE;
For a physical standby database, reduce the number of archiver (ARCn) processes to the minimum needed for both remote and local archiving. Additional archiver processes can take additional time to shut down, thereby increasing the overall time it takes to perform a switchover. After the switchover has completed you can reenable the additional archiver processes.
Set the LOG_FILE_NAME_CONVERT initialization parameter to any valid value for the environment, or if it is not needed set the parameter to null.
As part of a switchover, the standby database must clear the online redo log files on the standby database before opening as a primary database. The time needed to complete the I/O can significantly increase the overall switchover time. By setting the LOG_FILE_NAME_CONVERT parameter, the standby database can pre-create the online redo logs the first time the MRP process is started. You can also pre-create empty online redo logs by issuing the SQL*Plus ALTER DATABASE CLEAR LOGFILE statement on the standby database.
|
See Also: Support notes for switchover best practices for Data Guard Physical Standby (11.2.0.2):
|
A failover is typically used only when the primary database becomes unavailable, and there is no possibility of restoring it to service within a reasonable period. During a failover the primary database is taken offline at one site and a standby database is brought online as the primary database.
With Data Guard the process of failover can be completely automated using fast-start failover or it can be a manual, user driven process. Oracle recommends using fast-start failover to eliminate the uncertainty inherent in a process that requires manual intervention. Fast-start failover automatically executes a failover within seconds of an outage being detected.
For more on Data Guard failover best practices, see:
|
See Also: For a comprehensive review of Oracle Data Guard failover best practices, see:
|
There are two distinct types of failover: manual failover and fast-start failover. An administrator initiates manual failover when the primary database fails. In contrast, Data Guard automatically initiates a fast-start failover without human intervention after the primary database has been unavailable for a set period (the fast-start failover threshold).
Table 9-5 compares fast-start failover and manual failover.
Table 9-5 Comparing Fast-Start Failover and Manual Failover
| Points of Comparison | Fast-Start Failover | Manual Failover |
|---|---|---|
|
Benefits |
Allows you to increase availability with less need for manual intervention, thereby reducing management costs. |
Gives you control over exactly when a failover occurs and to which target standby database. |
|
Failover triggers |
The following conditions automatically trigger a fast-start failover:
|
A manual failover is user initiated and involves performing a series of steps to convert a standby database into a primary database. A manual failover should be performed due to an unplanned outage such as:
|
|
Management |
Use the following tools to manage fast-start failover failovers:
See Section 14.2.1.3, "How to Perform Data Guard Switchover". |
Use the following tools to perform manual failovers:
See Section 13.2.2.3, "Best Practices for Performing Manual Failover". |
|
Restoring the original primary database after failover |
Following a fast-start failover, Oracle Data Guard broker can automatically reconfigure the original primary database as a standby database upon reconnection to the configuration ( |
After manual failover, you must reinstate the original primary database as a standby database to restore fault tolerance. |
|
Restoring bystander standby databases after failover |
Oracle Data Guard broker coordinates the role transition on all databases in the configuration. Bystanders that do no require reinstatement are available as viable standby databases to the new primary. Bystanders that require reinstatement are automatically reinstated by the observer. |
A benefit of using Oracle Data Guard broker is that it provides the status of bystander databases and indicates whether a database must be reinstated. Status information is not readily available when using SQL*Plus statements to manage failover. See Section 13.3.2, "Restoring a Standby Database After a Failover". |
|
Application failover |
Oracle Data Guard broker automatically publishes FAN/AQ (Advanced Queuing) and FAN/ONS (Oracle Notification Service) notifications after a failover. Clients that are also configured for Fast Connection Failover can use these notifications to connect to the new primary database. You can also use the |
Oracle Data Guard broker automatically publishes FAN/AQ (Advanced Queuing) and FAN/ONS (Oracle Notification Service) notifications after a failover. Clients that are also configured for Fast Connection Failover can use these notifications to connect to the new primary database. You can also use the DB_ROLE_CHANGE system event to help user applications locate services on the primary database. (These events are also available for manual failovers performed by the broker. See Oracle Data Guard Broker.) |
To optimize failover processing:
Enable Flashback Database to reinstate the failed primary databases after a failover operation has completed. Flashback Database facilitates fast point-in-time recovery, if needed.
Use real-time apply with Flashback Database to apply redo data to the standby database as soon as it is received, and to quickly rewind the database should user error or logical corruption be detected.
Consider configuring multiple standby databases to maintain data protection following a failover.
Set the LOG_FILE_NAME_CONVERT parameter. As part of a failover, the standby database must clear its online redo logs before opening as the primary database. The time needed to complete this I/O can add significantly to the overall failover time. By setting the LOG_FILE_NAME_CONVERT parameter, the standby pre-creates the online redo logs the first time the MRP process is started. You can also pre-create empty online redo logs by issuing the SQL*Plus ALTER DATABASE CLEAR LOGFILE statement on the standby database.
Use fast-start failover. The MAA tests running Oracle Database 11g show that failovers performed using Oracle Data Guard broker and fast-start failover offer a significant improvement in availability. For more information, see Section 9.4.2.3, "Fast-Start Failover Best Practices".
For physical standby databases, do the following:
When transitioning from read-only mode to Redo Apply (recovery) mode, restart the database.
Go directly to the OPEN state from the MOUNTED state instead of restarting the standby database (as required in releases before Oracle Database 11g release 2).
See the MAA white paper "Oracle Data Guard Redo Apply and Media Recovery" to optimize media recovery for Redo Apply from the MAA Best Practices area for Oracle Database at
Fast-start failover automatically, quickly, and reliably fails over to a designated standby database if the primary database fails, without requiring manual intervention to execute the failover. You can use fast-start failover only in an Oracle Data Guard configuration that is managed by Oracle Data Guard broker.
The Oracle Data Guard configuration can be running in either the maximum availability or maximum performance mode with fast-start failover. When fast-start failover is enabled, the broker ensures fast-start failover is possible only when the configured data loss guarantee can be upheld. Maximum availability mode provides an automatic failover environment guaranteed to lose no data. Maximum performance mode provides an automatic failover environment guaranteed to lose no more than the amount of data (in seconds) specified by the FastStartFailoverLagLimit configuration property.
Use the following fast-start failover best practices in addition to the generic best practices listed in the Section 9.4.2.2, "Failover Best Practices (Manual Failover and Fast-Start Failover)":
Run the fast-start failover observer process on a host that is not located in the same data center as the primary or standby database.
Ideally, you should run the observer on a system that is equally distant from the primary and standby databases. The observer should connect to the primary and standby databases using the same network as any end-user client. If the designated observer fails, Oracle Enterprise Manager can detect it and automatically restart the observer. If the observer cannot run at a third site, then you should install the observer on the same network as the application. If a third, independent location is not available, then locate the observer in the standby data center on a separate host and isolate the observer as much as possible from failures affecting the standby database.
Make the observer highly available by using Oracle Enterprise Manager to configure the original primary database to be automatically reinstated as a standby database when a connection to the database is reestablished. Also, Oracle Enterprise Manager enables you to define an alternate hJ ost on which to restart the observer.
After the failover completes, the original primary database is automatically reinstated as a standby database when a connection to it is reestablished, if you set the FastStartFailoverAutoReinstate configuration property to TRUE.
Set the value of the FastStartFailoverThreshold property according to your configuration characteristics, as described in Table 9-6.
Table 9-6 Minimum Recommended Settings for FastStartFailoverThreshold
| Configuration | Minimum Recommended Setting |
|---|---|
|
Single-instance primary, low latency, and a reliable network |
15 seconds |
|
Single-instance primary and a high latency network over WAN |
30 seconds |
|
Oracle RAC primary |
Oracle RAC miscount + reconfiguration time + 30 seconds |
Test your configuration using the settings shown in Table 9-6 to ensure that the fast-start failover threshold is not so aggressive that it induces false failovers, or so high it does not meet your failover requirements.
You should perform a manual failover, which is user-driven, only in case of an emergency and the failover should be initiated due to an unplanned outage such as:
Site disaster that results in the primary database becoming unavailable
User errors that cannot be repaired in a timely fashion
Data failures, to include widespread corruption, which affects the production application
Use the following manual failover best practices in addition to the generic best practices listed in Section 9.4.2.2, "Failover Best Practices (Manual Failover and Fast-Start Failover)":
Reinstate the original primary database as a standby database to restore fault tolerance to your environment. The standby database can be quickly reinstated by using Flashback Database. See Section 13.3.2, "Restoring a Standby Database After a Failover."
If you have a license for the Oracle Active Data Guard option then you can open a physical standby database for read-only access while Redo Apply on the standby database continues to apply redo data received from the primary database. All queries reading from the physical standby database execute in real time and return current results, providing more efficient use of system resources and additional assurance that the standby is healthy without compromising data protection or extending recovery time if a failover is required. Hence, this capability is referred to as real-time query.
|
Note: A physical standby database can be open for read-only access while Redo Apply is active if a license for the Oracle Active Data Guard option has been purchased. This capability, known as real-time query also provides the ability to have block-change tracking on the standby database, thus allowing incremental backups to be performed on the standby. |
To deploy real-time query:
Ensure Active Data Guard is enabled.
The easiest and best way to view the status of Oracle Active Data Guard is on the Data Guard overview page through Oracle Enterprise Manager.
Alternatively, query the v$database view on the standby database and confirm the status of "READ ONLY WITH APPLY':
SQL> SELECT open_mode FROM V$DATABASE; OPEN_MODE -------------------- READ ONLY WITH APPLY
Use real-time apply on the standby database so that changes are applied as soon as the redo data is received.
Oracle Data Guard broker automatically enables real-time apply when the configuration is created. If you are using the SQL*Plus command-line to create your configuration, then enable real-time apply as follows:
Issue the statement:
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE
Enable Flashback Database on the standby database to minimize downtime for logical corruptions.
Monitor standby performance by using Standby Statspack. For complete details about installing and using Standby Statspack, see "Installing and Using Standby Statspack in 11g" in My Oracle Support Note 454848.1 at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=454848.1
When you deploy real-time query to offload queries from a primary database to a physical standby database, monitor the apply lag to ensure that it is within acceptable limits. See Oracle Data Guard Concepts and Administration for information about Monitoring Apply Lag in a Real-time Query Environment.
Create an Oracle Data Guard broker configuration to simplify management and to enable automatic apply instance failover on an Oracle RAC standby database.
|
See Also: The "Active Data Guard 11g Best Practices (includes best practices for Redo Apply)" white paper available from the MAA Best Practices area for Oracle Database at |
Beginning with Oracle Database release 11g, you can convert a physical standby database into a fully updatable standby database called a snapshot standby database.
To convert a physical standby database into a snapshot standby database, issue the SQL*Plus ALTER DATABASE CONVERT TO SNAPSHOT STANDBY statement. This command causes Oracle Data Guard to perform the following actions:
Recover all available redo data
Create a guaranteed restore point
Activate the standby database as a primary database
Open the database as a snapshot standby database
To convert the snapshot standby back to a physical standby, issue the ALTER DATABASE CONVERT TO PHYSICAL STANDBY statement. This command causes the physical standby database to be flashed back to the guaranteed restore point that was created before the ALTER DATABASE CONVERT TO SNAPSHOT STANDBY statement was issued. Then, you must perform the following actions:
Restart the physical standby database
Restart Redo Apply on the physical standby database
To create and manage snapshot standby databases:
Use the Oracle Data Guard broker to manage your Oracle Data Guard configuration, because it simplifies the management of snapshot standby databases. The broker will automatically convert a snapshot standby database into a physical standby database as part of a failover operation. Without the broker, this conversion must be manually performed before initiating a failover.
Create multiple standby databases if your business requires a fast recovery time objective (RTO).
Ensure the physical standby database that you convert to a snapshot standby is caught up with the primary database, or has a minimal apply lag. See Section 9.3.8, "Use Data Guard Redo Apply Best Practices" for information about tuning media recovery.
Configure a fast recovery area and ensure there is sufficient I/O bandwidth available. This is necessary because snapshot standby databases use guaranteed restore points.
|
See Also: Oracle Data Guard Concepts and Administration for complete information about creating a snapshot standby database |
To accurately assess the primary database performance after adding Data Guard standby databases, obtain a history of statistics from the V$SYSMETRIC_SUMMARY view or Automatic Workload Repository (AWR) snapshots before and after deploying Oracle Data Guard with the same application profile and load.
To assess the application profile, compare the following statistics:
Physical reads per transaction
Physical writes per transaction
CPU usage per transaction
Redo generated per transaction
To assess the application performance, compare the following statistics:
Redo generated per second or redo rate
User commits per second or transactions per second
Database time per second
Response time per transaction
SQL service response time
If the application profile has changed between the two scenarios, then this is not a fair comparison. Repeat the test or tune the database or system with the general principles outlined in the Oracle Database Performance Tuning Guide.
If the application profile is similar and you observe application performance changes on the primary database because of a decrease in throughput or an increase in response time, then assess these common problem areas:
CPU utilization
If you are experiencing high load (excessive CPU usage of over 90%, paging and swapping), then tune the system before proceeding with Data Guard. Use the V$OSSTAT view or the V$SYSMETRIC_HISTORY view to monitor system usage statistics from the operating system.
Higher I/O wait events
If you are experiencing higher I/O waits from the log writer or database writer processes, then the slower I/O effects throughput and response time. To observe the I/O effects, look at the historical data of the following wait events:
Log file parallel writes
Log file sequential reads
Log file parallel reads
Data file parallel writes
Data file sequential reads parallel writes
With SYNC transport, commits take more time because of the need to guarantee that the redo data is available on the standby database before foreground processes get an acknowledgment from the log writer (LGWR) background process that the commit has completed. A LGWR process commit includes the following wait events:
Log File Parallel Write (local write for the LGWR process)
LGWR wait on SENDREQ
This wait event includes:
Time to put the packet into the network
Time to send the packet to the standby database
RFS write or standby write to the standby redo log, which includes the RFS I/O wait event plus additional overhead for checksums
Time to send a network acknowledgment back to the primary database (for example, single trip latency time)
Longer commit times for the LGWR process can cause longer response time and lower throughput, especially for small time-sensitive transactions. However, you may obtain sufficient gains by tuning the log writer local write (Log File Parallel Write wait event) or the different components that comprise the LGWR wait on SENDREQ wait event.
To tune the disk write I/O (Log File Parallel Write or the RFS I/O), add more spindles or increase the I/O bandwidth.
To reduce the network time:
Tune the Oracle Net send and receive buffer sizes
Set SDU=65535 (for more information, see Section 9.3.7.2, "Set the Network Configuration and Highest Network Redo Rates")
Increase the network bandwidth if there is saturation
Possibly find a closer site to reduce the network latency
With ASYNC transport, the LGWR process never waits for the network server processes to return before writing a COMMIT record to the current log file. However, if the network server processes has fallen behind and the redo to be shipped has been flushed from the log buffer, then the network server process reads from the online redo logs. This causes more I/O contention and possibly longer wait times for the log writer process writes (Log File Parallel Write). If I/O bandwidth and sufficient spindles are not allocated, then the log file parallel writes and log file sequential reads increase, which may affect throughput and response time. In most cases, adding sufficient spindles reduces the I/O latency.
|
See Also:
|
This book provides high availability best practices for configuring and maintaining your Oracle Database system and network components.
This book is intended for chief information technology officers and architects, as well as administrators who perform the following database, system, network, and application tasks:
Plan data centers
Implement data center policies
Maintain high availability systems
Plan and build high availability solutions
For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.
Access to Oracle Support
Oracle customers have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.
For more information, see the Oracle database documentation set. These books may be of particular interest:
Oracle Data Guard Concepts and Administration and Oracle Data Guard Broker
Oracle Real Application Clusters Administration and Deployment Guide
The Oracle High Availability Best Practice white papers that can be downloaded from the Oracle Technology Network (OTN) at
The Oracle Enterprise Manager documentation library at
http://www.oracle.com/technetwork/oem/grid-control/overview/index.html
The following text conventions are used in this document:
| Convention | Meaning |
|---|---|
| boldface | Boldface type indicates graphical user interface elements associated with an action, or terms defined in text or the glossary. |
| italic | Italic type indicates book titles, emphasis, or placeholder variables for which you supply particular values. |
monospace | Monospace type indicates commands within a paragraph, URLs, code in examples, text that appears on the screen, or text that you enter. |
The steps described in this chapter assume that the client version and database version are 11.2.0.1 or higher. The 11.2.0.1 release provides for the following features when compared to 11.1.0.7 or below:
Role based services
Data Guard broker sending FAN ONS events to JDBC clients
Support for SCAN addresses
While previous versions do not have the above features it is possible to achieve similar results with manual configuration. For example:
Create triggers that manage stopping and starting a service based on the database role.
Utilize an external ONS publisher to send FAN events after a failover has occurred.
Creating Oracle Net aliases that include all hosts with the potential to become a primary.
The steps for configuring versions earlier than 11.2.0.1 are in the MAA white paper "Client Failover Best Practices for Highly Available Oracle Databases: Oracle Database 10g Release 2" at
http://www.oracle.com/goto/maa
Types of Failures
Unplanned failures of an Oracle Database instance fall into the general categories:
A server failure or other fault that causes the crash of an individual Oracle instance in an Oracle RAC database. To maintain availability, application clients connected to the failed instance must quickly be notified of the failure and immediately establish a new connection to the surviving instances of the Oracle RAC database.
A complete-site failure that results in both the application and database tiers being unavailable. To maintain availability users must be redirected to a secondary site that hosts a redundant application tier and a synchronized copy of the production database.
A partial-site failure where the primary database, a single-instance database, or all nodes in an Oracle RAC database become unavailable but the application tier at the primary site remains intact.
Configure Fast Connection Failover as a best practice to fully benefit from fast instance and database failover and switchover with Oracle RAC and Oracle Data Guard. Fast Connection Failover enables clients, mid-tier applications, or any program that connects directly to a database to failover quickly and seamlessly to an available database service when a database service becomes unavailable.
This chapter contains the following topics:
|
See Also:
|
The best practices for configuration to enable fast connection failover differs, depending on the type of your client: JDBC or OCI.
Fast Connection Failover for JDBC Clients
For JDBC clients, follow these best practices:
Enable Fast Connection Failover for JDBC clients by setting the DataSource property FastConnectionFailoverEnabled to TRUE.
Configure JDBC clients to use a connect descriptor that includes an address list that in turn includes the SCAN address for each site and connects to an existing service.
The JDBC client must set the oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR property. This property enables the JDBC client to quickly traverse an ADDRESS_LIST in the event of a failure.
Configure a remote Oracle Notification Service (ONS) subscription on the JDBC client so that an ONS daemon is not required on the client.
By default the JDBC application randomly picks three hosts from the setONSConfiguration property and creates connections to those three ONS daemons. You must change this default so that connections are made to all ONS daemons. This is done by setting the following property when the JDBC application is invoked to the total number of ONS daemons in the configuration:
java - oracle.ons.maxconnections=4
|
See Also:
|
Fast Connection Failover for OCI Clients
For OCI clients, follow these best practices:
Enable Fast Application Notification (FAN) for OCI clients by initializing the environment with the OCI_EVENTS parameter.
Link the OCI client applications with the thread library.
Set the AQ_HA_NOTIFICATIONS parameter to TRUE and configure the transparent application failover (TAF) attributes for services.
Configure an Oracle Net alias that the OCI application uses to connect to the database. The Oracle Net alias should specify both the primary and standby SCAN hostnames. For best performance while creating new connections the Oracle Net alias should have LOAD_BALANCE=OFF for the DESCRIPTION_LIST so that DESCRIPTIONs are tried in an ordered list, top to bottom. With this configuration the second DESCRIPTION is only attempted if all connection attempts to the first DESCRIPTION have failed.
|
See Also:
|
Oracle Database 11g provides the infrastructure to make your application data highly available with Oracle Real Application Clusters (Oracle RAC) and with the Oracle Data Guard. At the database tier you must configure fast application failover.
At a high level, automating client failover in an Oracle RAC configuration includes relocating database services to new or surviving instances, notifying clients that a failure has occurred to break the clients out of TCP timeout, and redirecting clients to a surviving instance (Oracle Clusterware sends FAN messages to applications; applications can respond to FAN events and take immediate action). For more information about FAN, see Section 6.1.1, "Client Configuration and Migration Concepts".
For services on an Oracle RAC database, Oracle Enterprise Manager or the SRVCTL utility are the recommended tools to manage services. A service can span one or more instances of an Oracle database and a single instance can support multiple services. The number of instances offering the service is managed by the DBA independent of the application.
Server-side callouts provide a simple, yet powerful integration mechanism with the High Availability Framework that is part of Oracle Clusterware. You can use server side callouts to log trouble tickets or page Administrators to alert them of a failure. For Up events, when services and instances are started, new connections can be created so the application can immediately take advantage of the extra resources
|
See Also:
|
To configure the Oracle Data Guard environment, do the following:
|
See Also: The MAA white paper "Client Failover Best Practices for Data Guard 11g Release 2" from the MAA Best Practices area for Oracle Database at |
In an Oracle Data Guard configuration you should only run primary application services on the primary database and run standby application services on the standby database. Beginning with Data Guard 11g Release 2, you can automatically control the startup of database services on primary and standby databases by assigning a database role to each service (roles include: PRIMARY, PHYSICAL_STANDBY, LOGICAL_STANDBY, and SNAPSHOT_STANDBY).
A database service automatically starts upon database startup if the management policy for the service is AUTOMATIC and if a role assigned to that service matches the current role of the database.
|
See Also:
|
The best practice is to configure Oracle Data Guard to manage the configuration with Oracle Data Guard Broker. Oracle Data Guard Broker is responsible for sending FAN events to client applications to clean up their connections to the down database and reconnect to the new production database. For more information about FAN, see Section 6.1.1, "Client Configuration and Migration Concepts".
Oracle Clusterware must be installed and active on the primary and standby sites for both single instance (using Oracle Restart) and Oracle RAC databases. Oracle Data Guard broker coordinates with Oracle Clusterware to properly fail over role-based services to a new primary database after a Data Guard failover has occurred. For more information, see
In Oracle Data Guard, the term switchover describes a planned event where a primary and standby database switch roles, usually to minimize the downtime while performing planned maintenance. The configuration best practices to address unplanned failovers also address most of the requirements for a planned switchover, except for several additional manual steps that apply to logical standby databases (SQL Apply).
|
Note: There are no additional considerations for switchovers using Oracle Active Data Guard. |
The following steps describe the additional manual switchover steps for Oracle Data Guard 11g Release 2:
The primary database is converted to a standby database. This disconnects all sessions and brings the database to the mount state. Oracle Data Guard Broker shuts down any read/write services.
Client sessions receive a ORA-3113 and begin going through their retry logic (TAF for OCI and application code logic for JDBC).
The standby database is converted to a primary database and any existing sessions are disconnected. Oracle Data Guard Broker shuts down read-only services.
Read-only connections receive an ORA-3113 and begin going through their retry logic (TAF for OCI and application code logic for JDBC).
As the new primary and the new standby are opened, the respective services are started for each role and clients performing retries now see the services available and connect.
For logical standby switchover:
Ensure that the proper reconnection logic has been configured (for more information, see Section 11.1, "Configure JDBC and OCI Clients for Failover" and Section 11.2, "Configure Oracle RAC Databases for Failover"). For example, configure TAF and RETRY_COUNT for OCI applications and code retry logic for JDBC applications.
Stop the services that the primary application uses and the read-only applications enabled on the standby database.
Disconnect or shutdown the primary and read-only application sessions.
Once the switchover has completed, restart the services used by the primary application and the read-only application.
Sessions that were terminated reconnect once the service becomes available as part of the retry mechanism.
Restart the application if an application shuts down.
Note that FAN is not needed to transition clients during a switchover operation if the application performs retries. FAN is only needed to break clients out of TCP timeout, a state that should only occur during unplanned outages.
The process of failing over an application that has a large number of connections may create a login storm. A login storm is a sudden spike in the number of connections to a database instance, which drains CPU resources. As CPU resources are depleted, application timeouts and application response times are likely to increase.
To control login storms:
Implement the Connection Rate Limiter
The primary method of controlling login storms is to implement the Connection Rate Limiter feature of the Oracle listener. This feature limits the number of connections that can be processed in seconds. Slowing down the rate of connections ensures that CPU resources remain available and that the system remains responsive.
Configure Oracle Database for shared server operations
In addition to implementing the Connection Rate Limiter, some applications can control login storms by configuring Oracle Database for shared server operations. By using shared server, the number of processes that must be created at failover time are greatly reduced, thereby avoiding a login storm.
Adjust the maximum number of connections in the mid tier connection pool
If such a capability is available in your application mid tier, try limiting the number of connections by adjusting the maximum number of connections in the mid tier connection pool.
|
See Also:
|
Currently, PeopleSoft Enterprise and Oracle WebLogic Server have support for FAN events.
PeopleSoft PeopleTools version 8.50.09 and higher supports FAN. This enables PeopleSoft applications to automatically failover database connections to a surviving instance in an Oracle RAC cluster or to a new primary database in an Oracle Data Guard configuration should its database connection be lost. If an Oracle RAC instance fails, a primary database fails, or the Oracle Database is shutdown or restarted, PeopleSoft servers and clients continue running and users are not required to login a second time.
In Oracle WebLogic Server 10.3.4, a single data source implementation has been introduced to support an Oracle RAC cluster. It responds to FAN events to provide Fast Connection Failover (FCF), run-time connection load-balancing (RCLB), and Oracle RAC instance graceful shutdown. XA affinity is supported at the global transaction ID level. The new feature is called WebLogic Active GridLink for Oracle RAC, which is implemented as the GridLink data source within Oracle WebLogic Server.
For applications that do not support FAN events, this includes a number of applications from Oracle (for example, Siebel and Oracle E-Business Suite), all of the steps described in this section should be completed for the fastest client failover possible. Even though FAN events cannot be used in such cases, applications can still be configured for efficient failover by using timeouts and application retries.
For more information see the MAA white paper "Client Failover Best Practices for Highly Available Oracle Databases: Oracle Database 11g Release 2" at