Oracle PDB Failover in a Data Guard environment

Oracle PDB Failover in a Data Guard environment:
Using Data Guard Broker to Unplug a Single Failed PDB from a Standby Database and
Plugging into a New Container or Migrate a Single PDB into a New Container.
Target:
In this post, I will explain about how operating simple process for either migration of a single PDB
or execution of a single PDB failover into another container database with minimal downtime to
the impacted PDB and zero impact to any remaining primary PDBs in the primary CDB.
The process takes advantage of new functionality added to the Data Guard broker command line
interface DGMGRL.
The steps involve changes being made to 2 independent CDBs, caution must be taken to
understand, validate and execute the steps documented below.
The functionality is supported in 12c,19c and later Oracle RDBMS versions.
Support for managing Transparent Data Encryption (TDE) keys is available for Oracle RDBMS
19.15 and later versions through a patch.
SOLUTION:
Prerequisites
•The source and destination CDBs must each be part of their own Data Guard con
fi
guration (not
in the same con
fi
guration).
The con
fi
gurations can consist of only a primary database with no standbys.
•The process requires either local connections (instances of both the source and destination
running on the same nodes as the broker command line session) or a shared
fi
le system that is
accessible to nodes running instances of the source and destination CDBs to ensure access to
the manifest
fi
le created for the PDB to be migrated.
This can be con
fi
gured either via tnsnames.ora aliases that point to speci
fi
c hosts and services or
through Oracle EZConnect connect strings when supplying the user credentials.
Oracle recommends if at all possible that the source and destination databases have instances
running on the same node(s) and the broker command line session be executed on one such
node.
•You must connect to the Broker command line process as SYSDBA
•Oracle recommends that you shutdown the services on both the primary and the standby that
are accessing the PDB prior to starting the migration process.
The PDB will be closed as part of migration so all connections to the PDB will be terminated.
•The destination CDB must already exist on the same environment as the source CDB (either the
primary database for migration or the standby database for failover) and it must run from an
ORACLE_HOME patched to the same level and with the same one-o
ff
patches as the source
environment.
This can be the same ORACLE_HOME being used by the source CDB database. The main
reason behind this is to avoid having to deal with extraneous violations after plugging the PDB
into the destination CDB.
•The destination CDB must have the same or higher COMPATIBLE setting as the source CDB.
The destination CDB cannot have a lower COMPATIBLE setting than the source.
•The destination CDB must have at least the same database options installed (e.g. Spatial, JVM,
etc) as the source CDB to avoid errors during plugin.
•If TDE is in use it must be con
fi
gured and enabled on both the source and destination databases.
It is not possible to encrypt/decrypt as part of this process.

Assuming a destination CDB is already in place and patched correctly on the standby site, the
entire process of moving the PDB can be completed in less than 15 minutes.
Additional Considerations
The following steps assume the source CDB database (either primary for migration or standby for
failover) and the destination CDB database have access to the same storage so no copying of
data
fi
les is required.
•The source CDB can be either an Oracle Real Application Cluster (RAC) database or a single
instance database.
•Active Data Guard is required for the source CDB standby for failover operations.
•The destination CDB can be either a RAC database or a single instance database.
•Storage for the source and destination databases can be either
fi
lesystem or ASM.
•If migrating a PDB from a standby database, ensure the TEMP
fi
le has already been created prior
to performing the migration.
•If the source PDB was closed immediate:
◦For migrate the PDB must be opened and recovered prior to attempting to migrate as the
unplug operation will fail.
◦For failover, close immediate of the PDB on the source standby will have no impact. Close
immediate of the PDB on the source primary may cause the unplug operation to fail and you
must perform manual cleanup to complete the process, however the source PDB will still be
failed over to the destination (assuming no other issues).
◦If the destination CDB is of a higher version, the PDB will be plugged in but left closed to
allow for manual upgrade as a post migration task.
◦Migration or failover to a CDB of a lower version is not supported.
◦PDB snapshot clones and PDB snapshot clone parents are not supported for migration or
failover.
◦After processing is completed, you may need to clean up leftover database
fi
les from the
source databases.
◦The plugin operation at the destination database is performed with STANDBYS=NONE, you
will need to manually enable recovery at any standby databases upon completion of the
migration.
Possible messages showing in migration:
NOTE: The following examples may generate di
ff
erent output as part of the migrate command
than you will see while executing the command based on the di
ff
erent states of PDBs and items
found by dgmgrl running prechecks in your environment.
In addition, Oracle does not ship message
fi
les with bug
fi
xes so instead of full messages being
displayed you may receive something lines similar to the following:
Message 17241 not found; product=rdbms; facility=DGM
This does not mean it's an error or a problem, it means the text we want to display is missing from
the message
fi
le. All messages will display in their entirety in the
fi
rst release fully containing all of
the
fi
xes.
The following is the list of messages possibly produced by the migrate function:
17180 - "Pluggable database %s must be open prior to starting a migration
operation."
17217 - "Migration cannot be performed when the source multitenant

container database (%(1)s) is a physical standby running a different
version of Oracle than %(2)s."
17235 - "Investigate why the pluggable database %s could not be unplugged."
17236 - "Resolve the issue and then manually unplug and drop the pluggable
database from database %s."
17237 - "Migration of pluggable database %s completed."
17238 - "Migration of pluggable database %s completed with warnings."
17239 - "Failed to migrate pluggable database %s."
17240 - "Media recovery is disabled for pluggable database %(1)s on
multitenant container database %(2)s."
17241 - "Warning: either source or destination multitenant container
database does not have local undo enabled."
17242 - "Migration from pluggable database %s not possible since it is
either a snapshot child or snapshot parent."
17243 - "Pluggable database %s could not be opened because it was migrated
to a database running a higher Oracle version."
17244 - "Please run the appropriate upgrade procedures prior to opening the
pluggable database."
17245 - "The file location specified (%s) is not accessible."
17246 - "A file name was not specified."
17247 - "An invalid file name (%s) was specified."
17248 - "Retry the command after the lag is resolved or use the IMMEDIATE
option to ignore the data loss."
17249 - "Media recovery is disabled for pluggable database %(1)s on
multitenant container database %(2)s."
17250 - "Warning: either source or destination multitenant container
database does not have local undo enabled."
17251 - "Migration from pluggable database %s not possible since it is
either a snapshot child or snapshot parent."
17413 - "Failed to open wallet of pluggable database %s."
17414 - "Keystore of pluggable database %s is not open."
17415 - "Master keys of the pluggable database %s need to be migrated."
17416 - "SECRET and KEYSTORE IDENTIFIED BY clauses are required."
17427 - "Unable to fetch wallet status of pluggable database %s."
17428 - "Keystore of pluggable database %s is open."
17429 - "Keystore password of pluggable database %s is not correct."
17430 - "Keystore password of multitenant container database %s is not
correct."
17536 - "Unable to fetch keystore mode of pluggable database %s."
17537 - "KEYFILE and SOURCE IDENTIFIED BY clauses are required."
Migration Steps
Migration is used to move a PDB from one container database to another.
The process will unplug the PDB from the primary database of a Data Guard con
fi
guration and
plug it into a primary database of another Data Guard con
fi
guration.
The source primary must share database
fi
le storage with the destination primary and the
manifest
fi
le created by the unplug operation on the source must be directly available to the
destination primary either through shared storage (such as an NFS mount) or by connecting to
source and destination instances running on the same node.
Migration allows you to move a PDB to a higher version database, but not to a database of a
lower version.
Note that during migration to a higher version database the migrate command leaves the PDB
closed at the destination to allow you to perform necessary steps required to complete any PDB
upgrades.

Migrations can occur to a database with the same or higher COMPATIBLE setting but migrating to
a database with a lower COMPATIBLE setting is not allowed.
For TDE enabled environments running Oracle RDBMS 19.15 and later with pre-requisite patches
applied, the broker will manage the TDE keys for the PDB, extracting them from the source CDB
and importing them into the keystore of the destination CDB.
Service creation in the destination database must be performed manually and, if using Oracle Grid
Infrastructure, can be added via srvctl prior to performing the migration.
Any changes required for application and end user connect strings must be performed manually
as well.
The steps to perform a migration are as follows:
1- Although not required, stop all services on both the source primary database and any standby
database(s) pertaining to the PDB to be migrated
2- If the source con
fi
guration has standby databases running in Active Data Guard mode and you
do not have Patch 25616359 applied, close the PDB on all instances of those standbys.
If they are not closed, redo apply will fail and cannot be successfully restarted until the PDB is
closed allowing redo apply to process the PDB drop.
3- Start a dgmgrl session connecting to the source con
fi
guration primary database.
You must connect to the source database as SYSDBA using something similar to the following:
$ dgmgrl
DGMGRL> connect sys as sysdba
4- Once connected, execute the migrate commandIf you leave o
ff
the connect clause, you will get
prompted for connection information. Be sure to specify the alias or Oracle EZConnect string on
the username prompt.
5-The location speci
fi
ed for the manifest
fi
le must be directly accessible to both the source
primary instance and the destination primary instance you are connecting to.
i- For non-TDE enabled environments:
DGMGRL> migrate pluggable database <PDBNAME1> to container <dest-cdb-name>
using '/<path>/<DEST_SID>/<PDBNAME1.xml>' connect as <USERNAME>@”<dest-cdb-
connect-identifer>”;
-or-
using '/<path>/<DEST_SID>/<PDBNAME1.xml>':
ii- For TDE enabled environments:
using '/<path>/<DEST_SID>/<PDBNAME1.xml>' connect as <USERNAME>@”<dest-cdb-
connect-identifer>” secret “<secret_word>” keystore identified by
“<destination_keystore_password>”;
-or-

using '/<path>/<DEST_SID>/<PDBNAME1.xml>' secret “<secret_word>” keystore
identified by “<destination_keystore_password>”;
SECRET is a value supplied to protect the keys for the PDB as part of the migration. The keys
are not accessible without the value for SECRET. This value is only required for the duration of
the migration process.
If SECRET or KEYSTORE are not speci
fi
ed in the command line, the migrate command will fail.
Once the connection to the destination is established the command will:
Perform all necessary validations for the migration operation
Unplug the PDB from the primary
NOTE: For all 12cR1 releases and 12cR2 releases without the
fi
x for Patch 25616359 installed, if
there is a standby database for the source primary and the PDB is open in that standby
database, redo apply will stop when it attempts to apply the unplug operation. To continue redo
apply processing, close the PDB and restart redo apply.
Create the PDB in the destination database using the primary's data
fi
les (NOCOPY clause) and
with STANDBYS=NONE.
Open the PDB in all instances of the destination database only if the destination is the same
software version as the source
Drop the PDB from the source primary database which also drops the PDB in all of the source
standby databases.
Once the command completes perform the following tasks.
Add services for the PDB as desired and start those services to resume connectivity to the PDB
Backup the PDB in the destination CDB to allow for recovery going forward
Follow the steps in Note 1916648.1 to enable recovery of the PDB at any standby databases to
establish availability and disaster recovery requirements
Note:
Advantage of using dgmgrl (Broker) is that you don't need manually close PDB and Unplug
it as xml
fi
le, all steps automatically was done by dgmgrl.
Migration example:
1- Connect to the source database and show the PDBs and the current role. For migration the
source database must be the primary
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
-- ------- --------- ---
2 <PDB$SEED> READ ONLY NO

5 <PDBNAME1> READ WRITE NO
8 <PDBNAME2> READ WRITE NO
SQL>select database_role from v$database;
DATABASE_ROLE
----------------
PRIMARY
-Connect to the source database primary using broker command line on the host shared by
instances of both the source database (<SOURCE DB>) and the destination database (<DEST
DB>)
DGMGRL> connect <USERNAME> as sysdba
Password:
Connected to "<SOURCE DB>"
Connected as SYSDBA.
-Execute the migrate command to migrate the PDB from <SOURCE DB> to <DEST DB> In the
following command /<path>/<SOURCE DB> is directly accessible to both <SOURCE DB> and
<DEST DB> on <DEST_HOSTNAME>.
using '/<path>/<SOURCE DB>/<PDBNAME1.xml>';
Username: <USERNAME>@"<dest-cdb-connect-identifer>"
Password:
Connected to "<DEST DB>"
Beginning migration of pluggable database <PDBNAME1>.
Source multitenant container database is <SOURCE DB>.
Destination multitenant container database is <DEST DB>.
Closing pluggable database <PDBNAME1> on all instances of multitenant
container database <SOURCE DB>.
Unplugging pluggable database <PDBNAME1> from multitenant container database
<SOURCE DB>.
Pluggable database description will be written to /<path>/<SOURCE DB>/
<PDBNAME1.xml>.
Dropping pluggable database <PDBNAME1> from multitenant container database
<SOURCE DB>.
Creating pluggable database <PDBNAME1> on multitenant container database
<DEST DB>.

Opening pluggable database <PDBNAME1> on all instances of multitenant
container database <DEST DB>.
Unresolved plug in violations found while migrating pluggable database
<PDBNAME1> to multitenant container database <DEST DB>.
Please examine the PDB_PLUG_IN_VIOLATIONS view to see the violations that
need to be resolved.
Migration of pluggable database <PDBNAME1> completed.
Succeeded.
Failover Steps
The following pictures show the original and
fi
nal projected con
fi
gurations after migrating a failed
PDB to the new container.
We have CDB1 with 3 PDBs (PDB1, PDB2, PDB3).
CDB1 also has a Data Guard physical standby. On the same environment as the CDB1 standby,
we also have CDB2 which is a read write database which will become the new host for one of the
targeted PDBs.
Before Image
In this image, CDB1 and all of its PDBs are running normally.
CDB1 Standby is not running Active Data Guard. CDB2 has its own PDB, PDB4.
After Image
PDB2 experiences a failure requiring a long recovery period, but the failure does not impact PDB1
and PDB3 and the standby for CDB1 continues to apply redo without error.
We will use PDB2’s
fi
les at the standby site to plug into CDB2 to restore read/write application
access and drop PDB2 from CDB1.
This will not be a native unplug operation as that requires a read/write CDB.

Process
Failover is used to move a PDB from a standby database to another container database when the
PDB at the primary database has become unusable and all other PDBs at the primary are still
operating normally.
Failover provides a method of failing over a single PDB without impacting other PDBs in the
primary.
The standby database must share database
fi
le storage with the destination primary and the
manifest
fi
le created by the DBMS_PDB.DESCRIBE operation at the standby must be directory
available to the destination primary either through shared storage (such as an NFS mount) or by
connecting to source and destination instances running on the same node.
Failover is an unplanned operation so the goal is to minimize downtime, ensure the
following prerequisites are met:
-The source and destination databases should be running the same patch levels so additional
scripts are not required to be executed. This requirement is enforced by the DGMGRL CLI
MIGRATE command. As a best practice the destination CDB should be running from the same
Oracle Home as the standby database.
-The destination CDB should minimally have the same database options installed (e.g. Oracle
Spatial, Oracle Text, Oracle Multimedia, etc) as the source database to ensure a smooth plug-in
operation.
The steps to perform a failover are as follows:
-Although not required, stop all services on both the source primary database and any standby
database(s) pertaining to the PDB to be migrated
-If the source con
fi
guration has standby databases running in Active Data Guard mode and you
do not have Patch 25616359applied, close the PDB on all instances of those standbys.
If they are not closed, redo apply will fail and cannot be successfully restarted until the PDB is
closed allowing redo apply to process the PDB drop.
-Start a dgmgrl session connecting to the source con
fi
guration standby database.
You must connect to the source database as SYSDBA using something similar to the following:
$ dgmgrl

Execute the migrate command to perform the failover.
NOTE: It is the same command as migrate
-If you leave o
ff
the connect clause, you will get prompted for connection information. Be sure to
specify the alias or Oracle EZConnect string on the user.
When doing failover, use of IMMEDIATE clause (e.g. MIGRATE PLUGGABLE DATABASE
IMMEDIATE) will cause a potential data loss failover.
Omitting the clause causes broker to validate that the standby
fi
les are current with the primary
(zero data loss failover).
-If data loss is detected (SCN in the header of the
fi
rst SYSTEM tablespace standby data
fi
le is
less than the corresponding SCN of the
fi
le in the primary) and IMMEDIATE has not been speci
fi
ed
the migrate command will fail. The most common reason is that the PDB in the primary is still
open, the PDB on the primary should be closed prior to attempting a failover.You must resolve the
SCN discrepancy or accept the data loss with the IMMEDIATE clause.
-For non-TDE enabled environments:
using '/<path>/<PDBNAME1>/<PDBNAME1.xml>' connect as <USERNAME>@”<dest-cdb-
connect-identifer>”;
-or-
using '/<path>/<PDBNAME1>/<PDBNAME1.xml>':
# This will prompt for user name and password
-For TDE enabled environments:
using '/<path>/<PDBNAME1>/<PDBNAME1.xml>' connect as <USERNAME>@”<dest-cdb-
connect-identifer>” secret “<some_value>” keystore identified by
��<destination_keystore_password>” keyfile ‘<path_filename>’ source keystore
identified by “<source_keystore_password>”;
-or-
using '/<path>/<PDBNAME1>/<PDBNAME1.xml>' secret <some_value> keystore
identified by “<destination_keystore_password>” keyfile ‘<path_filename>’
source keystore identified by “<source_keystore_password>”;
-SECRET is a value supplied to protect the keys for the PDB as part of the migration.
The keys are not accessible without the value for SECRET.
This value is only required for the duration of the migration process.

-Broker must export the TDE key for the PDB prior to extracting it from the source standby
database.
Broker requires a path and
fi
lename in which to store the keys, this location is speci
fi
ed in the
KEYFILE value.
-TDE requires the keystore password for both the source keystore and the destination keystore,
to extract from the source keystore and import into the destination keystore.
The destination keystore password is supplied using the KEYSTORE IDENTIFIED BY clause, the
source keystore password is supplied using the SOURCE KEYSTORE IDENTIFIED BY clause.
-If SECRET, KEYSTORE, KEYFILE or SOURCE KEYSTORE are not speci
fi
ed in the command
line, the migrate command will fail.
Once the connection to the destination is established the command will:
-Perform all necessary validations for the failover operation
-If TDE is enabled, export the TDES keys for the PDB from the source standby keystore
-Stop redo apply on the source standby if it is running
-Create the manifest on the standby at the location speci
fi
ed in the command using the
DBMS_PDB.DESCRIBE command
-Disable recovery of the PDB at the source standby
-If TDE is enabled, import TDE keys into the destination CDB keystore to allow the plugin to
succeed
-Plugin the PDB in the destination database using the standby's data
fi
les (NOCOPY clause) and
with STANDBYS=NONE.
-Open the PDB in all instances of the destination primary database
-If TDE is enabled, issue ADMINISTER KEY MANAGEMENT USE KEY in the context of the PDB
to associate the imported key and the PDB.
-Unplug the PDB from the source primary. If errors occur on unplug messaging is provided to
user to perform cleanup manually
-If unplug succeeds, drop the PDB from the source primary with the KEEP DATAFILES clause.
This will also drop the PDB in all of the source standby databases.
Once the command completes, add services for the PDB as desired and start those services to
resume connectivity to the PDB
-Backup the PDB in the destination CDB to allow for recovery going forward
-Follow the steps in Note 1916648.1 to enable recovery of the PDB at any standby databases to
establish availability and disaster recovery requirements.
Failover example:

Connect to the source database and show the PDBs and the current role. For
migration the source database must be the standby.
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
--- ---------- ---------- ----------
2 PDB$SEED READ ONLY NO
3 <PDBNAME1> READ ONLY NO
SQL> select database_role from v$database;
DATABASE_ROLE
----------------
PHYSICAL STANDBY
Connect to the broker command line on the host shared by instances of both the source
database (<CDB1STBY>) and the destination database (<CDB2>)
Password:
Connected to "<cdb1stby>"
Execute the migrate command to failover the PDB from cdb1stby to cdb2.
In the following command /home/oracle/cdb1stby is directly accessible to both cdb1stby and
cdb2 on desthost.
Using the IMMEDIATE command causes a potential data loss failover as this clause skips checks
to con
fi
rm that the standby copy of the PDB is consistent with the primary copy of the PDB.
DGMGRL> migrate pluggable database <pdb2> immediate to container cdb2 using
'/<path>/<cdb1stby>/<pdb2.xml>';
Username: USERNAME@"desthost/<cdb2>"
Password:
Connected to "<cdb2>"
Beginning migration of pluggable database <PDB2>.
Source multitenant container database is <cdb1stby>.
Destination multitenant container database is <cdb2>.

Connected to "cdb1"
Closing pluggable database <PDB2> on all instances of multitenant container
database <cdb1>.
Continuing with migration of pluggable database <PDB2> to multitenant
container database <cdb2>.
Stopping Redo Apply services on source multitenant container database
<cdb1stby>.
Succeeded.
Pluggable database description will be written to /<path>/<cdb1stby>/
<pdb2.xml>.
Closing pluggable database <PDB2> on all instances of multitenant container
database <cdb1stby>.
Disabling media recovery for pluggable database <PDB2>.
Restarting redo apply services on source multitenant container database
<cdb1stby>.
Succeeded.
Creating pluggable database <PDB2> on multitenant container database <cdb2>.
Opening pluggable database <PDB2> on all instances of multitenant container
database <CDB2>.
Unplugging pluggable database <PDB2> from multitenant container database
<cdb1>.
Pluggable database description will be written to /<path>/<pdb2.xml>.
Dropping pluggable database <PDB2> from multitenant container database
<cdb1>.
Unresolved plug in violations found while migrating pluggable database <PDB2>
to multitenant container database <cdb2>.
Please examine the PDB_PLUG_IN_VIOLATIONS view to see the violations that
need to be resolved.
Migration of pluggable database <PDB2> completed.
Succeeded
Best regards,
Alireza Kamrani.
https://www.linkedin.com/groups/8151826

Oracle PDB Failover in a Data Guard environment

Related slideshows

More Related Content

Oracle PDB Failover in a Data Guard environment