How to migrate a data protection relationship between two Protection Manager servers

[#1]

SCENARIO

A customer has an existing OnCommand 5.x Unified Manager server that runs Core services, Performance Advisor and Protection Manager (PM). Due to growth in their data protection environment, the customer elects to deploy a second PM server that will only be used only for PM services. They want to move some - but not all - data protection relationships from the current PM server to the new PM-only server. The customer has the following requirements:

• All data protection relationships being migrated must be kept in-tact. No re-baseline should be required.
• For migrated relationships, all existing backups created by the current PM server should be retained and capable of being used for restores.
• For migrated relationships, all existing backups created by the current PM server should expire according the backup policy and be deleted appropriately to reclaim space.

NOTE: In this scenario, the customer is electing to not restore the current DFM database to the new PM-only sever. It is a fresh install with a clean DFM database.

This document explains how to accomplish the customer's objectives in the above scenario. It only covers the migration process, not the instruction for building of the PM-only server.

PREPARATION CHECKLIST

Before migrating relationships between PM servers, one should perform a number of checks to ensure a smooth migration.

• Is hostname resolution working correctly on the new PM-only server?  Test both the hostname and FQDN of the storage controllers used in the data protection relationships.
• Have all of the storage controllers been discovered by the new PM-only server?  If not, add them now.
• Have the login credentials and transport protocols been specified for each storage controller in the new PM-only server? If not, specify them now.
• Have NDMP credentials been specified for each storage controller in the new PM-only server? If not, specify them now. Protection Manager cannot operate without proper NDMP credentials.
• It is highly recommended that you perform a "Diagnose Connectivity" on each and every storage controller on the new PM-only server. Verify that all protocols used pass their connectivity test. These are typically:  SSH, HTTP/HTTPS, NDMP, and SNMP (v1 or v3).
• Re-create any custom data protection policies, schedules, and throttles used by the data protection relationships. Compare the policies on both PM servers to verify they're identical.
• Create necessary resource pools.
• Verify all PM-specific server settings are identical between both servers.  For example, "dpMaxFanInRatio", "dpReaperCleanupMode" and "dpRebaselineMode".
• Copy any backup or fail-over scripts used by the existing PM server to the PM-only server. Verify they run correctly.
• Once all of the above have been addressed, give the PM-only server some time (15-30 minutes) and verify that it has discovered all of the existing data protection relationships. They will all be listed as "external relationships" in the PM-only server because it's not aware of the original PM server. If any relationships do not get discovered, run through the checklist again for the storage controllers in question.

MIGRATION STEPS

• Plan on migrating a single source:destination pair at a time. For example, if it's a VSM relationship, you'll only be migrating one relationship.  If its a SV relationship with multiple qtrees, you'll be migrating each relationship established between the source and destination volume in a single attempt.

NOTE:  If you have multiple source volumes & qtrees that fan-in to a single secondary volume, you would have to migrate all of the relationships at once. The reason is that we have to relinquish the secondary volume from the original dataset, so all relationships that rely on it must be migrated at the same time.

• Plan the migration so that it will not occur during a scheduled local or remote backup. Check the protection policy or dataset for the backup schedule. NOTE: Do not attempt to suspend the original dataset in order to avoid a backup window.
• If it doesn't exist already, plan ahead by creating an empty dataset on the PM-only server which will import the relationship. Create the dataset and do not assign any physical members to the primary node. Give it the correct protection policy, and do not assign any physical members to the backup node.  It should be a completely empty dataset.
• On the new PM-only server, verify that the relationship you're about to migrate is already shown as an "external relationship".  If not, review the preparation checklist again and resolve the issue.

1. On the original PM server, relinquish the data protection relationship(s) from the dataset that owns them. Do it in exactly this order:
   1. Relinquish each relationship that is part of the source:destination volume pair.  "dfpm dataset relinquish { [ <destination-volume-name-or-id> ] | [ <destination-qtree-name-or-id> ]"
   2. Edit the original dataset, and remove the primary volume from the primary node.
   3. Edit the original dataset, and remove the secondary volume from the backup/mirror node.
   4. Verify that the original PM server now shows the relationship(s) as an external relationship.
2. Do not delete the original dataset. Do not suspend the original dataset. The original dataset must remain to allow for restores from older backups. The original dataset also manages the retention & expiration of the local and remote backups created by the original dataset. Once all of the original backups have expired, then you can delete the dataset. NOTE: In order for the original dataset to perform restores and expire the backups, the original PM sever must still have connectivity to the controllers and volumes. If the customer's desire is to eventually remove the controllers from the original PM server, do not remove them until all of the original backups have expired and been deleted.
3. On the new PM-only server, import the relationship(s) into the appropriate dataset.
4. After importing the relationships, the dataset will show a status of Warning. When you investigate the warning, the dataset reports, "no backup history exists".  To clear this warning event, either run an on-demand backup job or wait for the next scheduled backup to occur. Either way, the backup job will force an update of the relationship and create a new backup snapshot. Once the new dataset creates the first backup for the imported relationship, the warning status will go away.

At this point, the relationship is now under the management of the new dataset on the PM-only server. It should run backups according to the dataset's protection policy and schedule. The original dataset on the original PM server will no longer be creating backups or updating the relationship(s). However, it will continue to expire and delete the backups it had created (on primary and secondary volumes) according to its protection policy. It will not expire or delete any new backups created by the new dataset because it won't be aware of them. The original PM server will continue to see the relationship, but only as an external relationship.

Based on the retention periods for your backups, set a calendar reminder to delete the original dataset once all of its backups have expired.  It will not be needed after that point.

NOTE:  Protection Policies have both a retention period AND a retention count for backups. Even if all backups have exceeded their retention period, Protection Manager will always retain [n] number of backups based on its retention count. On the original PM server, you may want to set the protection policy's retention count to 0 to ensure the original dataset deletes any/all backups it created. Be careful that this change doesn't impact other datasets using the same protection policy.

[#2]

This is really good. Thank you. I save all your posts. Many of these are tasks that would take hours or days on our own but much simpler having the example

[#3]

Many people have pointed out that you can accomplish this same task by restoring the existing DFM database to the new PM-only server, and deleting what you don't want. This is a valid method and it saves you a lot of time because all your protection policies and server settings get restored as well.  However, I have a real customer example where restoring the DFM database to the new PM server isn't an option. So we have to use the steps outlined above.

[#4]

The main disadvantages of using the import instead of db restore are the following.

1. We loose the dynamic referencing, and there by auto relationship creation of qtree(for QSM and SV) which are created after the dataset was created.
2. We also loose the ability of dynamic secondary sizing of the imported destination volumes as they are not marked dp_managed.
   • Though this could be over come using the Secondary Space Management feature by migrating the destination volumes.
3. We will have to juggle between two dfm server for restores
4. We will have unnecessary load on the old dfm server due to conformance and job
5. Also job will spawn and fail saying there are no member in the old dfm server.

[#5]

On import of existing PM relationships into a new PM server - why not allow imported relationships to be marked dp_managed?  Seems to be the perfect use case.

[#6]

Adai,

Good points. I don't envision this scenario for a long-term customer deployment.  It should be used for more of a short-term migration strategy (i.e. during a 6-month tech refresh of some hardware).

[#7]

Adai,

I have a question regarding dynamic referencing & DSS.  Assume a customer has a single Protection Manager server. Assume PM originally created a relationship and auto-provisioned the secondary volume from a resource pool. If we relinquish that relationship from the dataset, and later import the relationship back into a dataset (on the same PM server) will the dynamic referencing & DSS still be in-tact?  

[#8]

Hi Reid,

            Let me clarify two things here.

Dynamic referencing is applicable only for primary volumes, where as Dynamic Secondary Sizing is applicable only  for PM provisioned QSM and SV destination volumes. PM tracks this by using a flag and is applicable as long as you import the relationship into any dataset within the same server where this volume was provisioned.Whereas Dynamic referencing is specific to each dataset and is lost as soon as you move a relationship outside a dataset/import to another dataset within the same server.

To answer you specific question, Dynamic referencing will be lost where as DSS will be intact in your case.

Regards

adai

[#9]

BTW, a small clarification.Thanks to kevin, ryan for remaining the same.

Dynamic Secondary Sizing is applicable only  for PM provisioned QSM and SV destination volumes, until OC UM 5.0

Starting OC UM 5.1 DSS is applicable for VSM destination volumes as well. So the lone type of relationship for which PM doesn't create DSS enabled destination volume is OSSV destination volumes which remain the same as what we did in DFM version 3.5 for all kinds of relationship.(Aggr sized none guaranteed volumes)

Regards

adai