Protection Manager repeatedly deletes external SV relationship

[reide]

Running DFM 4.0.2

I imported an external SnapVault relationship into a dataset.  I later removed the relationships from the dataset and made them external again. 

The first odd thing I noticed was when I removed the relationships from the dataset. Protection Manager said it was going to delete the relationships (!!!!).  dpReaperCleanupMode is set to Orphans.  It is my understanding that a setting of "Orphans" will only delete orphaned relationships that had never been imported into a dataset. However, in this case it deleted my manually-created SnapVault relationship that was imported into and exported out of my Dataset. Thanks to Protection manager, there is no SnapVault relationships at this point.

So I manually re-created the exact same SnapVault relationship on the storage controller.  After a period of time, my manually-created SnapVault relationship disappears. I suspect that Protection Manager saw the exact same SnapVault relatiopnship that it had deleted previously and decided to delete it again.

First, why is Protection Manager deleteing this orphaned relationship that HAD been imported into a Dataset?

Second, why does Protection manager keep deleting the relationship when I manually re-create it on the controller?   How do I prevent it from doing this for this external relationship?

[adaikkap]

Hi Reide,

Can you get the output of the following commands ?

Dfm options list | findstr /I dp

Dfpm relationship list –x and highlight the deleted ones.

Did you change any other options ?

Having dfm diag output from the server will help.

Regards

adai

[reide]

Adai,

The "dfm diag" output is attached to this reponse as a text file.

I did set the dpReaperInterval to a much lower value so it cleans-up faster in my lab environment.  All other "dp" settings are the  defaults. 

dpDynamicSecondarySizing              Enabled

dpMaxFanInRatio                       1

dpReaperCleanupMode                   Orphans

dpReaperInterval                      5 minutes

dpReBaselineMode                      Confirm

dpRestoreTransfersPerHost             8

dfpm relationship list -x doesn't currently show the deleted relationships.  The relationship does not currently exist on the controllers because it keeps getting deleted.

dfpm relationship list -a DOES show the deleted relationships.

Relationship Id Relationship Type    Dataset Id Dataset Name         Source                                   Destination                              Deleted Deleted By

--------------- -------------------- ---------- -------------------- ---------------------------------------- ---------------------------------------- ------- ------------

            242 snapvault                     0                      chi-ntap-01:/vol0/-                      kc-ntap-03:/Test/Test                    Yes

            249 qtree_snapmirror              0                      chi-ntap-01:/vol0/-                      kc-ntap-03:/Test/My_Dataset              Yes

            323 snapvault                     0                      chi-ntap-01:/cifs_vol_1/qtree_y          kc-ntap-03:/cifs_backup/qtree_y          Yes

            325 snapvault                     0                      chi-ntap-01:/cifs_vol_1/-                kc-ntap-03:/cifs_backup/cifs_backup      Yes

            327 snapvault                     0                      chi-ntap-01:/cifs_vol_1/qtree_x          kc-ntap-03:/cifs_backup/qtree_x          Yes

            329 snapvault                     0                      chi-ntap-01:/cifs_vol_1/qtree_z          kc-ntap-03:/cifs_backup/qtree_z          Yes

            337 snapvault                   331 Local CIFS Backup    chi-ntap-01:/cifs_vol_1/qtree_z          kc-ntap-03:/cifs_backup_1/qtree_z        No

            339 snapvault                   331 Local CIFS Backup    chi-ntap-01:/cifs_vol_1/qtree_x          kc-ntap-03:/cifs_backup_1/qtree_x        No

            341 snapvault                   331 Local CIFS Backup    chi-ntap-01:/cifs_vol_1/qtree_y          kc-ntap-03:/cifs_backup_1/qtree_y        No

            343 snapvault                   331 Local CIFS Backup    chi-ntap-01:/cifs_vol_1/-                kc-ntap-03:/cifs_backup_1/cifs_backup_cifs_vol_1 No

            352 snapvault                   345 Local LUN Backups    chi-ntap-01:/lun_vol_1/-                 kc-ntap-03:/lun_backup/lun_backup_lun_vol_1 No

            354 snapvault                   345 Local LUN Backups    chi-ntap-01:/lun_vol_1/lun2              kc-ntap-03:/lun_backup/lun2              No

            356 snapvault                   345 Local LUN Backups    chi-ntap-01:/lun_vol_1/lun1              kc-ntap-03:/lun_backup/lun1              No

            365 snapvault                     0                      chi-ntap-01:/cifs_vol_2/qtree_u          kc-ntap-03:/cifs_backup_2/qtree_u        Yes

            366 snapvault                     0                      chi-ntap-01:/cifs_vol_2/qtree_v          kc-ntap-03:/cifs_backup_2/qtree_v        Yes

            367 snapvault                     0                      chi-ntap-01:/cifs_vol_2/qtree_w          kc-ntap-03:/cifs_backup_2/qtree_w        Yes

            381 snapvault                     0                      chi-ntap-01:/cifs_vol_3/qtree_r          kc-ntap-03:/cifs_backup_3/qtree_r        No

            382 snapvault                     0                      chi-ntap-01:/cifs_vol_3/qtree_s          kc-ntap-03:/cifs_backup_3/qtree_s        No

            383 snapvault                     0                      chi-ntap-01:/cifs_vol_3/qtree_t          kc-ntap-03:/cifs_backup_3/qtree_t        No

[rshiva]

Hi Reide,

You mentioned that you removed the relationships from the dataset and made them external again. How exactly did you do that? ( I guess I know how you did that, but just trying to make sure)

Thanks and regards

Shiva Raja

[reide]

Shiva,

I edited the dataset's primary physical resources and removed the qtrees for the desired relationships. I then edited the Dataset's secondary physical resources and removed the SV secondary volume.

NOTE: For these particular relationships, all the primary qtrees were SnapVaulting to the same secondary volume.  The secondary volume was dedicated to these qtrees and nothing else.  See earlier output of "dfpm relationship list -a".  The secondary volume was  chi-ntap-01:/cifs_vol_2

After removing them from the Dataset,  Protection Manager moved those relationships to the External Relationships page.  Shortly thereafter, the relationships were deleted by Protection manager.

[rshiva · Accepted solution]

Hi Reide,

When removing a relationship from a dataset (irrespective of the fact whether the relationship is imported or created by protection manager) always relinquish the relationship from the dataset, then remove the primary followed by the secondary members - so here are the brief steps:

Step - a: Relinquish the relationship from the dataset, so that protection manager can mark the relationship as external

                Use the command: "dfpm dataset relinquish <secondary qtree or volume id>"

Step-b: Remove the primary member (just like what you did)

Step-c: Remove the secondary member (just like what you did)

If you follow these steps, PM would never delete the relationships and you don't have to worry about the dpReaperCleanUpMode option.

Thanks and regards

Shiva Raja

[adaikkap]

Hi Shiva/Earls,

                 I suspect a regression. As per the functionality of reaper clean up mode this what its supposed to do when its value is orphan.

The default state for dpReaperCleanupMode will be Orphans which will only allow the reaper process to delete orphan relationships from non-imported relationships that are no longer in a dataset.

Imported relationship are not removed by repear cleanup mode except when its value is Automatic.

Regards

adai

[reide]

Shiva,

Thanks.  The worked as I expected.  The relationship immediately shows-up under "External Relationships" and I didin't see a message stating that it was going to delete anything.

[dscarbrough]

Interesting thread.  I'm running into a similiar issue with one of my customers that is a bit confusing.

Is there any way if you did not dfpm relinquish as the first step to clean up after the fact when the datasets no longer exist?  I'm seeing all of the deleted relationships with dfpm relationship list -a on my problem.  But, every time we recreate the snapmirrors, within 24 hours or less, the relationships disappear and the snap mirror.conf is edit'd by PM and removes all reference relationships. 

Repear is set to Orphan.

I'm looking to see if there is a deeper method to removing these orphans manually before Reaper comes along and whacks the relationship?  Will a brandnew baseline with new volumes (same name) work?  Because we're going across a WAN, I'm trying to avoid doing this unless absolutely necessary.

I'm hoping for a manual process involving dfpm commands.  Relinquish doesn't find anything to remove when I give it the previously deleted paths (datasets are long gone)

Thanks.