"snapmirror list-destinations" does not show recently created SnapVault relationships

JimRobertson · ‎2017-04-10

I am working with a customer that has three sites. Two are Snapmirrored hot/cold (A/B) and the third is a Snapvault destination (C). They want to be able to failover from A to B, reverse the Snapmirror to go from B to A, and then re-source the Snapvault to pull from the new prod data center (in this case, B).

After doing the failover, I

Delete the Snapvault relationship
Release it from the original source
Create a new relationship from the new prod data center
Resync the Snapvault

Once I do all of this, I can look at the Snapmirror relationships on the destinations and they appear as expected. I can run updates with no problem. But, if I run "snapmirror list-destinations" from the new source (B), it shows ONLY the Snapmirror relationships, and not the Snapvaults. Is this a bug? Does it just take time for the source to update from the destination? I've waited over an hour, but the list-destinations is still not showing the Snapvaults.

All three sites are on OnTap 9.0P2.

robinpeter · ‎2017-04-10

Restrictions/Limitations

The snapmirror show command will display values only for the following fields for relationships with "Relationship Capability" of "Pre 8.2", when run on the source cluster of a cross-cluster relationship: source-path, source-cluster, source-vserver, source-volume, destination-path, destination-cluster, destination-vserver, destination-volume, type, status, state, is-constituent, relationship-capability. You must issue the snapmirror show command on the destination cluster to have complete information about SnapMirror relationships.

Click Here for more info.

I did not find the above restrictions in Ontap 9.0 Documentation.

JimRobertson · ‎2017-04-11

Thanks, Robin. The relationship capability is "8.2 and above". Since I posted this yesterday, the Snapvault relationships are now showing up when I run the list-destinations. So, it took several hours for the source to get updated to what was happening on the destination. I wonder what updates this information, and whether or not I can force a manual update?

This is what I am seeing. The XDP relationships on "source-cluster-01" are the ones that were not showing up yesterday.

source-cluster-01::> snapmirror list-destinations -source-volume Volume*
Progress
Source Destination Transfer Last
Path Type Path Status Progress Updated
----------- ----- ------------ ------- --------- ------------
SOURCE_SVM:Volume XDP SV_SVM:Volume_SV Idle - -
DP SM_SVM:Volume Idle - -
SOURCE_SVM:Volume1 DP SM_SVM:Volume1 Idle - -
XDP SV_SVM:Volume1_SV Idle - -
4 entries were displayed.

snapvault-cluster-01::> snapmirror show -destination-path SV_SVM:Volume* -fields type,destination-path,state,status,total-progress,healthy,policy,schedule

source-path destination-path type schedule policy state status total-progress healthy
----------------- -------------------- ---- -------- ------------ ------------ ------ -------------- -------
SOURCE_SVM:Volume SV_SVM:Volume_SV XDP daily 30Day_SV Snapmirrored Idle - true
SOURCE_SVM:Volume1 SV_SVM:Volume1_SV XDP daily 30Day_SV Snapmirrored Idle - true
2 entries were displayed.

snapvault-cluster-01::> snapmirror show -destination-path SV_SVM:Volume_SV -instance

Source Path: SOURCE_SVM:Volume
Destination Path: SV_SVM:Volume_SV
Relationship Type: XDP
Relationship Group Type: none
SnapMirror Schedule: daily
SnapMirror Policy Type: vault
SnapMirror Policy: 30Day_SV
Tries Limit: -
Throttle (KB/sec): unlimited
Mirror State: Snapmirrored
Relationship Status: Idle
File Restore File Count: -
File Restore File List: -
Transfer Snapshot: -
Snapshot Progress: -
Total Progress: -
Network Compression Ratio: -
Snapshot Checkpoint: -
Newest Snapshot: daily.2017-04-11_0010
Newest Snapshot Timestamp: 04/11 00:10:00
Exported Snapshot: daily.2017-04-11_0010
Exported Snapshot Timestamp: 04/11 00:10:00
Healthy: true
Unhealthy Reason: -
Constituent Relationship: false
Destination Volume Node: snapvault-cluster-01-01
Relationship ID: ####################
Current Operation ID: -
Transfer Type: -
Transfer Error: -
Current Throttle: -
Current Transfer Priority: -
Last Transfer Type: update
Last Transfer Error: -
Last Transfer Size: 14.81KB
Last Transfer Network Compression Ratio: 1:1
Last Transfer Duration: 0:17:7
Last Transfer From: SOURCE_SVM:Volume
Last Transfer End Timestamp: 04/11 00:32:07
Progress Last Updated: -
Relationship Capability: 8.2 and above
Lag Time: 10:16:18
Identity Preserve Vserver DR: -
Volume MSIDs Preserved: -
Number of Successful Updates: 4
Number of Failed Updates: 0
Number of Successful Resyncs: 1
Number of Failed Resyncs: 0
Number of Successful Breaks: 0
Number of Failed Breaks: 0
Total Transfer Bytes: 15168
Total Transfer Time in Seconds: 1029

robinpeter · ‎2017-04-11

Some of the SnapMirror relationship information is cached. The snapmirror show command only returns the cached information, therefore there is a delay after the information is changed before it is reflected in the snapmirror show output. Other information, such as progress metrics during a transfer, is only updated periodically and can be very delayed in the snapmirror show output.

I don't know there is any way to force them to update faster. I'll look around.

If i find anything ill post here.

robin.

JimRobertson · ‎2017-04-12

Thanks, Robin. This is affecting our failover process, so I opened a ticket for it. If I get any definitive answers back from the ticket, I'll post them here.

xandervanegmond · ‎2017-04-12

Hi Jim,

Have you looked what happens when you use SVM_DR to replicate between A and B?

If you use personality preserve, you should be able to keep XDP alive when you failover.

I'll be honest, havent tested it myself, but that might be a better solution.

/Xander

JimRobertson · ‎2017-04-12

SVM_DR is definitely on our radar, but not something we can switch to quickly. Depending on what I find out about this issue, though, I may have to start pushing for it sooner rather than later. Thanks for the suggestion, Xander!

JimRobertson · ‎2017-05-01

In case anyone else was looking at this: I worked with support, and we discovered that the "list-destinations" does not update until Snapvault transfers actual changed data. So, once you change the source of the Snapvault relationship, the new source will not be aware of the relationship unitl a scheduled snapshot is taken and transferred to the destination. I have not tested this, but support said that if you manually create a snapshot with the same Snapmirror label as the other Snapvault snapshots, then do a Snapmirror update, the relationship will be updated at the source.

Manually adding an additional snapshot would reduce our required snapshot retention, however, so we are still discussing other solutions.

kamran1024 · ‎2017-07-18

Nothing constructive to add apart from thank you very much! Have been been scracthing my head all afternoon until I read this.

JimRobertson · ‎2017-07-19

I'm glad to see that I saved at least one person from having to try to figure this out

kamran1024 · ‎2017-07-19

Absolutely! The test of creating a 'test snap' with the correct snapvalt snap-label worked btw. i.e. created a snap on the source, ran snapmirror resync (type XDP) from the destination, and was able to see and confirm the relationship via list-destination on the source. Perfect!

Just wish I'd found this discussion earlier.

Thanks!