VSC with SVM-DR / Snapshot cannot age due "database record owner"

[pg2aude]

Hey folks,

we have some troubles with Snapshots of VM-Datastore via VSC in combination with SVM-DR relationship of the NFS-Datastore.

Snapshots cannot be deleted as configured in the vsc backup job because some snapshots have an record-owner entry. 

CL1::*> snapshot show -volume ESXxx -fields owners,busy,record-owner,create-time
vserver volume      snapshot                         create-time              busy  owners record-owner
------- ----------- -------------------------------- ------------------------ ----- ------ ----------------------------------------------------------------------
NFS1 ESXxx smvi__ESXxx_20180124213000 Wed Jan 24 21:43:27 2018 false -      snapmirrorDP.7eb76d1189f2e6119fd100a098c68937_2149105432.nonSMDstOwned
NFS1 ESXxx smvi__ESXxx_20180201213000 Thu Feb 01 21:43:06 2018 false -      snapmirrorDP.7eb76d1189f2e6119fd100a098c68937_2149105432.nonSMDstOwned
NFS1 ESXxx smvi__ESXxx_20180214213000 Wed Feb 14 21:43:06 2018 false -      snapmirrorDP.7eb76d1189f2e6119fd100a098c68937_2149105432.nonSMDstOwned
NFS1 ESXxx smvi__ESXxx_20180301213001 Thu Mar 01 21:43:16 2018 false -      -
[...]
NFS1 ESXxx smvi__ESXxx_20180329213001 Thu Mar 29 21:41:46 2018 false -      -
NFS1 ESXxx smvi__ESXxx_20180330213000 Fri Mar 30 21:42:16 2018 false -      snapmirrorDP.7eb76d1189f2e6119fd100a098c68937_2149105432.nonSMDstOwned
NFS1 ESXxx smvi__ESXxx_20180402213000 Mon Apr 02 21:41:26 2018 false -      -
NFS1 ESXxx smvi__ESXxx_20180403213001 Tue Apr 03 21:42:16 2018 false -      -
NFS1 ESXxx smvi__ESXxx_20180404213000 Wed Apr 04 21:53:26 2018 false -      snapmirrorDP.7eb76d1189f2e6119fd100a098c68937_2149105432.nonSMDstOwned
NFS1 ESXxx smvi__ESXxx_20180405213000 Thu Apr 05 21:42:06 2018 false -      -
NFS1 ESXxx smvi__ESXxx_20180406213000 Fri Apr 06 21:42:06 2018 false -      -
NFS1 ESXxx smvi__ESXxx_recent         Mon Apr 09 21:42:54 2018 false -      -
NFS1 ESXxx vserverdr.0.116db77e-f289-11e6-9fd1-00a098c68937.2018-04-10_080500
                                                     Tue Apr 10 08:05:02 2018 false -      VserverDR,snapmirrorDP.7eb76d1189f2e6119fd100a098c68937_2149105432.base
32 entries were displayed.

CL1::*> snapshot show -volume ESXxx -snapshot smvi__ESXxx_20180124213000 -instance

                              Vserver: NFS1
                               Volume: ESXxx
                             Snapshot: smvi__ESXxx_20180124213000
                 Snapshot Data Set ID: 914828035119
          Snapshot Master Data Set ID: 916977139480
                        Creation Time: Wed Jan 24 21:43:27 2018
                        Snapshot Busy: false
                       List of Owners: -
                        Snapshot Size: 319.6GB
           Percentage of Total Blocks: 3%
            Percentage of Used Blocks: 5%
              Consistency Point Count: 3352386
                              Comment: -
                  File System Version: 9.0
             File System Block Format: 64-bit
                     Physical Snap ID: 213
                      Logical Snap ID: 213
                Database Record Owner: snapmirrorDP.7eb76d1189f2e6119fd100a098c68937_2149105432.nonSMDstOwned
                        Snapshot Tags: VOPL_owner_snapmirrorDP.
                                       7eb76d1189f2e6119fd100a098c68937_
                                       2149105432.nonSMDstOwned=snapmirror
                        Instance UUID: b3fea069-1938-4c7f-8ef2-670eb364894f
                         Version UUID: b3fea069-1938-4c7f-8ef2-670eb364894f
                      7-Mode Snapshot: false
      Label for SnapMirror Operations: -
                       Snapshot State: -
                 Constituent Snapshot: false
                                 Node: CL1_N1
               AFS Size from Snapshot: 4.86TB
    Compression Savings from Snapshot: 1.78TB
          Dedup Savings from Snapshot: 2.94TB
       VBN Zero Savings from Snapshot: 199.4GB
             Snapshot Inofile Version: 4
                          Expiry Time: -
                     Compression Type: adaptive
                 SnapLock Expiry Time: -

Due to this entry, we cannot delete the snapshot in VSC, SystemManager and CLI (*force should work, but the vsc will not know about the deletion).

We are looking for a way to delete the record-owner entry or to prevent the vm backup process (VSC+SVM-DR) to create snapshots with stuck Database record owner.

Is there anyone out there, who already had such an behaviour? And maybe an solution? 

Thanks for any hints,

regards,

Pat

[GidonMarcus]

Hi Pat

i suggest you open a case with NetApp to try and find why the snapshot got stuck.

they should easily search in the syslog and snapmirorr.log for the snapshot name and figure out their history (you can also do it yourself in Active IQ. but they can maybe correlate it to a bug based on internal database faster).

once it found they could give you a full action plan for recovery. including when to force the deletions and what should be done before  / after.

Gidi

[pg2aude]

Hey Gidi,

we opened a case and will update the thread then the issue is solved.

Thanks.

Pat

[Giagnorio]

Hello,

if you check the IMT you will find out that the combination SVM-DR and VSC is not Supportet!

regards Piero