Re: Delete broken off snapmirror relationship after vFiler failover / reverse resync

BARMENIAIT · ‎2013-03-14

Hi!

Two FAS2240, vFiler (one volume: vol_test) on FAS1 and it's secondary/DR backup on FAS2. I stop the vFiler on FAS1, activate the vFiler von FAS2, then reverse resync. So basically I'm doing a failover while switching primary/secondary.

Now there's two snapmirror relationships for vol_test, one FAS2 -> FAS1 (active) and another one FAS1 -> FAS2 (broken off).

How do I delete the broken off relationship? Remove-Snapmirror returns Errors like "Destination not in snapmirrored state" (well, obviously!) or "No release-able destination found that matches those parameters" (strange, because Get-Snapmirror lists the broken off relationship).

I read something about deleting base snapshots but I'd like to avoid messing with them because I think the volumes still share the same base snapshot (remember I just reversed the sync direction, the data remained pretty much the same).

Thanks in advance!

Best regards

Alex

bsti · ‎2013-03-14

Hi there,

You have to remove two things to get the broken-off relationship to go away:

1) Remvoe the snapmirror.conf entry for the older SM relationship (if it exists)

2) You do need to remove the base snapshots associated with the old SM. They are pretty easy to spot. Look at teh snapshot list and you will see the base snapshots are named specifically:

[dest controler name](system_id)_[vol name].[number

The snapshots associated with the old (broken-off) SM relationship should be named:

FAS2(sysid)_vol_test.xxx

The new base snapshots should be: FAS1(sysid)_vol_test.xxx.

You should be able to confirm by the snapshot dates that the FAS1 snapshots are older. These need to be manually removed. Once they are, the SM relationship should go away. The Powershell cmdlet that usually does this for you is Invoke-NaSnapmirrorRelease. You can try that first, and it does not work, manually remove them.

Hope that helps.

EDIT- I had the snapshot names backward above in my original post. I just corrected that.

bsti · ‎2013-03-14

I forgot to comment on your not wanting to delete the old base snapshots. For one, nothing will ever remove those automatically. Since you reversed the snapmirror, they are now marooned out there until you remove them manually. So you have to do it eventually. For two, when you reversed the snapmirorr, a new set of baseline snapshots should have been created with FAS1(sysid)_... probably with new numbering starting at 0 or 1 again. THOSE are the new baselines you don't want to touch. You should be able to easily see both sets.

BARMENIAIT · ‎2013-03-14

Okay so removing the snapmirror relationship on FAS1 (old source) was no problem with

Invoke-NaSnapmirrorRelease -Source fas1:vol_test -Destination fas2:vol_test -Controller (Connect-NaController fas1.playground.local -Credential $creds) -Confirm:$false

but how can I remove the sm relation on FAS2? It's still there, no snapmirror.conf entry and commands like Invoke-NaSnapmirrorRelease (Error: No release-able destination found that matches those parameters.) or Remove-NaSnapmirror (Error: Snapmirror not found) don't work.

I'm still not sure what snapshot to delete.

PS C:\> Get-NaSnapshot -TargetName vol_test -Controller (Connect-NaController fas1.playground.local -Credential $creds)

Name Created Total Cumulative Dependency

---- ------- ----- ---------- ----------

fas1(xxxxxxxxxx)_vol_test.94 14.03.2013 84,0 KB 84,0 KB

fas1(xxxxxxxxxx)_vol_test.95 14.03.2013 0 0

hourly.0 14.03.2013 316,0 KB 400,0 KB

hourly.1 13.03.2013 220,0 KB 856,0 KB

hourly.2 13.03.2013 240,0 KB 1,1 MB

hourly.3 13.03.2013 260,0 KB 1,3 MB

hourly.4 13.03.2013 308,0 KB 1,6 MB

hourly.5 13.03.2013 224,0 KB 1,8 MB

nightly.0 13.03.2013 236,0 KB 636,0 KB

nightly.1 12.03.2013 260,0 KB 2,1 MB

PS C:\> Get-NaSnapshot -TargetName vol_test -Controller (Connect-NaController fas2.playground.local -Credential $creds)

Name Created Total Cumulative Dependency

---- ------- ----- ---------- ----------

fas1(xxxxxxxxxx)_vol_test.95 14.03.2013 200,0 KB 200,0 KB snapmirror

hourly.0 14.03.2013 316,0 KB 516,0 KB

hourly.1 13.03.2013 220,0 KB 972,0 KB

hourly.2 13.03.2013 240,0 KB 1,2 MB

hourly.3 13.03.2013 260,0 KB 1,4 MB

hourly.4 13.03.2013 308,0 KB 1,7 MB

hourly.5 13.03.2013 224,0 KB 2,0 MB

nightly.0 13.03.2013 236,0 KB 752,0 KB

nightly.1 12.03.2013 260,0 KB 2,2 MB

bsti · ‎2013-03-14

Can we see the output of:

Get-NaSnapmirrorDestination

or

snapmirror destinations -s

THis should give us a clue which snapshots are still holding the SM relationship.

BARMENIAIT · ‎2013-03-15

PS E:\> Get-NaSnapmirrorDestination -Controller (Connect-NaController fas1.playground.local)

DestinationLocation SourceLocation

------------------- --------------

PS E:\> Get-NaSnapmirrorDestination -Controller (Connect-NaController fas2.playground.local)

DestinationLocation SourceLocation

------------------- --------------

fas1:vol_test fas2:vol_test

fas1:vol_test->fas2:vol_test fas2:vol_test

(At the moment, fas2 is Primary and fas1 is DR backup.)

Strangely, it seems if you wait like 20 minutes, it will go away by itself, like there was some kind of garbage collector for broken off sm relationships. Will keep an eye on that to confirm.

Edit:

Okay, so after some time:

PS E:\> Get-NaSnapmirrorDestination -Controller (Connect-NaController fas2.playground.local)

DestinationLocation SourceLocation

------------------- --------------

fas1:vol_test fas2:vol_test

Still, there's a sm relationship visible on fas2 in OnCommand System Manager.

Now after sending the command

PS E:\> Invoke-NaSnapmirrorRelease -Source fas1:vol_test -Destination fas2:vol_test -Controller (Connect-NaController fas2.playground.local)

the sm relationship instantly disappears from OCSM on fas2 and everything is as it should be.

Who can explain that to me?

bsti · ‎2013-03-15

Glad you resolved it. Yes, sometimes there is a delay. The snapmirror process checks every few seconds for updates and takes a moment to register changes to the snapmirror.conf file and when relationships are added or removed.

Invoke-NaSnapmirrorRelease -Source fas1:vol_test -Destination fas2:vol_test -Controller (Connect-NaController fas2.playground.local)

the sm relationship instantly disappears from OCSM on fas2 and everything is as it should be.

When you run a release, you execute it on the source controller of the SM relationship you want to remove. When you ran it the first time, you ran it against FAS1, which you said successfully cleared the relationship on FAS1. When you ran it against FAS2, that cleared up the relationship on FAS2. All release does to my knowledge is remove the base snapshots tagged to that SM relationship. When you ran release the last time, it must have deleted the holdout snapshot for that old relationship. I can't figure out exactly which one that was given your output above, but if you look at the snapshot list again now you should see one missing. Normally, you only have to run release one time against the source controller, then on the destination controller delete the old destination volume or delete the baseline snapshots tied to the SM relationship.

The only two things that don't add up to me are:

1) Your get-snapmirrordestinations should have also exposed a SourceSnapShot column. I see that when I run it. Did you see it and just not print it? That was really what I was looking for.

2) Your get-snapshot outputs above don't appear to have had the baseline snapshot I was expecting. When you did a get-nasnapshot on FAS2, I really expected to see a FAS2(xxxx) snapshot. My only guess is that another snapshot became your baseline when you synched back. I don't do that very often so I don't recall exactly how that behaves.

BARMENIAIT · ‎2013-03-15

Invoke-NaSnapmirrorRelease -Source fas1:vol_test -Destination fas2:vol_test -Controller (Connect-NaController fas2.playground.local)

the sm relationship instantly disappears from OCSM on fas2 and everything is as it should be.

I forgot to mention the important part: If you send the above command it FAILS (Invoke-NaSnapmirrorRelease : SnapMirror error: No release-able destination found that matches those parameters.), yet still the connection disappears on fas2. Very weird.

1) Your get-snapmirrordestinations should have also exposed a SourceSnapShot column. I see that when I run it. Did you see it and just not print it? That was really what I was looking for.

I postet the exact output that was given to me. Maybe some parameter missing?

2) Your get-snapshot outputs above don't appear to have had the baseline snapshot I was expecting. When you did a get-nasnapshot on FAS2, I really expected to see a FAS2(xxxx) snapshot. My only guess is that another snapshot became your baseline when you synched back. I don't do that very often so I don't recall exactly how that behaves.

I could post the results of get-NaSnapshot and get-NaSnapmirrorDestination after another failover (i.e. making fas1 primary again with fas2 as DR backup).