Invoke-NcSnapMirrorBreak - Another SnapMirror operation is in progress.

[MRoney]

I am trying to script the snapmirror failover process using PTK.

When I run the following set of commands,

# Update Existing SnapMirror and then Break it off
Invoke-NcSnapMirrorUpdate -Controller $DstCtrl -DestinationVserver $DstSVM -DestinationVolume $DstVol -ErrorAction stop
Watch-Command { Get-NcSnapMirror -Controller $DstCtrl -DestinationVserver $DstSVM -DestinationVolume $DstVol | Select Status } -Until "idle"
Invoke-NcSnapMirrorQuiesce -Controller $DstCtrl -DestinationVserver $DstSVM -DestinationVolume $DstVol -ErrorAction stop
Watch-Command { Get-NcSnapMirror -Controller $DstCtrl -DestinationVserver $DstSVM -DestinationVolume $DstVol | Select Status } -Until "quiesced"
Invoke-NcSnapMirrorBreak -Controller $DstCtrl -DestinationVserver $DstSVM -DestinationVolume $DstVol -Confirm:$false -ErrorAction stop

the Invoke-NcSnapMirrorBreak sometime returns with an error:

Invoke-NcSnapMirrorBreak : Another SnapMirror operation is in progress.
At Z:\Software\NetApp\Powershell Toolkit\Scripts\SnapMirror-Reverse-v3.ps1:119 char:1
+ Invoke-NcSnapMirrorBreak -Controller $DstCtrl -DestinationVserver $Ds ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (apfas8200:NcController) [Invoke-NcSnapmirrorBreak], EANOTHER_OP_ACTIVE
+ FullyQualifiedErrorId : ApiException,DataONTAP.C.PowerShell.SDK.Cmdlets.Snapmirror.InvokeNcSnapmirrorBreak

Why does it report another operation is in progress?  I just checked the status, and it was quiesced.

It does not fail every time it runs, just some of the time.  I have many snapmirrors where I perform this operation repeatedly.  Many work just fine and then others seem to fail nearly all the time.  I can sometimes just resume the snapmirror and start the script again and it runs without any issues on the same snapmirror.

Is there some code I can add to check and try again... like maybe 3 times before aborting?

Or better yet, does anyone know what I am not doing correctly to make sure the snapmirror is not performing another operation before I request the snapmirror break?

[aladd · Accepted solution]

I'm not seeing anything wrong with the script. But adding a 3 second wait time following each watch-command may be helpful.

incorporating Python may help.

https://docs.netapp.com/us-en/ontap-select/concept_api_before_python.html

and I found the below on creating a time delay in Python.

https://realpython.com/python-sleep/#adding-a-python-sleep-call-with-timesleep

[NetApp_SR]

I agree with aladd that adding a delay after the "Watch-Command" lines is good practice. Power shell has a built in command to add the delay. The scripts I see add 15-30 seconds.

Start-Sleep -s 15

[MRoney]

Thank you for the feedback.  I am adding the delay and will let you know how that worked out.

[Luis_KawaZ750]

Hi MRoney,

did you resolve with delay?

Thanks a lot.

Bye

[MRoney]

Unfortunately, the delay did not resolve the issue.  I ended up re-writing the code.  Below is the replacement code I used.  Note: I am not a great PowerShell programmer, so I am sure there is a bunch of this that could be improved, but we have not had any EANOTHER_OP_ACTIVE issues with this new code..... basically, it just keeps retrying until success.

# Update Existing SnapMirror and then Break it off
Wr G "";Wr G "Updating Existing Snapmirror:"
do {
$jobstatus = Invoke-NcSnapMirrorUpdate -Controller $DstCtrl -Destination $DstLoc -Source $SrcLoc -ONTAPI
sleep 5
Write-Host ($jobstatus | Format-Table | Out-String)
} until ($jobstatus.Status -eq 'succeeded')

do {
$smstatus = Get-NcSnapMirror -Controller $DstCtrl -Destination $DstLoc -ontapi
sleep 5
Write-Host ($smstatus | Format-Table | Out-String)
} until ($smstatus.MirrorState -eq 'snapmirrored' -and $smstatus.Status -eq 'idle')

Wr G "";Wr G "SnapMirror Status after Update:"

$smstatus = Get-NcSnapMirror -Controller $DstCtrl -Destination $DstLoc -ontapi
Write-Host ($smstatus | Format-Table | Out-String)

$worked = $false
while (-not $worked) {
try {
Invoke-NcSnapMirrorQuiesce -Controller $DstCtrl -Destination $DstLoc -ONTAPI -ErrorAction Stop
$worked = $true # An exception will skip this
} catch {
Write-Host ("Error:", $_)
Write-Host ("Retrying...")
sleep 5
}
}
do {
$smstatus = Get-NcSnapMirror -Controller $DstCtrl -Destination $DstLoc -ontapi
sleep 5
Write-Host ($smstatus | Format-Table | Out-String)
} until ($smstatus.MirrorState -eq 'snapmirrored' -and $smstatus.Status -eq 'quiesced')

Wr G "";Wr G "SnapMirror Status after Quiesce:"

Get-NcSnapMirror -Controller $DstCtrl -Destination $DstLoc -ontapi

$worked = $false
while (-not $worked) {
try {
Invoke-NcSnapMirrorBreak -Controller $DstCtrl -Destination $DstLoc -Confirm:$false -ONTAPI -ErrorAction Stop
$worked = $true # An exception will skip this
} catch {
Write-Host ("Error:", $_)
Write-Host ("Retrying...")
sleep 5
}
}
do {
$smstatus = Get-NcSnapMirror -Controller $DstCtrl -Destination $DstLoc -ontapi
sleep 5
Write-Host ($smstatus | Format-Table | Out-String)
} until ($smstatus.MirrorState -eq 'broken-off' -and $smstatus.Status -eq 'idle')

do {
$smstatus = Get-NcSnapMirror -Controller $DstCtrl -Destination $DstLoc -ontapi
sleep 5
Write-Host ($smstatus | Format-Table | Out-String)
} until ($smstatus.MirrorState -eq 'broken-off' -and $smstatus.Status -eq 'idle')
Wr G "";Wr G "SnapMirror Status after Break:"

Get-NcSnapMirror -Controller $DstCtrl -Destination $DstLoc -ontapi

[Luis_KawaZ750]

Hi MRoney,

I'll try your solution!

Thanks a lot and happy new year!

Bye