Powershell NDU Invoke-NcClusterHaTakeover -AllowVersionMismatch not working as expected

[nathane]

When upgrading OnTap minor release versions (P1 -> P5), the following cmdlet works as expected:

Invoke-NcClusterHaTakeover -Controller $connectedController -Node $controllerBNode

When upgrading OnTap major release versions (9.2 -> 9.3), the following cmdlet produces the following error message:

Failed to initiate takeover. Reason: version mismatch. Note: If you are performing a Data ONTAP non-disruptive upgrade or downgrade, use the (privilege: advanced) 
"-option allow-version-mismatch" parameter. 

This is all well and good and so I add AllowVersionMismatch switch parameter as follows:

Invoke-NcClusterHaTakeover -Controller $connectedController -Node $controllerBNode -AllowVersionMismatch

But receive the same error message:

Failed to initiate takeover. Reason: version mismatch. Note: If you are performing a Data ONTAP non-disruptive upgrade or downgrade, use the (privilege: advanced) 
"-option allow-version-mismatch" parameter. 

I'm hoping it is obvious what I missing and/or how I elevate to "set -privilege advanced" from the Netapp Powershell CLI.  I have considered performing this takeover step with Invoke-NcSSH, but I am hoping there is a more elegant way and that I am just missing a step or something obvious.  (Invoke-NcSsh option mentioned here: https://community.netapp.com/t5/Software-Development-Kit-SDK-and-API-Discussions/Do-we-have-any-DataONTAP-powershell-toolkit-cmdlet-to-set-privilege-l...)

The parameter help talks about cleanly halting the node to be taken over but this doesn't quite make sense to me - is it a clue to someone else?  I am not familiar with proatively halting a node during a NDU:

    -AllowVersionMismatch
        If specified, allow takeover if the partner was running an incompatible operating system version and was cleanly halted.  This option is used for non-disruptive upgrade (NDU).

Thanks in advance for your insight!

[mbeattie]

Hi,

Did you try using "Invoke-NcSystemApi" using the "cf-failover" ZAPI?

Alternately you could use Invoke-NcSsh and pass mulitple commands using ampersand...EG:

[String]$command = "set -privilege advanced`;storage failover takeover -ofnode cluster1-02"
Try{
   Invoke-NcSsh -Command $command -Credential $credentials -ErrorAction Stop
   Write-Host "Executed Command`: $command"
}Catch{
   Write-Warning -Message $("Failed Executing Command`: $command. Error " + $_.Exception.Message)
   Break;
}

Whilst that might be a workaround it's not really a solution and it does sound like a bug to me. I'll raise the issue with our developers for you. Thanks

/Matt

[nathane]

Thanks for the quick reply Matt!

I should have mentioned in the first place I was using the Powershell Toolkit 4.3 and the issue was previously present in 4.0 -- I will see if 4.5 fixes the issue and report back when our next set of controllers comes up for upgrades (it will be a few weeks).

I'm not terribly familiar with the ZAPI but I came up with the following block - I won't be able to test on an HA cluster for a few weeks:

$Request = @"
<cf-takeover>
<node>$controllerBNode</node>
<version-mismatch-ok>true</version-mismatch-ok>
</cf-takeover>
"@
Invoke-NcSystemApi -Controller $connectedController -Request $Request

Does that look reasonable?

Unfortunately, Invoke-NcSsh isn't the best solution at the moment as it appears cDot 9.3 may affected by the same bug as 7mode 8.2.5 as discussed here here: http://community.netapp.com/t5/Microsoft-Cloud-and-Virtualization-Discussions/Invoke-NaSsh-not-working-anymore-with-DOT-8-2-5-7-mode/m-p/135824/highli...

(I replied to the Netapp empoyee mentioning the bug about this detail in case the bug should be expanded to include cDot - I upgraded to PSTK 4.5 and the issue persists)

The upgrade script we built uses Invoke-Ssh at the start and end to invoke autosupport and disable autogiveback.  Immediately after upgrading to 9.3, Invoke-NcSsh started returning:

Invoke-NcSsh : An established connection was aborted by the software in your host machine.

Best

n

[nathane · Accepted solution]

A quick follow-up to this issue. 

I finally have the zapi call workarounds in place to get around the broken cmdlets/broken support on ONTAP.  Included below are the various zapi blocks I have to workaround broken Invoke-NcSSH, and broken Invoke-NcClusterHATakeover cmdlets.

Disable AutoGiveback as suggested during an OnTAP NDU

$disableAutoGiveBackANodeApiCall = @"
                <cf-modify-iter>
                <query>
                <options-related-info-modify>
                <node>$controllerANode</node>
                </options-related-info-modify>
                </query>
                <attributes>
                <options-related-info-modify>
                <auto-giveback-enabled>false</auto-giveback-enabled>
                </options-related-info-modify>
                </attributes>
                </cf-modify-iter>
"@
Invoke-NcSystemApi -Controller $connectedController -Request $disableAutoGiveBackANodeApiCall -ErrorAction Stop | Out-Null

$disableAutoGiveBackBNodeApiCall = @"
                <cf-modify-iter>
                <query>
                <options-related-info-modify>
                <node>$controllerBNode</node>
                </options-related-info-modify>
                </query>
                <attributes>
                <options-related-info-modify>
                <auto-giveback-enabled>false</auto-giveback-enabled>
                </options-related-info-modify>
                </attributes>
                </cf-modify-iter>
"@

Invoke-NcSystemApi -Controller $connectedController -Request $disableAutoGiveBackBNodeApiCall -ErrorAction Stop | Out-Null

Invoke Takeover during an NDU with version mismatch

$Request = @"
                <cf-takeover>
                <node>$controllerBNode</node>
                <version-mismatch-ok>true</version-mismatch-ok>
                </cf-takeover>
"@
Invoke-NcSystemApi -Controller $connectedController -Request $Request -ErrorAction Stop | Out-Null

Revert LIFs not at home after an NDU

        ForEach ( $vServer in $vServers ) {

            $lifsNotHome = Get-NcNetInterface -Controller $connectedController -Vserver $vServer.Vserver | Where-Object { $_.IsHome -eq $FALSE } | Select-Object -ExpandProperty InterfaceName

            ForEach ( $lif in $lifsNotHome ) {

                $revertLifApiCall = @"
                    <net-interface-revert>
                    <vserver>$($vServer.Vserver)</vserver>
                    <interface-name>$lif</interface-name>
                    </net-interface-revert>
"@
                try {
                    if ( $vServer.VserverType -like "admin" ) {
                        Invoke-NcSystemApi -Controller $connectedController -Request $revertLifApiCall -ErrorAction SilentlyContinue | Out-Null
                    }
                    else {
                        Invoke-NcSystemApi -Controller $connectedController -Request $revertLifApiCall -ErrorAction Stop | Out-Null
                    }
                }
                catch {
                    Write-Warning "$controller`: LIF revert was not successful"
                }
            }
        }

I hope this helps out my fellow Netapp wrangers.

Best

n