Invoke-NaSsh not working anymore with DOT 8.2.5 7-mode

[mark_schuren]

Hi all,

I upgraded one of our lab systems from ONTAP 8.2.4 to 8.2.5 (7-mode). Since that some of my scripts fail when doing "Invoke-NaSsh" against that system.

No idea why, I already regenerated SSH keys but error persists. It used to work with 8.2.4. And it's definitely not a credential issue.

PS C:\Users\mark> invoke-nassh -Name ucnlabfiler07 -Command date

invoke-nassh : An established connection was aborted by the software in your host machine.
In Zeile:1 Zeichen:1
+ invoke-nassh -Name ucnlabfiler07 -Command date
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidResult: (:) [Invoke-NaSsh], SshConnectionException
    + FullyQualifiedErrorId : SshExecFailed,DataONTAP.PowerShell.SDK.Cmdlets.Toolkit.Ssh.InvokeNaSsh

Any ideas? Does ONTAP reject the client's key length? How can I make it work again?

Cheers,

Mark

[JGPSHNTAP]

How did you connect to the 7mode controller.  Invoke-nassh only supports via http/https, not rpc

[mark_schuren]

I connect using credentials that were stored using Add-NaCredential before the ONTAP update. It worked before when ONTAP was 8.2.4.

PS C:\Users\mark> connect-nacontroller -name ucnlabfiler07 -HTTPS

Name                 Address           Ontapi   Version
----                 -------           ------   -------
ucnlabfiler07        10.230.1.7        1.21     NetApp Release 8.2.5 7-Mode: Wed Jul 19 03:55:53 PDT 2017


PS C:\Users\mark> invoke-nassh -Name ucnlabfiler07 -Command date
invoke-nassh : An established connection was aborted by the software in your host machine.
In Zeile:1 Zeichen:1
+ invoke-nassh -Name ucnlabfiler07 -Command date
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidResult: (:) [Invoke-NaSsh], SshConnectionException
    + FullyQualifiedErrorId : SshExecFailed,DataONTAP.PowerShell.SDK.Cmdlets.Toolkit.Ssh.InvokeNaSsh

As you see the connection to ONTAPI works, but gets teared down when using SSH with the PSTK.

PuTTy works. OpenSSH clients work. Invoke-NaSSh does not longer work. But works with other systems including cDOT (or even third-party SSH servers), even when not connected to a NaController.

Toolkit version is 4.4.0. Can anyone check if invoke-nassh works with DOT 8.2.5?

Any hints? I suspect some new security related "feature" of ONTAP's sshd that blocks the connection request. Does anyone know what SSH client (wrapper) is embedded in the PSTK?

[skellner]

I think I found it. On 8.2.5 you have options for tls. Per default they are off. You have to enable tls.enable for connecting to the controller via https. Then invoke-nassh will work also. This is new in Ontap 8.2.5.

[skellner]

Unfortunately invoke-nassh is not working although I'm connected via https. Anybody an idea how to fix this?

[mark_schuren]

Thanks for trying. Exactly same issue here. The problem is not with SSL (https), but with SSH itself.

I guess the PSTK includes "outdated SSH client" code (using old / weak host key or KEX algorithms with Invoke-NaSsh).

ONTAP 8.2.5 seems to just reject connections from this "client".

Strange thing is, the same call works agains ONTAP 9.1, so not wure if the culprit is within PSTK or ONTAP 8.2.5.

Ideas, anyone?

[skellner]

I also tried the newest toolkit on another w2k12R2 Server with powershell 5. Same issue. So I think it is Ontap 8.2.5. Without a solution for the issue we will have to stick with 8.2.4. Anybody an idea how this could be solved?

[mbeattie]

Hi Guys,

I noticed the same issue in my lab on an old 7-Mode simulator (Invoke-NsSsh failed).

The fix (in my environment) was  to re-run secureadmin (ensuring the key length is set to 2048):

TESTNS01> secureadmin setup -f ssl
Country Name (2 letter code) [US]: AU
State or Province Name (full name) [California]: NSW
Locality Name (city, town, etc.) [Santa Clara]: Sydney
Organization Name (company) [Your Company]: NetApp
Organization Unit Name (division):  NetApp
Common Name (fully qualified domain name) [TESTNS01.testlab.local]:
Administrator email:  admin@testlab.local
Days until expires [5475] :3650
Key length (bits) [512] :2048
Tue Sep  5 10:20:38 AEST [TESTNS01:secureadmin.ssl.setup.success:info]: Restarting SSL with new certificate.

PS C:\> Invoke-NaSsh -Name testns01.testlab.local -Command version -Credential $credential
NetApp Release 8.2.3 7-Mode: Thu Jan 15 21:30:45 PST 2015

/Matt

[mark_schuren]

Hi Matt,

thanks for your reply, but this question is specific for ONTAP 8.2.5 - I'm aware about the certificate/key length issues.

8.2.5 is a security/maintenance release for 7-mode and obviously something has changed that breaks the PSTK cmdlet (and thereby my scripts...).

Could anyone from the PSTK devs give us feedback here please?

Mark

[skellner]

We did a pktt to compare ssh access between 8.2.4 and 8.2.5:

works -> 8.2.4:

"Server: Protocol (SSH-2.0-Data ONTAP SSH 1.0)"
"Client: Protocol (SSH-2.0-Renci.SshNet.SshClient.0.0.1)"

Doesn't work -> 8.2.5:

"Server: Protocol (SSH-2.0-OpenSSH_7.2 FreeBSD-20160310)"
"Client: Protocol (SSH-2.0-Renci.SshNet.SshClient.0.0.1)"

Putty to 8.2.5:

"Server: Protocol (SSH-2.0-OpenSSH_7.2 FreeBSD-20160310)"
"Client: Protocol (SSH-2.0-PuTTY_Release_0.63)"

So ssh server changed in 8.2.5 and the ssh client in the toolkit seems not to be supported anymore.

[mark_schuren]

Thanks for looking deeper into it, that makes total sense.

So I can think of two possible solutions:

- either the Renci.SshNet client within PSTK needs to get updated or replaced by something newer (will this happen, and when?)

- or can we tweak / configure DOT 8.2.5 to still allow for the older Renci.SshNet client built into PSTK?

Any ideas welcome...

Mark

[skellner]

The issue is getting worse. We have several workflows in WFA that don't work anymore because we are using invoke-nassh for a lot of workaround where there's no native cmdlet. So please if somebody of PSTK development looks into this, how can we workaround or solve this issue? To stay below 8.2.5 is not an option.

[JGPSHNTAP]

Did you redo your secureadmin ssl setup and chose a higher bit as was suggested?  Did that still break?  

[skellner]

Yes, I did a new secureadmin setup ssl with 2048 bits. However, this didn't change anything.

[asulliva]

Hello everyone,

First, thank you for your efforts in identifying the issue.  Second, a couple of questions:

• Just to be clear, this is 7-mode?
• I may have missed this, but which version of the module are you using (Get-NaToolkitVersion)?
• Has anyone enabled debug logging (Set-NaToolkitConfiguration -LogLevel DEBUG) and captured the output while trying to use Invoke-NaSsh to the 8.2.5 system?
• Has a bug been opened by any of the NetApp people in this thread?

Thanks!

Andrew

[advuni-dv1]

---

@asulliva wrote:

Hello everyone,

First, thank you for your efforts in identifying the issue.  Second, a couple of questions:

• Just to be clear, this is 7-mode?
• I may have missed this, but which version of the module are you using (Get-NaToolkitVersion)?
• Has anyone enabled debug logging (Set-NaToolkitConfiguration -LogLevel DEBUG) and captured the output while trying to use Invoke-NaSsh to the 8.2.5 system?
• Has a bug been opened by any of the NetApp people in this thread?

Thanks!

Andrew

---

Ontap 8.2.5 is 7-Mode only. 😉

Please see the attachment.

[JGPSHNTAP]

I agree with the thread owners that this is a big issue on those folks that are using 7-mode still.  We are working to get off 7-mode, but still have it in play and without a working way to execute invoke-nassh is a bit of a killer.

Has anyone filed a bug with Netapp yet, if not I will have to work with my SAM and TSE to open a bug on this.

We need this fixed, we know that 8.2.5 7-mode is the last copy of 7-mode to be released and it was primary a security releases as well as the option to turn off SMB 1.    

[asulliva]

Hello @JGPSHNTAP and everyone else,

Yes, a bug (id # 1123356) was opened for this.

Andrew

[skellner]

In the meantime I found a workaround for the missing cmdlet invoke-nassh. A colleague gave me the hint to use invoke-nasystemapi. I wrote a powershell function and a workflow Automation command. Below you can find the code. In my environment it works very well.

Powershell:

function Invoke-Cli
{
    <#
    .Synopsis
       Cli command ausführen
    .DESCRIPTION
       Die Funktion führt über die api ein cli command auf dem verbundenen Controller oder Vfiler aus
    .EXAMPLE
       Invoke-CliCommand "version"
    .EXAMPLE
       Invoke-CliCommand "sysconfig -a"
    #>

    $command = @()
    $command = $args.split(" ")
    
    Try
    {
        $api = $("<system-cli><args><arg>"+($command -join "</arg><arg>")+"</arg></args></system-cli>")
        $output = Invoke-NaSystemApi -Request $api -ErrorAction Stop
        If($output.results."cli-result-value" -ne 1){
            write-warning  $("Command output: '" + $output.results."cli-output" + "'. Result value: " + $output.results."cli-result-value")
            Throw $("Cli Command failed: "+ $output.results."cli-output")
        }
    }
    Catch
    {
        Write-Error $("Failed Executing Command`: $command. Error " + $_.Exception.Message)
    }
    return $($output.results."cli-output")
}

WFA:

param (
[parameter(Mandatory=$true, HelpMessage="Array name or IP address")]
[string]$Array,

[parameter(Mandatory=$false, HelpMessage="Vfiler Name")]
[string]$Vfiler,

[parameter(Mandatory=$true, HelpMessage="cli command")]
[string]$cli_cmd)

# Connect to Array or Vfiler
if ($Vfiler)
{
    Connect-WfaController -Array $Array -vfiler $Vfiler
    Get-WFALogger -Info -message $("Connecting to Vfiler: " + $Vfiler)
}
else
{    
    Connect-WfaController -Array $Array
    Get-WFALogger -Info -message $("Connecting to Array: " + $Array)
}


$command = @()
$command = $cli_cmd.split(" ")
    
Try
{
    $api = $("<system-cli><args><arg>"+($command -join "</arg><arg>")+"</arg></args></system-cli>")
    $output = Invoke-NaSystemApi -Request $api -ErrorAction Stop
    If($output.results."cli-result-value" -ne 1){
        Get-WFALogger -Info -Message  $("Command output: '" + $output.results."cli-output" + "'. Result value: " + $output.results."cli-result-value")
        Throw $("Cli Command failed: "+ $output.results."cli-output")
    }
}
Catch
{
    Get-WFALogger -Error -Message $("Failed Executing Command`: $command. Error " + $_.Exception.Message)
    Throw "Failed invoking cli command"
}

if ($output.results."cli-output")
{
    Get-WFALogger -Info -Message  $("Command output: ")
    Get-WFALogger -Info -Message  $($output.results."cli-output")
}

[JGPSHNTAP]

Nice job. I tested it and it works on older systems as well.

One suggestion, in your WFA function you use param feature.  I would suggest you carry this over to your standard function as well, just for ease of use and consistency.