Microsoft Virtualization Discussions

Invoke-NaSsh not working anymore with DOT 8.2.5 7-mode

mark_schuren
21,727 Views

Hi all,

 

I upgraded one of our lab systems from ONTAP 8.2.4 to 8.2.5 (7-mode). Since that some of my scripts fail when doing "Invoke-NaSsh" against that system.

 

No idea why, I already regenerated SSH keys but error persists. It used to work with 8.2.4. And it's definitely not a credential issue.

 

PS C:\Users\mark> invoke-nassh -Name ucnlabfiler07 -Command date

invoke-nassh : An established connection was aborted by the software in your host machine.
In Zeile:1 Zeichen:1
+ invoke-nassh -Name ucnlabfiler07 -Command date
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidResult: (:) [Invoke-NaSsh], SshConnectionException
    + FullyQualifiedErrorId : SshExecFailed,DataONTAP.PowerShell.SDK.Cmdlets.Toolkit.Ssh.InvokeNaSsh

 

Any ideas? Does ONTAP reject the client's key length? How can I make it work again?

 

Cheers,

Mark

57 REPLIES 57

JGPSHNTAP
15,769 Views

How did you connect to the 7mode controller.  Invoke-nassh only supports via http/https, not rpc

mark_schuren
15,734 Views

I connect using credentials that were stored using Add-NaCredential before the ONTAP update. It worked before when ONTAP was 8.2.4.

 

PS C:\Users\mark> connect-nacontroller -name ucnlabfiler07 -HTTPS

Name                 Address           Ontapi   Version
----                 -------           ------   -------
ucnlabfiler07        10.230.1.7        1.21     NetApp Release 8.2.5 7-Mode: Wed Jul 19 03:55:53 PDT 2017


PS C:\Users\mark> invoke-nassh -Name ucnlabfiler07 -Command date
invoke-nassh : An established connection was aborted by the software in your host machine.
In Zeile:1 Zeichen:1
+ invoke-nassh -Name ucnlabfiler07 -Command date
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidResult: (:) [Invoke-NaSsh], SshConnectionException
    + FullyQualifiedErrorId : SshExecFailed,DataONTAP.PowerShell.SDK.Cmdlets.Toolkit.Ssh.InvokeNaSsh

 

As you see the connection to ONTAPI works, but gets teared down when using SSH with the PSTK.

PuTTy works. OpenSSH clients work. Invoke-NaSSh does not longer work. But works with other systems including cDOT (or even third-party SSH servers), even when not connected to a NaController.

 

Toolkit version is 4.4.0. Can anyone check if invoke-nassh works with DOT 8.2.5?

 

Any hints? I suspect some new security related "feature" of ONTAP's sshd that blocks the connection request. Does anyone know what SSH client (wrapper) is embedded in the PSTK?

 

skellner
15,579 Views

I think I found it. On 8.2.5 you have options for tls. Per default they are off. You have to enable tls.enable for connecting to the controller via https. Then invoke-nassh will work also. This is new in Ontap 8.2.5.

skellner
15,576 Views

Unfortunately invoke-nassh is not working although I'm connected via https. Anybody an idea how to fix this?

mark_schuren
15,547 Views

Thanks for trying. Exactly same issue here. The problem is not with SSL (https), but with SSH itself.

 

I guess the PSTK includes "outdated SSH client" code (using old / weak host key or KEX algorithms with Invoke-NaSsh).

ONTAP 8.2.5 seems to just reject connections from this "client".

 

Strange thing is, the same call works agains ONTAP 9.1, so not wure if the culprit is within PSTK or ONTAP 8.2.5.

 

Ideas, anyone?

skellner
15,355 Views

I also tried the newest toolkit on another w2k12R2 Server with powershell 5. Same issue. So I think it is Ontap 8.2.5. Without a solution for the issue we will have to stick with 8.2.4. Anybody an idea how this could be solved?

mbeattie
15,219 Views

Hi Guys,

 

I noticed the same issue in my lab on an old 7-Mode simulator (Invoke-NsSsh failed).

The fix (in my environment) was  to re-run secureadmin (ensuring the key length is set to 2048):

 

TESTNS01> secureadmin setup -f ssl
Country Name (2 letter code) [US]: AU
State or Province Name (full name) [California]: NSW
Locality Name (city, town, etc.) [Santa Clara]: Sydney
Organization Name (company) [Your Company]: NetApp
Organization Unit Name (division):  NetApp
Common Name (fully qualified domain name) [TESTNS01.testlab.local]:
Administrator email:  admin@testlab.local
Days until expires [5475] :3650
Key length (bits) [512] :2048
Tue Sep  5 10:20:38 AEST [TESTNS01:secureadmin.ssl.setup.success:info]: Restarting SSL with new certificate.

PS C:\> Invoke-NaSsh -Name testns01.testlab.local -Command version -Credential $credential
NetApp Release 8.2.3 7-Mode: Thu Jan 15 21:30:45 PST 2015

/Matt

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

mark_schuren
15,179 Views

Hi Matt,

 

thanks for your reply, but this question is specific for ONTAP 8.2.5 - I'm aware about the certificate/key length issues.

 

8.2.5 is a security/maintenance release for 7-mode and obviously something has changed that breaks the PSTK cmdlet (and thereby my scripts...).

 

Could anyone from the PSTK devs give us feedback here please?

 

Mark

 

skellner
15,088 Views

We did a pktt to compare ssh access between 8.2.4 and 8.2.5:

works -> 8.2.4:

"Server: Protocol (SSH-2.0-Data ONTAP SSH 1.0)"
"Client: Protocol (SSH-2.0-Renci.SshNet.SshClient.0.0.1)"

Doesn't work -> 8.2.5:

"Server: Protocol (SSH-2.0-OpenSSH_7.2 FreeBSD-20160310)"
"Client: Protocol (SSH-2.0-Renci.SshNet.SshClient.0.0.1)"

Putty to 8.2.5:

"Server: Protocol (SSH-2.0-OpenSSH_7.2 FreeBSD-20160310)"
"Client: Protocol (SSH-2.0-PuTTY_Release_0.63)"

So ssh server changed in 8.2.5 and the ssh client in the toolkit seems not to be supported anymore.

mark_schuren
13,117 Views

Thanks for looking deeper into it, that makes total sense.

 

So I can think of two possible solutions:

 

- either the Renci.SshNet client within PSTK needs to get updated or replaced by something newer (will this happen, and when?)

- or can we tweak / configure DOT 8.2.5 to still allow for the older Renci.SshNet client built into PSTK?

 

Any ideas welcome...

Mark

skellner
12,953 Views

The issue is getting worse. We have several workflows in WFA that don't work anymore because we are using invoke-nassh for a lot of workaround where there's no native cmdlet. So please if somebody of PSTK development looks into this, how can we workaround or solve this issue? To stay below 8.2.5 is not an option.

JGPSHNTAP
12,945 Views

Did you redo your secureadmin ssl setup and chose a higher bit as was suggested?  Did that still break?  

skellner
12,942 Views

Yes, I did a new secureadmin setup ssl with 2048 bits. However, this didn't change anything.

asulliva
12,802 Views

Hello everyone,

 

First, thank you for your efforts in identifying the issue.  Second, a couple of questions:

 

  • Just to be clear, this is 7-mode?
  • I may have missed this, but which version of the module are you using (Get-NaToolkitVersion)?
  • Has anyone enabled debug logging (Set-NaToolkitConfiguration -LogLevel DEBUG) and captured the output while trying to use Invoke-NaSsh to the 8.2.5 system?
  • Has a bug been opened by any of the NetApp people in this thread?

Thanks!

 

Andrew

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

advuni-dv1
13,640 Views

@asulliva wrote:

Hello everyone,

 

First, thank you for your efforts in identifying the issue.  Second, a couple of questions:

 

  • Just to be clear, this is 7-mode?
  • I may have missed this, but which version of the module are you using (Get-NaToolkitVersion)?
  • Has anyone enabled debug logging (Set-NaToolkitConfiguration -LogLevel DEBUG) and captured the output while trying to use Invoke-NaSsh to the 8.2.5 system?
  • Has a bug been opened by any of the NetApp people in this thread?

Thanks!

 

Andrew


Ontap 8.2.5 is 7-Mode only. 😉

 

Please see the attachment.

JGPSHNTAP
13,628 Views

I agree with the thread owners that this is a big issue on those folks that are using 7-mode still.  We are working to get off 7-mode, but still have it in play and without a working way to execute invoke-nassh is a bit of a killer.

 

Has anyone filed a bug with Netapp yet, if not I will have to work with my SAM and TSE to open a bug on this.

 

We need this fixed, we know that 8.2.5 7-mode is the last copy of 7-mode to be released and it was primary a security releases as well as the option to turn off SMB 1.    

 

 

asulliva
13,613 Views

Hello @JGPSHNTAP and everyone else,

 

Yes, a bug (id # 1123356) was opened for this.

 

Andrew

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

skellner
13,320 Views

In the meantime I found a workaround for the missing cmdlet invoke-nassh. A colleague gave me the hint to use invoke-nasystemapi. I wrote a powershell function and a workflow Automation command. Below you can find the code. In my environment it works very well.

 

Powershell:

 

function Invoke-Cli
{
    <#
    .Synopsis
       Cli command ausführen
    .DESCRIPTION
       Die Funktion führt über die api ein cli command auf dem verbundenen Controller oder Vfiler aus
    .EXAMPLE
       Invoke-CliCommand "version"
    .EXAMPLE
       Invoke-CliCommand "sysconfig -a"
    #>

    $command = @()
    $command = $args.split(" ")
    
    Try
    {
        $api = $("<system-cli><args><arg>"+($command -join "</arg><arg>")+"</arg></args></system-cli>")
        $output = Invoke-NaSystemApi -Request $api -ErrorAction Stop
        If($output.results."cli-result-value" -ne 1){
            write-warning  $("Command output: '" + $output.results."cli-output" + "'. Result value: " + $output.results."cli-result-value")
            Throw $("Cli Command failed: "+ $output.results."cli-output")
        }
    }
    Catch
    {
        Write-Error $("Failed Executing Command`: $command. Error " + $_.Exception.Message)
    }
    return $($output.results."cli-output")
}

 

 

WFA:

 

param (
[parameter(Mandatory=$true, HelpMessage="Array name or IP address")]
[string]$Array,

[parameter(Mandatory=$false, HelpMessage="Vfiler Name")]
[string]$Vfiler,

[parameter(Mandatory=$true, HelpMessage="cli command")]
[string]$cli_cmd)

# Connect to Array or Vfiler
if ($Vfiler)
{
    Connect-WfaController -Array $Array -vfiler $Vfiler
    Get-WFALogger -Info -message $("Connecting to Vfiler: " + $Vfiler)
}
else
{    
    Connect-WfaController -Array $Array
    Get-WFALogger -Info -message $("Connecting to Array: " + $Array)
}


$command = @()
$command = $cli_cmd.split(" ")
    
Try
{
    $api = $("<system-cli><args><arg>"+($command -join "</arg><arg>")+"</arg></args></system-cli>")
    $output = Invoke-NaSystemApi -Request $api -ErrorAction Stop
    If($output.results."cli-result-value" -ne 1){
        Get-WFALogger -Info -Message  $("Command output: '" + $output.results."cli-output" + "'. Result value: " + $output.results."cli-result-value")
        Throw $("Cli Command failed: "+ $output.results."cli-output")
    }
}
Catch
{
    Get-WFALogger -Error -Message $("Failed Executing Command`: $command. Error " + $_.Exception.Message)
    Throw "Failed invoking cli command"
}

if ($output.results."cli-output")
{
    Get-WFALogger -Info -Message  $("Command output: ")
    Get-WFALogger -Info -Message  $($output.results."cli-output")
}

JGPSHNTAP
12,734 Views

Nice job. I tested it and it works on older systems as well.

 

One suggestion, in your WFA function you use param feature.  I would suggest you carry this over to your standard function as well, just for ease of use and consistency.

skellner
12,729 Views
 
Public