API invoke failed

[tdimarzio]

Running PSTK 3.2 on Windows 2008 R2 servers.

Many different PowerShell scripts using module DataOnTap.

Everything has been fine, running like clockwork on multiple servers.

Last night was production patching night and the following were installed:

****

https://support.microsoft.com/en-us/kb/3177186
- MS16-114: Description of the security update for Windows SMBv1 Server: September 13, 2016
https://support.microsoft.com/en-us/kb/3182203
- September 2016 time zone change for Novosibirsk
https://support.microsoft.com/en-us/kb/3175024
- MS16-111: Description of the security update for Windows Kernel: September 13, 2016
https://support.microsoft.com/en-us/kb/3184122
- MS16-116: Description of the security update for OLE Automation for VBScript Scripting Engine: September 13, 2016
https://support.microsoft.com/en-us/kb/3185911
- MS16-106: Description of the security update for Microsoft Graphics Component: September 13, 2016
https://support.microsoft.com/en-us/kb/3185319
- MS16-104: Security update for Internet Explorer: September 13, 2016

****

Today, on multiple servers, all of my powershell scripts are failing due to a failure in Connect-NaController:

PS C:\Users\admin> Connect-NaController -HTTPS server01
Connect-NaController : API invoke failed.
At line:1 char:21
+ Connect-NaController <<<< -HTTPS server01
+ CategoryInfo : InvalidResult: (msfcnf01:NaController) [Connect-NaController], NaException
+ FullyQualifiedErrorId : HttpConnectionFailed,DataONTAP.PowerShell.SDK.ConnectNaController

Connecting via RPC works fine but we use HTTPS.

Any help would be greatly appreciated.

[tdimarzio]

Update:

This is what I get when trying the same with PSTK 4.2 (latest version, just downloaded today)

****

PS C:\Users\admin> Connect-NaController -HTTPS server01
Connect-NaController : Connection to server01 using HTTPS failed - The request was aborted: Could not create SSL/TLS
secure channel.
The error may be resolved by generating a new certificate on the storage controller, with a longer key length.
At line:1 char:1
+ Connect-NaController -HTTPS server01
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidResult: (msfcnf01:NaController) [Connect-NaController], NaConnectionSSLException
+ FullyQualifiedErrorId : HttpConnectionFailed,DataONTAP.PowerShell.SDK.ConnectNaController

****

OnCommand System manager to the same exact filers is working fine from the same server that is exhibiting the PowerShell failures above.

[tdimarzio]

Another update:

The same exact error occurs when running PSTK 3.2 or 4.2 from Windows 7.

Fortunately, I had a snapshot from a few days ago on my Windows 7 VM, before the last round of Microsoft patches.

Restore to that VM snapshot and the PSTK connect-nacontroller -https works perfectly.

So, this is definitely the result of recent Microsoft patches.

Would be surprised if more people weren't experiencing this.

[tdimarzio]

We uninstalled the Microsoft patches on the server and that resolved the PowerShell ToolKit 3.2 connect-nacontroller.

We are now going to reinstall the patches, one-by-one, with reboots in-between, to figure out exactly which Microsoft patch is responsible.

[tdimarzio]

Update:

Found the MS patch responsible:

https://support.microsoft.com/en-us/kb/3175024

- MS16-111: Description of the security update for Windows Kernel: September 13, 2016

So, any ideas how to mitigate this while keeing KB3175024 installed?

[Frank_Hoeben]

Good find.

I ran into the same problem and worked around it that same way (removing the KB3175024 patch).

[tdimarzio]

I have a ticket in with our Microsoft TAM to provide a work-around so that we can re-apply KB3175024.

When I receive the work-around I'll post it here.

[lzweers]

Hello,

We had same problem and see that after updates that connect did not work.

We did some troubleshooting and it seems that on the controllers the key length was only 512.

This should be at least 2048 for security reasons.

I advice to renew certificates with key lenght to at least 2048.

After install certificate you will be able to connect again.

[UTrentowski]

Hi lzweers,

could you share/provide more details about the troubleshooting you've done?

I'm running PSTK 4.0 and have the hotfixes installed and have not a problem connecting to the filer.

Kind regards,

U.

[lzweers]

Hello,

This is the update that is causing the problem: https://support.microsoft.com/en-us/kb/3174644.

This update blocks certificates with keylenght 512.

For Windows 7 and Windows Server 2008 R2, this update is included in security update 3175024.

 https://support.microsoft.com/en-us/kb/3175024

For doing check or renewal on NetApp filer :

https://kb.netapp.com/support/s/article/how-to-renew-an-ssl-certificate-in-clustered-data-ontap

Hope this helps

Leon

[tdimarzio]

We are running 7-Mode but I can see that the private key uses a 512 bit keylength using:

keymgr list key

Unfortunately, refreshing certificates on 22 NetApp filers spread across the country is a daunting task.

I am still waiting for a response from our Microsoft TAM on a way to allow the 512 bit keylength either globally or for specified, trusted hosts.

[tdimarzio · Accepted solution]

I have finally received a working registry work-around from Microsoft. Here it is:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\PKCS]

"ClientMinKeyBitLength"=dword:00000200

The difference versus the non-working registry work-around is "PKCS" key versus previous "DiffieHellman" key

[lzweers]

Hi,

What Microsoft is doing with adjusting the registry key is a workarround so you kan work for now with certificates who are build with a keylength of 512.

If you want a more secure and lasting solution i suggest you create new certificates with keylength at least 1024 (2048 or 4096 are even better).

This will prevent a new Microsoft update to change the key again and some dissapointment in future.

Greetz Léon Zweers

[tdimarzio]

Yes, thank you for that.

I am aware that the registry key is simply loosening the restriction on the minimum key length.

That is obvious from the name of the registry key.

However, it will take some time to update 22 production NetApp filers with higher keylength certificates.

This registry key allows us to work-around the problem in the meantime.

Regards,

Tony

[axsys]

thank you, thank you, thank you!
thought I was driving mad! searched my ass of as suddenly all my scripts failed.
Just to add in case anyone is asking: dword is in hex, no reboot required.

cheers,

axsys