Powershell Toolkit "API invoke failed" after upgrade to ONTAP 9.6P9

[netapple]

Hi all,

we run a Powershell script to create snapshots from Windows 2012 R2 machine on a FAS2554. It ran flawlessly until last week when we upgraded the FAS Ontap Version. Upgrade path was as follows: 9.3P12 -> 9.5P18 -> 9.6P9.

Now the script fails and as I found it can't even connect to the FAS. A simple Connect-NcController fails with error "API invoke failed". Other facts are:

- Authentication works, because when I type wrong credentials, I get a corresponding error message

- I tried https and http and both are enable on the target FAS

- A "Test-Connection" either to port 80 or 443 is successful

- Certificates have been renewed and are not expired

After trying everything I can think of, I am completely at a loss. Any help would be greatly appreciated.

Thanks

Dan

[Ontapforrum]

Are there any ems logs (related to ssl) on the controller side giving any hints ?

Not sure if this is the cause, but no harm in checking this out.

BUG:
https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1186901

[netapple]

Thanks for your response. I doubt that this is the problem because OCUM for example (which is now AIQUM) is not affected, all certs have already been recreated and connection over HTTP (which should ignore the certs) gives the same error.

[netapple]

What is ems log? I can do an event show but it does not show any messages regarding my connection attempts. I don't know of any other logs to look at.

[Ontapforrum]

By EMS, I simply meant 'event logs'.

It may be worth checking as well.
/mlog/audit.log (As it contains API related events)

I suggest: Please follow the steps to set the debug logs and re-produce that error, this will give us more information to find the root cause.
How to collect debug logs for the Data ONTAP PowerShell Toolkit:
https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/How_to_collect_debug_logs_for_the_Data_ONTAP_PowerShell_Toolkit

[netapple]

Thanks for your further suggestions. Unfortunately there is nothing in the audit.log.

We already tried the debug level of Powershell toolkit but that did not reveal any helpful information.

Meanwhile I heared of someone else trying to use the toolkit against an Ontap 9.6P9 using an ontap simulator and he experiences the exact same problem. Seems to me, they have changed something in that release, because her reported it works fine with 9.6.P6 or P4 for example.

[netapple]

Btw debug log gives the following output (just tried it with specifying SSLVersion but without the output is the same):

2020-08-07 11:51:04,731 INFO DataONTAP.C.PowerShell.SDK.ConnectNcController - Cmdlet invocation: Connect-NcController 1
0.99.16.51 -SSLVersion "TLSv1.2"
2020-08-07 11:51:04,751 DEBUG NetApp.Ontapi.NaServer - Application name set to PowerShell Toolkit
2020-08-07 11:51:04,751 DEBUG DataONTAP.C.PowerShell.SDK.ConnectNcController - No credentials provided, consulting cache

2020-08-07 11:51:13,769 DEBUG DataONTAP.C.PowerShell.SDK.ConnectNcController - Trying HTTP/HTTPS
2020-08-07 11:51:13,775 DEBUG NetApp.Ontapi.NaServer - NaServer port changed to 443
2020-08-07 11:51:13,776 DEBUG NetApp.Ontapi.NaServer - NaServer Protocol changed to HTTPS
2020-08-07 11:51:13,776 DEBUG NetApp.Ontapi.NaServer - Proxy changed to
2020-08-07 11:51:13,777 DEBUG DataONTAP.C.PowerShell.SDK.ConnectNcController - Connecting to <ipaddress> via HTTPS
2020-08-07 11:51:13,845 DEBUG NetApp.Ontapi.NaServer - <netapp version='1.0' xmlns='http://www.netapp.com/filer/admin'><
system-get-version /></netapp>

2020-08-07 11:51:44,051 DEBUG NetApp.Ontapi.NaServer - NaServer port changed to 80
2020-08-07 11:51:44,051 DEBUG NetApp.Ontapi.NaServer - NaServer Protocol changed to HTTP
2020-08-07 11:51:44,051 DEBUG NetApp.Ontapi.NaServer - Proxy changed to
2020-08-07 11:51:44,051 DEBUG DataONTAP.C.PowerShell.SDK.ConnectNcController - Connecting to <ipaddress> via HTTP
2020-08-07 11:51:44,052 DEBUG NetApp.Ontapi.NaServer - <netapp version='1.0' xmlns='http://www.netapp.com/filer/admin'><
system-get-version /></netapp>

Connect-NcController : API invoke failed.
At line:1 char:1
+ Connect-NcController <ipaddress> -SSLVersion "TLSv1.2"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidResult: (<ipaddress>:NcController) [Connect-NcController], NaException
+ FullyQualifiedErrorId : HttpConnectionFailed,DataONTAP.C.PowerShell.SDK.ConnectNcController

2020-08-07 11:52:14,266 DEBUG DataONTAP.C.PowerShell.SDK.ConnectNcController - Failed to connect to <ipaddress>
2020-08-07 11:52:14,268 DEBUG DataONTAP.C.PowerShell.SDK.ConnectNcController - Setting value of global:CurrentNcControll
er
2020-08-07 11:52:14,268 DEBUG DataONTAP.C.PowerShell.SDK.ConnectNcController - Starting EMS logging
2020-08-07 11:52:14,268 DEBUG DataONTAP.C.PowerShell.SDK.ConnectNcController - Failed to complete EMS logging
System.NullReferenceException: Object reference not set to an instance of an object.
at DataONTAP.C.PowerShell.SDK.ConnectNcController.ProcessRecord(String name)

[Ontapforrum]

Output of the following command ?

::> system services web show

[netapple]

Good point but already checked. Here's the output anyway:

External Web Services: true
Status: online
HTTP Protocol Port: 80
HTTPS Protocol Port: 443
HTTP Enabled: true

Meanwhile I heard from Netapp support that toolkit 9.7.1 is missing some DLLs that are planned to be implemented in 9.8. Quite unspecific and vague but at least an explanation. It also means there is no point in further investigation for me/us.

Thanks all for your hints and help.

[Ontapforrum]

Interesting info, indeed vague. How did it work earlier before the 9.6P9? That means, those missing DLLs didn't really matter. So, something have change on 9.6P9 rather than PS Tool kit version.

Anyway, usually, i just look for up bugs in the BUG TOOL for that particular ONTAP version and then I would raise a ticket with NetApp, which I guess you already did and you heard from them as well.

For 9.6P9 there is only one bug listed

https://mysupport.netapp.com/site/bugs-online/product/ONTAP/BURT/1342292

I don't know if this is related anyway but have a good weekend!

[parisi]

Someone else (actually 2 in the same thread) mentioned a similar issue with P9:

https://community.netapp.com/t5/ONTAP-Discussions/ontap-api-not-working-with-veeam-backup-and-PRTG-after-updating-to-ontap9-6P9/td-p/158095

So yes, it seems like perhaps something got broken in that patch. A case is the best route here.

In the meantime I'll spin up a VSIM of the release and see if I can figure it out.

[parisi · Accepted solution]

Ok, this is a confirmed bug in P9.

When you open a case, have the TSE add it to bug 1342292. We'll have a public report soon.

There doesn't seem to be a workaround for this other than reverting to the previous P release or moving to the latest 9.7 release.

[netapple]

Just wanted to let you know. We rolled back to 9.6P8 and now the toolkit works again as expected.