API Invoke failed in workflow, command works by itself

[jont]

Hi all,

I'm having a strange issue with WFA 4.1 under Windows and hope this looks famailar to someone. To wit:

1) The first non-no-op command in a workflow (any workflow) runs to completion.

2) The next command in the workflow fails with the message "API invoke failed. Location: <location of the second command>"

I've tried both my own workflow as well as certified workflows and get the same result. It doesn't matter which workflow I pick or which specific command is second in line.

An example of the error, using the NFSv3 File Access certified workflow as an example:

workflow_exec log shows:

12:47:48.354 INFO  [NFSv3 File Access] ***** Workflow Execution Started *****
12:47:48.386 INFO  [Create export policy] ### Command 'Create export policy' in 'POWER_SHELL' ###
12:47:49.651 INFO  [Create export policy] Get-WfaCredentials -Host 10.39.224.200
12:47:49.683 INFO  [Create export policy] Credentials successfully provided for '10.39.224.200'
12:47:49.714 INFO  [Create export policy] Connect-Controller -Type CLUSTER -Name 10.39.224.200 -Credential System.Management.Automation.PSCredential -Vserver 
12:47:49.761 INFO  [Create export policy] Credentials successfully provided for '10.39.224.200'
12:47:49.839 INFO  [Create export policy] Connect-NcController (with credentials) -Name 10.39.224.200 -Timeout 60000 -ErrorAction Stop -Port 443
12:48:06.059 INFO  [Create export policy] Connected to cluster node
12:48:06.106 INFO  [Create export policy] Creating Export Policy 'rnd' on Storage Virtual Machine 'de03tst006fnt90'
12:48:06.324 INFO  [Create export policy] Command completed, took 17938 milliseconds
12:48:06.434 INFO  [Create export rule] ### Command 'Create export rule' in 'POWER_SHELL' ###
12:48:07.762 INFO  [Create export rule] Using cached cluster connection
12:48:13.153 INFO  [Create export rule] Creating policy : rnd
12:48:13.200 INFO  [Create export rule] Creating export rule : New-NcExportRule -ErrorAction Stop -VserverContext de03tst006fnt90 -Policy rnd -ClientMatch '192.168.194.133' -ReadOnlySecurityFlavor sys -ReadWriteSecurityFlavor sys -Protocol nfs -SuperUserSecurityFlavor sys
12:48:13.278 ERROR  [Create export rule] API invoke failed.
12:48:13.497 ERROR  [Create export rule] Command failed for Workflow 'NFSv3 File Access' with error : API invoke failed.
12:48:13.497 INFO  [Create export rule] ***** Workflow Execution Failed *****

wfa.log shows:

...


2017-05-23 12:48:06,059 INFO  [com.netapp.wfa.command.execution.instance.impl.ExecutionInstanceDaoImpl] (default task-22) [Create export policy]   Connected to cluster node
2017-05-23 12:48:06,106 INFO  [com.netapp.wfa.command.execution.instance.impl.ExecutionInstanceDaoImpl] (default task-25) [Create export policy]   Creating Export Policy 'rnd' on Storage Virtual Machine 'de03tst006fnt90'
2017-05-23 12:48:06,324 INFO  [com.netapp.wfa.command.execution.instance.impl.ExecutionInstanceDaoImpl] (Thread-57 (HornetQ-client-global-threads-975816356)) [Create export policy]   Command completed, took 17938 milliseconds
2017-05-23 12:48:06,434 INFO  [com.netapp.wfa.command.execution.instance.impl.ExecutionInstanceDaoImpl] (Thread-57 (HornetQ-client-global-threads-975816356)) [Create export rule]   ### Command 'Create export rule' in 'POWER_SHELL' ###
2017-05-23 12:48:07,778 INFO  [com.netapp.wfa.command.execution.instance.impl.ExecutionInstanceDaoImpl] (default task-28) [Create export rule]   Using cached cluster connection
2017-05-23 12:48:13,153 INFO  [com.netapp.wfa.command.execution.instance.impl.ExecutionInstanceDaoImpl] (default task-33) [Create export rule]   Creating policy : rnd
2017-05-23 12:48:13,200 INFO  [com.netapp.wfa.command.execution.instance.impl.ExecutionInstanceDaoImpl] (default task-35) [Create export rule]   Creating export rule : New-NcExportRule -ErrorAction Stop -VserverContext de03tst006fnt90 -Policy rnd -ClientMatch '192.168.194.133' -ReadOnlySecurityFlavor sys -ReadWriteSecurityFlavor sys -Protocol nfs -SuperUserSecurityFlavor sys
2017-05-23 12:48:13,278 ERROR [com.netapp.wfa.command.execution.instance.impl.ExecutionInstanceDaoImpl] (default task-37) [Create export rule]   API invoke failed.
2017-05-23 12:48:13,497 ERROR [com.netapp.wfa.engine.job.WorkflowExecutionJobExecutorImpl] (Thread-57 (HornetQ-client-global-threads-975816356)) API invoke failed.: CommandExecutionException{Message: API invoke failed., Cause: null}
 at com.netapp.wfa.command.execution.impl.executors.ScriptBasedCommandExecutor.executeScriptCommand(ScriptBasedCommandExecutor.java:200) [command-0.5.jar:]
 at com.netapp.wfa.command.execution.impl.executors.ScriptBasedCommandExecutor.executeScriptCommand(ScriptBasedCommandExecutor.java:69) [command-0.5.jar:]
 at com.netapp.wfa.command.execution.impl.executors.ScriptBasedCommandExecutor.execute(ScriptBasedCommandExecutor.java:36) [command-0.5.jar:]
 at com.netapp.wfa.engine.job.WorkflowExecutionJobExecutorImpl.executeCommandOrChildWorkflow(WorkflowExecutionJobExecutorImpl.java:358) [workflow-0.5.jar:]
 at com.netapp.wfa.engine.job.WorkflowExecutionJobExecutorImpl.execute(WorkflowExecutionJobExecutorImpl.java:179) [workflow-0.5.jar:]
 at com.netapp.wfa.engine.job.WorkflowExecutionJobExecutorImpl.execute(WorkflowExecutionJobExecutorImpl.java:75) [workflow-0.5.jar:]
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121]

...

Thanks in advance for any ideas/solutions!

(I've already tried restarting services, restarting the web browser, and rebooting the WFA Windows server) 

[mbeattie]

Hi Jon

Have you tried running powershell as admin on the WFA server and loading the WFA profile so you have access the WFA environment (but externally to WFA) then run the same powershell commands? EG:

PS C:>Set-Location -Path "C:\program files\netapp\wfa\posh"
PS C:\program files\netapp\wfa\posh>.\profile.ps1
PS C:\program files\netapp\wfa\posh>cd \
PS C:>Connect-NcController -Name 10.39.224.200 -HTTPS -Credential Get-Credential
PS C:>New-NcExportRule -VserverContext de03tst006fnt90 -Policy rnd -ClientMatch '192.168.194.133' -ReadOnlySecurityFlavor sys -ReadWriteSecurityFlavor sys -Protocol nfs -SuperUserSecurityFlavor sys -ErrorAction Stop

Does that give you the same error or does it complete succuessfully? A few other questions, have you rececently upgraded WFA. Are the credentials added in the WFA credential cache added by IP Address or FQDN and if so do the DNS records exist? I've seen WFA throw similar errors (which doesn't really describe the problem) but the issue was actually related to credentials or DNS. I would recommend your next point of troubleshooting (assuming that running the powershell commands above does work successfully) would be remove\re-add the WFA credentials for your cluster via the FQDN.

Hope that helps

/Matt

[jont]

Hi mbeattie,

I tried running commands from the CLI as you suggest and they worked. From the Powershell CLI I am not having any problems and can "emulate" the workflows that are failing in WFA.

As I've just discovered, the root cause of the problem stems from disabling TLSv1 on the storage nodes. As soon as the PowerShell client is forced to use TLSv1.1 or TLSv1.2, the first non-no-op WFA command in a workflow still runs but following commands fail. It would appear that WFA is incompatible with TLSv1.1+ (and thus FIPS-compliant mode as well, which is the desierd state for the environment in question).

[mbeattie]

Hi Jonathan

Thanks for the update, I've also noticed that the "Invoke-NcSsh" CmdLet fails if FIPS mode is enabled on ONTAP 9.X clusters.

I'd assume this also relates to WFA using TLS1. It's probably worth raising a case for, should be raises as a BURT and a KB article published on it.

/Matt

[sinhaa]

@jont

It would appear that WFA is incompatible with TLSv1.1+ (and thus FIPS-compliant mode as well, which is the desierd state for the environment in question).

----

WFA hasn't done anything on purpose here, but this is an interesting find. I'll get back to you on this.

sinhaa 

[sinhaa · Accepted solution]

This has been resolved via email. Posting it for the benefit of others who use ONTAP's FIPS compliance feature and use WFA for Storage Automation.

Enabling FIPS compliance disabled the SSL protocol TLS1.0 on the DataONTAP Cluster. So now the connectivity is only over TLS1.1, TLS1.2 

The WFA cmdlet Connect-WfaCluster (internally users DataONTAP PSTK's Connect-NcController ) always connects to Cluster using the default SSL protocol which is TLS1.0. hence the connection failure on systems with FIPS enabled.

To fix this, do the following:

For WFA4.0 and above

1. You need to modify the file WFA\PoSH\Modules\WFAWrapper\WFAWrapper.psm1

2. Attached is the code with the fix. Replace the existing code with this and save the file.

Done.

For WFA3.2 and below:

I'll provide the code if someone asks for it. 

sinhaa

[sinhaa]

I've updated the WFAWrapper_4X.txt file. Now WFA connection caching will also work.

sinhaa

[mbeattie]

Hi Jonathan,

This might be related to the issue:

https://kb.netapp.com/support/s/article/ka11A0000008hRGQAY/HTTPS-connections-to-controllers-or-clusters-fail-after-installing-Microsoft-KB-3174644-or-...

/Matt