Subscribe

API invoke failed on connect-ncController

Hi,

From time to time I get an error while trying to connect to some of my clusters.

 

The command used is: Connect-NcController $ControllerName -Credential $cred -HTTPS -Timeout 480000

 

And I get this error: 

 

Connect-NcController : API invoke failed.
At E:\Folders\Stats.ps1:1355 char:5
+ Connect-NcController $ControllerName -Credential $cred -HTTPS -Timeout
4800 ...
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidResult: (server.domain.com:NcController
) [Connect-NcController], NaException
+ FullyQualifiedErrorId : HttpConnectionFailed,DataONTAP.C.PowerShell.SDK.
ConnectNcController

 

 

If I try it later it works. Anyone know what could be wrong?

 

Thanks

Phil

Re: API invoke failed on connect-ncController

Hi Phil,

 

It sounds like a transient error which are always the most frustrating and often difficult to troubleshoot, so to help identify what might be causing it lets start by checking version of the DataONTAP powershell toolkit are you running and what OS version of ONTAP are you connecting to?

 

Check the powershell toolkit version using:

 

Import-Module -Name DataONTAP
Get-NaToolkitVersion

Can you check the following:

 

>version
>system service web show

What is your ONTAP version and configuration for:

 

  • External Web Services
  • Status
  • HTTP Protocol Port
  • HTTPs Protocol Port
  • TLSv1 Enabled
  • SSLv3 Enabled
  • SSL FIPS 140-2 Enabled

Is the configuration and ONTAP version between your clusters different? In particular the SSLv3 and FIPS configuration.

 

/Matt

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

Re: API invoke failed on connect-ncController

Hi Matt,

 

Here's the info you requested. Let me know if you need to know anything else.

 

Powershell toolkit version: 3.3.0.65

NetApp version: NetApp Release 8.3.1P2: Wed Dec 09 03:10:24 UTC 2015

System service web show:

  External Web Services: true
  Status: online
  HTTP Protocol Port: 80
  HTTPs Protocol Port: 443
  TLSv1 Enabled: true
  SSLv3 Enabled: false
  SSL FIPS 140-2 Enabled: false

 

The configuration and NetApp version are the same. The only difference is, one cluster is local and the other is remote(another site).

Re: API invoke failed on connect-ncController

Hi Philip,

 

Thanks, the powershell toolkit and ontap versions you are running are fine and you should be able to connect to each cluster without issue. The configuration doesn't indicate the potential issue i suspected (In ONTAP9.X you can have connectivity issues with FIPS enabled, specifically using HTTPS\Invoke-NcSsh). Given that is not the issue if you do a:

 

> certificate show

 

 

Are all the certificates still valid (not expired). I'd assume both local clusters have DNS A & PTR records created and you've verfied these are resolvable using nslookup from the host you are attempting to connect from? Does the host you are connecting from have a route to the remote cluster? Is the problem consistently repeatable?

 

/Matt

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

Re: API invoke failed on connect-ncController

Hi Matt,

 

Yes all the certificates are still valid.

Yes we have DNS A & PTR records, nslookup are working also.

 

The problem happens a couple time a week and not at the same time during the day or night.

Might be an issue not related to NetApp at all I guess. Network, Windows, powershell issue maybe.

Re: API invoke failed on connect-ncController

Hi Philip

 

I can't see any storage issue that would cause the API to fail to connect to the cluster. It's certainly possible it's an operating system or network issue in your environment or on the host you are running the script from rather than a storage issue. Does your script create any logs with timestamps so that you can trace what time the errors occurred in correlation to any event logs? I noticed you've set the timeout to 480000 (8 minutes) have you tried increasing this 3600000 (60 minutes)?. Also is E:\ drive that your script is running from a local disk, a mapped network share or a mapped LUN?

 

/Matt

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

Re: API invoke failed on connect-ncController

Hi Matt,

 

Yes the script is creating logs and that's why I'm pretty sure it's not a timeout issue. The command is sent to the cluster and we get a return almost right away that it failed to connect.

The script is running on a local drive.

 

I'll just add some error handling in the script and retry the connection when it fails.

 

Thanks for your help Matt.

 

Re: API invoke failed on connect-ncController

[ Edited ]

Hi Philip,

 

Here is an example of some error handling and logging functions that will add timestamps to help troubleshoot the issue.

The following example scripts does the following:

 

  • Accepts an input parameter for the cluster name to connect
  • Prompts for credentials to connect to the cluster
  • Imports the DataONTAP module
  • Attempts to connect to the cluster via HTTPS
  • Logs the result to the scripts working directory

 

A successful connection is logged to a .log file in YYYY-MM-DD format. Any errors are logged to a .err file

 

 

#'------------------------------------------------------------------------------
Param(
   [Parameter(Mandatory=$True, HelpMessage="The hostname or IP Address of the cluster to connect to")]
   [String]$ClusterName
)
#'------------------------------------------------------------------------------
Function Get-IsoDateTime{
   <#
   .SYNOPSIS
   This Function enumerates the date and time in ISO format.
   .DESCRIPTION
   Returns a date time in the form of "YYYY-MM-DD HH:MM:SS".
   .EXAMPLE
   Get-IsoDateTime
   #>
   Return (Get-IsoDate) + " " + (Get-IsoTime)
}#End Function
#'------------------------------------------------------------------------------
Function Get-IsoDate{
   <#
   .SYNOPSIS
   This Function enumerates the date in ISO format.
   .DESCRIPTION
   Returns a date in the format of "YYYY-MM-DD".
   .EXAMPLE
   Get-IsoDate
   #>
   Return Get-Date -uformat "%Y-%m-%d"
}#End Function
#'------------------------------------------------------------------------------
Function Get-IsoTime{
   <#
   .SYNOPSIS
   This Function enumerates the time in ISO format.
   .DESCRIPTION
   Returns a time in the format of "HH:MM:SS".
   .EXAMPLE
   Get-IsoTime
   #>
   Return Get-Date -uformat "%H:%M:%S"
}#End Function
#'------------------------------------------------------------------------------
Function Append-Message{
   <#
   .SYNOPSIS
   This function appends a message to log file based on the message type.
   .DESCRIPTION
   Appends a message to a log a file.
   .PARAMETER
   Accepts an integer representing the log file extension type
   .PARAMETER
   Accepts a string value containing the message to append to the log file.
   .EXAMPLE
   Append-Message -logType 0 -message "Command completed succuessfully"
   .EXAMPLE
   Append-Message -logType 2, -message "Application is not installed"
   #>
   [CmdletBinding()]
   Param(
      [Parameter(Position=0,
         Mandatory=$True,
         ValueFromPipeLine=$True,
         ValueFromPipeLineByPropertyName=$True)]
      [Int]$LogType,
      [Parameter(Position=1,
         Mandatory=$True,
         ValueFromPipeLine=$True,
         ValueFromPipeLineByPropertyName=$True)]
      [String]$Message
   )
   Switch($logType){
      0 {$extension = "log";}
      1 {$extension = "err";}
      2 {$extension = "err";}
      3 {$extension = "csv";}
      default {$extension = "log"}
   }
   If($logType -eq 1){
      $message = ("Error`: " + $error[0].Exception.Message + ". Code`: " + $error[0].Exception.ErrorCode + ". Message`: " + $Message)
   }
   $prefix = Get-IsoDateTime
   Try{
      ($prefix + "," + $message) | Out-File -filePath `
      ($scriptLogPath + "." + $extension) -encoding ASCII -Append -ErrorAction SilentlyContinue
   }Catch{
      Break;
   }
}#End Function
#'------------------------------------------------------------------------------
#'Initialization Section. Define Global Variables.
#'------------------------------------------------------------------------------
[String]$scriptPath     = Split-Path($MyInvocation.MyCommand.Path)
[String]$scriptSpec     = $MyInvocation.MyCommand.Definition
[String]$scriptBaseName = (Get-Item $scriptSpec).BaseName
[String]$scriptName     = (Get-Item $scriptSpec).Name
[String]$scriptLogPath  = $scriptPath + "\" + (Get-IsoDate)
#'------------------------------------------------------------------------------
Import-Module -Name DataONTAP -ErrorAction SilentlyContinue
$credentials = $host.ui.PromptForCredential("Connect to Cluster ""$ClusterName""", "Please enter the user name and password", "admin", "")
Try{
   Connect-NcController -Name $clusterName -HTTPS -Credential $credentials -ErrorAction Stop
   Append-Message 0 "Connected to ""$clusterName"""
}Catch{
   Append-Message 1 "Failed connecting to cluster ""$clusterName"""
   Exit -1
}
#'------------------------------------------------------------------------------

An example of running the script:

 

PS C:\Scripts\PowerShell\Projects\ConnectCluster> .\ConnectCluster.ps1 -ClusterName cluster1.testlab.local

An example of the log files for both successful and unsuccessful connections:

 

C:\Scripts\PowerShell\Projects\ConnectCluster>dir /b
2016-09-30.err
2016-09-30.log
ConnectCluster.ps1

C:\Scripts\PowerShell\Projects\ConnectCluster>type 2016-09-30.log
2016-09-30 14:58:01,Connected to "cluster1.testlab.local"

C:\Scripts\PowerShell\Projects\ConnectCluster>type 2016-09-30.err
2016-09-30 14:59:35,Error: No such host is known. Code: 11001. Message: Failed connecting to cluster "cluster10.testlab.local"

By logging the value of the "$error[0].Exception.ErrorCode" it might help to identify the cause of the issue.

Also If you look at the Methods for the "$error[0].Exception" object you'll notice you can do a stacktrace which might be useful. EG

 

PS C:\Scripts\PowerShell\Projects\ConnectCluster> $error[0].Exception | Get-Member

   TypeName: System.Net.Sockets.SocketException

Name             MemberType Definition
----             ---------- ----------
Equals           Method     bool Equals(System.Object obj), bool _Exception.Equals(System.Object obj)
GetBaseException Method     System.Exception GetBaseException(), System.Exception _Exception.GetBaseException()
GetHashCode      Method     int GetHashCode(), int _Exception.GetHashCode()
GetObjectData    Method     void GetObjectData(System.Runtime.Serialization.SerializationInfo info, System.Runtime.S...
GetType          Method     type GetType(), type _Exception.GetType()
ToString         Method     string ToString(), string _Exception.ToString()
Data             Property   System.Collections.IDictionary Data {get;}
ErrorCode        Property   int ErrorCode {get;}
HelpLink         Property   string HelpLink {get;set;}
HResult          Property   int HResult {get;}
InnerException   Property   System.Exception InnerException {get;}
Message          Property   string Message {get;}
NativeErrorCode  Property   int NativeErrorCode {get;}
SocketErrorCode  Property   System.Net.Sockets.SocketError SocketErrorCode {get;}
Source           Property   string Source {get;set;}
StackTrace       Property   string StackTrace {get;}
TargetSite       Property   System.Reflection.MethodBase TargetSite {get;}

PS C:\Scripts\PowerShell\Projects\ConnectCluster> $error[0].Exception.SocketErrorCode
HostNotFound

PS C:\Scripts\PowerShell\Projects\ConnectCluster> $error[0].Exception.ErrorCode
11001

PS C:\Scripts\PowerShell\Projects\ConnectCluster> $error[0].Exception.StackTrace
   at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
   at System.Net.Dns.GetHostAddresses(String hostNameOrAddress)
   at NetApp.Ontapi.NaServer..ctor(String name, String uri, Int32 port)
   at NetApp.Ontapi.Filer.C.NcController..ctor(String hostName)
   at DataONTAP.C.PowerShell.SDK.ConnectNcController.ProcessRecord(String name)
   at DataONTAP.C.PowerShell.SDK.ConnectNcController.ProcessRecord()
   at System.Management.Automation.CommandProcessor.ProcessRecord()

I know this doesn't answer your question as to why the connection is failing but hopefully this information will be helpful to troubleshoot and identify the root cause of the issue for you.

 

Cheers

 

/Matt

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

Re: API invoke failed on connect-ncController

Thanks Matt!!

 

I will run that script on a schedule and hopefully get some connection erreors.