Microsoft Virtualization Discussions
Microsoft Virtualization Discussions
Hi,
From time to time I get an error while trying to connect to some of my clusters.
The command used is: Connect-NcController $ControllerName -Credential $cred -HTTPS -Timeout 480000
And I get this error:
Connect-NcController : API invoke failed.
At E:\Folders\Stats.ps1:1355 char:5
+ Connect-NcController $ControllerName -Credential $cred -HTTPS -Timeout
4800 ...
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidResult: (server.domain.com:NcController
) [Connect-NcController], NaException
+ FullyQualifiedErrorId : HttpConnectionFailed,DataONTAP.C.PowerShell.SDK.
ConnectNcController
If I try it later it works. Anyone know what could be wrong?
Thanks
Phil
Hi Phil,
It sounds like a transient error which are always the most frustrating and often difficult to troubleshoot, so to help identify what might be causing it lets start by checking version of the DataONTAP powershell toolkit are you running and what OS version of ONTAP are you connecting to?
Check the powershell toolkit version using:
Import-Module -Name DataONTAP Get-NaToolkitVersion
Can you check the following:
>version
>system service web show
What is your ONTAP version and configuration for:
Is the configuration and ONTAP version between your clusters different? In particular the SSLv3 and FIPS configuration.
/Matt
Hi Matt,
Here's the info you requested. Let me know if you need to know anything else.
Powershell toolkit version: 3.3.0.65
NetApp version: NetApp Release 8.3.1P2: Wed Dec 09 03:10:24 UTC 2015
System service web show:
External Web Services: true
Status: online
HTTP Protocol Port: 80
HTTPs Protocol Port: 443
TLSv1 Enabled: true
SSLv3 Enabled: false
SSL FIPS 140-2 Enabled: false
The configuration and NetApp version are the same. The only difference is, one cluster is local and the other is remote(another site).
Hi Philip,
Thanks, the powershell toolkit and ontap versions you are running are fine and you should be able to connect to each cluster without issue. The configuration doesn't indicate the potential issue i suspected (In ONTAP9.X you can have connectivity issues with FIPS enabled, specifically using HTTPS\Invoke-NcSsh). Given that is not the issue if you do a:
> certificate show
Are all the certificates still valid (not expired). I'd assume both local clusters have DNS A & PTR records created and you've verfied these are resolvable using nslookup from the host you are attempting to connect from? Does the host you are connecting from have a route to the remote cluster? Is the problem consistently repeatable?
/Matt
Hi Matt,
Yes all the certificates are still valid.
Yes we have DNS A & PTR records, nslookup are working also.
The problem happens a couple time a week and not at the same time during the day or night.
Might be an issue not related to NetApp at all I guess. Network, Windows, powershell issue maybe.
Hi Philip
I can't see any storage issue that would cause the API to fail to connect to the cluster. It's certainly possible it's an operating system or network issue in your environment or on the host you are running the script from rather than a storage issue. Does your script create any logs with timestamps so that you can trace what time the errors occurred in correlation to any event logs? I noticed you've set the timeout to 480000 (8 minutes) have you tried increasing this 3600000 (60 minutes)?. Also is E:\ drive that your script is running from a local disk, a mapped network share or a mapped LUN?
/Matt
Hi Matt,
Yes the script is creating logs and that's why I'm pretty sure it's not a timeout issue. The command is sent to the cluster and we get a return almost right away that it failed to connect.
The script is running on a local drive.
I'll just add some error handling in the script and retry the connection when it fails.
Thanks for your help Matt.
Hi Philip,
Here is an example of some error handling and logging functions that will add timestamps to help troubleshoot the issue.
The following example scripts does the following:
A successful connection is logged to a .log file in YYYY-MM-DD format. Any errors are logged to a .err file
#'------------------------------------------------------------------------------ Param( [Parameter(Mandatory=$True, HelpMessage="The hostname or IP Address of the cluster to connect to")] [String]$ClusterName ) #'------------------------------------------------------------------------------ Function Get-IsoDateTime{ <# .SYNOPSIS This Function enumerates the date and time in ISO format. .DESCRIPTION Returns a date time in the form of "YYYY-MM-DD HH:MM:SS". .EXAMPLE Get-IsoDateTime #> Return (Get-IsoDate) + " " + (Get-IsoTime) }#End Function #'------------------------------------------------------------------------------ Function Get-IsoDate{ <# .SYNOPSIS This Function enumerates the date in ISO format. .DESCRIPTION Returns a date in the format of "YYYY-MM-DD". .EXAMPLE Get-IsoDate #> Return Get-Date -uformat "%Y-%m-%d" }#End Function #'------------------------------------------------------------------------------ Function Get-IsoTime{ <# .SYNOPSIS This Function enumerates the time in ISO format. .DESCRIPTION Returns a time in the format of "HH:MM:SS". .EXAMPLE Get-IsoTime #> Return Get-Date -uformat "%H:%M:%S" }#End Function #'------------------------------------------------------------------------------ Function Append-Message{ <# .SYNOPSIS This function appends a message to log file based on the message type. .DESCRIPTION Appends a message to a log a file. .PARAMETER Accepts an integer representing the log file extension type .PARAMETER Accepts a string value containing the message to append to the log file. .EXAMPLE Append-Message -logType 0 -message "Command completed succuessfully" .EXAMPLE Append-Message -logType 2, -message "Application is not installed" #> [CmdletBinding()] Param( [Parameter(Position=0, Mandatory=$True, ValueFromPipeLine=$True, ValueFromPipeLineByPropertyName=$True)] [Int]$LogType, [Parameter(Position=1, Mandatory=$True, ValueFromPipeLine=$True, ValueFromPipeLineByPropertyName=$True)] [String]$Message ) Switch($logType){ 0 {$extension = "log";} 1 {$extension = "err";} 2 {$extension = "err";} 3 {$extension = "csv";} default {$extension = "log"} } If($logType -eq 1){ $message = ("Error`: " + $error[0].Exception.Message + ". Code`: " + $error[0].Exception.ErrorCode + ". Message`: " + $Message) } $prefix = Get-IsoDateTime Try{ ($prefix + "," + $message) | Out-File -filePath ` ($scriptLogPath + "." + $extension) -encoding ASCII -Append -ErrorAction SilentlyContinue }Catch{ Break; } }#End Function #'------------------------------------------------------------------------------ #'Initialization Section. Define Global Variables. #'------------------------------------------------------------------------------ [String]$scriptPath = Split-Path($MyInvocation.MyCommand.Path) [String]$scriptSpec = $MyInvocation.MyCommand.Definition [String]$scriptBaseName = (Get-Item $scriptSpec).BaseName [String]$scriptName = (Get-Item $scriptSpec).Name [String]$scriptLogPath = $scriptPath + "\" + (Get-IsoDate) #'------------------------------------------------------------------------------ Import-Module -Name DataONTAP -ErrorAction SilentlyContinue $credentials = $host.ui.PromptForCredential("Connect to Cluster ""$ClusterName""", "Please enter the user name and password", "admin", "") Try{ Connect-NcController -Name $clusterName -HTTPS -Credential $credentials -ErrorAction Stop Append-Message 0 "Connected to ""$clusterName""" }Catch{ Append-Message 1 "Failed connecting to cluster ""$clusterName""" Exit -1 } #'------------------------------------------------------------------------------
An example of running the script:
PS C:\Scripts\PowerShell\Projects\ConnectCluster> .\ConnectCluster.ps1 -ClusterName cluster1.testlab.local
An example of the log files for both successful and unsuccessful connections:
C:\Scripts\PowerShell\Projects\ConnectCluster>dir /b 2016-09-30.err 2016-09-30.log ConnectCluster.ps1 C:\Scripts\PowerShell\Projects\ConnectCluster>type 2016-09-30.log 2016-09-30 14:58:01,Connected to "cluster1.testlab.local" C:\Scripts\PowerShell\Projects\ConnectCluster>type 2016-09-30.err 2016-09-30 14:59:35,Error: No such host is known. Code: 11001. Message: Failed connecting to cluster "cluster10.testlab.local"
By logging the value of the "$error[0].Exception.ErrorCode" it might help to identify the cause of the issue.
Also If you look at the Methods for the "$error[0].Exception" object you'll notice you can do a stacktrace which might be useful. EG
PS C:\Scripts\PowerShell\Projects\ConnectCluster> $error[0].Exception | Get-Member TypeName: System.Net.Sockets.SocketException Name MemberType Definition ---- ---------- ---------- Equals Method bool Equals(System.Object obj), bool _Exception.Equals(System.Object obj) GetBaseException Method System.Exception GetBaseException(), System.Exception _Exception.GetBaseException() GetHashCode Method int GetHashCode(), int _Exception.GetHashCode() GetObjectData Method void GetObjectData(System.Runtime.Serialization.SerializationInfo info, System.Runtime.S... GetType Method type GetType(), type _Exception.GetType() ToString Method string ToString(), string _Exception.ToString() Data Property System.Collections.IDictionary Data {get;} ErrorCode Property int ErrorCode {get;} HelpLink Property string HelpLink {get;set;} HResult Property int HResult {get;} InnerException Property System.Exception InnerException {get;} Message Property string Message {get;} NativeErrorCode Property int NativeErrorCode {get;} SocketErrorCode Property System.Net.Sockets.SocketError SocketErrorCode {get;} Source Property string Source {get;set;} StackTrace Property string StackTrace {get;} TargetSite Property System.Reflection.MethodBase TargetSite {get;} PS C:\Scripts\PowerShell\Projects\ConnectCluster> $error[0].Exception.SocketErrorCode HostNotFound PS C:\Scripts\PowerShell\Projects\ConnectCluster> $error[0].Exception.ErrorCode 11001 PS C:\Scripts\PowerShell\Projects\ConnectCluster> $error[0].Exception.StackTrace at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) at System.Net.Dns.GetHostAddresses(String hostNameOrAddress) at NetApp.Ontapi.NaServer..ctor(String name, String uri, Int32 port) at NetApp.Ontapi.Filer.C.NcController..ctor(String hostName) at DataONTAP.C.PowerShell.SDK.ConnectNcController.ProcessRecord(String name) at DataONTAP.C.PowerShell.SDK.ConnectNcController.ProcessRecord() at System.Management.Automation.CommandProcessor.ProcessRecord()
I know this doesn't answer your question as to why the connection is failing but hopefully this information will be helpful to troubleshoot and identify the root cause of the issue for you.
Cheers
/Matt
Thanks Matt!!
I will run that script on a schedule and hopefully get some connection erreors.