NetApp OnCommand Plugin for Microsoft 4.1.3P2 powershell script failed Exception calling "InvokeMeth

[DavidMahler]

Hello,

We are running SCOM 2012 R2 UR14 and NetApp OnCommand Plugin for Microsoft 4.1.3P2.

2 powershell scripts continue to fail.  Any thoughts on how to address?

Thank you for any help!

--David

1.

The PowerShell script failed with below exception

System.Management.Automation.MethodInvocationException: Exception calling "InvokeMethod" with "6" argument(s): "Execution of OC.Cluster.OM.Server.Monitoring.StorageMonitoring.GetDataVserveriSCSIPerformanceStatistics method resulted in exception being thrown. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> NetApp.Ontapi.NaUnknownErrnoException: For iscsi_lif:vserver object, no instances were found to match the given query.
at NetApp.Ontapi.NaApi`1.Invoke(INaServer server)
at OC.Common.ClusterLibrary.ZAPI.Performance.Performance.GetPerformanceInstanceData(NcController controller, String[] counters, String filterData, String[] instanceUUIDs, String[] instances, String objectName, String privilegeLevel)
at OC.Cluster.OM.Server.Monitoring.StorageMonitoring.GetDataVserverIscsiPerformanceStatistics(String dataVserverUuid)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker.InvokeMethod(Assembly assembly, String typeName, Object[] constructorParameters, String methodName, Object[] parameters)
at OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker.AppDomainCallback()"At line:64 char:5
+ $result = [OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker]::Invo ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
at Microsoft.EnterpriseManagement.Common.PowerShell.RunspaceController.RunScript[T](String scriptName, String scriptBody, Dictionary`2 parameters, Object[] constructorArgs, IScriptDebug iScriptDebug, Boolean bSerializeOutput)

Script Name: Monitoring.ps1

One or more workflows were affected by this.

Workflow name: DataONTAP.Cluster.Monitoring.GetDataVserveriSCSIPerformanceStatistics.Rule

Instance name: Storage virtual machine chnetapp35nas.

2.

The PowerShell script failed with below exception

System.Management.Automation.MethodInvocationException: Exception calling "InvokeMethod" with "6" argument(s): "Execution of OC.Cluster.OM.Server.Monitoring.StorageMonitoring.GetDataVserverFCPPerformanceStatistics method resulted in exception being thrown. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> NetApp.Ontapi.NaUnknownErrnoException: For fcp_lif:vserver object, no instances were found to match the given query.
at NetApp.Ontapi.NaApi`1.Invoke(INaServer server)
at OC.Common.ClusterLibrary.ZAPI.Performance.Performance.GetPerformanceInstanceData(NcController controller, String[] counters, String filterData, String[] instanceUUIDs, String[] instances, String objectName, String privilegeLevel)
at OC.Cluster.OM.Server.Monitoring.StorageMonitoring.GetDataVserverFcpPerformanceStatistics(String dataVserverUuid)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker.InvokeMethod(Assembly assembly, String typeName, Object[] constructorParameters, String methodName, Object[] parameters)
at OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker.AppDomainCallback()"At line:64 char:5
+ $result = [OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker]::Invo ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
at Microsoft.EnterpriseManagement.Common.PowerShell.RunspaceController.RunScript[T](String scriptName, String scriptBody, Dictionary`2 parameters, Object[] constructorArgs, IScriptDebug iScriptDebug, Boolean bSerializeOutput)

Script Name: Monitoring.ps1

One or more workflows were affected by this.

Workflow name: DataONTAP.Cluster.Monitoring.GetDataVserverFCPPerformanceStatistics.Rule

Instance name: Storage virtual machine chnetapp35nas.

[mbeattie]

Hi David,

Is it possible the error is being thrown because the script is attempting to call methods on objects on a vserver that might not exist? (IE performance statistics for ISCSI and FCP LIFS) on a vserver that might be restricted to CIFS or NFS protocols?

What's the output of:

net int show -vserver chnetapp35nas -fields lif, data-protocol, services

Does that vserver have any LIFs that use the ISCSI or FCP protcols?

/Matt

[DavidMahler]

Matt,

Thank you for your help!

Yes, you are correct.  We have 2 SVMs.  One does not use either ISCSI or FCP and the other does not use ISCSI.

Is there a way to configure SCOM to avoid these powershell script errors?

I appreciate your assistance.

--David

[StewartMilne]

We have this exact same error with 4.1.3P2, both with SCOM 2012 R2 and 2019. We get "Power Shell Script failed to run" alerts for every SVM we're monitoring:

The PowerShell script failed with below exception

System.Management.Automation.MethodInvocationException: Exception calling "InvokeMethod" with "6" argument(s): "Execution of OC.Cluster.OM.Server.Monitoring.StorageMonitoring.GetDataVserverFCPPerformanceStatistics method resulted in exception being thrown. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> NetApp.Ontapi.NaUnknownErrnoException: For fcp_lif:vserver object, no instances were found to match the given query.
at NetApp.Ontapi.NaApi`1.Invoke(INaServer server)
at OC.Common.ClusterLibrary.ZAPI.Performance.Performance.GetPerformanceInstanceData(NcController controller, String[] counters, String filterData, String[] instanceUUIDs, String[] instances, String objectName, String privilegeLevel)
at OC.Cluster.OM.Server.Monitoring.StorageMonitoring.GetDataVserverFcpPerformanceStatistics(String dataVserverUuid)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker.InvokeMethod(Assembly assembly, String typeName, Object[] constructorParameters, String methodName, Object[] parameters)
at OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker.AppDomainCallback()"At line:64 char:5
+ $result = [OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker]::Invo ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
at Microsoft.EnterpriseManagement.Common.PowerShell.RunspaceController.RunScript[T](String scriptName, String scriptBody, Dictionary`2 parameters, Object[] constructorArgs, IScriptDebug iScriptDebug, Boolean bSerializeOutput)


Script Name: Monitoring.ps1
One or more workflows were affected by this.
Workflow name: DataONTAP.Cluster.Monitoring.GetDataVserverFCPPerformanceStatistics.Rule
Instance name: Storage virtual machine svmvsv.
Instance ID: {BFE16D2A-4003-FF70-B6A0-6D145382AAA0}
Management group: SCOMMG

[mbeattie]

Hi Stewart,

It sounds like the "monitoring.ps1" is attempting to invoke methods to enumerate performance statistics for FCP LIFS which may not exist on your vserver if you have restricted the protocols to CIFS\NFS\ISCSI

NetApp.Ontapi.NaUnknownErrnoException: For fcp_lif:vserver object, no instances were found to match the given query

I'd suggest looking at the monitoring.ps1 script and updating it to enumerate the vservers allowed protocols then only attempt to invoke performance statistic methods for vservers configured protocols. Can you post the script?

/Matt

[StewartMilne]

Thanks for the response Matt.

The Monitoring.ps1 script is just the default one that comes bundled with the management pack. It's embedded in the DataONTAP.Cluster.807.xml file under C:\Program Files\Microsoft System Center\Operations Manager\Server\Health Service State\Management Packs on the management servers with the OnCommand plugin installed.

#####################
# Copyright (c) 2013 NetApp.
#
# Monitoring.ps1
# 
# This script to run clustered storage monitoring.
#
##############################################################################
Param($MonitoringMethodName, $VserverUUID)

##############################################################################
#Script utilities
function GetInstallPath
{
    $netappRegPath = "HKLM:\SOFTWARE\NetApp\OnCommand\MS_Plugin"
    $ibmRegPath = "HKLM:\SOFTWARE\IBM\OnCommand\MS_Plugin"

    if (Test-Path -path $netappRegPath)
    {
      # Check for the registry key value of the install location for NetApp
        $key = get-itemproperty $netappRegPath
    }
    elseif (Test-Path -path  $ibmRegPath)
    {
        # Check for IBM registry key since can't find NetApp
        $key= get-itemproperty  $ibmRegPath
    }
    else
    {
        throw ("OnCommand Plug-in for Microsoft is not installed on " + $env:COMPUTERNAME + "`r`n")
    }

	return $key.InstallPath
}

function CreateAppDomain
{
    $setup = [AppDomain]::CurrentDomain.SetupInformation;
    $setup.ShadowCopyFiles = $true
    return [AppDomain]::CreateDomain("OC.Cluster.OM.Powershell.NonDefaultAppDomain", 
        $null, 
        $setup);
}

function InvokeMethodInAppDomain([AppDomain]$appDomain,
    [string]$assemblyPath,
    [string]$typename,
    [array]$contructorParameters,
    [string]$methodName,
    [array]$methodParameters)
{
    #In SCOM a Powershell script is executed in a MonitoringHost.exe process.
    #Lifetime of a  MonitoringHost.exe is not controlled by us and it continues to run after the script is done, possibly
    #running other scripts or the same script again. A script is executed in a default AppDomain (by SCOM design) and
    #as a result any assemblies loaded by script can not be unloaded because the defualt AppDomain can not be unloaded.
    #Impossibility of unloading an assembly becomes a problem when then the assembly DLL is being updated 
    #(i.e. during an istallation), the problem is that even after the assembly DLL is updated, 
    #the new assembly code is not loaded if the assembly is already loaded.
    #The solution for this problem is to load an assembly to a non-default domain which can be unloaded when the script is done.
    
    #The OC.Cluster.OM.AppDomainInvoker assembly has to be in the GAC
    $AppDomainInvokerAssembly = [System.Reflection.Assembly]::LoadWithPartialName('OC.Cluster.OM.AppDomainInvoker')

    $result = [OC.Cluster.OM.AppDomainInvoker.AppDomainInvoker]::InvokeMethod($appDomain,
        $assemblyPath,
        $typename,
        $contructorParameters,
        $methodName,
        $methodParameters)

    return $result
}
##############################################################################

import-module operationsmanager

$Error.Clear();

$scriptName = "Monitoring.ps1"

try
{
    $api = new-object -comObject 'MOM.ScriptAPI'

    $logMessage = "Calling Clustered Data ONTAP monitoring method " + $MonitoringMethodName + " for storage system with UUID " + $VserverUUID
    $api.LogScriptEvent($scriptName, 3000, 0, $logMessage)

    $nonDefaultAppDomain = CreateAppDomain

    $installPath = GetInstallPath

    InvokeMethodInAppDomain $nonDefaultAppDomain `
        (Join-Path  $installPath 'OC.Cluster.OM.Server.dll') `
        "OC.Cluster.OM.Server.Monitoring.StorageMonitoring" `
        @("localhost") `
        $MonitoringMethodName `
        @($VserverUUID)

    $api.LogScriptEvent($scriptName, 3001, 0, "Monitoring succeeded.")
}
catch
{
	if ($api)
	{
		$api.LogScriptEvent($scriptName, 3002, 1, "Monitoring failed: `r`n" + $Error)
	}

	#throw the current error so that it will be displayed in the SCOM task output and status of the task will be set as failed
    throw
}
finally
{
    if ($nonDefaultAppDomain)
    {
         [AppDomain]::Unload($nonDefaultAppDomain)
         $nonDefaultAppDomain = $null
    }
}

[mbeattie]

Hi Stuart,

Thanks for posting the code, looking at the script it appears to create a COM object, load a DLL and invoke some methods. I suspect the issue exists within the .dll rather than the script. Did you open a support case? If so post the case number and i'll take a look at the notes.

/Matt

[StewartMilne]

Hi Matt,

I haven't raised a support case for this yet. I'll create one and send you the number.

Thanks!

[mbeattie]

Hi,

Just updating this thread as this issue is not a scripting\programming problem. The work around is to configure SCOM to override rules in the management pack as required (EG if the SVM’s are not running ISCSI or FCP then disable the following rules so the script doesn't fail attempting to enumerate performance statistics for protocols that are not applicable).

You must determine which rules are applicable to the SVM's that are being monitored based on the vservers allowed protocols and data LIFS and disable as required. The following link contains a list of the SCOM management pack rules:

https://library.netapp.com/ecmdocs/ECMP1650593/html/GUID-F4B0A0ED-7C10-479C-AD61-546008C6C1E9.html

EG if the SVM's are only serving NAS data (CIFS or NFS) then overide the following rules to disable them:

• Clustered Data ONTAP: Storage Virtual Machine (SVM) FCP Performance Statistics Collection Rule
• Clustered Data ONTAP: Storage Virtual Machine (SVM) iSCSI Performance Statistics Collection Rule

To override the rules see the documentation links below:

https://library.netapp.com/ecmdocs/ECMP1650593/html/GUID-D32AF2A8-C726-4D53-8E5E-E3BC6FBE23DA.html

https://library.netapp.com/ecmdocs/ECMP1650593/html/GUID-D84C2545-EF1F-4A8F-AD8C-C210E1DBC17D.html

Hope that helps

/Matt