OCUM vApp 9.4P1 alert script to make REST calls to WFA workflow

[cjautonetics]

Hi there,

I have been working on a script based on the Self healing whitepaper - https://www.netapp.com/us/media/tr-4585.pdf

The script is for OCUM inode alert to trigger a WFA work flow to fix inodes

Since we are using the vApp I had to use a perl scrip instead of a  power shell script

Thanks to the white paper and information available on this community i was able to get  WFA workflow working to fix the inodes , but having issues with the perl script on OCUM that is needed to make the REST API call.

 Took help from some of the debugging instructions shared by @mbeattie in a few different posts here and identified that the issue was because REST/Client.pm was not available on the OCUM vApp that is needed by the perl script.

Debug error message when the perl script is executed:

2019-07-03 11:01:43 [INFO ] [ScriptExecutor-2] [AlertScriptService:108] - Script execution of file : OCUM_WFA_Inode_Event.pl for EventID : 270028 in progress..
Can't locate REST/Client.pm in @INC (you may need to install the REST::Client module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /opt/netapp/ocum/scriptPlugin/OCUM_WFA_Inode_Event.pl line 7.

I tried installing REST::Client on the vApp using CPAN , but getting below error :

cpan[1]> install REST::Client
Fetching with HTTP::Tiny:
http://www.cpan.org/authors/01mailrc.txt.gz
HTTP::Tiny failed with an internal error: Could not connect to 'www.cpan.org:80': Network is unreachable

Trying with
/usr/bin/wget -O "/root/.cpan/sources/authors/01mailrc.txt.gz.tmp26476"
to get
http://www.cpan.org/authors/01mailrc.txt.gz
--2019-07-04 08:03:45-- http://www.cpan.org/authors/01mailrc.txt.gz
Resolving www.cpan.org (www.cpan.org)... 151.101.78.217, 2a04:4e42:12::729
Connecting to www.cpan.org (www.cpan.org)|151.101.78.217|:80... failed: Connection timed out.
Connecting to www.cpan.org (www.cpan.org)|2a04:4e42:12::729|:80... failed: Network is unreachable.

Function system("/usr/bin/wget -O "/root/.cpan/sources/authors/01mailrc.txt.gz.tmp26476" "http://www.cpan.org/authors/01mailrc.txt.gz" ")
returned status 4 (wstat 1024), left
/root/.cpan/sources/authors/01mailrc.txt.gz.tmp26476 with size 0
Warning: no success downloading '/root/.cpan/sources/authors/01mailrc.txt.gz.tmp26476'. Giving up on it.

I believe this is because the way vApp is designed . That is locked down to prevent outside softwares or patches being applied ? But shouldnt making REST API calls be one of the basic functionalities that must be built in into OCUM ?

Any help would be greatly appreciated!

[mbeattie · Accepted solution]

Hi CJ

Yes it is certainly frustrating, unfortunately the OCUM vApp is closed architecture (and i doubt that is going to change), installation of third party software in the vApp (whilst "technically" possible) is NOT supported. The vApp does not include PERL or the modules required to invoke REST API's so the script plugin isn't much use if using the OCUM vApp.

However as an alternative it is possilble to poll OCUM events via the REST API therefore if you have an orchetration application (such as WFA) you can configure a schedule to check for any events and process them as required. This isn't as quite as efficent as directly using the script plugin if OCUM is installed on Windows or RHEL to process events in near to real-time however in general it does suffice for the majority of requirements.

Here is some "example" source code to get you started if you wanted to poll the OCUM REST API events and process any new events that match a specific event type (in the example i'm process the "volume" resource type where the event name is "inodes full" or "inodes nearly full").

This method enables customers who are using the OCUM vApp to perform self-healing but on scheduled basis. EG you could poll OCUM ever 15-30 minutes (hourly is probably sufficent) using WFA (or worst case a schedule task). If new events are identified that match the types you want to process then the scheduled WFA workflow will invoke a sub workflow to resolve the event.

Hope this helps? Any questions please let me know.

/Matt

Param(
   [Parameter(Mandatory=$False, HelpMessage="The WFA Server Hostname, IP Address or FQDN")]
   [String]$WfaHostname,
   [Parameter(Mandatory=$False, HelpMessage="The WFA Server Port Number")]
   [Int]$WfaPortNumber=443,
   [Parameter(Mandatory=$True, HelpMessage="The OCUM Server Hostname, IP Address or FQDN")]
   [String]$OcumHostname,
   [Parameter(Mandatory=$False, HelpMessage="The OCUM Server Port Number")]
   [Int]$OcumPortNumber=443,
   [Parameter(Mandatory=$True, HelpMessage="The number of hours to enumerates OCUM events for")]
   [Int]$EventRangeHours,
   [Parameter(Mandatory=$True, HelpMessage="The WFA Event Severity to process. Supports a comma delimited list. Valid values are 'critical', 'error', 'warning', 'normal' and 'information'")]
   [String]$EventSeverity,
   [Parameter(Mandatory=$True, HelpMessage="The WFA Event Source Type to process. Supports a comma delimited list. Valid values are 'volume', 'inode'")]
   [String]$EventSourceType,
   [Parameter(Mandatory=$True, HelpMessage="The WFA Event Impact Level to process. Supports a comma delimited list. Valid values are 'incident', 'risk' and 'event'")]
   [String]$EventImpactLevel,
   [Parameter(Mandatory=$True, HelpMessage="The WFA Event State to process. Supports a comma delimited list. Valid values are 'new', 'acknowledged', 'resolved' and 'obsolete'")]
   [String]$EventState,
   [Parameter(Mandatory=$True, HelpMessage="The WFA workflow name for resizing volumes")]
   [String]$ResizeVolumeWorkflow,
   [Parameter(Mandatory=$True, HelpMessage="The Volume percentage increase")]
   [ValidateRange(1, 100)]
   [Int]$VolumePercentIncrease,
   [Parameter(Mandatory=$True, HelpMessage="The WFA workflow name for increasing inodes")]
   [String]$IncreaseInodesWorkflow,
   [Parameter(Mandatory=$True, HelpMessage="The WFA workflow name for restarting an SVM service")]
   [String]$StartSvmServiceWorkflow,
   [Parameter(Mandatory=$False, HelpMessage="If specified verbose logging will be enabled")]
   [Bool]$VerboseLogging=$False
)
#'------------------------------------------------------------------------------
Function ConvertFrom-UnixTimestamp{
   Param(
      [Parameter(Mandatory=$True, HelpMessage="The UNIX timestamp")]   
      [String]$Timestamp
   )
   Get-Date -Date $([TimeZone]::CurrentTimeZone.ToLocalTime(([DateTime]'1/1/1970').AddMilliseconds($Timestamp))) -Uformat "%Y-%m-%d %H:%M:%S"
}#End Function
#'------------------------------------------------------------------------------
Function ConvertTo-UnixTime{
   Param(
      [Parameter(Mandatory=$True, HelpMessage="The ISO timestamp to convert")]
      [String]$Timestamp
   )
   [DateTime]$Epoch = '1970-01-01 00:00:00' #ISO date time format.
   $epochTime = [Math]::Floor((New-TimeSpan -Start $epoch -End (Get-Date -Date $Timestamp)).TotalSeconds)
   return $epochTime;
}#End Function
#'------------------------------------------------------------------------------
Function Get-OCUMClusterAddress{
   Param(
      [Parameter(Mandatory=$True, HelpMessage="The OCUM server's hostname or IP Address")]   
      [String]$Hostname,
      [Parameter(Mandatory=$True, HelpMessage="The OCUM cluster ID number")]   
      [Int]$ClusterId,
      [Parameter(Mandatory=$True, HelpMessage="The OCUM credentials used to authenticate the REST API request")]
      [System.Management.Automation.PSCredential]$Credentials
   )
   [String]$username = $Credentials.GetNetworkCredential().UserName
   [String]$password = $Credentials.GetNetworkCredential().Password
   $encodedCredentials = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$username`:$password"))
   $headers = @{
      "Authorization" = "Basic " + $encodedCredentials
      "Accept"        = "application/vnd.netapp.object.inventory.hal+json"
   }
   [String]$uri = "https://$HostName/rest/clusters`?filter`=clusterId`%2Cis`%2C$ClusterId"
   [String]$clusterAddress = ""
   Try{
      $cluster = Invoke-RestMethod -Headers $headers -Uri $uri -ErrorAction Stop
   }Catch{
      Get-WFALogger -Error -Message $("Failed enumerating OCUM cluster ID $ClusterID using URI ""$uri"". Error " + $_.Exception.Message)
      Return $clusterAddress;
   }
   If($Null -ne $Cluster){
      [String]$clusterAddress = $cluster._embedded.'netapp:clusterInventoryList'.networkAddress
      Get-WFALogger -Info -Message "Enumerated OCUM cluster ID $ClusterID as ""$clusterAddress"" using URI ""$uri"". "
   }Else{
      Get-WFALogger -Error -Message "Failed enumerating OCUM cluster ID $ClusterID using URI ""$uri"""
   }
   Return $clusterAddress;
}
#'------------------------------------------------------------------------------
Function ConvertTo-WfaXml{
   Param(
      [Parameter(Mandatory=$True, HelpMessage="The WFA workflow inputs")]
      [HashTable]$WorkflowInputs,
      [Parameter(Mandatory=$False, HelpMessage="The WFA workflow comment")]
      [String]$Comment
   )
   $xml  = "<?xml version=""1.0"" encoding=""UTF-8""?>`n"
   $xml += '<workflowInput><userInputValues>'
   ForEach($key In $WorkflowInputs.keys){
      $xml += '<userInputEntry key="' + $key + '" value="' + $WorkflowInputs[$key] + '" />'
   }
   If($Comment){
      $xml += '</userInputValues><comments>' + $Comment + '</comments></workflowInput>'
   }Else{
      $xml += '</userInputValues><comments></comments></workflowInput>'
   }
   Return $xml;
}#End Function
#'------------------------------------------------------------------------------
Function Invoke-WFAWorkflow{
   Param(
      [Parameter(Mandatory=$True, HelpMessage="The WFA portal URI")]   
      [String]$Uri,
      [Parameter(Mandatory=$True, HelpMessage="The WFA workflow name")]   
      [String]$Workflow,
      [Parameter(Mandatory=$False, HelpMessage="The WFA workflow comment")]   
      [String]$Comment,
      [Parameter(Mandatory=$True, HelpMessage="The WFA workflow input parameters")]   
      [Hashtable]$Inputs,
      [Parameter(Mandatory=$True, HelpMessage="The crednentials used to authenticate to WFA with")]
      [System.Management.Automation.PSCredential]$Credentials
   )
   #'---------------------------------------------------------------------------
   #'Enumerate WFA environment variables.
   #'---------------------------------------------------------------------------
   [String]$workflowName = $(Get-WfaRestParameter "workflowName")
   [Int]$jobId           = $(Get-WfaRestParameter "jobId")
   [String]$userName     = $(Get-WfaRestParameter "userName")
   #'---------------------------------------------------------------------------
   #'Enumerate the WFA workflow REST execution URL.
   #'---------------------------------------------------------------------------
   If(-Not($Uri.EndsWith("/rest/workflows"))){
      [String]$Uri = "$Uri/rest/workflows"
   }
   #'------------------------------------------------------------------------------
   #'Ignore self signed SSL certificate on WFA server
   #'------------------------------------------------------------------------------
   Add-Type @"
      using System.Net;
      using System.Security.Cryptography.X509Certificates;
      public class TrustWFACertsPolicy : ICertificatePolicy {
         public bool CheckValidationResult(
         ServicePoint srvPoint, X509Certificate certificate,
         WebRequest request, int certificateProblem) {
         return true;
      }
   }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustWFACertsPolicy
   #'---------------------------------------------------------------------------
   #'Disable certificate CRL check and set TLS.
   #'---------------------------------------------------------------------------
   [System.Net.ServicePointManager]::CheckCertificateRevocationList = $False;
   #[ServerCertificateValidationCallback]::Ignore();
   [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$True}
   [System.Net.ServicePointManager]::SecurityProtocol = @("Tls12","Tls11","Tls","Ssl3")
   #[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
   #'---------------------------------------------------------------------------
   #'Enumerate the WFA workflow Execution URI.
   #'---------------------------------------------------------------------------
   Get-WFALogger -Info -Message "Enumerating URI for workflow ""$workflow"" using URI ""$Uri"""
   Try{
      [Xml]$workflowXML             = Invoke-RestMethod -Credential $credentials -Method Get -Uri "$Uri" -Body @{"name" = $Workflow;} -ErrorAction Stop
      [String]$workflowUuid         = $workflowXML.collection.workflow.uuid
      [String]$workflowExecutionUrl = "$Uri/$workflowUuid/jobs"
      Get-WFALogger -Info -Message "Enumerated execution URL for WFA workflow ""$workflow"" as URI ""$workflowExecutionUrl"""
   }Catch{
      Get-WFALogger -Error -Message $("Failed enumerating execution URL for WFA workflow ""$workflow"" using URI ""$Uri"". Error " + $_.Exception.Message)
      Throw "Failed enumerating execution URL for WFA workflow ""$workflow"" using URI ""$Uri"""
   }
   If([String]::IsNullOrEmpty($workflowUuid)){
      Throw "Failed enumerating Workflow UUID for workflow ""$Workflow"" using URI ""$Uri"""
   }
   ForEach($key In $Inputs.Keys){
      Get-WFALogger -Info -Message $("Parameter Name`: " + $key + ". Value`: " + $inputs[$key])
   }
   #'---------------------------------------------------------------------------
   #'Convert the WFA input paramaters into XML.
   #'---------------------------------------------------------------------------
   If(-Not($Comment)){
      [String]$Comment = "Invoked by workflow ""$workflowName"" job ID ""$jobId"" by user ""$userName"""
   }
   $restInput = ConvertTo-WfaXml -WorkflowInputs $Inputs -Comment $Comment
   Get-WFALogger -Info -Message "Execution Comment`: $Comment"
   #'---------------------------------------------------------------------------
   #'Set .NET service point manager defaults.
   #'---------------------------------------------------------------------------
   #[System.Net.ServicePointManager]::Expect100Continue     = $False
   [System.Net.ServicePointManager]::DefaultConnectionLimit = 500
   [System.Net.ServicePointManager]::MaxServicePoints = 10;
   [System.Net.ServicePointManager]::CheckCertificateRevocationList = $False;
   $errorActionPreference = "Stop"
   #'---------------------------------------------------------------------------
   #'Invoke the WFA workflow.
   #'---------------------------------------------------------------------------
   Try{
      $servicePoint      = [System.Net.ServicePointManager]::FindServicePoint($workflowExecutionUrl)
      [xml]$executionXML = Invoke-WebRequest -Method Post -Credential $Credentials -Uri $workflowExecutionUrl -body $restInput -ContentType "application/xml" -ErrorAction Stop
      Get-WFALogger -Info -Message "Invoked WFA workflow ""$Workflow"" using URI ""$workflowExecutionUrl"""
      $servicePoint.CloseConnectionGroup("")
   }Catch{
      $servicePoint.CloseConnectionGroup("")
      Get-WFALogger -Error -Message $("Failed invoking WFA workflow ""$workflow"" using URI ""$workflowExecutionUrl"". Error " + $_.Exception.Message)
      Throw "Failed invoking WFA workflow ""$workflow"" using URI ""$workflowExecutionUrl"""

   }
   [Int]$jobId        = $executionXML.job.jobId
   [String]$statusUri = "$workflowExecutionUrl/$jobId"
   Get-WFALogger -Info -Message $("WFA job ID`: "     + $jobID)
   Get-WFALogger -Info -Message $("WFA Status URI`: " + $statusUri)
   Return $statusUri;
}#End Function
#'------------------------------------------------------------------------------
#'Enumerate the OCUM server IP Address to connect to.
#'------------------------------------------------------------------------------
[String]$octet    = '(?:0?0?[0-9]|0?[1-9][0-9]|1[0-9]{2}|2[0-5][0-5]|2[0-4][0-9])'
[Regex]$ipv4Regex = "^(?:$octet\.){3}$octet$"
If($OcumHostname -NotMatch $ipv4Regex){
   [String]$ocumIp = [System.Net.Dns]::GetHostAddresses($OcumHostname).IPAddressToString
}Else{
   [String]$ocumIp = $OcumHostname
}
If([String]::IsNullOrEmpty($ocumIp)){
   Throw "Failed DNS IP Address resolution for ""$OcumHostname"""
}
Get-WFALogger -Info -Message "OCUM IP Address`: ""$ocumIp"""
#'------------------------------------------------------------------------------
#'Enumerate WFA Credentials to authenticate the REST API to invoke the workflow.
#'------------------------------------------------------------------------------
If(-Not($wfaHostname)){
   [String]$wfaHostname = $env:computername
   [String]$wfaIp       = [System.Net.Dns]::GetHostAddresses($wfaHostname)[1].IPAddressToString;
}Else{
   If($wfaHostname -NotMatch $ipv4Regex){
      [String]$wfaIp = [System.Net.Dns]::GetHostAddresses($WfaHostname).IPAddressToString
   }Else{
      [String]$wfaIp = $WfaHostname
   }
   If([String]::IsNullOrEmpty($wfaIp)){
      Throw "Failed DNS IP Address resolution for ""$WfaHostname"""
   }
   Get-WFALogger -Info -Message "WFA IP Address`: ""$wfaIp"""
}
$WfaCredentials = Get-WfaCredentials $wfaIp
#'------------------------------------------------------------------------------
#'Set the policy to trust self signed certificates for OCUM.
#'------------------------------------------------------------------------------
Add-Type @"
   using System.Net;
   using System.Security.Cryptography.X509Certificates;
   public class TrustOCUMCertsPolicy : ICertificatePolicy {
   public bool CheckValidationResult(
   ServicePoint srvPoint, X509Certificate certificate,
   WebRequest request, int certificateProblem) {
      return true;
   }
}
"@
#'------------------------------------------------------------------------------
#'Set credentials to authenticate the OCUM REST API request.
#'------------------------------------------------------------------------------
$ocumCredentials  = Get-WfaCredentials $ocumIp
[String]$username = $ocumCredentials.GetNetworkCredential().UserName
[String]$password = $ocumCredentials.GetNetworkCredential().Password
[System.Net.ServicePointManager]::SecurityProtocol  = [System.Net.SecurityProtocolType]'Tls12'
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustOCUMCertsPolicy
$encodedCredentials = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("$username`:$password"))
$headers = @{
   "Authorization" = "Basic " + $encodedCredentials
   "Accept"        = "application/vnd.netapp.object.inventory.hal+json"
}
#'------------------------------------------------------------------------------
#'Enumerate the OCUM Event.
#'------------------------------------------------------------------------------
[String]$ocumEventUri = $("https://$ocumIp/rest/events?triggeredTime=LAST`_" + $EventRangeHours + "h")
Try{
   $events = Invoke-RestMethod -Headers $headers -Uri $ocumEventUri -ErrorAction Stop
   Get-WFALogger -Info -Message "Enumerated OCUM events within the past $EventRangeHours hours using URI ""$ocumEventUri"""
}Catch{
   Get-WFALogger -Error -Message $("Failed enumerating OCUM Events within the past $EventRangeHours hours using URI ""$ocumEventUri"". Error " + $_.Exception.Message)
   Throw "Failed enumerating OCUM Events within the past $EventRangeHours hours using URI ""$ocumEventUri"""
}
#'------------------------------------------------------------------------------
#'Create hashtables of comma delimited input variables for comparison.
#'------------------------------------------------------------------------------
[HashTable]$eventSeverities = @{};
If($EventSeverity.Contains(",")){
   ForEach($element In $EventSeverity.Split(",")){
      [HashTable]$eventSeverities.Add($element, "")
   }
}Else{
   [HashTable]$eventSeverities.Add($EventSeverity, "")
}
[HashTable]$eventSourceTypes = @{};
If($EventSourceType.Contains(",")){
   ForEach($element In $EventSourceType.Split(",")){
      [HashTable]$eventSourceTypes.Add($element, "")
   }
}Else{
   [HashTable]$eventSourceTypes.Add($EventSourceType, "")
}
[HashTable]$eventImpactLevels = @{};
If($EventImpactLevel.Contains(",")){
   ForEach($element In $EventImpactLevel.Split(",")){
      [HashTable]$eventImpactLevels.Add($element, "")
   }
}Else{
   [HashTable]$eventImpactLevels.Add($EventImpactLevel, "")
}
[HashTable]$eventStates = @{};
If($EventState.Contains(",")){
   ForEach($element In $EventState.Split(",")){
      [HashTable]$eventStates.Add($element, "")
   }
}Else{
   [HashTable]$eventStates.Add($EventState, "")
}
#'------------------------------------------------------------------------------
#'Enumerate WFA variables for logging within the WFA workflow execution comment.
#'------------------------------------------------------------------------------
[String]$wfaUri       = "https://$WfaHostname"
If($WfaPortNumber){
   [String]$wfaUri    = $($wfaUri + "`:$WfaPortNumber")
}
[String]$wfaUserName  = $(Get-WfaRestParameter "userName")
[String]$WorkflowName = $(Get-WfaRestParameter "workflowName")
[Int]$WorkflowJobID   = $(Get-WfaRestParameter "jobId")
#'------------------------------------------------------------------------------
#'Process the OCUM event.
#'------------------------------------------------------------------------------
[Int]$eventCount = $events._embedded.'netapp:eventDtoList'.Count
If($eventCount -ge 1){
   Get-WFALogger -Info -Message "Processing $eventCount OCUM Events"
}Else{
   Get-WFALogger -Info -Message "There are no OCUM Events requiring processing. Exiting"
   Exit 0 
}
ForEach($event In $events._embedded.'netapp:eventDtoList'){
   [Int]$eventId              = $event.objectId
   [String]$eventName         = $event.name
   [String]$eventResourceType = $event.sourceResourceType.ToLower();
   [String]$eventSourceName   = $event.sourceFullName
   [Int]$clusterId            = $event.clusterId
   [String]$comment           = "Processing OCUM Event ID ""$eventId"" URI ""https`://$OcumHostname/events/$eventId"". Invoked by workflow ""$WorkflowName"" WFA job ID ""$WorkflowJobID"" by user ""$wfaUserName"""
   If($VerboseLogging){
      Get-WFALogger -Info -Message "Processing OCUM Event $eventId"
   }   
   Do{
      If(-Not($eventStates.ContainsKey($($event.state).ToLower()))){
         If($VerboseLogging){
            Get-WFALogger -Info -Message $("The event ID $eventId state is """ + $($event.state).ToLower() + """ and does not match the desired state of ""$EventState"". Exit processing")
         }
         Break;
      }
      If(-Not($eventSeverities.ContainsKey($($event.severity).ToLower()))){
         If($VerboseLogging){
            Get-WFALogger -Info -Message $("The event $eventId severity is """ + $($event.severity).ToLower() + """ and does not match the desired state of ""$EventSeverity"". Exit processing")
         }
         Break;
      }
      If(-Not($eventSourceTypes.ContainsKey($($eventResourceType).ToLower()))){
         If($VerboseLogging){
            Get-WFALogger -Info -Message $("The event $eventId source type is ""$eventResourceType"" and does not match the desired state of ""$EventSourceType"". Exit processing")
         }
         Break;
      }
      If(-Not($eventImpactLevels.ContainsKey($($event.impactLevel).ToLower()))){
         If($VerboseLogging){
            Get-WFALogger -Info -Message $("The event $eventId impact level is """ + $($event.impactLevel).ToLower() + """ and does not match the desired state of ""$EventImpactLevel"". Exit processing")
         }
         Break;
      }
      #'------------------------------------------------------------------------
      #'Log the OCUM event properties.
      #'------------------------------------------------------------------------
      If($VerboseLogging){
         Get-WFALogger -Info -Message $("Condition`: "    + $event.conditionMessage)
         Get-WFALogger -Info -Message $("Id`: "           + $eventId)
         Get-WFALogger -Info -Message $("Impact Area`: "  + $event.impactArea)
         Get-WFALogger -Info -Message $("Impact Level`: " + $event.impactLevel)
         Get-WFALogger -Info -Message $("Name`: "         + $eventName)
         Get-WFALogger -Info -Message $("Severity`: "     + $event.severity)
         Get-WFALogger -Info -Message $("Source Name`: "  + $eventSourceName)
         Get-WFALogger -Info -Message $("Source Type`: "  + $eventResourceType)
         Get-WFALogger -Info -Message $("State`: "        + $event.state.ToLower())
         Get-WFALogger -Info -Message $("Timestamp`: "    + $(ConvertFrom-UnixTimestamp -Timestamp $event.timestamp))
         Get-WFALogger -Info -Message $("Type`: "         + $event.source.objectType)
      }
      #'------------------------------------------------------------------------
      #'Process inode events.
      #'------------------------------------------------------------------------
      If($eventResourceType -eq "volume" -And ($eventName -eq "Inodes Full" -Or $eventName -eq "Inodes Nearly Full")){
         #'------------------------------------------------------------------------------
         #'Ensure the OCUM cluster address is valid.
         #'------------------------------------------------------------------------------
         [String]$clusterAddress = Get-OCUMClusterAddress -HostName $OcumIp -ClusterId $clusterId -Credentials $ocumCredentials
         If([String]::IsNullOrEmpty($clusterAddress)){
            Throw "Failed enumerating OCUM cluster address for cluster ID`: $clusterId"
         }
         [String]$VserverName = $eventSourceName.Split(":")[0]
         [String]$VolumeName  = $eventSourceName.Split(":")[1].SubString(1)
         [HashTable]$inputs = @{
            "Cluster"       = $clusterAddress;
            "VserverName"   = $VserverName;
            "VolumeName"    = $VolumeName;
         }
         #'------------------------------------------------------------------------
         #'Invoke the WFA workflow to increase the volumes inodes.
         #'------------------------------------------------------------------------
         Get-WFALogger -Info -Message "Invoking workflow ""$IncreaseInodesWorkflow"" for vserver ""$VserverName"" on cluster ""$clusterAddress"""
         Try{
            Invoke-WFAWorkflow -Uri $wfaUri -Workflow $IncreaseInodesWorkflow -Comment $comment -Inputs $inputs -Credentials $wfaCredentials -ErrorAction Stop
            Get-WFALogger -Info -Message "Invoked workflow ""$IncreaseInodesWorkflow"" using URI ""$wfaUri"" for volume ""$VolumeName"" on vserver ""$VserverName"" on cluster ""$clusterAddress"""
         }Catch{
            Get-WFALogger -Error -Message $("Failed Invoking workflow ""$IncreaseInodesWorkflow"" using URI ""$wfaUri"" for volume ""$VolumeName"" on vserver ""$VserverName"" on cluster ""$clusterAddress"". Error " + $_.Exception.Message)
            Throw "Failed invoking workflow ""$IncreaseInodesWorkflow"" using URI ""$wfaUri"" for volume ""$VolumeName"" on vserver ""$VserverName"" on cluster ""$clusterAddress"""
         }
      }
   }Until($True)
}
#'------------------------------------------------------------------------------

[cjautonetics]

Thanks Matt,

It does sound like that's a road block on performing real time self healing using OCUM vApp. Hope the modules and plugins needed to invoke REST API will be included in future vApp releases.

Thanks for the sample code to poll  the OCUM REST API events. Yes the concern would be that this might not be as efficient as OCUM invoking a WFA workflow directly. As the OCUM by itself includes the 15 mins polling interval for Cluster events , An added  schedule to poll OCUM events would add into the resolution time. Don't know how this will fare in instances where the inode count increasing rapidly with a bulk of very small files being copied into volumes. Might have to reduce the threshold of "Inodes Nearly full" events maybe.

I will test polling the OCUM REST API events and update how it goes. Will also explore the option of migrating the OCUM instance over to a Unix server.

Cheers

CJ