Subscribe
Accepted Solution

IOPS / Latency alert not working in OC5

Hi ,

We are using Oncommand 5 for our FAS3140 filers. We want to setup IOPS threshholds (FCP , iscsi , nfs and lun/volume wise ) & high disk latency (average,read & write ) alerts for these filers - protocol wise but I am not getting these alerts. We are gettling alerts for high CPU utilisation. Please advice to resolve this issue. If possible, provide me proper steps for creating these alram alerts. I went thru configuration pdf but in vain.

Thanks & Regards,

Vivek

Re: IOPS / Latency alert not working in OC5

Vivek,

Kindly let us know in case even after setting the IOPs thresholds you are not receiving the alerts or you need the procedure to set the thresholds so that the alerts are sent to you as when the threshold levels are breached.

Please be informed that the alerts are sent to the user as when the threshold levels set are breached.

Regards,

-Seshu

Re: IOPS / Latency alert not working in OC5

Hi Seshu Please provide me exact steps, I can setup again & confirm . Also wanted to know what should be threshhold values for FAS3140 controllers. Thanks & Regards, Vivek

Re: IOPS / Latency alert not working in OC5

Hi Vivek,

     Can you get the  output of the following cli ?

Use dfm perf threshold list to get the list of thresholds you have set. Then from that get the id of the threshold in question and run the following cli.

dfm perf threshold list -x <threshold id>

Your output will be like this.

[root@ ~]#  dfm perf threshold list -x 105

Threshold Id: 105

Event Name: system:cpu_busy

Interval: 60

Enabled: Yes

Object Info:

        Id: 1273

        Name: sim-4-1-182-99-sf

        Type: filer

Counter Info:

        Counter: system:cpu_busy

        Threshold Type: upper

        Threshold Value: 20

        Threshold Unit: percent

[root ~]#

Similarly also get the output of the following cli to see if the values really breached.

[root@ ~]#  dfm perf data retrieve -o 1273 -V "CPU_USAGE"

Timestamp       1273:cpu_busy

-------------------------------------------------------------------------------

2011-09-07 05:42:38     1.085

2011-09-07 05:42:54     0.802

2011-09-07 05:44:33     1.292

2011-09-07 05:45:54     1.132

2011-09-07 05:46:08     0.521

2011-09-07 05:46:34     1.743

2011-09-07 05:47:34     1.128

2011-09-07 05:49:24     0.820

2011-09-07 05:49:53     2.020

In order to get the object id( the value for -o, use the info from threshold list -x and use the Id under object Info).Similarly for the view name( the name of the view in which this counter is used or data is being collected)

use the cli below or use the name from NMC.

[root ~]# dfm perf view list CPU_USAGE

View Name

--------------------

CPU_USAGE

[root ~]#

Regards

adai

Re: IOPS / Latency alert not working in OC5

C:\>dfm perf threshold list Id    Status    Event Name                                    Template ----- ---------- --------------------------------------------- ----------------- --- 4    enabled    system:cpu_busy 5    enabled    system:sys_avg_latency 10    enabled    Latency 11    disabled  Volume_Write_Read_Latency 12    enabled    LUN_Read_Write_Latency 13    enabled    FCP_Latency                                  FCP_Latency 14    enabled    iSCSI_Latency                                iSCSI_Latency 15    enabled    NFS_Latency                                  NFS_Latency 16    enabled    CIFS_Latency                                  CIFS_Latency 17    enabled    CPU_Utilization                              CPU_Utilization 18    enabled    Disk_Utilization                              Disk_Utilization 19    enabled    Network_Utilization                          Network_Utilizati on 22    enabled    system:cpu_busy 23    enabled    system:cpu_busy 24    enabled    IOPS                                          IOPS =============== C:\>dfm perf threshold list -x 24 Threshold Id: 24 Event Name: IOPS Interval: 60 Enabled: Yes Template Info:         Id: 9         Name: IOPS Object Info:         Id: 77         Name: MUMCLDSTR02         Type: filer Counter Info:         Counter: system:pa_total_ops         Threshold Type: upper         Threshold Value: 2000         Threshold Unit: per_sec         Counter: system:fcp_ops         Threshold Type: upper         Threshold Value: 1000         Threshold Unit: per_sec         Counter: system:iscsi_ops         Threshold Type: upper         Threshold Value: 1000         Threshold Unit: per_sec         Counter: system:nfs_ops         Threshold Type: upper         Threshold Value: 1000         Threshold Unit: per_sec ===================== C:\>dfm perf data retrieve -o 1273 -V "CPU_USAGE" Error: Failed in getting object:counter from view "CPU_USAGE". C:\>dfm perf data retrieve -o 77 -V "IOPS_USAGE" Error: Failed in getting object:counter from view "IOPS_USAGE". C:\>dfm perf data retrieve -o 77 -V "pa_total_ops" Error: Failed in getting object:counter from view "pa_total_ops". C:\>dfm perf data retrieve -o 77 -V "fcp_ops" Error: Failed in getting object:counter from view "fcp_ops". C:\>dfm perf data retrieve -o 77 -V "MUMCLDSTR02" Error: Failed in getting object:counter from view "MUMCLDSTR02". C:\>dfm perf data retrieve -o 77 -V "iops" Error: Failed in getting object:counter from view "iops". C:\>dfm perf data retrieve -o 77 -V "IOPS" Error: Failed in getting object:counter from view "IOPS". C:\>dfm perf data retrieve -o 77 -V "system:fcp_ops" Error: Failed in getting object:counter from view "system:fcp_ops". ======================== C:\>dfm perf view list View Name -------------------- Aggregate Summary View Dataset Summary View Disk Summary View Group - Top Objects Group - Top vFilers Group Summary View Lun Alignment View Lun Summary View Network Interface Summary View Processor Summary View Qtree Summary View Resource Pool Summary View Storage System - Disk Throughput Storage System - Network Throughput Storage System - Per Protocol Latencies Storage System - Per Protocol Ops Storage System - Per Protocol Sizes Storage System - Top Logical Objects Storage System - Top Physical Objects Storage System - Top vFilers Storage System Summary View Target Basic Top Aggregates Top CPUs Top LUNs Top Network Interfaces Top Volumes vFiler Basic vFiler Summary View Volume IOPs View Volume Latency View Volume Summary View CPU Average Load fcp-mum02 HYD - CPU Average Load iscsi-mum02 MUM-filerA-performance MUM-filerB-performance nfs-mum02 total ops-mum02 Latency Storage System Summary View-bbiek top vols by ops-latency-mum02 C:\> =============================

Re: IOPS / Latency alert not working in OC5

Vivek,

Thanks for the output sent, Can you please run dfm perf threshold list –x to get the output in better verbose format and send the same.

C:\>dfm perf threshold list -x

From the output you have sent, looks like the threshold values you have set seems to be in correct and the user need to configure the threshold values based on the given time period( mean, min , max and the suggested percentile value) for a particular counter.

Regards,

-Seshu

From: c-xdl-communities

Sent: Tuesday, January 03, 2012 3:45 PM

To: Reddy, Seshu

Subject: - Re: IOPS / Latency alert not working in OC5

<https://communities.netapp.com/index.jspa>

Re: IOPS / Latency alert not working in OC5

created by viveksingh183<https://communities.netapp.com/people/viveksingh183> in OnCommand Mgmt Software - View the full discussion<https://communities.netapp.com/message/70819#70819>

C:\>dfm perf threshold list Id Status Event Name Template

Re: IOPS / Latency alert not working in OC5

hi Sheshu, o/p as requested C:\>dfm perf threshold list -x 24 Threshold Id: 24 Event Name: IOPS Interval: 60 Enabled: Yes Template Info:         Id: 9         Name: IOPS Object Info:         Id: 77         Name: MUMCLDSTR02         Type: filer Counter Info:         Counter: system:pa_total_ops         Threshold Type: upper         Threshold Value: 2000         Threshold Unit: per_sec         Counter: system:fcp_ops         Threshold Type: upper         Threshold Value: 1000         Threshold Unit: per_sec         Counter: system:iscsi_ops         Threshold Type: upper         Threshold Value: 1000         Threshold Unit: per_sec         Counter: system:nfs_ops         Threshold Type: upper         Threshold Value: 1000         Threshold Unit: per_sec Thanks & Regards, Vivek

Re: IOPS / Latency alert not working in OC5

Hi Vivek,

     You have set a combination threshold of 4 counters. Your event will be generated only when the values for all the 4 counters are breached and stay for more than 60 sec, as per your threshold.

Regards

adai

Re: IOPS / Latency alert not working in OC5

Hi Vivek,

 

From the output you have sent it looks like there are four counters which are being configured for the alerts:

Counter Info:        

Counter: system:pa_total_ops        

Threshold Type: upper        

Threshold Value: 2000        

Threshold Unit: per_sec       

Counter: system:fcp_ops        

Threshold Type: upper        

Threshold Value: 1000       

Threshold Unit: per_sec        

Counter: system:iscsi_ops       

Threshold Type: upper       

Threshold Value: 1000      

Threshold Unit: per_sec        

Counter: system:nfs_ops        

Threshold Type: upper        

Threshold Value: 1000        

Threshold Unit: per_sec

As mentioned previously by Adai, please get the output of the following to proceed further with this issue:

Kindly check for the Views for which these counters are configured for and get the out put accordingly.

To get the view list run dfm perf view list on the command line and one can get the list of available views.

To see the list of available counters configured in a view please run C:\>dfm perf view describe "View Name"

For example : C:\>dfm perf view describe "Storage System Summary View"

Once you get to see the counters you have configured for a particular view run :

[root@ ~]#  dfm perf data retrieve -o id -V "View Name"

-o is the object id

-V is the View name for which the counters configuired are listed.

Note : Please mention the right View name to retrieve the data for the counters configured.

Please share us the output of the above command to verify the values if they are really breached as per  the configuration.

Re: IOPS / Latency alert not working in OC5

Thanks Adai & Seshu for your help. I have modied my threshhold settings & now its working fine . Just curious if you have IOPS limit (threshhold values ) for FAS3140 R5 filers. Regards, Vivek