Subscribe

Do IOPS on Aggregate Performance Report include IO from cache Hits?

Hi - I'm looking to find (for each aggregate in a FAS3070 Filer) how many IOPS are being delivered at peak.

I'm working on the theory that each data disk should be capable of delivering approx 180 IOPS (they're 15k FC disks), hence I can create a "theoretical" performance limit for each aggregate, (180 x number of data disks), beyond which latency is likely to suffer.

The reason I want to know is that we are considering moving to 300GB disks, but there is little point in getting extra capacity if I can't practically make use of it (I get double the capacity but I get no more performance).

My storage admnistrator has produced an Operations Manager aggregate report (OPS Manager – ‘dfm report view aggregates-performance-summary’) showing at peak a level of IOPS that would equate to approx 600 IOPS per data disk.

600 IOPS per disk doesn't sound realistic (nobody is complaining of poor latency/performance)

Could it be that the aggregate performance report is including IOPS from Cache Hits??? (In which case this is the wrong report for me - I am purely interested in Disk IOPS).

Can anyone help?

Stu

P.S. Summary below:

Filer Name

Aggregate

# Data
Disks

Theoretical
IOPS
maximum

Monitored
IOPS

% utilised

Filer1

aggr1

70

12601

10748

85%

Filer2

aggr1

68

12241

8841

72%

Filer3

aggrebg

47

8461

10239

121%

Filer4

aggr1

37

6661

21168

318%

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

I have been looking at this in the past. The documentation states the following for the total_transfers:

total_transfers: Total number of disk operations serviced by the aggregate per second

You would not expect caching to be in this stat, but to check this assumption I used several other commands to verify that this is the sum of  disk transfers.

Aggregate total transfers: stats show aggregate:*:total_transfers

Per disk total transfers: stats show disk:*:total_transfers

Statit (old fashioned disk utilisation report): statit -b  / statit -e

The sum of per disk transfers matched the total aggregate transfers. Also the statit output confirmed the number of disk transfers. Therefore I can safely assume that aggregate total transfers are actual disk transfers.

Then there is the question why your numbers seem too high. How where the peak numbers determined? It could be that there was a one time freak peak.

The peformance advisor has for each aggreagate a graph of total transfers over the last week. Take a look and make your own estimate of the peak disk transfers.

An other useful number is disk busy: stats show disk:*:disk_busy

If your disks are over 50% busy all the time you will normally see performance starting to degrade.

Message was edited by: pascalduk

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

Hi,

OpsMgr disks-performance-summary report tracks IOPS at individual disk level. We can aggregate these values for 'Filer3:aggrebg, Filer4:aggr1 ' objects and compare against

value reported  in 'aggregate-performance-summary for the same duration. Even NMC has 'Disk Summary View'  which captures IOPS at disk level, but aggregating for 37 disks is

a tedious task.

And also please let me know if you see any discrepancies between NMC Aggregate Summary View's  total_transfers counter and OpsMgr aggregate-performance-summary report

IOPS value.

regards,

nagesh

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

Hi - and many thanks for your response.

After initial pleasure at your explanation that cache was not included in the aggregate IOPS report, I thought i'd better do a doublecheck by comparing Aggregate stats and Disk Stats.

The commands my administrator used are:

IOPS Command - dfm report view aggregates-performance-summary

Disk Performance - dfm report view disks-performance-summary

Unfortunately there seem to be some discrepancies.

In many cases the IOPS are very very similar (but not exactly the same). For a particular aggregate however, I see 4685.13 IOPS (agg view) but a total of 2473 IOPS when adding up the constituent disk IOPS.

There was only a 0.3 second difference in the capture time - surely not enought to account for such a discrepancy?

We understand that these commands get their data from the DFM database (rather than direct from the Filer). Is tehre any reason that woudl skew the results?

Aggregate View:

Object ID Aggregate Storage System                   Total Ops/Sec

--------- --------- -------------------------------- ------------- ------------------------------ -------------------------------------

87        aggr1     filer1002       4685.13                                                                    

Per Disk View

Aggregate

Read OPS/Sec

Wrote OPS/Sec

CP Read OPS/Sec

TOTAL

aggr1

66.2

1.25

2.12

aggr1

65.53

1.08

2.17

aggr1

65.15

1.88

2.22

aggr1

64.93

1.82

2.2

aggr1

64.37

1

1.9

aggr1

64.25

0.93

2

aggr1

63.83

1.82

2.33

aggr1

63.47

1.37

2.23

aggr1

63.37

1.27

2.37

aggr1

63.23

1.23

2.22

aggr1

63.17

1.38

2.32

aggr1

63.13

1.83

2.2

aggr1

63.05

1.45

2.18

aggr1

63

1.27

2.33

aggr1

62.85

1.2

2.15

aggr1

62.72

1.32

2.35

aggr1

62.63

1.87

2.07

aggr1

62.62

1.42

2.17

aggr1

62

1.82

2.12

aggr1

61.98

1.85

2.07

aggr1

61.97

1.85

2.25

aggr1

61.93

1.87

2.32

aggr1

61.93

1.32

2.42

aggr1

61.87

1.83

1.98

aggr1

61.33

1.83

1.88

aggr1

61.1

1.28

2.22

aggr1

60.95

1.3

2.25

aggr1

60.95

1.33

2.4

aggr1

60.8

1.28

2.2

aggr1

60.8

1.28

2.07

aggr1

60.7

1.23

1.97

aggr1

60.57

1.23

2.07

aggr1

60.57

1.22

2.28

aggr1

60.5

1.18

2.2

aggr1

60.48

1.85

2.27

aggr1

59.78

1.9

2.35

aggr1

58.72

1.8

2.2

aggr1

0.38

3.12

3.6

aggr1

0.38

2.82

3.6

aggr1

0

2.17

1.68

aggr1

0

2.13

2.73

aggr1

0

2.17

1.67

aggr1

0

2.13

2.72

2307.19

69.18

97.05

2473.42

Any ideas/explanations??

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

Have a quick look at this powerpoint presentation as people have told me it was helpful.

Bren

http://communities.netapp.com/people/BrendonHiggins/blog/2009/10/21/sept-nug-meeting

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

That is a nice one!

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

husselbees wrote:

There was only a 0.3 second difference in the capture time - surely not enought to account for such a discrepancy?

I don't expect these reports to be real time and with a very variable workload this can give these differences.

You will get a better idea of your workload with the netapp cli commands I posted earlier or the performance advisor graphs.

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

Hi,

I'd like to create a report that will monitor aggregate IOPS. I'm not sure how to do this. Can you please provide some guidance?

I do have DFM but I'm unable to either create or find report that contains this information.

Thanks

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

Hello Ivan,

Ops manager provides for performance reports that can help you in monitoring aggregate IOPS.

From the CLI,

The report catalog that contains all aggregate related reports is :

[root@lnx218-33 ~]# dfm report catalog list | grep -i aggr

Aggregate                           Aggregates                        

[root@lnx218-33 ~]#

We can get the fields in the catalog "Aggregate":

These are the regular report fields :

[root@lnx218-33 ~]# dfm report catalog list Aggregate

Aggregate Catalog

Default Display Tab: Aggregates

Fields:

Field                               Default Name                                  Default Format

----------------------------------- --------------------------------------------- --------------

All                                 All fields of this catalog                                 

Name                                Aggregate Name                                             

FullName                            Aggregate Full Name                                        

Id                                  Aggregate Id                                               

GroupId                             Aggregate Group Id                                         

StorageSystem                       Aggregate Storage System (A)                               

Cluster                             Aggregate Cluster (A)                                      

Comment                             Aggregate Comment                                          

Type                                Aggregate Type                                             

Block Type                          Aggregate Block Type                                       

RAID                                Aggregate RAID Type                                        

State                               Aggregate State                                            

Snaplock                            SnapLock                                                   

Status                              Aggregate Status                                           

Mirrored                            Aggregate Mirrored                                         

DeletedWhen                         Aggregate Deleted When                        DD MMM 24H   

DeletedBy                           Aggregate Deleted By                                       

Used                                Aggregate Used Capacity                       KB           

UsedPct                             Aggregate Used Capacity %                    1            

FullThreshold                       Aggregate Full Threshold                      1            

NearlyFullThreshold                 Aggregate Nearly Full Threshold               1            

FullThresholdInterval               Aggregate Full Threshold Interval                          

OvercommittedThreshold              Aggregate Overcommitted Threshold             1            

NearlyOvercommittedThresh           Aggr. Nearly Overcommitted Thresh.            1            

AggrOverDeduplicatedThreshold       Aggregate Over Deduplicated Threshold         1            

AggrNearlyOverDeduplicatedThreshold Aggregate Nearly Over Deduplicated Threshold  1            

AggrSpaceSavings                    Aggregate Space Savings - Deduplication and LUN/File Clones KB           

AggrSpaceSavingsPct                 Aggregate Space Savings (%) - Deduplication and LUN/File Clones 1            

AggrDedupeVolCount                  Aggregate Volume Count - Deduplication and LUN/File Clones              

AggrPhysicalUsed                    Aggregate Physical Used                       KB           

AggrEffectiveUsed                   Aggregate Effective Used                      KB           

TotalSpace                          Aggregate Total Space                         KB           

SpaceAvailable                      Aggregate Space Available                     KB           

AvailablePct                        Aggregate Space Available in %                1            

BytesCommitted                      Aggregate Bytes Committed                     KB           

BytesCommittedPct                   Aggregate Bytes Committed %                   1            

DailyGrowthRate                     Aggregate Daily Growth Rate                   KB           

DailyGrowthPct                      Aggregate Daily Growth (%)                    1            

DaysToFull                          Aggregate Days to Full                                     

SnapshotDisabled                    Aggregate Snapshot Disabled                                

SnapshotAutoDelete                  Aggregate Snapshot Autodelete                              

SnapReserveTotal                    Aggregate Snap Reserve Total                  KB           

SnapReserveUsed                     Aggregate Snap Reserve Used                   KB           

SnapReserveUsedPct                  Aggregate Snap Reserve Used %                 1            

[root@lnx218-33 ~]#

These are the performance specific fields :

[root@lnx218-33 ~]# dfm report catalog list -P Aggregate

Aggregate Catalog

Default Display Tab: Aggregates

Fields:

Field                               Default Name                                  Default Format

----------------------------------- --------------------------------------------- --------------

TotalTransfers                      Aggregate Total Transfers Ops/Sec                          

UserReads                           Aggregate User Reads Ops/Sec                               

UserWrites                          Aggregate User Writes Ops/Sec                              

CPReads                             Aggregate CP Reads Ops/Sec                                 

PerfViolationCount                  Aggregate Perf Threshold Violation Count                   

PerfViolationPeriod                 Aggregate Perf Violation Period (Sec)                      

[root@lnx218-33 ~]#

Using report create CLI we can create a Custom report with aggregate IOPS.

[root@lnx218-33 ~]# dfm report create -R Aggregate -f FullName,StorageSystem,UserReads,UserWrites,CPReads,TotalTransfers Aggregate_IOPS

Report Aggregate_IOPS created.

Check the created report here :

[root@lnx218-33 ~]# dfm report view Aggregate_IOPS help

Aggregate_IOPS Report (Aggregate_IOPS)

Catalog Name: Aggregate

Display Tab: Aggregates

Catalog Field                                 Field Name                                    Format  

--------------------------------------------- --------------------------------------------- ---------

Aggregate.FullName                            Aggregate Full Name                                   

Aggregate.StorageSystem                       Aggregate Storage System (A)                          

Aggregate.UserReads                           Aggregate User Reads Ops/Sec                          

Aggregate.UserWrites                          Aggregate User Writes Ops/Sec                         

Aggregate.CPReads                             Aggregate CP Reads Ops/Sec                           

Aggregate.TotalTransfers                      Aggregate Total Transfers Ops/Sec                     

Default sort order is Aggregate.FullName.

[root@lnx218-33 ~]#

Hope this helps.

Thanks,

Akshay

Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

Thanks Akshay for spelling this out for me. Unfortunately I was unable to create these reports and I think that's due to the fact that I have DFM 3.6. It looks like none of the Aggregate reports you referred in your post were available. I hope Operations Manager 3.8 has these Aggregate reports so I’ll go on download and install it.

What version of DFM are you running?

Thanks,

Ivan