Re: Do IOPS on Aggregate Performance Report include IO from cache Hits?

husselbees · ‎2009-10-20

Hi - I'm looking to find (for each aggregate in a FAS3070 Filer) how many IOPS are being delivered at peak.

I'm working on the theory that each data disk should be capable of delivering approx 180 IOPS (they're 15k FC disks), hence I can create a "theoretical" performance limit for each aggregate, (180 x number of data disks), beyond which latency is likely to suffer.

The reason I want to know is that we are considering moving to 300GB disks, but there is little point in getting extra capacity if I can't practically make use of it (I get double the capacity but I get no more performance).

My storage admnistrator has produced an Operations Manager aggregate report (OPS Manager – ‘dfm report view aggregates-performance-summary’) showing at peak a level of IOPS that would equate to approx 600 IOPS per data disk.

600 IOPS per disk doesn't sound realistic (nobody is complaining of poor latency/performance)

Could it be that the aggregate performance report is including IOPS from Cache Hits??? (In which case this is the wrong report for me - I am purely interested in Disk IOPS).

Can anyone help?

Stu

P.S. Summary below:

Filer Name	Aggregate	# Data Disks	Theoretical IOPS maximum	Monitored IOPS	% utilised
Filer1	aggr1	70	12601	10748	85%
Filer2	aggr1	68	12241	8841	72%
Filer3	aggrebg	47	8461	10239	121%
Filer4	aggr1	37	6661	21168	318%

pascalduk · ‎2009-10-21

I have been looking at this in the past. The documentation states the following for the total_transfers:

total_transfers: Total number of disk operations serviced by the aggregate per second

You would not expect caching to be in this stat, but to check this assumption I used several other commands to verify that this is the sum of disk transfers.

Aggregate total transfers: stats show aggregate:*:total_transfers

Per disk total transfers: stats show disk:*:total_transfers

Statit (old fashioned disk utilisation report): statit -b / statit -e

The sum of per disk transfers matched the total aggregate transfers. Also the statit output confirmed the number of disk transfers. Therefore I can safely assume that aggregate total transfers are actual disk transfers.

Then there is the question why your numbers seem too high. How where the peak numbers determined? It could be that there was a one time freak peak.

The peformance advisor has for each aggreagate a graph of total transfers over the last week. Take a look and make your own estimate of the peak disk transfers.

An other useful number is disk busy: stats show disk:*:disk_busy

If your disks are over 50% busy all the time you will normally see performance starting to degrade.

Message was edited by: pascalduk

nagesha · ‎2009-10-21

Hi,

OpsMgr disks-performance-summary report tracks IOPS at individual disk level. We can aggregate these values for 'Filer3:aggrebg, Filer4:aggr1 ' objects and compare against

value reported in 'aggregate-performance-summary for the same duration. Even NMC has 'Disk Summary View' which captures IOPS at disk level, but aggregating for 37 disks is

a tedious task.

And also please let me know if you see any discrepancies between NMC Aggregate Summary View's total_transfers counter and OpsMgr aggregate-performance-summary report

IOPS value.

regards,

nagesh

husselbees · ‎2009-10-21

Hi - and many thanks for your response.

After initial pleasure at your explanation that cache was not included in the aggregate IOPS report, I thought i'd better do a doublecheck by comparing Aggregate stats and Disk Stats.

The commands my administrator used are:

IOPS Command - dfm report view aggregates-performance-summary

Disk Performance - dfm report view disks-performance-summary

Unfortunately there seem to be some discrepancies.

In many cases the IOPS are very very similar (but not exactly the same). For a particular aggregate however, I see 4685.13 IOPS (agg view) but a total of 2473 IOPS when adding up the constituent disk IOPS.

There was only a 0.3 second difference in the capture time - surely not enought to account for such a discrepancy?

We understand that these commands get their data from the DFM database (rather than direct from the Filer). Is tehre any reason that woudl skew the results?

Aggregate View:

Object ID Aggregate Storage System Total Ops/Sec

--------- --------- -------------------------------- ------------- ------------------------------ -------------------------------------

87 aggr1 filer1002 4685.13

Per Disk View

Aggregate	Read OPS/Sec	Wrote OPS/Sec	CP Read OPS/Sec	TOTAL
aggr1	66.2	1.25	2.12
aggr1	65.53	1.08	2.17
aggr1	65.15	1.88	2.22
aggr1	64.93	1.82	2.2
aggr1	64.37	1	1.9
aggr1	64.25	0.93	2
aggr1	63.83	1.82	2.33
aggr1	63.47	1.37	2.23
aggr1	63.37	1.27	2.37
aggr1	63.23	1.23	2.22
aggr1	63.17	1.38	2.32
aggr1	63.13	1.83	2.2
aggr1	63.05	1.45	2.18
aggr1	63	1.27	2.33
aggr1	62.85	1.2	2.15
aggr1	62.72	1.32	2.35
aggr1	62.63	1.87	2.07
aggr1	62.62	1.42	2.17
aggr1	62	1.82	2.12
aggr1	61.98	1.85	2.07
aggr1	61.97	1.85	2.25
aggr1	61.93	1.87	2.32
aggr1	61.93	1.32	2.42
aggr1	61.87	1.83	1.98
aggr1	61.33	1.83	1.88
aggr1	61.1	1.28	2.22
aggr1	60.95	1.3	2.25
aggr1	60.95	1.33	2.4
aggr1	60.8	1.28	2.2
aggr1	60.8	1.28	2.07
aggr1	60.7	1.23	1.97
aggr1	60.57	1.23	2.07
aggr1	60.57	1.22	2.28
aggr1	60.5	1.18	2.2
aggr1	60.48	1.85	2.27
aggr1	59.78	1.9	2.35
aggr1	58.72	1.8	2.2
aggr1	0.38	3.12	3.6
aggr1	0.38	2.82	3.6
aggr1	0	2.17	1.68
aggr1	0	2.13	2.73
aggr1	0	2.17	1.67
aggr1	0	2.13	2.72
	2307.19	69.18	97.05	2473.42

Any ideas/explanations??

BrendonHiggins · ‎2009-10-21

Have a quick look at this powerpoint presentation as people have told me it was helpful.

Bren

http://communities.netapp.com/people/BrendonHiggins/blog/2009/10/21/sept-nug-meeting

pascalduk · ‎2009-10-22

That is a nice one!

pascalduk · ‎2009-10-22

husselbees wrote:

There was only a 0.3 second difference in the capture time - surely not enought to account for such a discrepancy?

I don't expect these reports to be real time and with a very variable workload this can give these differences.

You will get a better idea of your workload with the netapp cli commands I posted earlier or the performance advisor graphs.

ivan_huter · ‎2009-12-29

Hi,

I'd like to create a report that will monitor aggregate IOPS. I'm not sure how to do this. Can you please provide some guidance?

I do have DFM but I'm unable to either create or find report that contains this information.

Thanks

nakshay · ‎2009-12-29

Hello Ivan,

Ops manager provides for performance reports that can help you in monitoring aggregate IOPS.

From the CLI,

The report catalog that contains all aggregate related reports is :

[root@lnx218-33 ~]# dfm report catalog list | grep -i aggr

Aggregate Aggregates

[root@lnx218-33 ~]#

We can get the fields in the catalog "Aggregate":

These are the regular report fields :

[root@lnx218-33 ~]# dfm report catalog list Aggregate

Aggregate Catalog

Default Display Tab: Aggregates

Fields:

Field Default Name Default Format

----------------------------------- --------------------------------------------- --------------

All All fields of this catalog

Name Aggregate Name

FullName Aggregate Full Name

Id Aggregate Id

GroupId Aggregate Group Id

StorageSystem Aggregate Storage System (A)

Cluster Aggregate Cluster (A)

Comment Aggregate Comment

Type Aggregate Type

Block Type Aggregate Block Type

RAID Aggregate RAID Type

State Aggregate State

Snaplock SnapLock

Status Aggregate Status

Mirrored Aggregate Mirrored

DeletedWhen Aggregate Deleted When DD MMM 24H

DeletedBy Aggregate Deleted By

Used Aggregate Used Capacity KB

UsedPct Aggregate Used Capacity % 1

FullThreshold Aggregate Full Threshold 1

NearlyFullThreshold Aggregate Nearly Full Threshold 1

FullThresholdInterval Aggregate Full Threshold Interval

OvercommittedThreshold Aggregate Overcommitted Threshold 1

NearlyOvercommittedThresh Aggr. Nearly Overcommitted Thresh. 1

AggrOverDeduplicatedThreshold Aggregate Over Deduplicated Threshold 1

AggrNearlyOverDeduplicatedThreshold Aggregate Nearly Over Deduplicated Threshold 1

AggrSpaceSavings Aggregate Space Savings - Deduplication and LUN/File Clones KB

AggrSpaceSavingsPct Aggregate Space Savings (%) - Deduplication and LUN/File Clones 1

AggrDedupeVolCount Aggregate Volume Count - Deduplication and LUN/File Clones

AggrPhysicalUsed Aggregate Physical Used KB

AggrEffectiveUsed Aggregate Effective Used KB

TotalSpace Aggregate Total Space KB

SpaceAvailable Aggregate Space Available KB

AvailablePct Aggregate Space Available in % 1

BytesCommitted Aggregate Bytes Committed KB

BytesCommittedPct Aggregate Bytes Committed % 1

DailyGrowthRate Aggregate Daily Growth Rate KB

DailyGrowthPct Aggregate Daily Growth (%) 1

DaysToFull Aggregate Days to Full

SnapshotDisabled Aggregate Snapshot Disabled

SnapshotAutoDelete Aggregate Snapshot Autodelete

SnapReserveTotal Aggregate Snap Reserve Total KB

SnapReserveUsed Aggregate Snap Reserve Used KB

SnapReserveUsedPct Aggregate Snap Reserve Used % 1

[root@lnx218-33 ~]#

These are the performance specific fields :

[root@lnx218-33 ~]# dfm report catalog list -P Aggregate

Aggregate Catalog

Default Display Tab: Aggregates

Fields:

Field Default Name Default Format

----------------------------------- --------------------------------------------- --------------

TotalTransfers Aggregate Total Transfers Ops/Sec

UserReads Aggregate User Reads Ops/Sec

UserWrites Aggregate User Writes Ops/Sec

CPReads Aggregate CP Reads Ops/Sec

PerfViolationCount Aggregate Perf Threshold Violation Count

PerfViolationPeriod Aggregate Perf Violation Period (Sec)

[root@lnx218-33 ~]#

Using report create CLI we can create a Custom report with aggregate IOPS.

[root@lnx218-33 ~]# dfm report create -R Aggregate -f FullName,StorageSystem,UserReads,UserWrites,CPReads,TotalTransfers Aggregate_IOPS

Report Aggregate_IOPS created.

Check the created report here :

[root@lnx218-33 ~]# dfm report view Aggregate_IOPS help

Aggregate_IOPS Report (Aggregate_IOPS)

Catalog Name: Aggregate

Display Tab: Aggregates

Catalog Field Field Name Format

--------------------------------------------- --------------------------------------------- ---------

Aggregate.FullName Aggregate Full Name

Aggregate.StorageSystem Aggregate Storage System (A)

Aggregate.UserReads Aggregate User Reads Ops/Sec

Aggregate.UserWrites Aggregate User Writes Ops/Sec

Aggregate.CPReads Aggregate CP Reads Ops/Sec

Aggregate.TotalTransfers Aggregate Total Transfers Ops/Sec

Default sort order is Aggregate.FullName.

[root@lnx218-33 ~]#

Hope this helps.

Thanks,

Akshay

ivan_huter · ‎2009-12-31

Thanks Akshay for spelling this out for me. Unfortunately I was unable to create these reports and I think that's due to the fact that I have DFM 3.6. It looks like none of the Aggregate reports you referred in your post were available. I hope Operations Manager 3.8 has these Aggregate reports so I’ll go on download and install it.

What version of DFM are you running?

Thanks,

Ivan

nakshay · ‎2009-12-31

Hello Ivan,

AFAIK, there is not much changes in the Aggregate reports from 3.6 to 3.8

Thanks,

Akshay

ivan_huter · ‎2009-12-31

Akshay,

I have a problem that I do not have the fields you refer to. These are the field that I see when I list the Aggregate catalog.

X:\>dfm report catalog list Aggregate
Aggregate Catalog

Default Display Tab: Aggregates

Fields:

Field                          Default Name                        Default Format
------------------------------ ----------------------------------- -------------
All                            All fields of this catalog
Name                       Aggregate Name
FullName                  Aggregate Full Name
Id                             Aggregate Id
GroupId                    Aggregate Group Id
Filer                         Aggregate Filer (A)
Comment                 Aggregate Comment
Type                        Aggregate Type
RAID                        Aggregate RAID Type
State                       Aggregate State
Snaplock                 SnapLock
Status                     Aggregate Status
Mirrored                   Aggregate Mirrored
DeletedWhen           Aggregate Deleted When              DD MMM 24H
DeletedBy                Aggregate Deleted By
Used                       Aggregate Used Capacity             KB
UsedPct                  Aggregate Used Capacity %           1
FullThreshold            Aggregate Full Threshold            1
NearlyFullThreshold   Aggregate Nearly Full Threshold     1
FullThresholdInterval Aggregate Full Threshold Interval
OvercommittedThreshold         Aggregate Overcommitted Threshold   1
NearlyOvercommittedThresh      Aggr. Nearly Overcommitted Thresh. 1
TotalSpace                     Aggregate Total Space               KB
SpaceAvailable                 Aggregate Space Available           KB
AvailablePct                   Aggregate Space Available in %      1
BytesCommitted                 Aggregate Bytes Committed           KB
BytesCommittedPct              Aggregate Bytes Committed %         1
DailyGrowthRate                Aggregate Daily Growth Rate         KB
DailyGrowthPct                 Aggregate Daily Growth (%)          1
DaysToFull                     Aggregate Days to Full
SnapshotDisabled               Aggregate Snapshot Disabled
SnapshotAutoDelete             Aggregate Snapshot Autodelete
SnapReserveTotal               Aggregate Snap Reserve Total        KB
SnapReserveUsed                Aggregate Snap Reserve Used         KB
SnapReserveUsedPct             Aggregate Snap Reserve Used %       1

So, when I run a report create command I get an error.

X:\>dfm report create -R Aggregate -f FullName,StorageSystem,UserReads,UserWrites,CPReads,TotalTransfers Aggregate_IOPS
Error: StorageSystem does not exist.
Use "dfm report catalog list" to see the list of available catalogs and fields.

Do you know what's the cause? I tried creating this report with CLI and web GUI with no luck.

Thanks for your help.

nakshay · ‎2010-01-01

Hello Ivan,

Wish you a very happy and successful year ahead

Since you are running DFM 3.6, you will have to refer to the old name of the storage system."Filer"

>>>> Filer Aggregate Filer (A)

Also you obtain the performance fields using :

"dfm report catalog list -P Aggregate"

dfm report catalog list Aggregate , will only provide you with normal report fields and not performance fields.

So your final command will look like :

dfm report create -R Aggregate -f FullName,Filer,UserReads,UserWrites,CPReads,TotalTransfers Aggregate_IOPS

Note the change in the highlighted word above.

Hope that helps.

Regards,

Akshay

ivan_huter · ‎2010-01-01

Happy New Year to you too Akshay. Thanks for taking time to help me out get to the bottom of this. Even though you say there's not much difference between 3.6 and 3.8, I came to conclusion there is. All the fields that I'm looking were introduced in version 3.7 and above. Below is the result of dfm catalog create command. As you can see dfm command and syntax you suggested does not produce desired results.

X:\>dfm report catalog list -P Aggregate
Catalog                             Default Tab
----------------------------------- -----------------------------------
Agent                               FileSRM
Appliance                           Appliances
Aggregate                           Aggregates
Disk                                Appliances
Event                               Events
Filer                               Appliances
FCPTarget                           LUNs
HBAPort                             LUNs
HostRole                            Appliances
HostUsergroup                       Appliances
HostUser                            Appliances
HostLocalUser                       Appliances
HostDomainUser                      Appliances
Interface                           Appliances
InitiatorGroup                      LUNs
IPAddress                           vFilers
LUN                                 LUNs
PrimaryDirectory                    Filesystems
Qtree                               Filesystems
SANHost                             LUNs
Script                              Scripts
ScriptJob                           Scripts
ScriptSchedule                      Scripts
SnapmirrorRelationship              Filesystems
SnapReserve                         Filesystems
Snapshot                            Filesystems
SRMPath                             FileSRM
SRMFile                             FileSRM
SRMDir                              FileSRM
User                                FileSRM
UserQuota                           FileSRM
vFiler                              vFilers
Volume                              Filesystems

However, if I type following I get aggregate fields that are available. As you can see the parameter -P is not supported in 3.6 version.

X:\>dfm report catalog list help

NAME
list -- List the catalogs and fields available for creating a report

SYNOPSIS
dfm report catalog list [ <catalog> ]

X:\>dfm report catalog list Aggregate
Aggregate Catalog

Default Display Tab: Aggregates

Fields:

Field                          Default Name                        Default Format
------------------------------ ----------------------------------- -------------
All                            All fields of this catalog
Name                           Aggregate Name
FullName                       Aggregate Full Name
Id                             Aggregate Id
GroupId                        Aggregate Group Id
Filer                          Aggregate Filer (A)
Comment                        Aggregate Comment
Type                           Aggregate Type
RAID                           Aggregate RAID Type
State                          Aggregate State
Snaplock                       SnapLock
Status                         Aggregate Status
Mirrored                       Aggregate Mirrored
DeletedWhen                    Aggregate Deleted When              DD MMM 24H
DeletedBy                      Aggregate Deleted By
Used                           Aggregate Used Capacity             KB
UsedPct                        Aggregate Used Capacity %           1
FullThreshold                  Aggregate Full Threshold            1
NearlyFullThreshold            Aggregate Nearly Full Threshold     1
FullThresholdInterval          Aggregate Full Threshold Interval
OvercommittedThreshold         Aggregate Overcommitted Threshold   1
NearlyOvercommittedThresh      Aggr. Nearly Overcommitted Thresh. 1
TotalSpace                     Aggregate Total Space               KB
SpaceAvailable                 Aggregate Space Available           KB
AvailablePct                   Aggregate Space Available in %      1
BytesCommitted                 Aggregate Bytes Committed           KB
BytesCommittedPct              Aggregate Bytes Committed %         1
DailyGrowthRate                Aggregate Daily Growth Rate         KB
DailyGrowthPct                 Aggregate Daily Growth (%)          1
DaysToFull                     Aggregate Days to Full
SnapshotDisabled               Aggregate Snapshot Disabled
SnapshotAutoDelete             Aggregate Snapshot Autodelete
SnapReserveTotal               Aggregate Snap Reserve Total        KB
SnapReserveUsed                Aggregate Snap Reserve Used         KB
SnapReserveUsedPct             Aggregate Snap Reserve Used %       1

I should've been more specific in my previous post. So, long story short - I'm handicapped to what I want to do with the version 3.6 and upgrade is eminent.

Thanks and regards,

Ivan

nagesha · ‎2010-01-03

Performance Reports are introduced in DFM 3.7.

regards,

nagesh

ivan_huter · ‎2010-01-08

Akshay,

I upgraded DFM to 3.8, created reports successfully but I'm not getting any data. Do you know what could be the problem and can you point me out in right direction. This stop-n-go is really frustrating. I'd thing this would be the one thing that should come right out of the box. It's the same thing like if you were to buy a very nice sports fast car without an od0motar, right?

Anyways, thanks for reading my rants and have a nice weekend.

Regards,

Ivan

adaikkap · ‎2010-01-08

Hi Ivan,

Pls check the perf status for each of the filer where you dont find any performance data.

This can be done by either running dfm host diag <filername/id> or in the NMC> Host>Setup.

Only if the perf data is collected this report will be able to show data for the performance fields.

Regards

adai

harish · ‎2010-01-11

Hi Ivan,

Please let me know if your problem is still not resolved.

Regards

Harish

ivan_huter · ‎2010-01-11

Adai,

thanks for your input. Here's the result of DFM command. I removed some of the proprietary info, like filer, account and domain names.

Let me know what do you think. Thanks.

X:\>dfm host diag MYFILER

Network Connectivity
IP Address             10.170.112.241
Network                10.170.112.0/22 (last searched 11 Jan 15:19)
DNS Aliases            Failed
DNS Addresses          Failed
SNMP Version in Use    SNMPv1
SNMPv1                 Passed (391 ms)
SNMP Community         XXXXXXXXX
SNMP sysName           MYFILER.com
SNMP sysObjectID       .1.3.6.1.4.1.789.2.3 (Clustered Filer)
SNMP productId         0118069635
SNMPv3                 Failed: Unknown user name
SNMPv3 Username        DOMAIN\svc_acc
ICMP Echo              Passed (0 ms)
HTTP                   Passed (0 ms)
NDMP (login not set)   Skipped
RSH                    10.170.112.241: Permission denied.
rsh: can't establish connection
SSH                    Passed (1516 ms)
RLM                    Skipped (hostRLMAddress is empty)
XML (https port 443)   Passed (78 ms)

Appliance Details
According to:   DataFabric Manager server       Host
Host Name       MYFILER.com MYFILER.com
System ID       0118069635                     0118069635
Model           FAS6080                        FAS6080
Type            Clustered Storage System       Clustered Storage System
OS Version      7.3.2                          7.3.2
Revisions       320,7.3.1,2.0.1                320,7.3.1,2.0.1

Monitoring Timestamps
Timestamp Name       Status   Interval     Default      Last Updated     Status   Error if older than ...
ccTimestamp          Normal   4 hours      4 hours                                11 Jan 11:20
cfTimestamp          Normal   5 minutes    5 minutes    11 Jan 15:18     Normal   11 Jan 15:15
cpuTimestamp         Normal   5 minutes    5 minutes    11 Jan 15:20     Normal   11 Jan 15:15
dfTimestamp          Normal   30 minutes   30 minutes   11 Jan 15:03     Normal   11 Jan 14:50
diskTimestamp        Normal   4 hours      4 hours      11 Jan 12:59     Normal   11 Jan 11:20
envTimestamp         Normal   5 minutes    5 minutes    11 Jan 15:20     Normal   11 Jan 15:15
fcTimestamp          Normal   5 minutes    5 minutes    11 Jan 15:17     Normal   11 Jan 15:15
fsTimestamp          Normal   15 minutes   15 minutes   11 Jan 15:19     Normal   11 Jan 15:05
hostPingTimestamp    Normal   1 minute     1 minute     11 Jan 15:19     Normal   11 Jan 15:19
ifTimestamp          Normal   15 minutes   15 minutes   11 Jan 15:05     Normal   11 Jan 15:05
licenseTimestamp     Normal   4 hours      4 hours      11 Jan 12:53     Normal   11 Jan 11:20
lunTimestamp         Normal   30 minutes   30 minutes   11 Jan 15:20     Normal   11 Jan 14:50
opsTimestamp         Normal   10 minutes   10 minutes   11 Jan 15:13     Normal   11 Jan 15:10
qtreeTimestamp       Normal   8 hours      8 hours                                11 Jan 07:20
rbacTimestamp        Normal   1 day        1 day        11 Jan 12:54     Normal   10 Jan 15:20
userQuotaTimestamp   Normal   1 day        1 day        11 Jan 14:01     Normal   10 Jan 15:20
sanhostTimestamp     Normal   5 minutes    5 minutes                              11 Jan 15:15
snapmirrorTimestamp Normal   30 minutes   30 minutes                             11 Jan 14:50
snapshotTimestamp    Normal   30 minutes   30 minutes   11 Jan 14:59     Normal   11 Jan 14:50
statusTimestamp      Normal   10 minutes   10 minutes   11 Jan 15:18     Normal   11 Jan 15:10
sysInfoTimestamp     Normal   1 hour       1 hour       11 Jan 14:56     Normal   11 Jan 14:20
svTimestamp          Normal   30 minutes   30 minutes                             11 Jan 14:50
svMonTimestamp       Normal   8 hours      8 hours                                11 Jan 07:20
xmlQtreeTimestamp    Normal   8 hours      8 hours      11 Jan 14:03     Normal   11 Jan 07:20
vFilerTimestamp      Normal   1 hour       1 hour                                 11 Jan 14:20

Performance Advisor Checklist
perfAdvisorEnabled     Passed
hostType               Passed
hostRevision           Passed
hostLogin              Passed
perfAdvisorTransport   Passed

X:\>

adaikkap · ‎2010-01-11

Hi Ivan,

All host setting are correct, and perf status is also fine you should be able to see the aggregate performance report.

Regards

adai

harish · ‎2010-01-11

Hi Ivan,

Please let me know if you still cannot see performance data.

Regards

Harish