Hello all,
I'm in the process of running some stress/performance tests on a new set of 6280 controllers attached to SAS 15K disks. I'm testing using NetApp SIO on an 84-disk aggregate current. In my latest batch of tests, I'm hammering it with 48 threads, 64k block sizes, 50 GB file size sample, 50% read, 100% random data samples. I've launched this process against multiple files on the aggregate.
My question is I'm seeing a large disparity between the OS reported Avg. Disk sec/Transfer and avg_latency reported by the stats command on the controller.
The OS is reporting some 7800 IOPS, 470 MB/s, and sec/transfer of a solid 33 msec (reported as .036 seconds in perfmon).
The stats command reports: 7800 IOPS, 480 MB/s, and only 1.57 ms of avg latency.
I'm liking the matching IOPS and MB/s, but am unsure why the large disparity in latency.
My first assumption is that PAM and read caching are frogging up my results, so I set this test to 100% random, use a file size of 50 GB ( larger than the 48 GB of RAM recognized by the system), and have disabled PAM (options flexscale.enable off). There is NO other traffic on this SAN, and the server is totally idle but for this testing.
My second assumption is latency introduced by the switch stack and or FC transport (I'm 100% FC, no iSCSI, NAS, or CiFS). I'm on a brand new Brocade switch with no other traffic, I'm using NetApp FC Host Utilities 5.3 (so my FC Adapter queues should be correct), NetApp MPIO, and SnapDrive. My zoning is correct, to my knowledge.
Any ideas on things I should check?