Dan,
The array has both SAS, 300GB and 450GB 15K, and 1TB SATA drives. The 300GB SAS is in 9+1 RAID5 DP Pools, 450GB SAS in 6+1 RAID5 DP Pools, and the 1TB SATA is in 8+2 RAID6 DP Pools. There are 1:1 DP Pools to NetApp aggregates. Some of the LUNs are 300GB and some are 783GB. In most cases there are between 3-5 parity groups per DP Pool. Total disk count is above 400.
So as an example, we have a DP Pool with 4x parity groups of 9+1 300GB drives. In this dp pool we have 32x 300GB LUNs allocated to a V3240 pair, all owned by one filer in the same aggregate with RG size of 8.
Our AMS has 4 front-end ports per controller, 8 total. Each port is only dedicated to one filer, with each filer having a front-end port on each controller and thus two paths to each device.
I have opened cases in the past with both HDS and NetApp. NetApp told me that the latencies were caused by the HDS disk and HDS told me that the latencies I saw for the SAS LUNs at the time were expected. The latencies were between 25-50ms from the AMS on these SAS LUNs.
The higher than expected latencies are primarily on VMware NFS datastores. The majority of our volumes perform between 2-10ms host latency on average. The VMware datastores average ~20ms with peaks going above 100ms and these peaks are fairly common. These are host latencies as measured from both vcenter and NetApp Performance Advisor. We've been round and round with VMware performance, vmdk alignment, network best practices, etc. Things don't seem to change and all the evidence continue to point to the back-end disk.
Am I seeing disk queuing on the AMS? I think the answer would be yes here, but I'm not sure how to quantify this. HDS offers a metric called "tag count" and they show it on a per drive and per LUN basis. The tag count can range from 2-25.
These systems do send autosupports. I have run perfstat in the past and have provided this for NetApp's troubleshooting.
I have a NetApp Account Team and I do know who my SE is.
I don't expect this community to solve my problems, but I am mostly just asking to see what other folks may be doing in a similar setup. Most of the settings I've listed above have very little documentation on their own, let alone on a V-series on AMS.