How are the interfaces configured (single nic - vif - same subnet or routed)
Assuming flexvol on an aggregate - how many spindles and of what disk type?
External disk shelf? What type?
Is this the only workload on the aggregate or is it shared with other applications / processes?
During your IOMeter test - how about profiling the system workload a little with:> sysstat -x 1
This should give us an impression of what you're working with and general performance on the box. It's worth dumping some of the sysstat output here for people to look at.
NB - relating to the flow control issue - my understanding is that unless specified, the default flow control is set to advertise at full however, the operational flow control varies on an number of factors (autonegotiation etc). ifstat should give this value.
Are there separate aggregates on different loops here (i.e. more disk other than the 1xds14 mk2 sata?) I'm seeing more IOPS than I actually expected and there is some iSCSI activity running in parallel. Assuming dual parity plus 1 spare = 11 data disks, I would be happy if the SATA shelf was pulling 1000 iops. Sysstat is displaying over 2000 iops at times without stressing the disk at all. Large cache hit plus low disk utilisation would lead me to expect low latency rather than the results you've been seeing. It's only chucking out 15MB per second. Hmmm.
If I were at the console, my next step would be:
* sysconfig -r (to check the raid config)
* ifconfig -a (for interface setup)
* vif status (check vif status - surprise)
* ifstat -a (looking for errors)
I would also be interested in the routing. I know you've mentioned that the traffic should be same subnet / non routed but I always like to take a look: To make sense - if you're running multiple vlans etc - you'll need to identify which one the CIFS traffic is supposed to be served on. * route -sn
If you're happy to share this info, stick it in notepad and append it to the thread.
I'd spend some time in Performance Advisor as well....one of my default first places to go for performance issues (as covers CPU, interfaces, disks, volumes, aggregates, etc.). You can get all this stuff via stats/statit but Performance Advisor is just much easier to wander through when you don't know exactly what you're looking for.
Also, can you temporarily license NFS and see if any difference? Basically just trying to think of a way to take a variable out of the picture... (i.e. the CIFS protocol and focus on network/backend disks/etc.)