I'm seeing a weird issue that seems to be specific to a particular application.
First, this application is a 3rd party/customized application which processes data sets around 100GB in size. The raw data is read, analyzed, and written back out.
We moved this data set from physical servers to VMware with NFS datastores. For various reasons, it's a single server which is processing this data, and in some cases, we only have 1 GbE available for the storage. I don't expect to see more than 1 Gbps throughput, but I do expect to see it in each direction as the network is Full Duplex.
However, when this data set is processed, I see almost exactly 1 Gbps of throughput... half read, half write... as if there's a Half Duplex link somewhere in the mix. After enlisting the help of our server and network teams and not finding the smoking gun, I fired up a synthetic workload (iometer) and was able to validate that we get 1 Gbps in each direction simultaneously. So I know there is no misconfiguration - the infrastructure is capable of Full Duplex 1 Gbps.
However, this application reaches a very suspicious plateau at 500 Mbps read & 500 Mbps write. Unfortunately, when this occurs, something gets so congested that disk queue length on the guest shoots up to about 25, which is roughly 1 per spindle in the aggregate and the server becomes extremely sluggish and unresponsive. Takes roughly 5 minutes to log in via RDP.
While testing, I moved the workload to a 10 GbE network and again, verified I could push more than 5 Gbps, and I can with iometer. However, the application does not seem to be able to. It doesn't have the disk queue length issue and sluggishness, but I have a feeling if I increase the workload, I'll find that "50% limit" on the 10 GbE network, just like on the 1 GbE network.
I moved the server's C:\ disk to block storage and left the data drive on the NFS datastore, and this seems to have resolved the sluggish responsiveness of the guest, even though disk queue length on the guest is 25 on the data drive. So I think I found the workaround... what I'd like to understand better is why disk queue length jumps so high when the application is only filling the wire 50%.
Has anyone ever seen this before? Can anyone offer an explanation?
