Not 100% the answer to your question, but maybe it is helping you a little bit:
(I am not an expert with netapp, but had similar problems in the past, too)
When you use VMWARE, maybe you are using VMWare-snapshot-based backup-software like VEEAM? This was causing high latency in my system when VEEAM made snapshots at times after midnight.
Best thing you can do: if you have a maintenance contract, open a case with Netapp!
I would check if the times of the latency correlate with backup- or other regular events. Is it around midnight, when deduplication and/or reallocation are running? This adds more work to the filer and disks, too.
I also would check the CPU-load in times of high latency.
Furthermore I would look for b2b-CPs (with the command "sysstat -x -1 " , 1 is the time in seconds - every second (or every 5, 10, ... seconds if you use another number) you get information about actual CPU-Load and other interesting things).
There is a tool "perfmon" or "perfstat", which is collecting informations, but I guess you will not be able to interpret the resulting file - the netapp support team can do that, if you have a maintenance subsription.
"The 7200 drives in a FAS2040 are still going to be a possible bottleneck when working with VMs and with this type of filer. This system has a small NVRAM and it is split up into buffers. The filer has two buffers for accepting and logging write data. With a filer that has a small amount of NVRAM, the buffers will fill up quickly and with the slower disks, you may still hit the b2b CPs that I mentioned in an earlier email, during high load times. That is because it will take longer to write things to the slower disks, even when they are working optimally, and since the 2nd buffer is not that large, it will fill up pretty quickly when there is a lot of work coming into the filer. "
The FAS2040 with 7.2 k drives is limited in performance, I had to learn.
Adding DS14 MK 4 Shelfs with 15 k FC-Drives helped in my case, but I am still not really happy with the speed and the performance of my filer. Not enough memory, not enough NVRAM, not enough CPU-Power, slow 7.2 k-SATA-Disks with approx. 80 IOPs. The 15k FC-Disks are a lot faster (approx 180 IOPs), and can be attached to the FAS2040 if you have the FC-ports still available. 15k SAS-Drives could be used, too, but are normally more expensive and smaller in size.