Hi,
Two ideas:
(1) The command reallocate start -f /vol/esxprod2_nfs_sys will indeed give an error if you have snapshots (because of the high rate of changed blocks that might result in filling up your vol) but you can run it per LUN/file like reallocate start -f /vol/esxprod2_nfs_sys/data.vmdk. Maybe give it a try on some VMs that you think are most impacted.
(2) If you have a metrocluster you can enable reads to hit both plexes as by default its set to local plex only. If you have a long distance between sites, or know you'll saturate your ISLs in a fabric MC then no solution but for many situations it allows you to double your read IOPs at no cost. The behavour is configured by a system-wide option, can be done online, and takes effect immediately:
filer> options raid.mirror_read_plex_pref alternate
From the manpage:
raid.mirror_read_plex_pref
Specifies the plex preference when reading from a mirrored traditional volume or aggregate on a metrocluster-configured system. There are three possible values -- `local' indicates that all reads are handled by the local plex (plex consisting of disks from Pool0), `remote' indicates that all reads are handled by the remote plex (plex consisting of disks from Pool1), and `alternate' indicates that the handling of read requests is shared between the two plexes. This option is ignored if the system is not in a metrocluster configuration, i.e., cluster_remote is not licensed. The option setting applies to all traditional volumes and aggregates on the filer.
Cheers,
Chris