With fastpath enabled, the filer will respond to the same MAC address and tries to send the packet back thru the same interface as it came in. This allows for loadbalancing on multiple trunk interfaces on the same subnet. Disabling this means that ONTAP has to look up its routing table to figure out where to stick the packet in and *might* introduce a delay. In other scenarios, if you have the filers connected to switches running HSRP, then it might conflict with the fastpath setting.
I would start by checking to see if there are any errors reported on the filer (netstat -nr, netdiag commands may be useful to check settings)
Another thing to check if you have multiple interfaces in the same subnet, then try disabling fastpath.