Do you see a lot of "other" IOPs? Is the flashcache in default mode (data and metadata)? It might help to set to metadata only if a lot of directory and metadata lookups. A perfstat would be interesting to have analyzed to see performance.
nfsv3:nfs:nfsv3_op_latency.readdirplus:6041.57us - DLHFS15 directory scanning – This build does a lot of directory scanning (11% of the operation is directory scanning) and the latency for the same on DLHFS15 is much higher than that of BNRFS04
The number of files in BNRFS04 is 9604456whereas in DLHFS15, it is 191026996. This is the root cause of it. The solution for this problem is to have a system with more memory. We have already tried this with different configuration of PAM on FAS6040, but its caching is not helping with your workload. This is the configuration that we tried in your environment for PAM
flexscale.enable on (same value in local+partner recommended)
flexscale.lopri_blocks off (same value in local+partner recommended)
flexscale.normal_data_blocks off (same value in local+partner recommended)
flexscale.pcs_high_res off (same value in local+partner recommended)
flexscale.pcs_size 0GB (same value in local+partner recommended)
But just want to confirm why Flash Cache is not usefull in such type of environment.
Also I believe the number of file available in filer doesn't matters. In fact the thing which matter is the the number of files or indoes for which lookup requests is generated.
Did any one know how we find the the number of lookup request generated by the filer?