Environment with average file size is then 16k

[#1]

In my environment, I have multi terra bytes of file data which I access using NFS from the grid. Now average file size is less than 16k. Just to give you an idea, the 80% of files are <16k, and 50% of the files are 4kb or less, as a result my performance figues are not that great. Most of the data are write intrisic, and since Netapp do all sequential writes, so wondering how can i improve my storage performance further. The bottle neck is huge metadata becuase of so many files and I have crazy inodes figures. network bandwidth is not an issue becuase of the file size.

Can someone who have experienced similar enviroment can give me some insight for tuning any parameters,which will improve the storage performance. Also is there any upper limits of NFS IOPS that 3200 series filers can handle without PAM module, also will the PAMII module be some some help to me, becuase of huge metadat the filer needs to handle even though the read operations are much less compared to writes.

Thanks in advance to all.

[#2]

In environments like this (another grid customer) we used FlashCache (PAM) and turned off normal blocks leaving it in MetaData only mode.  So the FlashCache can be set to metadata only which made a big difference compared to both data and metadata. 

You can run predictive cache statistics (PCS) to see the benefit of doing this by running in both modes (there is a third lopri mode but doesn't apply here) and work with your NetApp team to analyze the results to see if FlashCache makes sense... PCS does take some resources but good to see if it helps and I bet it does.  You can also set the predictive cache size to simulate the amount of cache you add (1 or 2x 512GB for example depending on your model...or even the new 1TB flashcache now if supported on your model).

[#3]

With the workload you have (write intensive) I'd be curious to see your disk utilization.  This can be viewed within performance advisor or by running the stats commands from the filer. I'd be curious to specifically see the output from this command:

filer> stats show disk:*:disk_busy

Also, how many disks do you have within your aggregate? What aggregate raid group size? What speed & type (FC, SAS, SATA)?

[#4]

Thanks Scott, I also share similar view, but I think it is good idea to check and the PCS thing before going for it. I will update once I collect the data and see the results.

[#5]

We found that offloading the reads helped the heavy write. Also the disk stats post is a good idea along with a perfstat for more details.

Sent from my iPhone 4S

[#6]

My filer has two type of disk.

FC 10krpm - 2 shelves

SATA - 1 shelve (24 disk shelf)

One aggr is for all the 28 FC disk, out of which 2 spare,and one raid group of 26 disks(RAID-DP)

One aggr is for all 24 SATA disk, out of which 1 spare, two raid group of 11 and 12 disk(RAID-DP)

Also here is the output of stats show for disk busy..Can you please explain, how to read this output? and how it is relevant?

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

disk:500605BA:0003FF84:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003A224:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FA18:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FAC4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00041CF4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0004161C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00035744:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00040024:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FE88:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FE34:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003F7E0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FC78:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003A208:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00041E90:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FFB0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003A1F4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003A200:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:001CD208:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003F7A4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FE78:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FF24:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003A094:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003F7F8:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003F930:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500000E0:1E2CDAB0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:2F3831D4:397C63FB:5B4312C4:BAD41B7E:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0019F128:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0005FE08:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:22839F05:352F40F6:05E07CE8:C0FBEAF4:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:63420954:81B80AE3:9C610BB0:361092BD:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:000613E8:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:903904DED478DEA:3EC9C10A:9433728D:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0005EE88:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:C4C15D4E:C1CA0912:7E2D6A77:568B44B1:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:E95D47D4:21A2A109:41DEC12F:AEEB7DB7:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:3848BB84:A9CF3BFA:1542E5CB:561BAF8F:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00060AB8:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:EE5FB9DB:C35D92EF:FEBE1687:3EAF7D65:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:AA3589E38EAA86E:86946D93:17170746:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00042310:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:73561E3FB70B560:C5EB8DC5:9861DF45:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:4%

disk:500605BA:00041EC0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:18%

disk:62E93B6C:FDE764DD:FF64AF6F:E107B290:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:4EB4CB50:34913C42:C449C924:7E10EA93:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0005E6C0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FEA4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:29%

disk:500605BA:0003FCB8:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:23%

disk:500605BA:00060E74:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003F7C4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:20%

disk:500605BA:0003FBE4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:19%

disk:02BA94FC:B1DEECC90A7EC6D:4B215ED0:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0005E00C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:3FE786D8:1139B204:BEEDA40EBECD2A8:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00042EB4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00040058:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:21%

disk:2DAFD344:A0FA25D0:FFF634BB:880CB094:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:433495A4:EEB7E28B:ED342B48:848BC1F6:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FF7C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:40%

disk:2D491B03:9CC34AE9:89C8031A:EFDB88C0:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0005E688:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:C6F9EA17:2F00B26F:049682DC:70FBD427:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00040068:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:18%

disk:500605BA:0005F074:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00040050:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:25%

disk:500605BA:0003A230:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:11%

disk:56FA1E46:C4637CB3:1D9B17D0:212346ED:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0005E6D0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:7%

disk:500605BA:0003F920:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:22%

disk:500605BA:0003F7D8:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:22%

disk:500000E0:19EA0FE0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:46%

disk:500605BA:0003FFC8:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:38%

disk:7F34CBDB:E658CADC:19894A03:67C89263:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:C098521265908CB:2956F478:A96A8CB4:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00060E14:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00049A64:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00041ED8:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:4%

disk:500605BA:0005E588:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:13%

disk:799C40A0:B9A8B9E0:6A40DA4C:C3BAA403:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00041768:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:3%

disk:FA9B4B27:0313EDCD:6F4FBFC84A730DF:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:00060A9C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0004166C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:24%

disk:500605BA:00041EF4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:20%

disk:500605BA:00060C10:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:9%

disk:500605BA:0003FE80:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:1%

disk:500605BA:000612FC:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:11%

disk:500000E0:1B0AEB20:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:48%

disk:500605BA:00060A84:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:3%

disk:500605BA:0003FCB0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:23%

disk:500605BA:0005EEF8:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:7%

disk:500605BA:0003F7B4:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:29%

disk:500000E0:1E17D5E0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:44%

disk:500000E0:1B0BCDE0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:52%

disk:500000E0:1C654420:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:46%

disk:500605BA:0005E6F0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:CDAFECC1F562F5C:97F2736E:EB9D0115:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0003FA38:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:2%

disk:500605BA:0005E60C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:5%

disk:500605BA:0003F7EC:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:21%

disk:500000E0:1B0BB570:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:51%

disk:2000001D:38BDE5B0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:49%

disk:500000E0:1E35B150:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:45%

disk:500000E0:1695C6F0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:47%

disk:500000E0:17CC1AC0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:61%

disk:200C000C:CAD2CD9C:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:44%

disk:500000E0:1B0C2650:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:54%

disk:5BC93A72:C7BC4F1E:7B04F41E:3CED20C3:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500605BA:0005E694:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:2%

disk:500000E1:13BF7F60:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:21%

disk:500000E0:1B0C7800:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:43%

disk:500000E1:12FF70B0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:52%

disk:500000E0:1B0C3FE0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:48%

disk:2000001D:38BF4D84:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:54%

disk:0A762205:A5E5D053:68E66006:A728D86B:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500000E0:17CBC340:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:49%

disk:500000E0:17B2BBA0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:42%

disk:500000E1:13022A90:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:38%

disk:500000E0:17CC1910:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:45%

disk:20000000:877E28C5:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500000E0:1BF9AA10:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:43%

disk:20000000:877D1F3A:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:54%

disk:500000E0:17C23C50:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:44%

disk:500000E1:13A4ACB0:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:22%

disk:500605BA:00060B00:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:280FDA39:A8B4DC19:0C7F7499:FF0FDE81:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:56BD0741:5D09549E:B06055E2:8C35E339:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%

disk:500000E0:1C903500:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:48%

disk:500605BA:000F0B80:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:20%

---------------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks