Hi Casey,
Here's some info from memory...
Firstly, the cache hit rate shown in sysstat output isn't showing the same thing as the flexscale stats. Ignore the sysstat one for now...
With a low hit rate you should look into what type of data you are missing, and what you are caching. If you use stats show ext_cache_obj this will give you more info. Better still run it over a few minutes using stats start -I cache ext_cache_obj ...wait for a minute or two during a busy period... stats stop -I cache.
This will give an output like this (notes added for some counters to check):
StatisticsID: cache
ext_cache_obj:ec0:type:IOMEM-FLASH
ext_cache_obj:ec0:blocks:67108864
ext_cache_obj:ec0:size:256 <------- size of cache in GB
ext_cache_obj:ec0:usage:92% <------- How full the cache is (should be >90% unless you've just booted)
ext_cache_obj:ec0:accesses:32047/s <----- number of times the cache was accessed per sec
ext_cache_obj:ec0:disk_reads_replaced:3768/s <---- how many disk reads were not replaced by the cache.
ext_cache_obj:ec0:hit:17898/s <---- ave number of hits per sec (add to the number of misses below to get total)
ext_cache_obj:ec0:hit_normal_lev0:16517/s <---- this is 'normal' data hits. here you can see most of the hits are 'normal' data
ext_cache_obj:ec0:hit_metadata_file:907/s <--- this and the next few are the types of hit
ext_cache_obj:ec0:hit_directory:42/s
ext_cache_obj:ec0:hit_indirect:435/s
ext_cache_obj:ec0:total_metadata_hits:1385/s
ext_cache_obj:ec0:total_metadata_misses:434/s <--- if this is high relative to the total operations, you should consider metadata cache mode
ext_cache_obj:ec0:miss:14149/s <---- number of misses per sec
ext_cache_obj:ec0:miss_metadata_file:85/s <---- this and the next few are the types of miss
ext_cache_obj:ec0:miss_directory:1/s
ext_cache_obj:ec0:miss_indirect:347/s
ext_cache_obj:ec0:hit_percent:55% <--- % hits (hits / miss)
ext_cache_obj:ec0:inserts:426/s
ext_cache_obj:ec0:inserts_normal_lev0:337/s
ext_cache_obj:ec0:inserts_metadata_file:49/s
ext_cache_obj:ec0:inserts_directory:6/s
ext_cache_obj:ec0:inserts_indirect:39/s
ext_cache_obj:ec0:evicts:11/s <---- this indicates the number of blocks evicted (ie not accesses frequently) If this is very high you *may* need more flash caches
ext_cache_obj:ec0:evicts_ref:6/s
ext_cache_obj:ec0:readio_solitary:1524/s
ext_cache_obj:ec0:readio_chains:3768/s
ext_cache_obj:ec0:readio_blocks:15788/s
ext_cache_obj:ec0:readio_max_in_flight:511
ext_cache_obj:ec0:readio_avg_chainlength:4.19
ext_cache_obj:ec0:readio_avg_latency:0.56ms
ext_cache_obj:ec0:writeio_solitary:0/s
ext_cache_obj:ec0:writeio_chains:6/s
ext_cache_obj:ec0:writeio_blocks:426/s
ext_cache_obj:ec0:writeio_max_in_flight:182
ext_cache_obj:ec0:writeio_avg_chainlength:64.00
ext_cache_obj:ec0:writeio_avg_latency:2.14ms
ext_cache_obj:ec0:invalidates:514/s <--- data in the cache that's been overwritten on disk (and thus 'invalidated')
Based on this you may need to adjust your cache settings (options flexscale) to cache different data.
You should also check you're VM alignment - misaligned blocks may affect caching efficiency. If that looks OK, dedupe may also help.
If you have Performance Advisor you can also collect these stats and review regularly if required.
Hope that helps,
Craig
