Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have a 512GB PAM II card
and when I run a stats show -p flexscale-pcs it shows a high % (92%) usage, a low Hit % (9%) but it doesnt show a high evict or insert rate.
But when I run a sysstat -u 1 I get a cache hit rate (94%) but my disk utilization is high (91%)
We run VMs on this server. My question: is my system cache maninly being used a very little making it to the PAM ? It shows a lot of information in PAM but the hit rate is very low.
Or is this to be expected because the VMs are large files etc?
2 REPLIES 2
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Casey,
Here's some info from memory...
Firstly, the cache hit rate shown in sysstat output isn't showing the same thing as the flexscale stats. Ignore the sysstat one for now...
With a low hit rate you should look into what type of data you are missing, and what you are caching. If you use stats show ext_cache_obj this will give you more info. Better still run it over a few minutes using stats start -I cache ext_cache_obj ...wait for a minute or two during a busy period... stats stop -I cache.
This will give an output like this (notes added for some counters to check):
StatisticsID: cache
ext_cache_obj:ec0:type:IOMEM-FLASH
ext_cache_obj:ec0:blocks:67108864
ext_cache_obj:ec0:size:256 <------- size of cache in GB
ext_cache_obj:ec0:usage:92% <------- How full the cache is (should be >90% unless you've just booted)
ext_cache_obj:ec0:accesses:32047/s <----- number of times the cache was accessed per sec
ext_cache_obj:ec0:disk_reads_replaced:3768/s <---- how many disk reads were not replaced by the cache.
ext_cache_obj:ec0:hit:17898/s <---- ave number of hits per sec (add to the number of misses below to get total)
ext_cache_obj:ec0:hit_normal_lev0:16517/s <---- this is 'normal' data hits. here you can see most of the hits are 'normal' data
ext_cache_obj:ec0:hit_metadata_file:907/s <--- this and the next few are the types of hit
ext_cache_obj:ec0:hit_directory:42/s
ext_cache_obj:ec0:hit_indirect:435/s
ext_cache_obj:ec0:total_metadata_hits:1385/s
ext_cache_obj:ec0:total_metadata_misses:434/s <--- if this is high relative to the total operations, you should consider metadata cache mode
ext_cache_obj:ec0:miss:14149/s <---- number of misses per sec
ext_cache_obj:ec0:miss_metadata_file:85/s <---- this and the next few are the types of miss
ext_cache_obj:ec0:miss_directory:1/s
ext_cache_obj:ec0:miss_indirect:347/s
ext_cache_obj:ec0:hit_percent:55% <--- % hits (hits / miss)
ext_cache_obj:ec0:inserts:426/s
ext_cache_obj:ec0:inserts_normal_lev0:337/s
ext_cache_obj:ec0:inserts_metadata_file:49/s
ext_cache_obj:ec0:inserts_directory:6/s
ext_cache_obj:ec0:inserts_indirect:39/s
ext_cache_obj:ec0:evicts:11/s <---- this indicates the number of blocks evicted (ie not accesses frequently) If this is very high you *may* need more flash caches
ext_cache_obj:ec0:evicts_ref:6/s
ext_cache_obj:ec0:readio_solitary:1524/s
ext_cache_obj:ec0:readio_chains:3768/s
ext_cache_obj:ec0:readio_blocks:15788/s
ext_cache_obj:ec0:readio_max_in_flight:511
ext_cache_obj:ec0:readio_avg_chainlength:4.19
ext_cache_obj:ec0:readio_avg_latency:0.56ms
ext_cache_obj:ec0:writeio_solitary:0/s
ext_cache_obj:ec0:writeio_chains:6/s
ext_cache_obj:ec0:writeio_blocks:426/s
ext_cache_obj:ec0:writeio_max_in_flight:182
ext_cache_obj:ec0:writeio_avg_chainlength:64.00
ext_cache_obj:ec0:writeio_avg_latency:2.14ms
ext_cache_obj:ec0:invalidates:514/s <--- data in the cache that's been overwritten on disk (and thus 'invalidated')
Based on this you may need to adjust your cache settings (options flexscale) to cache different data.
You should also check you're VM alignment - misaligned blocks may affect caching efficiency. If that looks OK, dedupe may also help.
If you have Performance Advisor you can also collect these stats and review regularly if required.
Hope that helps,
Craig
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I feel a bit silly - it was because we had a degraded raid set and it was reb rebuilding...
