How to configure flash cache cards to work on a flash pool aggregate

[as]

Hi,

I would like to know what is the best practices and your expirience on setting and how to configure (NVMe) flash cache card to work on a flash pool aggregate.

Some facts about the platform:

Ontap 9.1p13

FAS9000 with NVMe card (caching currently disabled)

two sata aggregates both of them are fabric pools

the application hosted on those aggregates is mail application with 95% read and 5% write workload

One KB articel about enabling flash cache on a flash pool aggregate states this command:

clusthershell> node run <node_name>
nodeshell> options flexscale.flash_pool_caching on

another sources says enable it like this:

system node external-cache modify -is-hya-enabled true

Does it matter which command to use? Are they doing both the same = enable flash cache on flash pool aggregates?

And there are a couple other caching options like mbuf-inserts, pcs-size, rewarm

Which setting for a flash cache/flash pool combination works best?

The current configuration looks like this:

netapp::*> system node external-cache show -node netapp-*

Node        enabled    rewarm    mbuf-inserts    pcs-size    hya-caching

----------    -------    ------    ------------    --------    -----------

netapp-01    true      true      true            4096        false

netapp-02    true      true      true            4096        false

netapp-03    true      false     true            894         false

netapp-04    true      false     true            894         false

netapp-05    true      false     true            894         false

netapp-06    true      false     true            894         false

netapp-07    true      false     true            894         false

netapp-08    true      false     true            894         false

netapp-09    true      false     true            894         false

netapp-10    true      false     true            894         false

netapp-11    true      false     true            894         false

netapp-12    true      false     true            894         false

12 entries were displayed.

Highly appriciate any input, so far I only got feedback "it depends" and unfortunately that option is not available 😉

Thanks

Alex

[D_BEREZENKO]

Short answer: you cannot use both simultaneously.

FabricPool is only for SSD aggregates.
FlashCache does not give benefits for SSD aggregates; therefore FlashCache is not working with FabricPool aggregates.

The same applies to FlashPool: if you have a hybrid aggregate consists of HDD + SSD (FlashPool), then FlashCache not going to work for that aggregate.

PS.

Do not mix up FabricPool (Data Tiering Technology to S3 object storage) with FlashPool (Hybrid Disk Caching Technology) for read/write caching, while FlashCache is almost the same as FlashPool, except for the fact it is a card (PCIe or M2 with NVMe memory on it), it works controller-wide, and only for read caching.

[as]

to my knowledge they work together since ONTAP 9.1

the option "- Is hya caching enabled" is the one to enable flash cache card caching on an hybrid aggregate.

Also mentioned in the documentation and in an internal KB articel from NetApp.

From the ONTAP 9 documentation:

Description

The system node external-cache modify command can be used to modify the following attributes of external-cache for a node:

• is-enabled
• is-rewarm-enabled
• is-mbuf-inserts-enabled
• pcs-size
• is-hya-enabled

[-is-hya-enabled {true|false}] - Is hya caching enabled

Specifies whether the external-cache module allows caching of blocks targeted for hybrid aggregates. This option is set to true by default when the external-cache is enabled.

Hence my question if someone has expirience with it and has it enabled in production.

Cheers

Alex

[D_BEREZENKO]

Alex, they don't.

Hybrid aggregate means FlashPool. The technology has two names: Marketing name is "FlashPool" while in command line arguments uses "Hybrid Aggregate" name.

[as]

Well, I have an internal NetApp document infront of me called "How to configure flash cache cardsto work on a Flash pool aggregate.pdf" which states:

Starting with Ontap 9.1 onwards, Flash Cache could be configured to be used on a Flash pool aggregate together; this is controlled by a node shell option flexscale.flash_pool_caching, which is by default set to off.

To enable this:

clusthershell> node run <node_name>
nodeshell> options flexscale.flash_pool_caching on

To find out how much Flash Chache helps in a Flash pool aggregate:

ext_cache_obj:ec0:disk_reads_replaced: Estimated number of
disk reads per second replaced by cache
ext_cache_obj:ec0:hya_reads_replaced: Estimated number of HyA
reads per second replaced by cache

And I have several confirmations that this works from netapp engineers.

Cheers

Alex

[D_BEREZENKO]

Its not working this way.

Just try on real system and you will understand.

[AlexDawson]

Hi @as - To help make your question clearer, I think you need to clarify - do you mean fabric pools or flash pools? I know the names are similar, but the technology is not and it seems you're using them interchangably in your question and haven't addressed this in your subsequent followups

Also.. internal documents are internal for good reasons. They may refer to functionality which is not available or supported publicly, and which may impact data integrity or availability. Some functionality requires undocumented boot flags to be set, or even license codes applied.

If you are having performance concerns, please open a ticket with our support team and they will work through it with you, including enabling other functionality if required.

[as]

Hi Alex,

flash pool aggregates (HDD+SSDs).

And yes, I understand they are internal documents and I was thinking for a while until I posted it - maybe I should not have done it...

Fully undertand that it might have impact and requires undocumented options etc., hence my question if someone has expirience with it and has it running in production.

I tried to involve NetApp support via our SAM but no luck so far, only answer I got is that this is not a support question. I tried to involve pre-sale, also no luck and no answer. That's why I asked in the end the community...

[AlexDawson · Accepted solution]

I've done some digging - this functionality was available undocumented in ONTAP 8.3.1 and 9.0, and became default for any new install of 9.1

The command is indeed "options flexscale.flash_pool_caching on"

So.. it's on on your system, since FAS9000's can run a min ONTAP of 9.1

The read and write policy for any new volumes you create will be "auto", but if you transitioned shelves from an older system, or copied with snapmirror, it might not be, so check out this KB article for how to set it to auto - https://kb.netapp.com/app/answers/answer_view/a_id/1031068/loc/en_US#__highlight

As the KB article says - we do not suggest you change it from auto except under specific advice from performance specialists.

Hope this helps!

[D_BEREZENKO]

Alex, you just mixing things up.

That KB explains how to use caching policies in a FlashPool aggregate and not related to FlashCache in any way. Though FlashCache & FlashPool have very similar names, algorithms, and some similar read caching policies, they are different not compatible things. FlashCache, FlashPool & FabricPools are technologies which currently mutually exclude each other.

Maybe, just maybe at some point in the future, when NetApp will have Storage Class memories (SCM) in AFF systems, then will be a new generation of FlashCache cards based on SCM memory which will work with FabricPool or All Flash aggregates. It's not available yet though.

TR-4070, page 17:

"Data from volumes that are provisioned in a FlashPool aggregate or an all-SSD aggregate is automatically excluded from being inserted into FlashCache cache. Data from volumes provisioned in a FlashPool aggregate use the FlashPool cache, and data from volumes provisioned on an SSD aggregate does not benefit from using FlashCache."

Though NVMe interface is more powerful than SAS, FlashCache has already been PCIe cards, so there should be no or small difference in performance between FlashCache based on PCIe vs. FlashCache based on NVMe interface. While media itself for FlashCache for old & new cards was and are nearly the same NAND technology, therefore no significant performance difference and consequently no significant caching difference there too. Only SCM have big performance difference which can give a boost for caching for all-SSD aggregates.

That is precisely the reason why All Flash systems do not have FlashCache cards on-board.

[AlexDawson]

Hi @D_BEREZENKO thank you for this, and all your contributions on this forum, but I have triple checked (I double checked before I posted 🙂 and I stand by my original comments. The TR you linked to pre-dates ONTAP 9.1 where this was released, and so that piece of information is technically invalid.

It is not a major piece of functionality and so it is not broadly covered by marketing or other material. Flashcache and Flashpool SE Presentation v2.0, slide 28 if you have access and you'd like to read more on the subject.

[D_BEREZENKO]

Alex, you are right about it, FlashCache now can work together with FlashPool/all-SSD aggregates. I didn't know it changed, thanks for the research.

Though, as I would expect, it is recommended to disable FlashCache for volumes on all-SSD aggregates.

[as]

thanks for your inputs and verifications so far.

Coming back to my original questions, has anyone experience with flash cache in combination with flashpool in production?

Did you see improvements on the caching side by using flashpool and flash cache in combination?e.g. higher cache hit rate, etc.

What are the best practices setting for the rest of the external-cache options (rewarm, mbuf-inserts, pcs-size) in a flash cache+flashpool combination?

netapp::*> system node external-cache show -node netapp-*

Node        enabled    rewarm    mbuf-inserts    pcs-size    hya-caching

----------    -------    ------    ------------    --------    -----------

netapp-01    true      true      true            4096        false

netapp-02    true      true      true            4096        false

netapp-03    true      false     true            894         false

netapp-04    true      false     true            894         false

netapp-05    true      false     true            894         false

netapp-06    true      false     true            894         false

netapp-07    true      false     true            894         false

netapp-08    true      false     true            894         false

netapp-09    true      false     true            894         false

netapp-10    true      false     true            894         false

netapp-11    true      false     true            894         false

netapp-12    true      false     true            894         false

12 entries were displayed.

No idea why the topic was set to solved, it is not actually...my questions were around best practices settings and expirience in the field - not if it is possible or not, I knew it is 😉

Thanks

Alex

[D_BEREZENKO]

In my understanding, there are few potential areas for benefit when using (current gen) FlashCache + FlashPool. Apparently in this case you'll get the bigger total amount of cache, which meant nothing by itself, but for certain data pattern it might be beneficial:

1. When you have more read operations & those read operations are small random blocks
2. Optimization for certain read caching operations 
3. Some benefits for situations for short bursts of sequential operations and/or big blocks of data been read

[COMITSUPPORT]

Integrated NetApp Flash Cache and Flash Pool Caching

Volumes in Flash Pool aggregates and all-SSD aggregates use NetApp Flash Cache

Flash Cache is used for all policies except “none”

It is recommended to set policy to “none” for volumes contained within all-SSD aggregates

Enables longer-term retention of frequently-read blocks in Flash Pool cache

Dual-stage caching prevents blocks that are not really hot from entering Flash Pool cache

NetApp ONTAP 9 enhancements

Common set of caching policies for both Flash Cache and Flash Pool caching

One policy per volume applies to both caches

14 caching policies that are combinations of:

Random and sequential read block insertion

Random and sequential write blocks inserted via a read caching policy (write-through caching)

blocks are cached after being written to HDD

Random overwrites are inserted directly into cache (write-back caching) – Flash Pool only policy

Most useful polices are:

“auto” – default policy

“none”

“all_read”

“all”

hope it helps

[Baiju625]

This thread was really helpful. We had disk utilization going beyond 50% and after I enabled the  following option, the disk utilization came down to below 10% average.

system node external-cache modify -is-hya-enabled true  -node *

Also please note that you need to wait for cache to warm up to see the benefit, so check utilization next day. 

Please do these guys to use the flash cache you have purchased (along with flash-pool/hybrid aggregate)