Subscribe
Accepted Solution

How to configure flash cache cards to work on a flash pool aggregate

[ Edited ]

Hi,

 

I would like to know what is the best practices and your expirience on setting and how to configure (NVMe) flash cache card to work on a flash pool aggregate.

 

Some facts about the platform:

Ontap 9.1p13

FAS9000 with NVMe card (caching currently disabled)

two sata aggregates both of them are fabric pools

the application hosted on those aggregates is mail application with 95% read and 5% write workload

 

One KB articel about enabling flash cache on a flash pool aggregate states this command:

 

clusthershell> node run <node_name>
nodeshell> options flexscale.flash_pool_caching on

 

another sources says enable it like this:

 

system node external-cache modify -is-hya-enabled true

 

Does it matter which command to use? Are they doing both the same = enable flash cache on flash pool aggregates?

 

And there are a couple other caching options like mbuf-inserts, pcs-size, rewarm

Which setting for a flash cache/flash pool combination works best?

 

The current configuration looks like this:

 

netapp::*> system node external-cache show -node netapp-*

Node        enabled    rewarm    mbuf-inserts    pcs-size    hya-caching

----------    -------    ------    ------------    --------    -----------

netapp-01    true      true      true            4096        false

netapp-02    true      true      true            4096        false

netapp-03    true      false     true            894         false

netapp-04    true      false     true            894         false

netapp-05    true      false     true            894         false

netapp-06    true      false     true            894         false

netapp-07    true      false     true            894         false

netapp-08    true      false     true            894         false

netapp-09    true      false     true            894         false

netapp-10    true      false     true            894         false

netapp-11    true      false     true            894         false

netapp-12    true      false     true            894         false

12 entries were displayed.

 

Highly appriciate any input, so far I only got feedback "it depends" and unfortunately that option is not available ;-)

 

Thanks

Alex

 

Re: How to configure flash cache cards to work on a flash pool aggregate

[ Edited ]

Short answer: you cannot use both simultaneously.

FabricPool is only for SSD aggregates.
FlashCache does not give benefits for SSD aggregates; therefore FlashCache is not working with FabricPool aggregates.

 

The same applies to FlashPool: if you have a hybrid aggregate consists of HDD + SSD (FlashPool), then FlashCache not going to work for that aggregate.

PS.

Do not mix up FabricPool (Data Tiering Technology to S3 object storage) with FlashPool (Hybrid Disk Caching Technology) for read/write caching, while FlashCache is almost the same as FlashPool, except for the fact it is a card (PCIe or M2 with NVMe memory on it), it works controller-wide, and only for read caching.

Re: How to configure flash cache cards to work on a flash pool aggregate

[ Edited ]

to my knowledge they work together since ONTAP 9.1

the option "- Is hya caching enabled" is the one to enable flash cache card caching on an hybrid aggregate.

Also mentioned in the documentation and in an internal KB articel from NetApp.

 

From the ONTAP 9 documentation:

Description

The system node external-cache modify command can be used to modify the following attributes of external-cache for a node:
  • is-enabled
  • is-rewarm-enabled
  • is-mbuf-inserts-enabled
  • pcs-size
  • is-hya-enabled

[-is-hya-enabled {true|false}] - Is hya caching enabled

Specifies whether the external-cache module allows caching of blocks targeted for hybrid aggregates. This option is set to true by default when the external-cache is enabled.

 

Hence my question if someone has expirience with it and has it enabled in production.

 

Cheers

Alex

Re: How to configure flash cache cards to work on a flash pool aggregate

Alex, they don't.

 

Hybrid aggregate means FlashPool. The technology has two names: Marketing name is "FlashPool" while in command line arguments uses "Hybrid Aggregate" name.

Re: How to configure flash cache cards to work on a flash pool aggregate

Well, I have an internal NetApp document infront of me called "How to configure flash cache cardsto work on a Flash pool aggregate.pdf" which states:

 

Starting with Ontap 9.1 onwards, Flash Cache could be configured to be used on a Flash pool aggregate together; this is controlled by a node shell option flexscale.flash_pool_caching, which is by default set to off.

 

To enable this:

 

clusthershell> node run <node_name>
nodeshell> options flexscale.flash_pool_caching on

 

To find out how much Flash Chache helps in a Flash pool aggregate:

 

ext_cache_obj:ec0:disk_reads_replaced: Estimated number of
disk reads per second replaced by cache
ext_cache_obj:ec0:hya_reads_replaced: Estimated number of HyA
reads per second replaced by cache

 

And I have several confirmations that this works from netapp engineers.

 

Cheers

Alex

Re: How to configure flash cache cards to work on a flash pool aggregate

Its not working this way.

Just try on real system and you will understand.

Re: How to configure flash cache cards to work on a flash pool aggregate

Hi @as - To help make your question clearer, I think you need to clarify - do you mean fabric pools or flash pools? I know the names are similar, but the technology is not and it seems you're using them interchangably in your question and haven't addressed this in your subsequent followups

 

Also.. internal documents are internal for good reasons. They may refer to functionality which is not available or supported publicly, and which may impact data integrity or availability. Some functionality requires undocumented boot flags to be set, or even license codes applied.

 

If you are having performance concerns, please open a ticket with our support team and they will work through it with you, including enabling other functionality if required.

Re: How to configure flash cache cards to work on a flash pool aggregate

Hi Alex,

 

flash pool aggregates (HDD+SSDs).

 

And yes, I understand they are internal documents and I was thinking for a while until I posted it - maybe I should not have done it...

Fully undertand that it might have impact and requires undocumented options etc., hence my question if someone has expirience with it and has it running in production.

 

I tried to involve NetApp support via our SAM but no luck so far, only answer I got is that this is not a support question. I tried to involve pre-sale, also no luck and no answer. That's why I asked in the end the community...

Re: How to configure flash cache cards to work on a flash pool aggregate

[ Edited ]

I've done some digging - this functionality was available undocumented in ONTAP 8.3.1 and 9.0, and became default for any new install of 9.1

 

The command is indeed "options flexscale.flash_pool_caching on"

 

So.. it's on on your system, since FAS9000's can run a min ONTAP of 9.1

 

The read and write policy for any new volumes you create will be "auto", but if you transitioned shelves from an older system, or copied with snapmirror, it might not be, so check out this KB article for how to set it to auto - https://kb.netapp.com/app/answers/answer_view/a_id/1031068/loc/en_US#__highlight

 

As the KB article says - we do not suggest you change it from auto except under specific advice from performance specialists.

 

Hope this helps!

Re: How to configure flash cache cards to work on a flash pool aggregate

[ Edited ]

Alex, you just mixing things up.

That KB explains how to use caching policies in a FlashPool aggregate and not related to FlashCache in any way. Though FlashCache & FlashPool have very similar names, algorithms, and some similar read caching policies, they are different not compatible things. FlashCache, FlashPool & FabricPools are technologies which currently mutually exclude each other.

 

Maybe, just maybe at some point in the future, when NetApp will have Storage Class memories (SCM) in AFF systems, then will be a new generation of FlashCache cards based on SCM memory which will work with FabricPool or All Flash aggregates. It's not available yet though.

 

 

TR-4070, page 17:

"Data from volumes that are provisioned in a FlashPool aggregate or an all-SSD aggregate is automatically excluded from being inserted into FlashCache cache. Data from volumes provisioned in a FlashPool aggregate use the FlashPool cache, and data from volumes provisioned on an SSD aggregate does not benefit from using FlashCache."

 

Though NVMe interface is more powerful than SAS, FlashCache has already been PCIe cards, so there should be no or small difference in performance between FlashCache based on PCIe vs. FlashCache based on NVMe interface. While media itself for FlashCache for old & new cards was and are nearly the same NAND technology, therefore no significant performance difference and consequently no significant caching difference there too. Only SCM have big performance difference which can give a boost for caching for all-SSD aggregates.

 

That is precisely the reason why All Flash systems do not have FlashCache cards on-board.