V-Series and FAST Cache

[tom_maddox]

Hopefully I don't start a religious war here, but I have a question about interoperability between the V-Series filers (a V3160 in our case) and FAST Cache on the EMC Clariion CX4 series, specifically whether it's a good idea to enable FAST Cache on LUNs which are presented to the filers. More specifically, we have two classes data by and large, unstructured data (files) and virtual machines. Would enabling FAST Cache for either class of data potentially improve performance?

Thanks,

Tom

[isaacs]

Hi Tom,

The short answer is that we don't yet fully understand the performance implications of a FAST-Cache enabled storage pool.

The long answer is that our testing to this point has been primarily to ensure interoperability with that feature.  My job this summer is to gauge the performance benefits and develop a set of recommendations for optimal performance.  We should have a Tech Report (TR) early this Fall/late Summer.  I'll be sure to let you know as soon as we do.

Dan Isaacs

TME - V-Series
isaacs@netapp.com

[tom_maddox]

Thanks, Dan, it's good to know you guys are working on this. My understanding of how FAST Cache works (and this is as a non-specialist) is that the SAN analyzes usage of individual blocks in a FAST Cache-enabled LUN and promotes frequently-used blocks to the flash drive cache. EMC also mentions that FAST Cache works best with i/o that is non-sequential but has medium to high locality. Given what you know about WAFL, would you guess that enabling FAST Cache could be worthwhile, or would it be a waste of resources? I ask because we have immediate performance problems, and waiting 3-6 months for a recommendation is not ideal.

Thanks again,

Tom

[isaacs]

My hypothesis is that it would not help that much with WAFL.  Or at least, not as much as having a FlashCache module in the V-Series would.  But I really want to test it to be sure.

Has a NetApp support case been opened for your performance issue?  Has a bottleneck been identified?

[tom_maddox]

The bottleneck is the back end disks, which is leading to high DRAM cache utilization and degraded performance during usage spikes. We can throw more disks at the problem, but we would prefer to find a less wasteful solution.

[isaacs]

Is there a FlashCache card already installed in the V-Series?  Are these primarily reads that are a problem?

[tom_maddox]

We do have FlashCache installed, which has somewhat remedied the performance issues, but we still see write cache saturation on the filers from time to time, which significantly impacts latency.

[radek_kubka]

My 2 cents:

EMC FAST Cache is about random read & *write* caching, whilst NetApp Flash Cache works for reads only.

NetApp Flash Pool in ONTAP 8.1.1 will be a direct equivalent of EMC FAST Cache, but it is limited to NetApp native disks.

Regards,

Radek

[tom_maddox]

Here is my experience so far. I turned on FAST Cache for a single-plex aggregate hosting an NFS-mounted VMware datastore, and the write hit ratio is hanging out at about .125, and the read hit ratio is hanging out around .65 (both estimates are very rough based on a quick eyeball of the performance chart), with variances down to .5 and up to .89 for reads and .05 and .36 for writes. What's interesting (to me, anyway), is that the ratios are almost precisely reversed for the SP Cache: read hits are down in the same area as FAST Cache write hits, while write hits are actually even higher than FAST Cache read hits. Looking at the hits per second, FAST Cache read hits are significant, at >40, while everything else pales in comparison.

Overall, though, it seems as though roughly 80% of reads are coming out of either SP Cache or FAST Cache (mostly the latter), while almost all of the writes are hitting cache at some point. I know that conventional wisdom is that WAFL doesn't really benefit from write caching (or so I have read), but given that the LUN service time is relatively miniscule (topping out at 4 ms and generally staying between 1-2 ms), while the response time is >10x that amount generally, most of the data must be coming from cache, so FAST caching does seem to be boosting read performance. That's my conclusion, anyway, speaking as very much a novice when it comes to storage performance tuning.

Anyone have any feedback, critique, etc.?

[isaacs]

Interesting!

With any perfomance test, it's important to understand the variables in play.

1.  How big was the dataset?

2.  What tools were used to drive the load?

3.  What options were set on the load generator?  (block size, # of threads)

We would also be interested in seeing Perfstat output comparing the same tests run with FAST Cache enabled vs. disabled.  This at least lets us look at how WAFL is benefiting from the faster storage.

Let us know if you need help getting Perfstat running, or need more information abnout it.

Thanks Tom!