New DS2246 SAS/SSD mix - How best to add Flash pool to existing controllers/aggregates?

[911Eric]

Hi all,

I recently added a DS2246 to our existing FAS3220 HA pair running 7-mode.  The new shelf contains 20 SAS and 4 SSD disks with the intent of adding some needed IOPS to our existing infrastructure which was originally built solely wth SATA (2 DS4246 full of SATA drives).

My question is how best to "engineer" the flash pool(s) so that the performance benefit is seen across the aggregates and volumes that already exist.

Our setup is rather simple at this point, we have an aggr0 on each controller with VMFS volumes contained in each.  My thought is to do something such as adding 2 SSD and 3 SAS to each of the aggregates and then putting them in a flash pool so that the benefits of our new SSD and SAS drives are equally spread across the entire system.

Does this make sense?  Is there a better way to do this?

Thanks!

[GidonMarcus]

Hi

adding (forcing actually) the SAS disks to SATA AGGR will not improve perf. as the AGGR will work anyway up to the speed of the slowest disk.

The flash pool is only a caching technology and SSD's are not actually participate in the AGGR. you maybe want to read this:

https://kb.netapp.com/support/index?page=content&id=3014300&locale=en_US&access=s.

if you already got the SAS disks now. i will suggest to create a new AGGR for these. use the existing SATA-flashpool AGGR for data that has good DEUP ratio and need fast read Perf. and move data that doesn’t or one that require fast write to the new SAS disks AGGR (for example the root vol, DB transaction log, unique OS or virtual appliances)

Good luck

[911Eric]

Thanks for your reply.

I actually have looked at that very article you referenced.  I read some other information as well that mentioned as a best practice that NetApp recommends at least 5 disks in a flash pool (can be SSD and SAS according to what I've read) which is why I suggested the 2 SSD and 3 SAS per aggregate flash-pool.

So if I were to do as mentioned above and take the 2 SSD and 3 SAS, add them to the aggregate and then to the flash pool, the aggregate and it's contained volumes would not benefit from the 2 SSD and SAS drives?  I was hoping to be able to split the 4 SSD in the new shelf between the 2 root aggregates we have on each of the controllers in order to gain some needed IOPS for the FC Lun's contained therein.

I should reitirate that we have two existing aggr0 between the 2 controllers and I was hoping to be able to use these new SSD and SAS across both aggr for an even gain in IOPS.  From what I've read it would not be recommended to add only 2 SSD in a flash pool for an aggregate, which is  why I suggested the additional SAS drives in the flash pool.

[GidonMarcus]

You have a reference for the SAS can participate in the flash pool?

the main doc about it http://www.netapp.com/us/media/tr-4070.pdf not mentions it

[911Eric]

https://community.netapp.com/fukiw75442/attachments/fukiw75442/data-ontap-discussions/17208/1/43347_TR-4070_NetApp_Flash_Pool_Design_and_Implementatio...

First chapter, second paragraph.

"A Flash Pool is built from a Data ONTAP aggregate in a two-step process, described in section 3,1, "7- Mode CLI." Essentially it is the addition of SSDs into an aggregate to provide a high-bandwidth, lowlatency location that is capable of caching random reads and random overwrites. The feature does not require a license and works with any NetApp SSDs and one type of HDD per Flash Pool. That is, SSD and SAS performance drives can be combined to make a Flash Pool, or SSD and SATA capacity drives can be combined to make a Flash Pool. You cannot combine SSD, SAS, and SATA into a single Flash Pool"

[bobshouseofcards]

Well - yes, that's the idea on the combination on type of drive in practice.  That quoted text doesn't say anything about count of disks, and that's where you appear to be combining two true statements incorrectly.  The minimum recommendation of 5 disks in an aggregate is not combined SAS + SSD total, but 5 of the base type alone.  A five disk aggregate gets you 2 parity plus three data disks.  

Consider - with RAID-DP - 3 SAS disks has two parity disks and 1 data disk in a raid group.  So your aggregate size is limited to the useable size of the one SAS disk.  Your effective IO rate is also limited to the performance of one SAS disk.  That's why the minimum disk is 5 - to at least get you 3 data disks.  

SSD's are added to a base aggregate in raid groups.  They don't live in isolation or get just added on to the base disks in an aggregate.  You can (at least in cDot 8.2, not sure about 7-mode) create the SSD cache as RAID-4 even if the main aggregate is RAID-DP which gives you 2 cache data, 1 cache parity, and 1 SSD spare to add onto whatever aggregate you want.

With the available disks I concur with previous comments.  Create one aggregate of all the SAS (leaving at least 1 spare).  If you do that, be sure to specify the proper raid group size during the aggregate creation so that you don't end yup with a raid group of 16 disks and another one of 3 disks (19 total) for instance - that would just as quickly destroy performance.  Better to define a raid group size of 19 up front.

You're definitely not going to split the SSD's between both nodes with only 4 available, if that was your intent.

I hope this helps you.

Bob Greenwald

Lead Storage Engineer

Huron Legal | Huron Consulting Group

NCDA, NCIE - SAN Clustered, Data Protection

[911Eric]

Thanks Bob, that helps some.

I can't find the article off-hand but I read somewhere else as well that the recommended flash-pool size is also a raid group of five.  Our current raid group is RAID-DP as you mentioned, which would mean that the flash pool would also be raid-dp, so I would think that would give us 2 cache disks, 2 parity and one spare if I were to use the 2 SSD and 3 SAS.  To answer your last question, yes, my intent was to split the 4 SSD between both controllers, but, with the addition of the 3 SAS drives in the pool.  So you're saying that won't work and/or I shouldn't do it though?

That essentially leaves me having one of the two current aggregates solely running SATA drives if I can't spread the SSD/SAS drives somewhat evenly between the two aggregates in a flash pool.

So I have the option of creating a 19 disk SAS Aggregate and then adding my 4 SSD in a flash pool to either that aggregate or one of the two SATA aggregates....but no way of splitting the SSD's between the two existing aggregates in a combination SSD/SAS flash pool?

[bobshouseofcards]

Yes - you're getting closer.

To be clear, there is no such thing as a "combination SAS/SSD flash pool".  Flash Pool is SSD disks only that are added to an already established aggregate of either SATA or SAS disks.  You cannot make a flash pool of 2 SSD + 3 SAS disks to get to 5 disks for the recommended minimum aggregate size.  You have to count disk type separately to establish a raid group, as in 5 disks of each type.  The absolute minimum is 3 disks of each type in RAID-DP.  

You start with an aggregate of SATA disks or an aggregate of SAS disks, which stand alone.  To that you add a separate raid group of SSD disks which can function as a cache mechanism.  The combination of the base aggregate plus the add on SSD's makes it a "flash pool" aggregate.   The key point is that SSDs are not combined with the other disks - each type is counted and stands alone in a raid group.  The raid groups, when combined in a single aggregate, make the aggregate a flash pool.

That's the high level process.  The standard NetApp Data ONTAP documentation will have the exact steps to follow for both 7-mode and cDot.  TR-4070 describes the separate raid groups or the base disk type and the SSD in section 3.2.

Because you have only 4 SSDs, the best option to maximize the capacity of the flash pool is to use RAID-4 (single parity) on the SSDs so you can have 2 data, 1 parity and 1 spare SSD.  And yes, you are limited to one flash pool aggregate - which ever one you choose.  

I hope this helps.

Bob Greenwald

Lead Storage Engineer

Huron Legal | Huron Consulting Group

NCDA, NCIE - SAN Clustered, Data Protection

Kudos and accepted solutions are always appreciated.

[JamesIlderton]

Actually, once you create the Storage Pool with the SSDs, it will allow you to assign the pool to any aggregates in the HA pair in allocation units (25% per unit, so the pool can be split to as many as 4 different aggregates).

https://library.netapp.com/ecmdocs/ECMP12458210/html/GUID-E7778CC5-20E6-48CC-B5B4-8C548C1237FF.html

I've got several clusters using Flash Pool combined with SAS disks in this fashion.  What I would suggest would be to determine the actual performance requirements of your various data sets and use the pool's allocation units to accelerate the highest random write volume(s), combining them with either the existing SATA or the new SAS aggregate based on the capacity requirement for the same data (there are various tools from NetApp to do so, your SE can help you determine which best fits your environment).  If all of the data is VMware as you indicated, you may also be able to put a little more effort into it (or Pro Services time if necessary) to move your root volume for one of the nodes to the SAS aggregate so you can destroy a SATA aggregagate and add the disks to the other one to get the best performance and capacity from it, essentially making one of your nodes host the SATA and the other host the SAS.

[bobshouseofcards]

James - 

Did they add storage pools in 7-mode as well for 8.2?  I stand corrected then certainly.  I'm not as up on my 7-mode anymore these days.

Bob Greenwald

Lead Storage Engineer

Huron Legal | Huron Consulting Group

NCDA, NCIE - SAN Clustered, Data Protection

[JamesIlderton]

No, I mis-read the original post, I was describing how SSDs can be used to create Flash Pool aggregates in cDOT 8.3+ (using ADP)...but since he is still using 7-mode it's not as flexible so he'll need to allocate the cache to a single aggregate.  You really should look at moving to cDOT, maybe once 8.3.2 goes GA (RC1 is out now) and you can use the copy free transition with with SAS disks hosting the root vols on new/swing controllers.  Trust me, you'll never want to go back to 7-mode once you've started using all the power and flexibility of cDOT (don't get me started on tab-complete!)

[GidonMarcus]

Hi,

In page 12 of the tr I linked to earlier there's the recommended amount of ssd in flash pool.

At the same doc there also the phrase

Starting with the Data ONTAP 8.2 software release, the RAID policies for the SSD RAID group (or groups) are independent of the policies for the HDD RAID groups within a Flash Pool aggregate. For example, a SSD RAID group in a Flash Pool aggregate can be configured with RAID 4 and a group size of 8, and the HDD RAID groups in the same Flash Pool aggregate can use RAID-DP with a group size of 16.



So. Yes add your ssd to one controller, and the SAS to another. Note that the flashpool will take some CPU resources, so seat and think about the sizing you do from all the aspects.

[911Eric]

Thanks for all of the replies guys, it has been enlightening to say the least.

We do plan on going to CDoT later this year but for now I need to get this new shelf added and get the flash pools going so that I can resolve some latency issues I've been seeing.

The documentation I've been reading specifically says that you can combine SSD and ONE other disk drive be it SAS or SATA but not all 3, in a flash pool, but I guess at this point that won't really help me much.

So, then, to clarify (sorry to keep asking) my best option at this point is going to be to create a new aggregate with the 20 SAS disks and then add the 4 SSD as the flash pool (obviously minus disks for spare and parity)?  Or would it still make sense to just add all 4 of the flash disks to one of the existing SATA aggregates so that I could get the performance benefit on one of the larger existing aggregates?

Thanks again for all of the responses, you have all been helpful to me, a somewhat amateur storage admin 😉

[bobshouseofcards]

Hi Eric -

Question on best benefit depends on a number of factors.  Without having specific details of your situation, the "generic" advice is to use AWA (Automated Workload Analyzer) built into Data ONTAP.  Assuming of course that your 7-mode system is at a level that includes it.  Even without the cache installed and configured, Data ONTAP can collect the statistics on whether it would cache certain data and whether it might be available to re-read at the appropriate time.  This doesn't add any real workload as AWA just pretends you have the cache, then let's DoT do its thing internally.  After you run it for an hour or three during times of decent workload, AWA will pull the statistics and tell you how many cacheable IOs are present.  It will then recommend cache sizes to meet various cache hit percentage levels.  That way if you choose to add SSD flash pool to an existing SATA aggregate, you can figure out which one might get the best benefit given your current configuration.  I'll meantion that in cDot 8.3, AWA can be run on each volume separately, so that even if an aggregate has a flash pool, you can adjust which volumes on the aggregate use it.

Most likely, the best benefit will be to reorganize your volumes into a new aggregate structure, moving some to a new SAS aggregate based on the IO profiles and needs of each volume, then using AWA to determine where the SSD would be most effective.  Down side of course is that, in 7-mode, this isn't easy and is certainly disruptive.  I'm guessing you don't have the luxury to do this now, so the AWA plan if available to you is the best option to determine how effective SSD might be and for which SATA aggregate.

I hope this helps you.

Bob Greenwald

Lead Storage Engineer

Huron Legal | Huron Consulting Group

NCDA, NCIE - SAN Clustered, Data Protection

Kudos and accepted solutions are always appreciated.