Assigning round robin dedupe schedule to new volumes in WFA

[Jim_Robertson]

I am trying to setup a volume creation workflow for a customer.  They would like to have background deduplication enabled by default, but only want it to run once/week.  I am trying to spread out the schedule so that they are starting after hours and not all kicking off at the same time, so I've created policies with matching schedules similar to the following:

Dedupe_Mon_2135

Dedupe_Mon_2235

Dedupe_Mon_2335

Dedupe_Tue_2135

Dedupe_Tue_2235

Dedupe_Tue_2335

etc...

Does anyone know a way that I could setup a round robin type of policy selection so that each time a volume is created, it assigns a different policy?  The multiple volumes would not be created at the same time.  They may create one volume and then days or weeks later, run the workflow again and create another volume.  Ideally, it would select from the existing list of policies and then start over from the beginning of the list once a volume that uses each policy is created.

Thanks for any help anyone can provide!

[NICKBARTON]

@Jim_Robertson

The best way I can think of to do this would be to create your own finder to return the efficiency policy with the least number of volumes assigned to it you can then create a filter to use that finder and include it in the efficiency field of your workflow. 

Now getting to this count is kind of a hoops game, you need to work backwards from your aggregate the new volume was created on to determine your node (you need this to drill down into dedupe count per node because that is where the overhead would be. 

I'm assuming this is a cluster mode based question. 

Ultimately the answer is you don't need to do this, standard policy now is that all volumes should have an automatic dedupe schedule as the overhead for this kind of work is not what it used to be back in the 7Mode days when this feature was first introduced. That being said if this is still a requirement based on bad past experiences the above route is the best way I can think of to accomplish what you want to do. Are you familiar with how to put the SQL query together to get the policy counts based on volumes on the current affected nodes? 

-Nick 

[Jim_Robertson]

@NICKBARTON,

Thanks for the reply.  I understand what you're saying about creating the finder and filter.  I think I could possibly make that work.  I'll give it a try and let you know if I have any issues.

I am very curious about your other statement though.  You said that, "standard policy now is that all volumes should have an automatic dedupe schedule as the overhead for this kind of work is not what it used to be back in the 7Mode days when this feature was first introduced".  I just found this in the OnTap 9 documentation.  Is this what you are referring to?  I am definitely going to have to look into this.  I'm a little nervous about these kicking off during production hours, but I guess if I set the policy to make them a "background" process, it should have minimal impact, correct?

From docs at:  http://docs.netapp.com/ontap-9

You can modify the efficiency operation schedule to run deduplication or data compression when the number of new blocks written to the volume after the previous efficiency operation (performed manually or scheduled) exceeds a specified threshold percentage.

About this task

If the schedule option is set to auto, the scheduled efficiency operation runs when the amount of new data exceeds the specified percentage. The default threshold value is 20 percent. This threshold value is the percentage of the total number of blocks already processed by the efficiency operation.

Step

1. Use the volume efficiency modify command with the auto@num option to modify the threshold percentage value.

   num is a two-digit number to specify the percentage.

Example

The following command modifies the threshold percentage value to 30 percent for the volume VolA:volume efficiency modify -vserver vs1 -volume -VolA -schedule auto@30

[NICKBARTON · Accepted solution]

@Jim_Robertson

Correct. When dedupe and compression were both first introduced they were pretty intensive processes that weren't multi-threaded very well making most people want to stager their schedule and only run it after hours. That is no longer the case it is actually now considered best practice to enable deduplication and compression inline and set to auto, you can change the percentage if you want but 20% change seems to be the sweet spot. Since OnTap version 8.1.2+ more and more work has been done to multithread and background a lot of the system level process including, snapshot management, deduplication, compression, etc. By the time you hit 8.3.2+ these are really just after thoughts, just have it all enabled as your default template for volumes and forget all about it. 

-Nick 

[Jim_Robertson]

Thanks, @NICKBARTON! 

I am going to work with the customer to implement this.  They seemed receptive to rolling it out to their RND and QA environments, so we'll test it there first and see how it goes.  Definitely seems much easier than trying to schedule throughout the week.

As far as the inline dedupe/compression though, I thought this was really only best practice for All Flash arrays, or has that changed in OnTap 9.x?  This customer is still using FAS8040/8060s with Flash Pools with a mixture of SAS and BSAS disks.  Is it wise to use inline processes in a configuration like that?

[Mike_B]

I have most of these filters/finders built if you want something to reference.

[NICKBARTON]

@Jim_Robertson,

So its dependent on workload as inline dedupe/compression only run in memory and not on disk so it can create some slight overhead on write I/O. It's just enabled by default on the AFF but you can use it elsewhere depending on workload requirements. 

-Nick 

[Jim_Robertson]

---

@Mike_B wrote:

I have most of these filters/finders built if you want something to reference.

---

That could be helpful if you don't mind sharing.  Thanks!