Subscribe

A function to generate a deduplication schedule

I have a number of customers who want to run staggered deduplication scedules through the night, or spread over the week, or both. To address this requirement in a workflow, I wrote this simple function which pseudorandomly selects a day@hour string for deduplication schedules. As is, it selects any day of the week and a time between 22:00 - 5:00, but this can be easily customized to different customer requirements.

The idea can of course be expanded to other "random" scheduling tasks (reallocate...).

Message was edited by: Jonathan Tourtellot - replaced DAR with cleaned-up version

Re: A function to generate a deduplication schedule

Hi Jonathan,

This is great stuff.  Thanks for sharing!

-Kevin

WFA Team.

Re: A function to generate a deduplication schedule

Hi Jon,

Awesome function. Very clear and simple to use. Thank you.

One comment though on sharing functions - Due to the complexity of checking function-related dependencies (Inner and otherwise)

WFA will export all non-certified functions in the same DAR file.

It may be advised to deploy the function code on a fresh installation to make sure that the function is exported by itself and will not introduce

a plethora of unrelated functions when imported by someone else.

I took the liberty of sending you a cleaned up DAR of your function that includes only it so you could update your post :-)

In the meantime - Here's my take on this: Based on the digits of the volume name (Expecting 3 digits for sequencing) I am selecting

the day of the week and using 1am as a pre-selected time to start.

Hope that would also be helpful. Obviously one can import this and make any modifications as he sees fit.

Best,

Yaron Haimsohn

WFA Team 

Re: A function to generate a deduplication schedule

Could someone just post the ASCII text of this function if they have it? This DAR won't import into WFA-2.2. I can peek inside DAR (.zip) and see the code but it's distorted by being in XML form that way. It needs some manual editing.

Re: A function to generate a deduplication schedule

You don't need any manual editing. A quick way is the following.

1. Extract the dar file using 7-Zip file manager or some other SW. You'll be able to see an XML file and its contents. You said you already have reached till here. So its half done.

2. Open a powershell console and do use this:

[xml]$xmldoc='<_Entire_contents_of_the_XML_file_>'

3. Now : $xmldoc.functionDefinition.mvelDefinition to see the function definition.

4. $xmldoc.functionDefinition.description  to see the function description.

sinhaa

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

Re: A function to generate a deduplication schedule

Thanks for that tip Sinhaa. I used that to re-create WFA 2.2 versions of both these original functions by mont and Yaron. I can't see how to attach right now so here is the source to both:

==========

def dedupeDayAndHour(none)

{

        quasirandom = System.currentTimeMillis()%1000;

        hour = quasirandom%8;

        if (hour == 6){

                hour= 22;

        }

        if (hour == 7){

                hour = 23;

        }

        day = (quasirandom%100)%7;

        if (day==0){

                return "mon@" + hour

        }

        if (day==1){

                return "tue@" + hour

        }

        if (day==2){

                return "wed@" + hour

        }

        if (day==3){

                return "thu@" + hour

        }

        if (day==4){

                return "fri@" + hour

        }

        if (day==5){

                return "sat@" + hour

        }

        if (day==6){

                return "sun@" + hour

        }

}

=================

def setDedupeSchedule(name)

{

    String num_str = name.substring(name.length()-3);

    int vol_num = Integer.parseInt(num_str);

    int day = vol_num % 7;

    if (day == 1)

        return "sun@1";

    if (day == 2)

        return "mon@1";

    if (day == 3)

        return "tue@1";

    if (day == 4)

        return "wed@1";

    if (day == 5)

        return "thu@1";

    if (day == 6)

        return "fri@1";

    else

        return"sat@1";

}

=================

Re: A function to generate a deduplication schedule

Hello all,

I just came across this post, and figured I will throw my version into the mix. My version is actually a command and not a function. It's based on Cron jobs already being setup with a naming convention of "maint_##_r" for replication, and "maint_##_d" for deduplication.  The "_d" will run an hour before the "_r"(see below).  It will choose the appropriate "_r" job based on the least number of snapmirror relationships using those cron schedules.   You can easily take the script and turn it into one that looks at the capacity of what is replicated as well, but I chose not to look at that as it isn't a true measurement of the change rate.

Once it chooses the replication cron job, it will also choose the associated deduplication job, and add those variables into the workflow parameters for later use down the line of the workflow. 

It also checks to see if the efficiency schedule is already setup on the source side, and creates the effeciency policy if it doesn't exist.

----- Example Cron Jobs -----

Name                Description

----------------    -----------------------------------------------------

maint_01_d          @21:00

maint_01_r          @22:00

maint_02_d          @22:00

maint_02_r          @23:00

maint_03_d          @23:00

maint_03_r          @0:00

maint_04_d          @0:00

maint_04_r          @1:00

maint_05_d          @1:00

maint_05_r          @2:00

maint_06_d          @2:00

maint_06_r          @3:00

maint_07_d          @3:00

maint_07_r          @4:00

maint_08_d          @4:00

maint_08_r          @5:00

-------------------------- Command -----------------------------

param (

  [parameter(Mandatory=$true, HelpMessage="Name or IP address of a cluster where the schedule exists.")]

  [string]$Cluster,

  [parameter(Mandatory=$false, HelpMessage="Name of the SVM where the volume is being created.")]

  [string]$vServer

)

Get-WFALogger -Info -message $("Connecting to the cluster: " +$Cluster)

#Connect to the Cluster

Connect-WfaCluster $Cluster

#Get Current Snapmirror Schedules

$mirrorschedules = Get-NcJobCronSchedule | Where-Object {$_.JobScheduleName -like 'maint_??_r'}

#Get Listing of Current SnapMirrors

$mirrors = Get-NcSnapmirror

#Sort Snapmirror Schedules and find the schedule with the fewest number of snapmirrors

$mirrorscheduleinfo = @()

foreach ($mirrorschedule in $mirrorschedules)

{

    $scheduleinfo = @{

    schedule = $mirrorschedule.JobScheduleName

    mirrorCount = $($mirrors | Where-object {$_.Schedule -eq $mirrorschedule.JobScheduleName} | measure).Count}

    $newObject = New-Object PSObject -Property $scheduleinfo

    $mirrorscheduleinfo += $newObject

}

$selectedmirror = $($mirrorscheduleinfo | Sort-Object MirrorCount | Select-Object -Index 0).Schedule

Get-WFALogger -Info -message $("The selected mirror schedule is: " +$selectedmirror)

$dedupeschedule = $selectedmirror -replace "_r", "_d"

Get-WFALogger -Info -message $("The selected dedupe schedule is: " +$dedupeschedule)

##Check to see if Effeciency Policy Exists and if it doesn't create it.

if (!$(Get-NcSisPolicy -vserver $vServer -Name $dedupeschedule))

{

       Get-WFALogger -Info -message $("No Effeciency Policy Exists. Creating one now." )

       New-NcSisPolicy -name $dedupeschedule -Schedule $dedupeschedule -VserverContext $vServer

       Get-WFALogger -Info -message $("Created Effeciency Policy: " +$dedupeschedule + " on: " +$vServer)

}

Add-WfaWorkflowParameter -Name SelectedMirror -Value $selectedmirror -AddAsReturnParameter $true

Add-WfaWorkflowParameter -Name DedupeSchedule -Value $dedupeschedule -AddAsReturnParameter $true