Repeating rows using MVEL arrays - Workflow Example

[abrian]

This example uses MVEL array types to avoid doing a splitByDelimiter inside the iteration, which is pretty costly with the regex compile and match going on each time, and multiple times for nested splitByDelimiter calls.  Rather than return a single value, I created a new function, split, which returns an array.  You can then index the array in each iteration.

The user enters the controller, vFiler, optional volume (one is created if an existing one isn't chosen), the number of qtrees to create, and a CSV list of qtree sizes.  If the number of sizes provided is less than the number of qtrees requested, the last size value is used for all remaining qtrees.  Thus, to create 5 qtrees if the same size, you only need enter one size value.

For existing volumes, the find chart verifies that the containing aggregate contains enough space to accommodate the qtrees without exceeding a maximum utilization percentage (a unique requirement of this particular customer).  If no volume is specified, the most utilized aggregate that can accommodate the request, again without exceeding the maximum utilization, is chosen.   Volume and qtree names follow a simple naming standard.  Finally, this customer wanted to specify the NFS export security for the qtrees as a copy/pasted string containing all the options, which is why the export creation and security setting commands are separated.

One problem I found with the existing Create QTree command is that it attempts to initialize quotas each time the QTree is created.  First, there is a potential problem with using the quota resize command, rather than reinitializing the quotas, because resize won't pick up the quotas correctly of no existing quota already exists (which would happen if the volume is newly created).  Second, once the quota resize (or reinitialization) starts, it runs for some time, causing subsequent Create QTree attempts to fail.  So, I moved the quota initialization code out of Create QTree (my Create Qtree 2, clever naming, huh?), and created a new command to reinitialize the quotas (turn them off then on again), which is called at the very end of the flow.  This also speeds up qtree creation considerably, since there are several fewer PS toolkit calls and no wait loop.

I'm still testing this, so if you see anything fishy, let me know and I'll either explain it or fix it

[yaronh]

Hi Brian,

Really nice stuff there!

I get some of your points in regards to the quota management, though I'd like to review the use case further before sharing

my exact thoughts. When we created the qtree create and Quota resize commands we tried to take those cases into account.

and especially maintaining control inside the command while this is executed so following commands will not get stuck.

With that said - I'd like to comment on one point. You mentioned you used CSV and MVEL array to avoid costly regex manipulation,

and that's a point i'd like to counter.

WFA's flow dictates that a planning/preview phase is happening prior to any execution.

It is in that phase that WFA resolves all resource selection activities, handles user inputs and creates the execution plan.

When we get to the execution phase, WFA would have a set and clear execution plan.

The planning phase is short from user perspective. It may take 2,5 or even 15 seconds. The order of magnitude here

is quite negligible. My point is - You can do 100 costly regex operations and the result would be that the planning would

grow from 10 to 15 seconds. I agree that we should aim to save time while the commands work on the arrays as that can

be costly (Including networking costs), but doing a more complex function (Even one which is very inefficient) is simply

negligible.

As a techie to techie I agree that we like to keep our "code" tight and clean and rocket-science efficient.

In regards to planning - Sometime simpler is just better.

Just my 2 cents...

Yaron Haimsohn

WFA team

[abrian]

Hi Yaron,

Yes, I definitely see that, in the overall scheme of things, these performance issues are not significant, especially when compared to actual flow execution, where connection establishment, PS toolkit command execution, etc. are taking place.   I actually found it more intuitive and simpler to do the array manipulation rather than the iterative splitByDelimiter , but that's just my perspective.

My experience with Create QTree was that it was indeed getting errors back from ONTAP when trying to create the 2nd qtree.  By putting the reinitialization at the end, I didn't encounter that error.  I'm by no means a quota expert, so I may have misinterpreted the cause.  In any case, reinitialization is likely to take quite a bit longer than resizing, and is required in the case where no quota (default or otherwise) already exists (at least according to the 7.3 Storage Management Guide I was referencing).   Waiting for reinitialization (as opposed to resizing) to complete may be undesirable in cases where more than a couple of QTrees are being created in the same volume.

I am really enjoying learning and working with WFA.  I think it is a big win for customers, and it'll be fun creating workflows for them when we are called upon.  I really like the way it's put together, and how incredibly flexible it is!   Excellent work by everyone involved!

Brian Atkins

NetApp RRE