Constraints based tool for volume migration suggestions to optimize IO & space

[fletch2007]

Hi,

Does there exist a tool to make volume/vFiler migration suggestions based on optimizing overall performance and space goals of a set of filers?

Analogous to how VMware DRS makes prioritized vMotion suggestions based certain criteria (CPU usage being primary criteria) - in the case of the Netapp tool the primary resources would be IOPs (observed average and peak) and storage space (observed current + growth rate).

Inputs (all known by DFM):

1) aggregates: sizes, IO characteristics (how many IOPs can this aggr do at 5,10,20ms latency?) (based on type, number of disks)

2) volumes: sizes, IO characteristics (avg and avg peak IOPS)

3) administrative goals (similar to vmware DRS rules) where the admin can configure rules to force a vFiler/volume stick on a certain aggr, or vFilers/volumes not be located on the same aggr or cluster (eg for fault tolerance)

Output:

The system would use a contraints based system to perform combinatorial optimization on the inputs to  recommend migrations based on optimizing the volume/vFiler set of aggregates calculated ability to provide IOPs at 5,10,20ms etc

Assumptions:

- temp space (to rearrage volumes/vFilers on a fully allocated or  nearly fully allocated set of aggrs a certain amount of temporary space  will be needed)

- all IOs are equal (the VMware concept of prioritizing via shares is not addressed directly) - instead the admin can dictate this volume/vFiler sticks here (eg

This tool could work in different use cases:

1) fully allocated systems (suggest migrations to increase IO efficiency)

2) upgrades (how best to re-arrange volumes on new filer aggrs

3) presales sizing (inputs would be estimates of dataset IO and sizes)

is there a tool like this in the works?

thanks!

Fletcher.

http://vmadmin.info

[adaikkap]

As of today, there is no tool to do an automated way of advise on doing migration.But as you said all this is available in dfm and the dfm SDK are available,

using which customers can build their own migration suggestor as per there needs.

Also the dfm db access is given via sql view which can again be used in making this decision.

Even the Performance Advisor data can be exported, to do IO profiling.

Have you taken a look at the Performance Advisor view in the NMC that has in-depth performance information about the controller and its objects.

Some ppl create custom view like the ones below help them in making the decision.

The outputs below are of the cli dfm perf view  describe of the custom views created.

The Custom_system_summary_view is very similar tothe default system summary view, except that it has some extra countersincluded to show  read/write information for network, ops, and latency

View Name: Custom System Summary

View

Applies To: Object type (filer)

Chart Details:

Chart Name: Network Throughput

Chart Type: simple chart

Counters in this Chart:

Counter: system:net_data_recv

Counter: system:net_data_sent

ChartName: Average Latency per Protocol

Chart Type: simple chart

Counters in this Chart:

Counter: nfsv3:nfsv3_read_latency

Counter: nfsv3:nfsv3_write_latency

Counter: cifs:cifs_latency

Counter: nfsv3:nfsv3_avg_op_latency

Chart Name: All Protocol Ops

Chart Type: simple chart

Counters in this Chart:

Counter: system:nfs_ops

Counter: system:cifs_ops

Counter: nfsv3:nfsv3_write_ops

Counter: nfsv3:nfsv3_read_ops

Chart Name: CPU Utilization

Chart Type: simple chart

Counters in this Chart:

Counter: system:cpu_busy

The all_volumes_summary_view is a bar chart thatsummarizes throughput, ops, and latency for all volumes on a physical storagesystem.  Set the perfMaxObjectInstancesInBarChart option to 500 so that we make sure to get all volumes included.  These bar charts can be converted  to line graphs so that they can historically see which volumes on a given physical storage system are driving the most I/O over time.

View Name: All Volumes Summary

View

Applies To: Object type (filer)

Chart Details:

Chart Name: IOPs

Chart Type: bar

Number of object instances: All

Top or Bottom Instances: Top

Counters in this Chart:

Counter: volume:total_ops

Chart Name: Throughput

Chart Type: bar

Number of object instances: All

Top or Bottom Instances: Top

Counters in this Chart:

Counter: volume:throughput

Chart Name: Latency

Chart Type: bar

Number of object instances: All

Top or Bottom Instances: Top

Counters in this Chart:

Counter: volume:avg_latency

.The all_aggregates_summary_view is similar to volume summary, but at an aggregate level. The idea here is to compare which aggregates are the busiest, both in terms of total transfers and disk busy percentage.  This is to help identify if disk utilization is a potential bottleneck on the system.

View Name: All Aggregates Summary View

Applies To: Object type (filer)

Chart Details:

Chart Name: Transfers

Chart Type: bar

Number of object instances: All

Top or Bottom Instances: Top

Counters in this Chart:

Counters: aggregate:total_transfers

Chart Name: Avg Disk Busy

Chart Type: bar

Number of object instances: All

Top or Bottom Instances: Top

Counters in this Chart:

Counter: aggregate:pa_avg_disk_busy

Hope this help, nevertheless its a tool to build right away.

Regards

adai