Solved: reallocate on a metrocluster - split the mirror?!

abuchmann · ‎2010-04-14

Hi there

First of all, read the following threads:

http://communities.netapp.com/thread/7456

http://communities.netapp.com/message/20969#20969

Both threads we're informative, but the solution to decrease the latency does not work on our systems.

We've got two FAS6030 in a metrocluster configuration on 7.3.2P2. As the metro is using local_syncmirror and we are using snapshots (nearly everywhere :-)) i get the following:

US-S001MUR> reallocate status /vol/esxprod2_nfs_sys
Reallocation scans are on
/vol/esxprod2_nfs_sys:
        State: Idle
     Schedule: n/a
     Interval: 1 day
Optimization: 5 [hot-spots: 26]

US-S001MUR> reallocate start -f -p /vol/esxprod2_nfs_sys
Unable to start reallocation scan on '/vol/esxprod2_nfs_sys': Aggregate is mirrored
US-S001MUR> reallocate start -f /vol/esxprod2_nfs_sys
Unable to start reallocation scan on '/vol/esxprod2_nfs_sys': target has snapshots

whats the easiest way to reallocate on our configuration?

Thank you in advance!

regards,

Adrian

abuchmann · ‎2012-11-21

Physical reallocation of mirrored aggregates is now supported. I found the following in the release notes of 8.1.1 (SE told me to have a look at it):

Reallocation scans, read reallocation, and extents with FlexVol volumes contained in mirrored aggregates

In prior releases, you could enable reallocation scans, read reallocation, and extents on FlexVol volumes contained in mirrored aggregates. However, some command options were not supported with this type of volume.

Starting in Data ONTAP 8.1.1, you can use the following command options to optimize the layout of FlexVol volumes that are contained in mirrored aggregates:

Reallocation scans	You can use the -p option when you run the reallocate start command. The -p option (also called physical reallocation) reallocates user data on the physical blocks in the aggregate while preserving the logical block locationswithin a FlexVol volume.
*Read reallocation*	You can set the read_realloc option to space_optimized when you run the vol options command. space_optimized conserves space but results in degraded Snapshot read performance.
Extents	You can set the extent option to space_optimized when you run the vol options command. space_optimized conserves space but results in degraded Snapshot read performance.

View solution in original post

pascalduk · ‎2010-04-14

I have run into the same issue with a test system. Like you the only option I could think of is destroying the mirror, reallocate and then build up the mirror from scratch again.

abuchmann · ‎2010-04-14

Oh, that sounds really painfull.

does anyone know the reason for this limitation?

What do you recommend to do? Is it possible to split an aggr, reallocate about 150 volumes and resync the aggregate within 4 days?

As I have no metrocluster to test with, i would be thankful for every advice 🙂

kind regards,

adrian

pascalduk · ‎2010-04-15

My guess is it has something to do with the syncmirror snapshot on aggregate level. Reallocating with the -p option on volume level will still cause a large aggregate snapshot.

No idea how long reallocating will take. Perhaps netapp support can give you an estimate.

amiller_1 · ‎2010-04-18

Any further details on this from NetApp by any chance? Especially when running VMware environments with dedupe, I find reallocate to be critical....so losing it when running MetroCluster (as I have one customer doing) is a serious issue IMHO.

pascalduk · ‎2010-04-18

Andrew Miller wrote:

Any further details on this from NetApp by any chance?

Nothing, it is still on my very long to do list

amiller_1 · ‎2010-04-20

He he....I understand what long to do lists are like. I'd be curious to hear whatever you find out...

abuchmann · ‎2010-04-21

our key account manager advised me to open a case because of that.

The support told me, that there's a request for enhancment, but - as usual - with no eta.

So I have to cross my fingers that I don't get caught by murphy's law while the plexes are splitted.

madden · ‎2010-04-22

Hi,

Two ideas:

(1) The command reallocate start -f /vol/esxprod2_nfs_sys will indeed give an error if you have snapshots (because of the high rate of changed blocks that might result in filling up your vol) but you can run it per LUN/file like reallocate start -f /vol/esxprod2_nfs_sys/data.vmdk. Maybe give it a try on some VMs that you think are most impacted.

(2) If you have a metrocluster you can enable reads to hit both plexes as by default its set to local plex only. If you have a long distance between sites, or know you'll saturate your ISLs in a fabric MC then no solution but for many situations it allows you to double your read IOPs at no cost. The behavour is configured by a system-wide option, can be done online, and takes effect immediately:

filer> options raid.mirror_read_plex_pref alternate

From the manpage:

raid.mirror_read_plex_pref
Specifies the plex preference when reading from a mirrored traditional volume or aggregate on a metrocluster-configured system. There are three possible values -- `local' indicates that all reads are handled by the local plex (plex consisting of disks from Pool0), `remote' indicates that all reads are handled by the remote plex (plex consisting of disks from Pool1), and `alternate' indicates that the handling of read requests is shared between the two plexes. This option is ignored if the system is not in a metrocluster configuration, i.e., cluster_remote is not licensed. The option setting applies to all traditional volumes and aggregates on the filer.

Cheers,
Chris

amiller_1 · ‎2010-04-23

Thanks for the info -- very helpful. Is it safe to assume that "reallocate measure" works the same in MC as otherwise? (so could diagnose which LUNs would be helped by reallocate)

pascalduk · ‎2010-04-25

Yes, the measure works like normal. Only the reallocate -p does not work in a metrocluster.

joostvandrenth · ‎2010-11-21

We have the same issue with a customer over at the IBM side of things, will try to get a case opened as well - let's see if we can get this fixed.

In the mean time we use the -f option, which pretty much delivers the same result right? I mean if it will rewrite current data across all new and existing RGs the layout will be fixed as well.

abuchmann · ‎2012-11-21

Physical reallocation of mirrored aggregates is now supported. I found the following in the release notes of 8.1.1 (SE told me to have a look at it):

Reallocation scans, read reallocation, and extents with FlexVol volumes contained in mirrored aggregates

In prior releases, you could enable reallocation scans, read reallocation, and extents on FlexVol volumes contained in mirrored aggregates. However, some command options were not supported with this type of volume.

Starting in Data ONTAP 8.1.1, you can use the following command options to optimize the layout of FlexVol volumes that are contained in mirrored aggregates:

Reallocation scans	You can use the -p option when you run the reallocate start command. The -p option (also called physical reallocation) reallocates user data on the physical blocks in the aggregate while preserving the logical block locationswithin a FlexVol volume.
*Read reallocation*	You can set the read_realloc option to space_optimized when you run the vol options command. space_optimized conserves space but results in degraded Snapshot read performance.
Extents	You can set the extent option to space_optimized when you run the vol options command. space_optimized conserves space but results in degraded Snapshot read performance.

BENJAMIN_BOHEC · ‎2014-07-29

Hello,

Thanks for the info that physical volume reallocate is available on syncmirror.

Anyone know if aggr option free_space_realloc can be used on syncmirror aggregates?

I read on Reallocate Best Practices that the free_space_realloc is an inline physical reallocate task but I didn't tried it.

Regards,

Benjamin.