Moving a Snapmirror destination volume to another aggregate

[KDEAN1961]

We are running Ontap 8.1 and need to move a Snapmirror destination volume (13TB) to a new aggregate without re-initializing.

Is the best solution to snapmirror the destination volume to a new volume/break SM between old and new destination/Resync from source to new destination?

I do notice when I do this the orginal SM job goes into a "pending" state because the destination is now busy Snap Mirroring to the new destination.  Being that the volume is 13TB the new SM job (old destination to new destination but in same DataCenter) will take about 3 days to complete  and the existing SM source -> desination  will stay in a pending state

Just checking if there are any other options.

Thanks

Kathy

[radek_kubka]

Hi Kathy,

I'd say it is a valid approach.

I think you can quiesce / resume the transfer from old to new destination, so source gets a chance to send incremental updates before the whole migration is completed.

Regards,
Radek

[bsti]

Does that work?  I've actually tried that without success.  What I see happen is when I pause the intialize from B - > C, then trigger an update from A -> B, then B is flagged as changed and the initialize starts all over again next time I trigger it.  I'd really really love to be wrong about this, so if you can fill in any detail on that I'd be ingratiated to you. 

[radek_kubka]

Nasty - I thought that as long as B->C uses common snapshot, new A->B update shouldn't cause full reinitialisation?

[bsti]

Is the aggregate you are relocating to on the same controller?  If so, there vol copy or vol move may work out better for you.

If you are moving to another controller, then I'm pretty sure SM is the best option.  I think you may be able to speed up your initialize time though by looking at a few things:

1) Consider snapmirror multipathing.  We've seen a 30% or more increase in throughput by doing this.

2)  Make sure as little as possible is occuring on the controllers.  SM performance is very much so affected by business on the filer.  IN my case, I had to reduce the frequency of SM Updates to those controllers as I moved the volume.  Frequent updates caused several volume deswizzling scans to run almost nonstop, driving my CPU % busy to about 60% at all times.  When I reduced this, my throughput increased by more than 6x.

Hope that helps!

[KDEAN1961]

Unfortunatley it is on a different controller.

Can you provide more info on snapmirror multipathing?

Thanks for your reply!

[bsti]

https://library.netapp.com/ecm/ecm_get_file/ECMP1113957

Starting on page 137.

One thing to note on this setup:

You need two different network interfaces on each controller, each in its own subnet and VLAN. This is probably the most difficult part of the setup.  However, once setup it works really well.  I'm not sure why, but even against a Interface Group of 2+ interfaces it performs better.  You might want to test it in your environment to be sure, but we swear by it.

Also, be sure to add the proper snapmirror.conf entries for multipathing, and add your entry for your new SM relationship BEFORE starting your SM initialization, or it will NOT multipath.  You can verify multpathing as it's running using ifstat -a.

[ryanbeaty]

This is how I normally do it. Like bsti@plex.com wrote, ideally you can use vol move.

Example Environment Equipment

• Netapp01 (original / production source)
• Netapp02 (original SnapMirror destination)
• Netapp03 (new SnapMirror destination)
• Volume named exchange

Configure SnapMirror Relationship between NetApp02 and Netapp03

1. Log into netapp03
2. Create a volume named "sm_exchange"
3. Restrict the volume "sm_exchange"
4. Edit the snapmirror.conf file
   1. Netapp02:sm_exchange netapp03:sm_exchange - * * * *
5. Snapmirror initialize the "netapp03:sm_exchange" volume

Break the SnapMirror Relationship between netapp01 and netapp02

1. Log into netapp02
2. Quiesce the snapmirror "netapp02:sm_exchange"
3. Break the snapmirror "netapp02:sm_exchange"
4. Edit the snapmirror.conf file to remove the relationship "netapp01:exchange netapp02:sm_exchange"

Break the SnapMirror Relationship between netapp02 and netapp03

1. Log into netapp03
2. Quiesce the snapmirror "netapp02:sm_exchange"
3. Break the snapmirror "netapp02:sm_exchange"
4. Edit the snapmirror.conf file to change the relationship from "netapp02:sm_exchange netapp03:sm_exchange" to "netapp01:exchange netapp03:sm_exchange"
5. Snapmirror resync "netapp03:sm_exchange"

Delete SnapShots and Volume on netapp02

1. Log into netapp02
2. Snap list the sm_exchange volume
3. Snap delete the sm_exchange (snapmirror) snapshot
4. Offline the "sm_exchange" volume
5. Destroy the "sm_exchange" volume
6. Verify the relationship between "netapp01:exchange netapp02:sm_exchange" is gone

Delete SnapShots on netapp01

1. Log into netapp01
2. Snap list the exchange volume
3. Snap delete the exchange (netapp02 snapmirror) snapshot
4. Verify the relationship between "netapp01:exchange netapp02:sm_exchange" is gone

[KDEAN1961]

Thanks for the detailed respone!  I will let you know the outcome.

[cfbnetapp]

I liked the detail on that reply, too, and the general strategy.  But, you might have been hoping to keep the mirrored volume current on your original target during the three days it takes to re-initialize the new target.  I'm not sure there's a good way to do that.

[KDEAN1961]

I followed your steps exactly and when I tried to do the resync from original to new destination I got the following

"Snapmirror resynchronization of snapmirror_test_newdest to sdcnasnap2n1:snapmirror_test : no common snapshot to use as the base for resynchronization

Aborting resync"

[bsti]

Now that I look, the steps I take are slightly different. 

I don't do a resync.  Where you have A -> B then B -> C, then once you've finished initializing B -> C, then you edit the snapmirror.conf  file on C to reflect the new relationship from A -> C.

After that, run a snapmirror update -S source_filer:source_vol new_dest_filer:new_dest_vol

Also, I don't delete snapshots on either destination until the relationship from A ->C is going.  THis may be why you get your error (you are deleting the base snapshot).

Try this basic procedure:

1) Create your destination volume on C as he states above.

2) Perform an initialize from B -> C.  Whiile this is happening you cannot update A -> B.

3) Once that is done, edit snapmirror.conf on C to reflect A -> C

4)  Do an update on C:  as in my example above.

5)  If that works, then clean up by removing snapshots on A that refer to B, then updating to C again.

6) Once you are 100% happy, delete volume on B.

Hope that helps.

[KDEAN1961]

Ok..I remember chaging a snapmirror destination  over a year ago and I know recall performing an UPDATE and not a Resync.  (It's all coming back to me now!)

I will try it again.  I did NOT delete any snapshots.  I think the UPDATE command will do the trick!

Thanks!

[ryanbeaty]

The only reason why you would need to delete the snapshots is to remove them from the "snapmirrorr status" list. Until you delete those old relationship snapshots they will always show up when you run snap mirror status. This is more of a cleanup phase after you are done.

[KDEAN1961]

I followed your steps:

1) I  quiesce/break both snapmirror jobs A ->B and B->C

2) Change the snapmirror.conf for A ->C

3) Ran an update -S from A ->C and got the following error. 

snapmirror update -S cwbnasnap2n1:snapmirror_test cwbnasnap4n1:snapmirror_test_repldest

SnapMirror: destination transfer from cwbnasnap3n1:snapmirror_test_repldest to snapmirror_test_repldest : destination is not in snapmirrored state.

If I did not quiesce/break the jobs first I got the following error:

snapmirror may be misconfigured, the source volume may be busy or unavailable.

[bsti]

Im not sure the full effect of breaking before doing your update.  You're timing is great because i literally just did this operation this weekend with success so i know it works.  I think breaking the sm breaks things.  To fix it, id try resyncing A to B, then running another update to fix b to c.

The error you got before breaking smells a lot like a typo.  When you edited snapmirror.conf prior to switching over, which controller did you edit it on?  You need to edit the file on controller C.  You then trigger your updates from C from there on.  Is there a typo in the file referring to controller A?  Can C resolve the name you specify for A?  Did you try multipathing?  That might introduce some extra complexity.

Basically, sm will fail with that error if the controller does not resolve and/or the volume you specify in the destination controller does not exist.

[KDEAN1961]

I did edit the snapmirror.conf file on controller C.  Multipathing is not an option at this time.

I'm glad I'm testing this before I cutover my real SM job!  I agree with you and I know it works.  I will re-do my testing and will not break the jobs prior to running my update.

Thanks again for you help.

[KDEAN1961]

I followed your steps and did NOT break any existing SM jobs.

Step1

updated the snapmiror.conf on the "C" controller: 

cwbnasnap2n1:snapmirror_test cwbnasnap4n1:snapmirror_test_repldest - - - - -

Step 2

run the update  -S on the C controller:

snapmirror update -S cwbnasnap2n1:snapmirror_test cwbnasnap4n1:snapmirror_test_repldest  

I received the error again:

SnapMirror: destination transfer from cwbnasnap2n1:snapmirror_test to snapmirror_test_repldest : transfer from source not possible; snapmirror may be misconfigured, the source volume may be busy or unavailable.

cwbnasnap2n1:snapmirror_test (source) -> cwbnasnap4n1:snapmirror_test_repldest (final destination)

I verified  controllers can resolve each other's names.. they are in the same DataCenter

Ugggh...any thoughts?

[bsti]

Can you check your snapmirror.allow file?  Is there an entry on the SOURCE controller for the new destination you are updating to?

[KDEAN1961]

By any change are you doing volume snapmirror or qtree.  I am using volume snapmirror and that may be the issue?

I did update my snapmirror.allow file on the source (thank you) but I am now getting the message that the "destination is offline, busy, or not initialized"

[bsti]

All volume snapmirror.  And I'm moving to a separate controller.  It's really strange you are having all of these issues.  I moved 4 volumes today.  Let me outline, more specifically, all of the steps I take.  I'm hoping you are just missing one small step:

Assume  the following:

Original Snapmirror Source:    A:snapmirror_src

Original SM Destination:         B:snapmirror_test

New SM Destination:              C:snapmirror_test_repldest

First:  Verify controller B has an entry for controller C in the snapmirror.allow file.  Make sure controller A also has an entry for controller C.  The entry should simply be the hostname of the controller.

1) Create a new volume on controller B.  Let's call it snapmirror_test_repldest.  Make it the same size as the original destination volume:  snapmirror_test.

2) Restrict the volume snapmirror_test_repldest.

3) Add an entry to controller C snapmirror.conf:

  B:snapmirror_test C:snapmirror_test_repldest - - - - -

4) STOP any transfers from A -> B.  They will fail during the initalization and mess things up.

5)  Initialize B -> C: 

  snapmirror initialize -S B:snapmirror_test C:snapmirror_test_repldest

6) Wait for the initialize to finish.

7) Edit the snapmirror.conf file on controller C again.  Change the entry you added from:

  B:snapmirror_test C:snapmirror_test_repldest - - - - -

To this:

  A:snapmirror_src C:snapmirror_test_repldest - - - - -

😎  Perform an update from A -> C:

  snapmirror update snapmirror_test_repldest

This *should* work.  Once this is complete. Then you can cleanup the following:

- On controller B, break the snapmirror relationship from A -> B.

- Remove all snapshots on B:snapmirror_test that refer to snapmirror_test_repldest.  I'm referring specifically to the snapshots SM creates automatically.  Not user-created snaps.  You can leave those alone.

-  Remove any entries on B in the snapmirror.conf file that refer to the the A -> B or B -> C relationships.

- Offline and/or remove the volume snapmirror_test on controller B.

- On the source controller A, remove any snapshots on snapmirror_src that refer to snapmirror_test on controller B. 

-  Perform one last update from C, updating A - > C.

At this point, you should be done.