PM 4.0 Non-disruptive migration of NFS and iSCSI workloads?

[aborzenkov]

Just read in announcement. Could someone give more pointers about it?

NFS transparent migration would be killer selling point for some projects. Thank you!

[adaikkap · Accepted solution]

Provisioning Manager is the interface for using the Data ONTAP 7.3.3 feature of Data Motion using Vfiler for Non-disruptive Migration.

You will need Data Ontap 7.3.3 minimum and Aynsc and Sync SnapMirror license on the filers for supporting DataMotion.

Regards

adai

[benjstarratt]

We have used vfiler migrate which is the technology this is built upon to manually move NFS workloads transparently.  As long as the applications can live through the cutover (usually 30-ish seconds) it is fine--we have had good success with VMware and Oracle though some challenges with Oracle in a data guard configuration.

There is a good TR-3814 that goes over the use cases and compatibility.  There are a few errors about supported models, but an email to our local rep got an immediate response back that our models (3140's) were covered and that the documentation was incorrect and will be changed.

The biggest issue is the lack of support for migration of vfilers with de-duplicated volumes.  I was told that the developers were very aware that was a desired feature.  You can use vfiler migrate with de-duplicated volumes, but the TR authors said that there is a slight difference in that the new technology uses a semi-synchronous method for transferring the data which is why they recommend against it.

There are limits to the number of volumes that can be attached to a vfiler--my experience with vfiler migrate is that the more volumes the longer the cutover.

All that said--this technology is wonderful for transparent data migration.

[arndt]

Check the latest version of TR-3814 for updates around Data Motion on systems that have deduplicated data. The original version of this TR that came out stated a lack of support, but a new version was quickly released that states support for Data Motion with deduplication as long as the systems involved are not under high load.

Mike

[fletch2007]

I just wrote up our experiences using non-disruptive vFiler migrate to upgrade our 3040's 25Tb (in 15 vFilers) to a new 3170 with zero downtime:

http://www.vmadmin.info/2011/02/vfiler-non-disruptive-migration.html

I agree, this is a killer feature and I am pleased with the robust implementation - seems like magic at first, but it just works (like vmware vMotion )

Cheers,

Fletch.

[sprakash]

Excellent writeup!

Always great to hear real world feedback.

Regards,

Prakash Subramanian

Product Manager, NetApp Manageability Software Group

[tanmoy]

Hello Fletch,

It is indeed a nice article and gives us a good feedback on the product datamotion. It's good know that the customer could do successful datamotion using our technology.

Howevere, we do find few issues the article faced by customer during the course of moving the data. Following is one of them :

"but there were a couple problematic vFilers erroring out for unspecified reasons, which were remaining preventing us from getting off the old 3040 hardware. For these we found running the same vFiler migrate from the command line actually proceeded without error."

We would like to know

a) What kind of errors were those and did they capture those error strings/messages, logs for the same?

b) Did they face those errors during the actual cutover time or in the dry-run results?

c) How did they get rid off those errors; could they run into successful cutover after facing those errors?

It's important for us to know those errors and related queries so that we can avoid such issues in field.

I have one more query; although I think I understood the problem but still want to make sure. This is on the following:

"However, we were less than excited that the existing vFiler migration had no way to import our existing multi-terabyte snapmirror relationships. To utilize PM's vFiler migration, we had to delete these snapmirror relationships with the DR vfiler, and recreate them (sometimes taking 18 hours) re-initializing from scratch."

a) Did you mean to say that they tried to use DR vfiler initially to accomplish the migration task and resulted into some issues? If yes, what kind of issues they faced there?

b) And while using datamotion ( Provisioning Manager) did customer want to use the existing snapmirror-relationships ( out of DR vfiler) for baselining the volumes?

Your answer to all the above queries will help us building a more robust solution.

Thanks

Tanmoy

[fletch2007]

Hi, I¹ll review my notes and logs to see what the error was from PM, but not

from the command line

1. Did you mean to say that they tried to use DR vfiler initially to

accomplish the migration task and resulted into some issues? If yes, what

kind of issues they faced there?

Yes, we encountered an unclean arp cache clear resulting in a duplicate IP

after executing a vfiler dr activate command ­ we had a case open on this

for weeks

2. And while using datamotion ( Provisioning Manager) did customer want

to use the existing snapmirror-relationships ( out of DR vfiler) for

baselining the volumes?

Yes, we would like to re-use those existing snapmirror relationships for

baselining the volumes

thanks

[scottgelb]

We noticed the arp issue with 7.3.4 but it may have been introduced earlier.  The workaround we use is ifconfig 0.0.0.0 or ifconfig -alias to remove the ip from the interface (even though stopping the vfiler should be enough but isn't with this bug).

For vFiler DR, there is a workaround to use existing mirrors... create the vfiler manually with all volumes mirrored manually, then stop the vfiler and then vfiler dr resync (instead of using vfiler dr configure to initate the mirrors).  ONTAP 7.3.5 just added a vfiler dr configure -u feature to allow the configure command to not re-initialize mirrors that already exist, so the workaround isn't required with that.  Any vfiler dr command still updates snapmirror.conf to 0-59/3 for 3 minute updates (even with existing entries) so save a copy of snapmirror.conf.

With vfiler migrate and data motion, I don't know of any way around not re-initializing the mirrors.  Hopefully we get a -u option similar to what vfiler dr just added in 7.3.5.

[niels]

Hi Fletch,

I'm cusious about your vFiler DR experiance and what you actually wanted to accomplish.

During my personal testing, all vFiler DR relationships remained intact, without the need for a re-baseline.

I tested the following scenario:

Existing:

Source A --> vFiler DR --> Dest B

- Replacing Source A with new System C

- Use DataMotion for vFilers to move vFilers from A to C

Resulting:

Source C --> vFiler DR --> Dest B

As said, no re-baseline for vFiler DR required.

Main difference, during my testing I used only some GB of data. As you migrated several TB, I assume the vFiler DR SnapMirror update conflicted with the newly established SnapMirror for the migration, perhaps losing a common SnapShot between the source(s) and the DR destination. Perhaps disabling the SnapMirror updates during the migration would have helped to keep the relationships in tact - but that's just a guess...

Generally DataMotion keeps all existing SnapMirror relationships while migrating vFilers, and as Provisioning and Protection Manager don't know anything about vFiler DR (which is sad but another story), those relationships are treated as a simple SnapMirror.

A different thing though is moving the DR vFiler. AFAIK that's not possible with DataMotion.

regards, Niels

[tanmoy]

I think what Fletch has tried to do is

a) He wanted to migrate vfilers using DR vfiler feature; did not use PM's datamotion feature at first shot.

b) While doing migration using DR vfiler he faced an issue; duplicate IP address issue.

c) Then he tried to accomplish migration using PM's datamotion feature for the same set of vfilers and it was successful.

d) But then PM datamotion again initialized the snapmirror baseline while doing migrating the vfilers and their volumes and it took time. He wanted to use the existing snapmirror relationships ( which were created for DR vfiler technique). That's what he is trying to explain there; I hope so.

Btw, I did not get the following :

"

A different thing though is moving the DR vFiler. AFAIK that's not possible with DataMotion."

Although Provisioning and Protection Manager don't trigger DR vfiler from their interface but during datamotion Provisioning Manager maintains the DR vfiler states and transfer the same. I did not get what do you mean by moving the DR vfiler.

PM can move a vfiler which is DR'd with other vfiler in some other storage system; you also explained the same in your update. Probably I am not getting the point here,it will be helpful if you can explain the use case in detail.

Thanks

Tanmoy

[niels]

Hi Tanmoy,

let's say I've the afore mentioned setup:

system A / vFiler A --> vFilerDR --> system B / vFiler A(DR)

With DataMotion I'm able to move vFiler A from system A to another HA pair, system C and the vFiler DR relationship stays intact:

system A

    |

    |

DataMotion

    |

    |

    V

system C / vFiler A --> vFilerDR --> system B /vFiler A(DR).

But in case I need to replace my secondary/DR system B with a new system D, rather than my system A, I'm not able to move the vFilerDR destination vFiler A(DR) from system B to system D. Thus the following is not possible with DataMotion:

                                     system B

                                         |

                                         |

                                     DataMotion

                                         |

                                         |

                                         V

system A / vFiler A --> vFilerDR --> system D / vFiler A(DR)

Provisioning Manager errors out with the following message:

Conformance Results

=== SEVERITY ===

Error:    vFiler unit 'NR-vFiler3'(43200) is not in running state.

=== REASON ===

Migration not supported (Error 23509)

=== SUGGESTION ===

No suggested corrective action is available.

As a consequence, migrating the destination DR vFiler always requires manual handling. Either manual SnapMirror from system B to new system D and manually re-establishing the relationship from system A to system D, which can get quite complicated. Or building a new vFiler DR from system A to system D which requires a complete baseline transfer.

Hope that makes it clearer.

regards, Niels

[tanmoy]

Hello Niels,

Thanks a lot for making the use case clear; I got your points now.

Yes, we don't support migration of DR destination and unfortunately one has to re-baseline or entirely do a mirroring from A to D. It's not automated in our tools now.

You can get in touch with our product management for this requirement.

Thanks

Tanmoy

[scottgelb]

vfiler dr uses vfiler0 for the mirror (same as data motion and vfiler migrate). When you vfiler migrate the source does the vfiler dr update mirrors after? You may have to update the target vfiler0 snapmirror.conf for the new source vfiler0. Or did it update snapmirror.conf for you with data motion?

On the dr vfiler, if isolated or if a different ip address, you could dr activate, then migrate, then dr resync from the source after migrate to make it dr again (snapmirror.conf is updated to every 3 min is a pesky thing but easy to edit). Not sure if that is a feasible workaround depending on the network implication.

Typos Sent on Blackberry Wireless

[niels]

Hi Scott,

the snapmirror.conf file is altered during the DataMotion Process and the updates of the vFiler DR relationship will correctly occur from the new source system.

Although Protection and Provisioning Manager don't know anything about vFiler DR, they at least treat the involved SnapMirrors as such and just move the relationships correctly.

regards, Niels