Re: Vmware Snapshot without pausing VM

SVOLLRAT1 · ‎2012-12-03

Hi All

We are on Implementing a "Backup"-Solution involving SnapCreator taking Snaps from our VMware Enviroment.

Snapshots handling works well so far but it seems VMs are paused during Backup-Creation.

Cannot logon to any VM involved in the Snapshot.

Since this is a Productive Enviroment this is not tolerated.

Which Switch did I miss to set so that SnapCreator takes online Snapshots of a VM?

Other Backup-Software buffers Changes to a Delta-File, not saving it during the Backup so that only the crash-consistent vmdks are included.

Similar thing is expected from SnapCreator but seems I don't get it right.

Thanks for help

BR Stefan

Arora_Kapil · ‎2012-12-06

You can set VIBE_VMWARE_SNAPSHOT =N in your configuration.

Check the config file/admin guide for a description on this param.

VMware snapshots will not be taken in this case.

Kapil

SVOLLRAT1 · ‎2012-12-10

Hi Kapil

We do want to have the VM to be snapped by VMware to have it at least Crash-Consistent (and having it update the Descriptor-Files properly).

What we don’t want is that the VM is paused to do a Memory-Dump too.

Schould I set SnapshotDisable=No and and VMCrashConsistent=Yes to achieve that?

The SC Parameters are quite complicated and reverse thinking in many cases, so I have my problems understanding how they link together most times.

(ok, I’m a SC Noob too)

Freundliche Grüsse / Kind Regards

Stefan Vollrath

T-Systems Schweiz AG

Storage & Backup Operations

Stefan Vollrath

Storage Engineer

Murgenthalstrasse 12, CH-4901 Langenthal

+41 (0) 62 916 0453 (phone)

+41 (0) 78 667 2069 (mobile)

+41 (0) 62 916 0201 (fax)

E-Mail: stefan.vollrath@t-systems.com

http://www.t-systems.ch <http://www.t-systems.ch/>

Notice: This transmittal and/or attachments may be privileged or confidential. If you are not the intended recipient, you are hereby notified that you have received this transmittal in error; any review, dissemination, or copying is strictly prohibited. If you received this transmittal in error, please notify us immediately by reply and immediately delete this message and all its attachments. Thank you.

Von: Kapil Arora

Gesendet: Freitag, 7. Dezember 2012 04:59

An: Vollrath, Stefan

Betreff: - Re: Vmware Snapshot without pausing VM

<https://communities.netapp.com/index.jspa>

Re: Vmware Snapshot without pausing VM

created by Kapil Arora <https://communities.netapp.com/people/akapila> in Snap Creator - View the full discussion <https://communities.netapp.com/message/96274#96274>

You can set VIBE_VMWARE_SNAPSHOT =N in your configuration.

Check the config file/admin guide for a description on this param.

VMware snapshots will not be taken in this case.

Kapil

Reply to this message by replying to this email -or- go to the message on NetApp Community <https://communities.netapp.com/message/96274#96274>

Start a new discussion in Snap Creator by email <mailto:discussions-community-products_and_solutions-databases_and_enterprise_apps-snapcreator@communities.netapp.com> or at NetApp Community <https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=2481>

Arora_Kapil · ‎2012-12-10

As far as I know the VMware plug-in does not pause VMs or do memory dumps.

The VMware plug-in only takes the VMware snapshot of the VM that is being backed up/

If as part of the VMware snapshot vCenter is performing that operation, SC is unaware of it.

Can you take a VMware snapshot of the VM manualy through vCenbter and see if the VM is paused?

You don not need to change the default settings for SnapshotDisable and VMCrashConsistent params.

Thanks,

Kapil

SVOLLRAT1 · ‎2012-12-10

Hi Kapil

Retried the thing in the Meantime.

Seems it is not the initial VM Snapshot that creates the Problem but the removal of it.

Applying the Changes done during the NetApp Snap takes massively to long.

Creating the Snap of just 6 VMs is done in around 6 Min, acceptable for me but could be faster (using parallel connects instead of just one to vC for example)

Removing the Snaps of the same Machines takes horrible 30 Min to complete, way too long for some MB of Delta-Data created in that 15 Sec passed between.

Most inacceptable Part is that the System that has the “remove all Snapshots” task active stops responding. Active RDP Session just hangs, new ones cannot be opened and so on.

I assume this has either to do with the way the VIBE-Plugin executes the Removal or comes down to a Configuration Error on NetApp-Storage.

Any Ideas on that?

Freundliche Grüsse / Kind Regards

Stefan Vollrath

T-Systems Schweiz AG

Storage & Backup Operations

Stefan Vollrath

Storage Engineer

Murgenthalstrasse 12, CH-4901 Langenthal

+41 (0) 62 916 0453 (phone)

+41 (0) 78 667 2069 (mobile)

+41 (0) 62 916 0201 (fax)

E-Mail: stefan.vollrath@t-systems.com

http://www.t-systems.ch <http://www.t-systems.ch/>

Notice: This transmittal and/or attachments may be privileged or confidential. If you are not the intended recipient, you are hereby notified that you have received this transmittal in error; any review, dissemination, or copying is strictly prohibited. If you received this transmittal in error, please notify us immediately by reply and immediately delete this message and all its attachments. Thank you.

Von: Kapil Arora

Gesendet: Montag, 10. Dezember 2012 12:57

An: Vollrath, Stefan

Betreff: - Re: Vmware Snapshot without pausing VM

<https://communities.netapp.com/index.jspa>

Re: Vmware Snapshot without pausing VM

created by Kapil Arora <https://communities.netapp.com/people/akapila> in Snap Creator - View the full discussion <https://communities.netapp.com/message/96450#96450>

As far as I know the VMware plug-in does not pause VMs or do memory dumps.

The VMware plug-in only takes the VMware snapshot of the VM that is being backed up/

If as part of the VMware snapshot vCenter is performing that operation, SC is unaware of it.

Can you take a VMware snapshot of the VM manualy through vCenbter and see if the VM is paused?

You don not need to change the default settings for SnapshotDisable and VMCrashConsistent params.

Thanks,

Kapil

Reply to this message by replying to this email -or- go to the message on NetApp Community <https://communities.netapp.com/message/96450#96450>

Start a new discussion in Snap Creator by email <mailto:discussions-community-products_and_solutions-databases_and_enterprise_apps-snapcreator@communities.netapp.com> or at NetApp Community <https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=2481>

ktenzer · ‎2012-12-11

SC follows this process if VIBE_VMWARE_SNAPSHOT=N

1) discover what netapp volumes datastores are located on

2) take netapp snapshot of datastores

SC follows this process if VIBE_VMWARE_SNAPSHOT=Y

1) discover what netapp volumes datastores are located on

2) take vmware snapshot of VMs (this can be slow, nothing netapp can do, we are calling vmware APIs)

3) take netapp snapshot of datastores

4) remove vmware snapshot of VMs (this can be slow, nothing netapp can do, we are calling vmware APIs)

If things are too slow with vmware snapshots then I am afraing only option is to do without them i.e. VIBE_VMWARE_SNAPSHOT=N. You may also want to try VSC which is another product from NetApp for backing up vmware. I would expect it to perform the same (they make same API calls as we do) but certainly worth a try.

Another thing you can try is just create a vmware snapshot and delete it manually. How long does it take? I would expect it to also be slow.

Keith

SVOLLRAT1 · ‎2012-12-12

Hi Keith

In the meantime we found something to fix the main issue.

The Pausing or simply not reacting only appeared during VMware Snapshot Removal.

OS-Guy found out that this is a Bug/Issue in vSphere in combination with NFS Datastores.

Seems to be fixed in ESXi 4.1.0 8003800.

Now the Snapshot Removal is done in 3min instead of 30.

Still could be faster when SnapCreator would trigger the VMware API parallel instead of sequential.

Hope there will be a Variable Included in SC in a future release to accomplice that, does speed up things quite a bit.

Freundliche Grüsse / Kind Regards

Stefan Vollrath

T-Systems Schweiz AG

Storage & Backup Operations

Stefan Vollrath

Storage Engineer

Murgenthalstrasse 12, CH-4901 Langenthal

+41 (0) 62 916 0453 (phone)

+41 (0) 78 667 2069 (mobile)

+41 (0) 62 916 0201 (fax)

E-Mail: stefan.vollrath@t-systems.com

http://www.t-systems.ch <http://www.t-systems.ch/>

Notice: This transmittal and/or attachments may be privileged or confidential. If you are not the intended recipient, you are hereby notified that you have received this transmittal in error; any review, dissemination, or copying is strictly prohibited. If you received this transmittal in error, please notify us immediately by reply and immediately delete this message and all its attachments. Thank you.

Von: Keith Tenzer

Gesendet: Dienstag, 11. Dezember 2012 20:00

An: Vollrath, Stefan

Betreff: - Re: Vmware Snapshot without pausing VM

<https://communities.netapp.com/index.jspa>

Re: Vmware Snapshot without pausing VM

created by Keith Tenzer <https://communities.netapp.com/people/ktenzer> in Snap Creator - View the full discussion <https://communities.netapp.com/message/96591#96591>

SC follows this process if VIBE_VMWARE_SNAPSHOT=N

1) discover what netapp volumes datastores are located on

2) take netapp snapshot of datastores

SC follows this process if VIBE_VMWARE_SNAPSHOT=Y

1) discover what netapp volumes datastores are located on

2) take vmware snapshot of VMs (this can be slow, nothing netapp can do, we are calling vmware APIs)

3) take netapp snapshot of datastores

4) remove vmware snapshot of VMs (this can be slow, nothing netapp can do, we are calling vmware APIs)

If things are too slow with vmware snapshots then I am afraing only option is to do without them i.e. VIBE_VMWARE_SNAPSHOT=N. You may also want to try VSC which is another product from NetApp for backing up vmware. I would expect it to perform the same (they make same API calls as we do) but certainly worth a try.

Another thing you can try is just create a vmware snapshot and delete it manually. How long does it take? I would expect it to also be slow.

Keith

Reply to this message by replying to this email -or- go to the message on NetApp Community <https://communities.netapp.com/message/96591#96591>

Start a new discussion in Snap Creator by email <mailto:discussions-community-products_and_solutions-databases_and_enterprise_apps-snapcreator@communities.netapp.com> or at NetApp Community <https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=2481>

ktenzer · ‎2012-12-12

Glad things are working better

Removing snapshots and creating them in parallel is something we are looking into. We havent done this yet because vmware has or had issues in past when you did deletes or creates in parallels. We observed timeouts and strange behavior and vmware did not recommend this at time. I have heard in 5.x things are much smoother and a lot of these issues may be fixed but we are waiting on guidance from vmware as to what is supported and what are the recommendations. I will see if I can get engineering to follow up on this again but wanted to communicate that we were thinking about it and already tried to do this in past, it would help a lot!

Keith

rmatt · ‎2012-12-15

Hi, Stefan. I think Keith and Kapil answered most of this already, but just some follow-ups as I wrote the plug-in:

1) We don't do VMware snapshot creations or removals in parallel as we've seen in large-scale environments that some snapshot creation and deletions hang indefinitely depending on the ESX version. This is entirely an issue with VMware and not something we can arbitrarily control or fix. We do take steps to remove old VMware snapshots we see that didn't get deleted during previous backup operations, just in case something was left around, but this is cleaning up and not actually fixing the real issue in the first place.

2) The default for the next version of Snap Creator is to set VIBE_VMWARE_SNAPSHOT=N by default. While taking VMware snapshots is recommended, the larger the scale of the environment, the more this can push out the backup window. Most OSes are running crash-consistent filesystems with appropriate logging mechanisms today, and adding VMware snapshots simply extends the backup window and introduces tremendous amounts of additional I/O based on how VMware does snapshots. A VMware snapshot creates a delta disk, and any/all I/O to that delta disk for new writes could add an additional 2 - 3x more I/O to your storage array (and this is ANY array vendor, not NetApp -- it's a side effect of their snapshots). Imagine if you're snapshotting 50 VMs, and you keep creating more and more of these delta disk files, so that by the time you reach the last VM, you're already doing 50 x 2.5 more I/Os for every VM to that point. Then you start removing the VMware snapshots, but you're already hammering your storage array. It's painful and it's why we turned the option off.

Let me know if this helps provide some context, Stefan. Thanks,

--Matt