SMVI VSS Error

[marklomas]

We're using Netapp Virtual Storage Console 2.0 to create backups of virtual machines (vSphere 4.1, ESXi 4.1 hosts).  Everything is working fine other than for one VM which is a Domain Controller (Windows 2008 R2, VMWare tools is installed).

When the VSC / SMVI backup is taken, we get the error in VMWare "Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine"

Interestingly enough, if I create a snapshot in vSphere manually, I do not get an error.

In Windows - I get a few events logged at the time of the snapshot attempt;

Application Log Event ID: 482, Source: ESENT, Type: Error

lsass (480) An attempt to write to the file "\\?\Volume{c85ad070-d2c3-11df-80e1-005056830007}\Windows\NTDS\edb.log" at offset 7145472 (0x00000000006d0800) for 512 (0x00000200) bytes failed after 0 seconds with system error 19 (0x00000013): "The media is write protected. ". The write operation will fail with error -1032 (0xfffffbf8). If this error persists then the file may be damaged and may need to be restored from a previous backup.

Application Log Event ID: 408. Source: ESENT, Type: Error

lsass (480) Unable to write to logfile \\?\Volume{c85ad070-d2c3-11df-80e1-005056830007}\Windows\NTDS\edb.log. Error -1032 (0xfffffbf8).

Application Log Event ID: 8229, Source: VSS, Type: Warning

A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,

the error is likely to reoccur.

. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation:

PostSnapshot Event

Context:

Execution Context: Writer

Writer Class Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}

Writer Name: NTDS

Writer Instance ID: {1e9eb060-32e2-4931-8935-477950a5b121}

Command Line: C:\Windows\system32\lsass.exe

Process ID: 480

I would appreciate any insight to this - many thanks

--

Mark Lomas

[amritad]

Hi

This is how we take a quiesced VMware snapshot. Could you check if this works?

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb55068

Usually frozen snapshots is an issue if VMware tools are not installed or if the VM is not aligned.

REgards

Amrita

[marklomas]

This triggers the error yes.  Oddly it does not trigger if I tick the box to snapshot the virtual machine's memory.

--

Mark Lomas

[Darkstar]

This is to be expected. If you snapshot with the memory, ESX doesn't need to quiesce the vmdk because all the not-yet-written data still is in the host memory so no data is lost.

Usually these errors are from the Sync driver that gets installed with the VMware tools. Try disabling and uninstalling it from the device manager (the name is "sync driver" IIRC...). Also, check if a VSS sync driver is available in the vmware tools for your OS and install that (if VSS is installed, it will be used instead of the sync driver). This resolves random backup problems like this most of the time.

Be aware, however, that in rare cases you might end up with a nonconsistent snapshot. But IMHO a nonconsistent snapshot is almost always better than no snapshot at all (or a VM that crashes because of a bug in the sync driver)

-Michael

[basvanberkel]

I thought that SMVI did not yet support quiesced snapshots (application consistent) in Windows 2008

http://media.netapp.com/documents/tr-3737.pdf

page 30

[marklomas]

Well - I'm using SMVI as integrated with VSC 2.0 - not sure if that's different to the standalone SMVI

--

Mark Lomas

[basvanberkel]

don't know.

The issue with quiesced snapshots should be solved in vSphere 4.1.

But could find anything on NetApp confirming this.

[marklomas]

Yeah, we're on 4.1 with ESXi hosts, and that's how the VMs started life, with the 'latest' version of VMWare tools installed.

--

Mark Lomas

[basvanberkel]

maybe this will help?

http://www.blueshiftblog.com/?p=473

[marklomas]

Our Virtual Machines (including the affected one) were all created on 4.1 - so the option variable mentioned in that blog is already set to 'true'.

--

Mark Lomas

[basvanberkel]

no raw device mapping or software iscsi initiators etcetera right?

[marklomas]

No RDMs - there is an iSCSI volume mapped using SnapDrive - but then, there are plenty of these on other VMs in the same environment and we have no problems backing them up.

--

Mark Lomas

[basvanberkel]

Weird, well one way to find out.

unmount iscsi volume and try again.

or create a clone, unmount iscsi volume and try again.

hopefully someone from netapp will shed some light on this

[HendersonD]

I thought that VMWare snapshots of domain controllers and any VMs that had iSCSI drives attached through the Microsoft iSCSI initiator, you could not take a VMWare snapshot, only a filer snapshot.

[marklomas]

Can anyone verify if this is the case?  documentation etc?  VSC 2.0, 2008 R2 DC - is this a no-go for a consistency snapshot?

[amritad]

By default, SMVI uses quiesced VMware snapshots of virtual machines to capture the consistent

state of the virtual machines prior to making a Data ONTAP Snapshot copy of the backing storage.

According to VMware KB article #1009073, VMware Tools are unable to create quiesced

snapshots of virtual machines that have NPIV RDM LUNs or Microsoft iSCSI Software Initiator

LUNs mapped to them (this often results in timeout errors during snapshot creation). Therefore,

customers using the Microsoft iSCSI Software Initiator in the guest and running SMVI with VMware

snapshots turned on, which is not recommended, are at high risk of experiencing SMVI backup

failures due to snapshot timeouts caused by the presence of Microsoft iSCSI Software Initiator

LUNs mapped to the virtual machines.

And here's the VMware KB

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009073

Regards

Amrita

[marklomas]

Sorry, I'm a little confused by this.  Firstly, we have plenty of example of customers with a VMWare Virtual Machine, with an iSCSI LUN mapped using the Microsoft iSCSI initiator (setup via SnapDrive) where we don't have a problem.

What we want is to:

a) Get LUNs mapped for Windows in the Virtual Machine and;

b) use SMVI in VSC 2.0 (ie, NOT the standalone SMVI) to back up the VMs and have those backups be consistent, useable backups

What approach are we supposed to take to marry up the above requirements?

Finally this is all related to a specific error we are getting backing up a Windows 2008 R2 VM which is a DC (see my original post on this thread).  It's the NTDS VSS provider that fails during the VMWare consistency snapshot, and Active Directory is NOT in this case stored on an iSCSI LUN.

--

Mark Lomas