SMVI backup consistency

[igor]

Hello everyone!

My client has vSphere environment set up where they run 2008 VMs, mostly with transactional applications such as MS SQL and Exchange. In that respect, they would do best to rely on application-consistent backups. I'm currently studying backup consistency using SMVI and I came across the following text in SMVI Best Practices document:

Since  SMVI  (with  VMware  snapshots  turned  on)  relies  on  VMware  quiescing  of  virtual  machines  when  creating  backups,  it  is  able  to  provide  application-consistent  backup  through  VMware  VSS requester/provider  components  for  the  applications running  inside  the  virtual  machines.  In  fact,  all  SMVI backups  with  VMware  snapshots  turned  on  are  “application-consistent.”  The  limitations  of  this  application-consistent  backup  methodology  are  explained  in  the “Application-Consistent  Recovery”  section  of  this  document. 

Note:  For  VMs  running  Windows  Server  2003  as  the  guest  operating  system,  the  VSS  snapshots  are application  consistent.  For  VMs  running  Windows  Server  2008  and  Windows  Vista  the  snapshots  are  file system consistent. For more information refer to  http://www.vmware.com/pdf/vsphere4/r40/vsp_vcb_15_u1_admin_guide.pdf

I'm a bit of a noob on SMVI-VSS relations and I'm confused by this...

Are all SMVI backups indeed "application-consistent" (why the quotes?) - or just that of 2003 VMs running VSS aware applications?

Thanks!

[paulstringfellow]

So this may not be the exact answer and will want double checking…

But basically SMVI (as does SMHV with Hyper-V) when they backup the virtual machine, they trigger a VSS call inside the VM… when VSS is triggered it ensures that all vss aware components in the VM are in a ready for backup state, so logs flushed, quisced and writes cached in memory and not committed to disk…

So in that way the contents of the VM are consistent…if the VM has exchange in it for instance, Exchange would be sat in hot standby ready to be backed up, so in the way would be consistent within the VM when its snapshotted…

I’m not quite sure of the difference between 2008 and 2003 as I would of assumed they all ran the same, so I would think this is a difference between how 2008 and 2003 handle those VSS calls rather than anything to do with SMVI.

Couple of caveats I’ve found with this…if you want proper consistency in the VM ensure you’re running snapdrive, then the machine is aware its installed on a NetApp based array… and also I’ve seen problems when you use the vmware tools VSS provider in the vm’s…this is a known issue with SnapDrive and VMware, so its probably wise to reconfigure vmtools to remove the vmware vss provider…and let the Microsoft one handle the work…

Last thing..is although the contents of the vm are consistent, this does not mean you have a recoverable backup point for your application, the whole VM still needs recovering really if you wanted to role back…so if you have Exchange in a VM…just because its consistent it doesn’t mean you can recover Exchange on its own, it has to be the whole VM… its not a replacement for using SME or SMSQL etc in the vm’s…

Hope that helps, its my take on it…and would be worth confirming if you can…

Regards

Paul.

[igor]

Hello Paul,

That's my understanding as well. Except, what you've described sounds a lot like file-system consistency to me. And yes, I believe it's not a SMVI issue as well. Later on in the document (page 30) it says "VMware VSS components can’t interact with VSS application writers in Windows 2008". That would explain the different level of backup integrity between Windows 2003 and  Windows 2008/Vista.

As for SnapDrive, it would also provides just file-system consistency... I should've mentioned before, my clients decided on FC datastores for their VMs, since some of them must handle burst-like workloads, which means they use VMDKs for storing application data.

So according to the above table, without switching to RDMs or iSCSI LUNs (ethernet links) for application data storage, the most they can hope to achieve through SMVI backups would be application-consistency for 2003 VMs and file-system consistency for 2008 VMs. Is this correct?

[paulstringfellow]

Yep so looking at what you’ve said here…is that the user has his machine and applications installed in a VMDK then they are pretty limited anyway In what they can do… and what is supported as per the table…

That table is accurate as it’s a limitation of the vmware VSS provider not working with the application vss components in server 2008…

However this probably comes down to what they want to achieve…what you’re basically looking at here is using vmware snapshot capability (although taking advantage of the NetApp backend) to give you vm consistent snapshots of virtual machines…

It would seem that because in server 2008 the vmware VSS writers can’t talk to the apps then you can’t guarantee app consistency inside of the vm with server 2008, but you can in server 2003.

However this application consistency, as we said earlier, only gives you a point in time recovery of a consistent app by recovering the entire VM (so no roll forward of logs to give you an up to date recovery of Exchange for example)… so really not a great solution…

It goes back to what we said…if they want to take full benefit of the NetApp solution to protect their apps…they will need to present RDM’s to the VM’s use SnapDrive and appropriate snapmanager to get application backup and recovery features…it would seem odd not to to be fair!

Don’t know if that helps…but your assumption looks correct…and it’s a matter of maybe reviewing how they built there VM’s if they want to take full advantage of what they have… if they do…top tips!.. remove vmware tools VSS components and use SnapDrive otherwise it will break!

[igor]

Exactly, I need to present the differences in backup consistencies. It's crucial for their disaster recovery scenario, since they do a lot of factory acceptance testing for their own products!

One question regarding roll forward... SMVI can restore an entire VM. OK. But it can also restore individual VMDKs, right? So if they keep the database on one disk and it's logs on another, in case of corruption, couldn't they restore only the disk containing the database to a heathly state and then use the (unchanged) logs to roll forward?

[paulstringfellow]

potentially that may work....but wouldn't like to bank on it!

i think the issue of that is you are hoping that the logs sync in with the retrieved databases etc...i'm not sure i'd want to go down that route...

personally i don't find the SnapManager tools cumbersome although i don't use SM for SharePoint... but if you have lots of Vm instances, then setup a central SMSQL stations and point it at the instances and manage it from one console...

but i would certainly be looking seriously at using those tools...but will depend on the environment and Lukes responses have been really useful i think...but still the issue with Vm only backups, is still its a vm recovery to bring a restore back on line and the way SMVI and Vm recovery of a single disk works, is from my experience is that can be very slow...with SMVI bringing a copy of the datastore on line then copying the vmdk from the backup store to the live one...this can be hugely slow...especially when compared to bringing the app back on line form RDMS in the virtual machine...

that's just my experience... its difficult throwing ideas in when not working in the environment, just sharing some recent experience, hope it helps.

[lmunro_hug]

Hi Igor

Just speaking from my experience here… I was under the impression that the limitation of 2008 and above only being filesystem consistent was an issue with the older versions of vSphere. These used a sync driver instead of VSS with the older versions of VMware tools.  If you look at the new features of vSphere 4.1 it states that this is now possible. Search for “application-consistent”

http://www.vmware.com/support/vsphere4/doc/vsp_41_new_feat.html

vStorage APIs for Data Protection (VADP) Enhancements. VADP now offers VSS quiescing support for Windows Server 2008 and Windows Server 2008 R2 servers. This enables application-consistent backup and restore operations for Windows Server 2008 and Windows Server 2008 R2 applications.

There is also this KB

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1028881

Personally I have done VM level snaps on my MS SQL on 2008 servers and sat monitoring the SQL Current logfile. You can see when you do this it is flushing out the DBs and putting them in a backup prepared state. If you do not see this maybe it’s on an older server that has had the previous versions of VMware tools installed and was using the sync driver and has not upgraded to VSS.

I have seen the same in the event log for Exchange VMs. Take a look and see if you see this?

Luke

[paulstringfellow]

That’s a useful update as well…

An interesting topic… still think if you are going to look at App consistent snaps in a NetApp world…its SnapManager in the VM…

Glad the question was asked though!

[lmunro_hug]

paulstringfellow,

We tried the snapmanager tools, not just SnapDrive and found them far too cumbersome for our environment (Snapmanager for SharePoint is the worst…) Especially so when you have a large number of MS SQL servers and multiple instances on single VMs on the same ESX hosts. When looking at it from a VMware RDM perspective you waste far too many LUNs when the environment already has imposed limits.

Maybe it should be revisited and tried on VMFS

[igor]

Hi Luke,

This is encouraging info. I will try the same with my client, good idea.

P.S. You had SQL on VMDKs, or RDMs?

[lmunro_hug]

Igor,

I had the VM disks as VMDK files, we are no longer using RDMs