VSC - Backups Hanging (In progress)

[penningtonkr]

Hi,

I have a situation where I have set VSC to backup at datastore level with a mixture of virtual machines including RDMS and VMDK drives.

All RDM mappings are set to physical so VSC wont look at backing these up, which is expected we use snapmanger for the data drives.

For some reason the virtual centre is "hung" on the following tasks:

Create virtual machine snapshot
TMLIVEDWHSRV-01
95%
Administrator
TMVCentreSrv01

01/04/2011 00:02:41
01/04/2011 00:02:41

NetApp Create Backup
VMFS5_OS
45%
Administrator
TMVCentreSrv01

01/04/2011 00:00:18
01/04/2011 00:00:18

Reconfigure virtual machine
TMLIVEDWHSRV-01
In Progress
Administrator
TMVCentreSrv01

01/04/2011 00:15:01
01/04/2011 00:15:01

* anyone know why there is a reconfigure event in this backup???

So i am in a situation where the datastore VMFS5_OS still has snapshots for every server that resides on there (20 VMS). There is 462.18GB free on that datastore totall size 900GB LUN/ 1.14TB VOL, so if this task doesnt clear up before the weekend we may experience a situation where by the datastore could be filled.

Bit of info about the backup job:

Enabled options: Inititate snapmirror update, Perform VMWare consistency snapshot

Bit of info about DWH:

Server 2008

18GB RAM

SnapDrive 6.3 x64

SnapManager 5.1 SQL

Bit of Info on storage system:

FAS2040 Active-Active configuration

3 x DS4243 shelves SAS 600

I can see a link between large RAM size and quiesced snapshots, but why would this cause this issue in this case when previous backups have always worked?

Anyone experienced these hanging jobs before or know a fix or manual overide to stop the job? Im concerned that with no abort option we could get in a situation where the job is waiting to complete, but datastores could fill?

Any help is appreciated!

[keitha]

Ah I see, you are using the script to disable the SnapDrive VSS component. That would make sense then... The SnapManagers don't use the VMware VSS writer and in fact do get along with ti at all but I can see the logic of disabling the VSS for the image level backup then use the VSS writer for the Application level backup.

The part I am still not sold on is the better file level backup using the VMware snapshot. Since VMs don't cache IO locally I don't see how the files will actually be any better off having a VMware snapshot than without. I might be wrong about this but I don't think the VMware snap improves the file quality. (Application consistentcy is a whole different ball of wax and I agree with you there!)

Good discussion though!

Keith

[support_2]

Yeah, not sure about that, you may be right there may never be a problem in this enviroment, as some of the things in vmware / sans may be enough to save you, have to do some reading here.  Just like vmware vss provider may not be doing everything I think it does.

I sort of go by this:

… that can coordinate with backup applications to produce aconsistent and reliable copy of data, a VSS backup will be application consistent as opposed to Crash consistent, a good analogy would be to think of application consistent backup as a manual shut down of all services and then a copy and a crash consistent backup as simply to press the power off button onyour server (good luck),

Not my own words pretty common analogy of what you are doing by unclicking that option.  Without VSS it's like pulling the plug on your server when you go to restore,  may come up may not (anymore like I said 9/10 it will, just like pulling the plug won't cause a problem) just depends on what is going on at the time of backup.

So this is like hitting the power option on your vm without shutting the OS down or esx server crashing bringing down the guests - either way doesn't make you feel comfortable.  Don't like having in the back of my mind that my backups are in this state, especially if I need to do a restore.

I just not willing to take that risk here if there are other options, does depends on the enviroment some people less concerned about OS (which in my head is your app here also other things flat files being written to etc).   Been a year or so since I have looked into this as all of products are working together now.  Which means the scripts and disabling the quiesce isn't really necessary, just me suggesting that you don't turn that option off lightly.

… that can coordinate with backup applications to produce a consistent and reliable copy of data, a VSS backup will be application consistent as opposed to Crash consistent, a good analogy would be to think of application consistent backup as a manual shut down of all services and then a copy and a crash consistent backup as simply to press the power off button on your server (good luck),