SMHV Puts Some VMs into Saved State During Backup

[israelmmi]

I just started using SMHV and noticed that for some of the VMs on my servers (not all) they are put into "Saved State" while the Volume Snapshot is taking place. This used to happen with MS's regular VSS backup of VMs in order to be able to quietly backup a VM VHD adn config file.

Anyone know why this happens to only some VMs? I am told this should not happen at all? Where should I look to see why this is happening?

In happens during this point from the job log, where there is a large gap of time:

[06/02/2010 00:34:43, (13)] Preparing for offline backup of component \74FBCF40-8C3F-4654-8A8E-B69D21EB3FF7
[06/02/2010 00:34:51, (13)] Preparing for offline backup of component \8F013759-ABC7-42C2-A88F-A1AF144D299B
[06/02/2010 00:35:00, (13)] Creating snapshot set
[06/02/2010 00:45:04, (13)] Saving VSS writer metadata
[06/02/2010 00:45:04, (13)] Generating backup information

[amritad]

Hi

VM's can go into saved state for the reasons listed here

http://msdn.microsoft.com/en-us/library/dd405549(VS.85).aspx

Could you also check if VDS service s running inside the VMs?

REgards

Amrita

[israelmmi]

I just backed up a 2003 x32 Enterprise SP2 VM which went into Saved State. The VDS service there is set to manual, but it looks it was started before/during the backup process, and then after it finished it was stopped.

I didn't really follow the link as to why offline should be required to backup these VMs. I could understand in older OSs but not here. What am I missing?

[amritad]

Hi

Is this the error message

Backup Jobs fail to complete because all virtual machines cannot be backed up online.
SMHV: Failed to take backup of VM HYPERV_VM[Hyper-V] as it cannot be backed up online.

SMHV is unable to create the Child Partition Snapshot due to an issue between Hyper-V Writer in the host and the Hyper-V writer in the Guest OS. The type of backup selected by Hyper-V writer is done outside of SMHV and is not something that SnapManager can change.

There are several reasons that Hyper-V Writer would not be able to create the Child Partition Snapshot:
* Backup (volume snapshot) Integration Service is installed and running in the child VM.The service name is "Hyper-V Volume Shadow Copy Requestor".
Windows 2000: Backup Integration Service is not supported.
* The child VM must be in the running state.
* The Snapshot File Location for the VM is set to be the same volume in the host operating system as the VHD files for the VM.
* All volumes in the child VM are basic disks and there are no dynamic disks.
* All disks in the child VM must use a file system that supports snapshots (for example, NTFS).
* Virtual Disk Service is disabled in the Guest OS
http://msdn.microsoft.com/en-us/library/dd405549%28VS.85%29.aspx

So can you check if these are enabled?

REgards

Amrita

[israelmmi]

No. The backup is successfull. It is just that it needs to put some of my VMs in Save State in order to back them up. Here is the SMHV portion of the log:

[22/02/2010 15:28:06, (9)] Initializing VSS for backup
[22/02/2010 15:28:06, (9)] Collecting VSS writer information
[22/02/2010 15:28:20, (9)] Adding components for backup
[22/02/2010 15:28:20, (9)] Preparing for offline backup of component \7ADF4636-E9D3-430C-B680-A207EBACC25A
[22/02/2010 15:28:27, (9)] Preparing for offline backup of component \96ABC08A-39D2-4278-82E4-232F997E15AC
[22/02/2010 15:28:35, (9)] Preparing for offline backup of component \E2945BF2-39B6-4CFB-A47E-67CDD5A84A02
[22/02/2010 15:28:43, (9)] Creating snapshot set
[22/02/2010 15:33:33, (9)] Saving VSS writer metadata
[22/02/2010 15:33:33, (9)] Generating backup information
[22/02/2010 15:33:33, (9)] Sending backup complete notification to VSS writers
[22/02/2010 15:33:36, (9)] Saving VSS backup metadata
[22/02/2010 15:33:36, (9)] Execution of VSS backup completed successfully.
[22/02/2010 15:33:36, (9)] Renaming snapshots after backup operation

It keeps my VMs offline for 5 minutes or so while taking the snapshot. See the screenshot.

[amritad]

Hi

1) open cmd on one of the hyper-v hosts

2) increase screen buffer for cmd prompt

3) run command - diskshadow

4) run command - list writers

5) copy the screen output into a notepad and search for the vms that are going into saved state

Once the vm is found then it will show what vss thinks it can do with a backup.

Another important thing to confirm is that all of the disks in the vm have at least 300mb worth of space available.

REgards

Amrita

[barve]

Hi,

You need to check a number of things in the VM (which is getting saved during backup):

1. Integration services are installed and running.

2. All Windows volumes in the VM are formatted using NTFS.

3. There are no dynamic disks in the VM.

4. There is at least 300 MB free space available on each Windows volume in the VM.

5. There are no offline volumes in the VM.

6. Type command 'vssadmin list shadowstorage' in the VM. Verify that each Windows volume in the VM has a shadow copy storage association.

If any of the above conditions is not met, Hyper-V writer decides to save the VM during backup. It does not log the reason for doing a saved state backup. So none of the SMHV or Windows logs will show us the reason. We must login to the VM and check for each of the above issues manually.

If you want SMHV to fail the backup if VMs will be backed up using saved state, you can uncheck the "Allow saved state backup" dataset policy.

Thanks,

Anagha

[israelmmi]

Ok. I have taken one VM to work with here.

Configuration:

• Windows 2003 Enterprise Edition x32 Service Pack 2
• Has one disk, C:, 13.6GB with 5.0GB Free
• The C: drive is a basic disk

From inside the VM:

C:\Documents and Settings\admin>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Shadow Copy Storage association
   For volume: (C:)\\?\Volume{99a1847a-e9e0-11d8-93d4-806e6f6e6963}\
   Shadow Copy Storage volume: (C:)\\?\Volume{99a1847a-e9e0-11d8-93d4-806e6f6e69
63}\
   Used Shadow Copy Storage space: 0 B
   Allocated Shadow Copy Storage space: 0 B
   Maximum Shadow Copy Storage space: 1.364 GB

Results from the Diskshadow command on the Host Physical Server:

Component "Microsoft Hyper-V VSS Writer:\E2945BF2-39B6-4CFB-A47E-67CDD5A84A02"
                        - Name: E2945BF2-39B6-4CFB-A47E-67CDD5A84A02
                        - Logical path:
                        - Full path: \E2945BF2-39B6-4CFB-A47E-67CDD5A84A02
                        - Caption: Backup Using Saved State\Dev1
                        - Type: VSS_CT_FILEGROUP [2]
                        - Is selectable: TRUE
                        - Is top level: TRUE
                        - Notify on backup complete: FALSE
                        - Paths affected by this component:
                                - C:\ClusterStorage\Virtual\Dev1\Virtual Hard Disks\
                                - C:\ClusterStorage\Virtual\Dev1\Virtual Machines\
                                - C:\ClusterStorage\Virtual\Dev1\Virtual Machines\E2945BF2-39B6-4CFB-A47E-67CDD5A84A02
                        - Volumes affected by this component:
                                - \\?\Volume{ba592545-9ff2-11de-8284-806e6f6e6963}\ [C:\]
                        - Component Dependencies:

So I am still at a loss as to why this is getting backed up using Saved State.

[israelmmi]

Also, again comparing a VM that doesn't work vs. one that does:

• Both have NTFS
• Both have the Virtual Disk Service set to manual (not disabled)

And if hotfixes are of interest, this is what I installed on the problem VM:

Windows Server 2003 Service Pack 1                                  KB889101
Windows Server 2003 Service Pack 2                                  KB914961
Update for Windows Server 2003 (KB923845)                   KB923845
Hotfix for Windows Server 2003 (KB927435-v2)                KB927435-v2
Hotfix for Windows Server 2003 (KB932039)                      KB932039
Hotfix for Windows Server 2003 (KB940349-v3)                 KB940349-v3
Update for Windows Server 2003 (KB943295)                   KB943295
Update for Windows Server 2003 (KB951072-v2)             KB951072-v2
Update for Windows Server 2003 (KB955839)                  KB955839

[barve]

Dynamic VHD is ok. The issue happens if Windows disks in the VM are dynamic. Or, if volumes are not "simple" volumes. Can you please check the following?

1. Log in to the VM, open a command prompt.

2. Execute command 'diskpart'. You will have a diskpart prompt.

3. Look at the output of 'list volume' and 'list disk' commands. Make sure that all disks are basic, all volumes are simple volumes and none of the volumes is offline.

Also, please check if there are any errors in Hyper-V VMMS event log on the parent Hyper-V host.

Thanks,

Anagha

[israelmmi]

Checked all of the problem VMs. They all are Layout "Partition" and Type "Basic" (as seen in disk management).

I just started to run a defrag of all  the disks on these machines maybe that is the problem, as that did help me with VSS errors in the past.

Still, I also looked at the server and VM event logs and did not see anything problematic.

Any other ideas? If I do not allow Saved State Backups, will it tell me why it cannot do a child snapshot backup?

[vkarinta]

Hi,

Can you also please check the version of the 'virtual machine integration component service' inside the VM

This service binary can be found at the following location c:\windows\system32\vmicsvc.exe' path inside the VM.

if we don't have the correct integration component service version saved state backup can happen as well.

Thanks,

Vineeth

[barve]

Please also check the Windows Application / System event log in the VM for any errors.

Thanks,

Anagha

[israelmmi]

My version is 6.1.7600.20542.

No errors in the event log.

[barve]

We have checked everything that Microsoft has documented as a reason for doing saved state backup. Is it possible to get access to the VMs and Hyper-V host?

Thanks,

Anagha

[amritad]

Hi

Would it be possible to log a support case for this? It would be easier for us to debug the exact cause.

REgards

amrita

[israelmmi]

Will do.

[israelmmi]

Hi Amrita,

I opened a case with support and in the end result they wanted me to turn to MS support since this was looking more and more to be a VSS issue.

Then I found the following link from MS regarding DPM, which ends up was the problem:

http://technet.microsoft.com/en-us/library/dd637102.aspx

The following line caught my attention: A ShadowStorage assignment of a volume inside the VM is explicitly set to a different volume other than itself.

Then I took a look at my VM VSS configuration and noticed for all of the problem VMs that the C drive (and E whenever applicable) was configured to use Drive D as the snapshot location within VSS (see screenshot attached).

Once I fixed that all my VMs are now backed up properly - in Online mode!

Also - I asked the technician working on my case to update this KB article as well:

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb55974

Good luck!

Reuv

[amritad]

Hi Reuv,

I'll make sure our support knows about this so that we can point othe customers to this too. And we will update our documentation with this as well.

Thanks for this great input.

Regards

Amrita