VSS Components Conflicting?

[trace3_matt]

Hello,

I have a question about VSS providers potentially conflicting when backup up a VM via VIBE.

Using the following software:

ESX 3.5 U3 + current patches

Guest: Windows 2003 Standard SP2

SQL 2005 SP2

SnapDrive 5.01

SM:SQL 2.1.2

VIBE 1.08

VMware Tools are up to date, with the VSS sync driver installed.

OnTap 7.2.4 and 7.3.1 both produce the issue

When VIBE kicks off, and calls VC to snap the SQL VM, it seems to trigger the OnTap VSS Providor to clone the LUNs and try to present them to the OS.  The VC snapshot process hangs at 95% until whatever is happening below completes.  The VC snapshot eventually completes successfully.  VIBE exits w/o errors - with status 0.

Check these logs from /etc/messages on the filer:

Tue Feb 17 20:00:33 PST [lun.offline:warning]: LUN /vol/cc_vcsql_snapinfo/{be54441b-f1fa-4336-89ab-67428b359cf5}.aux has been taken offline
Tue Feb 17 20:00:33 PST [lun.destroy:info]: LUN /vol/cc_vcsql_snapinfo/{be54441b-f1fa-4336-89ab-67428b359cf5}.aux destroyed
Tue Feb 17 20:00:34 PST [lun.offline:warning]: LUN /vol/cc_vcsql_db_master/{0ee0a747-56dc-438b-a473-f05a8182d8bb}.aux has been taken offline
Tue Feb 17 20:00:34 PST [lun.destroy:info]: LUN /vol/cc_vcsql_db_master/{0ee0a747-56dc-438b-a473-f05a8182d8bb}.aux destroyed
Tue Feb 17 20:00:34 PST [lun.offline:warning]: LUN /vol/cc_vcsql_db_virtualcenter/{1e6045d3-1f5f-4d22-8619-be4315669ff0}.aux has been taken offline
Tue Feb 17 20:00:34 PST [lun.destroy:info]: LUN /vol/cc_vcsql_db_virtualcenter/{1e6045d3-1f5f-4d22-8619-be4315669ff0}.aux destroyed
Tue Feb 17 20:00:34 PST [lun.offline:warning]: LUN /vol/cc_vc4sql_db_virtualcenter4/{08286ed9-e0a2-4409-981a-bd9dc012a3d8}.aux has been taken offline
Tue Feb 17 20:00:34 PST [lun.destroy:info]: LUN /vol/cc_vc4sql_db_virtualcenter4/{08286ed9-e0a2-4409-981a-bd9dc012a3d8}.aux destroyed
Tue Feb 17 20:00:36 PST [lun.newLocation.offline:warning]: LUN /vol/sdw_cc_vcsql_snapinfo_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_snapinfo has been taken offline to prevent map conflicts after a copy or move operation.
Tue Feb 17 20:01:12 PST [lun.map:info]: LUN /vol/sdw_cc_vcsql_snapinfo_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_snapinfo was mapped to initiator group cc_vcsql=4
Tue Feb 17 20:01:14 PST [lun.newLocation.offline:warning]: LUN /vol/sdw_cc_vcsql_db_master_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_db_master has been taken offline to prevent map conflicts after a copy or move operation.
Tue Feb 17 20:01:20 PST [lun.map:info]: LUN /vol/sdw_cc_vcsql_db_master_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_db_master was mapped to initiator group cc_vcsql=5
Tue Feb 17 20:01:22 PST [lun.newLocation.offline:warning]: LUN /vol/sdw_cc_vcsql_db_virtualcenter_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_db_virtualcenter has been taken offline to prevent map conflicts after a copy or move operation.
Tue Feb 17 20:01:58 PST [lun.map:info]: LUN /vol/sdw_cc_vcsql_db_virtualcenter_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_db_virtualcenter was mapped to initiator group cc_vcsql=6
Tue Feb 17 20:01:59 PST [lun.newLocation.offline:warning]: LUN /vol/sdw_cc_vc4sql_db_virtualcenter4_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vc4sql_db_virtualcenter4 has been taken offline to prevent map conflicts after a copy or move operation.
Tue Feb 17 20:02:35 PST [lun.map:info]: LUN /vol/sdw_cc_vc4sql_db_virtualcenter4_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vc4sql_db_virtualcenter4 was mapped to initiator group cc_vcsql=7
Tue Feb 17 20:02:53 PST [lun.map.unmap:info]: LUN /vol/sdw_cc_vcsql_snapinfo_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_snapinfo unmapped from initiator group cc_vcsql
Tue Feb 17 20:03:04 PST [lun.map.unmap:info]: LUN /vol/sdw_cc_vcsql_db_master_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_db_master unmapped from initiator group cc_vcsql
Tue Feb 17 20:03:16 PST [lun.map.unmap:info]: LUN /vol/sdw_cc_vcsql_db_virtualcenter_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vcsql_db_virtualcenter unmapped from initiator group cc_vcsql
Tue Feb 17 20:03:54 PST [lun.map.unmap:info]: LUN /vol/sdw_cc_vc4sql_db_virtualcenter4_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vc4sql_db_virtualcenter4 unmapped from initiator group cc_vcsql

SM:SQL snapshots are not scheduled at this time.

"vssadmin list providers" lists the OnTap Hardware Provider.

I do get duplicate disk signature messages from "partmgr" in the system error log, as well as emails from SnapDrive:

Virtual Disk Manager Event Detail:

Computer Name: CC-VC-SQL2005
Event ID: 310
Type: Warning
Category: Generic event

Description: Failed to enumerate virtual disk.
Device path: '\\?\mpio#disk&ven_netapp&prod_lun&rev_0.2_#1&7f6ac24&0&36304139383030343333343635373234363334344536443246373134313635#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
LUN path: '/vol/sdw_cc_vc4sql_db_virtualcenter4_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vc4sql_db_virtualcenter4'
SCSI address: (3,0,0,7)
Error code: 0xc0040375
Error description: 'LUN is not recognized by Windows. Possible reasons: this disk is not formatted, or this disk is under delete/disconnect/restore operation. Storage System 'goliath', path '/vol/sdw_cc_vc4sql_db_virtualcenter4_cl_3e54ae3f1fd2452f9d7a70d972e26a51_ss_0/cc_vc4sql_db_virtualcenter4', s/n 'C4erF4Nm/qAe'.

The production LUN never goes offline, nothing seems impacted.  I'd just rather avoid this thrash if possible.  Can I somehow specify which volumes apply to each VSS Hardware Provider, or disable the SQL Volumes from being quiesced by the VMware Tools VSS script?

I did lots of searching tonight with little luck, I'm sure someone has seen this before?

Thank you,

Matt Hallmark

[sigallcap]

Hello,

We've exactly the same problems!

Do you have any solutions?

thx

[trace3_matt]

Is no one else is running SnapDrive or Snap Manager products inside VM's, AND taking VIBE or SMVI snapshots?

Any ideas?

Thx,

Matt Hallmark

[peterl]

Have you tried this with the VMware Tools sync driver instead of VSS?  I don't think you want two VSS tools in the same VM.  (Could be wrong, but that's what I've been told.)

Peter

[fhpaschen]

I'm having the same issue.  As noted, it doesn't appear to affect the snapshots being created, just fires off hundreds of emails regarding the issue.  Did anyone figure this out?  Just to add my own information.  It *appears* to happen when a SnapManager for Exchange backup is taking place during a BackupExec backup.

[__COLIN_GRAHAM_14270]

Yes we have the same issue with VIBE and Snapdrive-iscsi luns mounted in the vmware guests.

Ive not noticed it happen with SMVI (which we have but are not using due to it snapshotting the "swap" volume too) - ill try to test a bit more next week.

[manawatudc]

I am having the same event 58 from partmgr occuring everytime snapmanager trys to take a snapshot.  everything seems to work ok, snapshots are fine, but i just get a raft of these warnings in the event log.

this happens just after VSS starts, the Data ONTAP VSS Hardware Provider Starts and then Microsoft Volume Shodow Copy Provider Starts.

I am also getting my backups with arcserve begin to fail when it uses VSS to back up the exchange databases, same services start then same warnings occur then the VSS snapshots fail.

anyone have any ideas?

[dwilsberg]

Hi,

i also have a Problem with SnapManager.

SnapManager doesent shows any Disk and the System Log shows the following error:

Failed to enumerate LUN.
Device path: '\\?\scsi#diskprod_lun1588251b000200#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}'
Storage path: '/vol/NavisionSQLBackup/Navision_Backup'
SCSI address: (1,0,2,0)
Error code: 0xc00402f7
Error description: A LUN with device path \\?\scsi#diskprod_lun1588251b000200#{53f56307-b6bf-11d0-94f2-00a0c91efb8b} and SCSI address (1, 0, 2, 0) is exposed through an unsupported initiator.

The Luns are working fine under Windows.

System:

SnapDrive 6.1 on an Windows 2003 x64 Virtual Machine running under VMWare ESX4 with 2 RAW LUNs.

Newest Host Utilities are installed on the ESX Servers.

Any Ideas?

BR

Dominik

[kris_boeckx]

Hi all,

I had this issue a year ago. We have VMware running on FC (Fiber Channel) VMFS datastores and FC RDM's.(FC is needed because snapdrive needs a FC VMFS datastore to put the RDM mapping file on, NFS datastore for mapping file is not supported)

We where thinking of implementing SMVI as a backup solution for our VMware enviroment and we also saw the issue that when doing a smvi (later we use vibe) backup, snapdrive wants to take a snapshot of the RDM(s) which will not work.

We opened a case @ Netapp and Vmware. VMware points to netapp and netapp told us: this is not supported.

As a best practice you should:

1. Disable VSS in the vmware tools of the machines where snapdrive is installed on.

2. Disable the "sync driver" (in the vmware tools) in ALL vmware machines.

3. Use snapdrive (or snapmanager for specific applications) to make snapshot's of you RDM's

Point 1 also means that you can't make a consistent snapshot of a non RDM disk when snapdrive is installed in the virtual machine.

Point 1 also means that you have to put data on a RDM if you what a consistent snapshot of this data

For the moment, we use vibe to backup our vmware enviroment with the knowledge that not the whole backup is consistent.

There is no way arround, just live with it.

Greetings,

Boeckx Kris

Pidpa

[#10]

Here is what I did to fix it similar problem and get consistent backups.

I'm not used VIBE to do backups but it should work close to the same, we are using netapps SMVI.

When vmware takes a snapshot (vmware snapshot) you can add pre freeze and post thaw script to run (before the snapshot and after).

Basically the flow is this unregistering the vss provider, take the vm snapshot, take the backup, remove the snapshot, register the vss provider

::*************************************************************
:: This script disables the Data ONTAP VSS Service so
:: you can take a vmware snapshot, this should be ran
:: as pre-freeze script
::*************************************************************

echo off

:: Disable the following services, SnapDrive, SnapDrive Management Service, Data ONTAP VSS Hardware Provider
NET Stop SWSvc
NET Stop SDMgmtSvc
NET Stop navssprv

:: Unregister the Data ONTAP VSS Service
"E:\Program Files\NetApp\SnapDrive\navssprv.exe" -r service /u

::*************************************************************
:: This script enables the Data ONTAP VSS Service so
:: you can take a snap manager snapshot, you should run this
:: script as the thaw
::*************************************************************

echo off

:: Enable the following services, SnapDrive, SnapDrive Management Service, Data ONTAP VSS Hardware Provider
NET Start SDMgmtSvc
NET Start SWSvc

:: Register & Start the Data ONTAP VSS Service, you should use the same service account as when you installed snapdrive
"E:\Program Files\NetApp\SnapDrive\navssprv.exe" -r service -a Domain\ServiceAccount -p <PASSWORD>
NET Start navssprv

In 2003 / 2000 you could also use the alternative to the vss provider the name escapes me at the moment, my understanding is it is what vm ware uses in 2000 OS since vss isn't available.  The option wasn't available for 2008 so I just came up with this.  Hope it helps.

[#11]

This should fix kris.boeckx@pidpa.be  problem from above this was our setup also.

VM with both virtual drives and RDMs.  Using SMVI and Snapmanager for SQL.  RDMs hold sql databases and VM's hold the apps. Running everything throug FC.

[karlssontaa]

Hi,

I have run this on a windows 2003 SP2 server.

Manually everything goes well but when i put the script as descibed in KB Article: 1006671 from vmware in

C:\Program Files\VMware\VMware Tools\backupScripts.d\ it does not run. I also tested to put pre-freeze-script.bat and post-thaw-script.bat in c:\windows with the same result.

The solution was to put only vcb.bat in C:\Program Files\VMware\VMware Tools\backupScripts.d, and vcb.bat point to c:\windows where I put pre-freeze-script.bat and post-thaw-script.bat.

C:\Program Files\VMware\VMware Tools\backupScripts.d\vcb.bat

echo off
if "%1" == "freeze" goto doFreeze
goto doThaw
:doFreeze
call c:\windows\pre-freeze-script.bat
goto EOF
:doThaw
call c:\windows\post-thaw-script.bat
:EOF

C:\WINDOWS\pre-freeze-script.bat
net stop swsvc
"C:\Program Files\NetApp\SnapDrive\navssprv.exe" -r service /u"

C:\WINDOWS\post-thaw-script.bat

NET Start SWSvc
"C:\Program Files\NetApp\SnapDrive\navssprv.exe" -r service -a  Domain\ServiceAccount -p <PASSWORD>

Thanks,

Tony