Solved: Re: Backup Verification Error with SME 6.0 against an Exchange 2010

mrphoenix · ‎2010-02-22

Hi folks,

After some initial problems with installing SME 6.0 I finally got this solved. It was a permission problem on our DCs that drove our Exchange nuts ...

Finally I got the backup configured and running, now I'm having some other erros and don't know how so solve them. Maybe somebody can help?

I'm seeing these errors for each DB that has to be verified:

----------

EventSource: SnapDrive

EventID: 1116

Failed to connect to LUN. VDS volume object for the LUN s/n 'C4iAWJV2-xA3' has not been found
.

LUN Name = mec-eu02
Storage Path = /vol/vol2imx/.snapshot/exchsnap__framsexm10_02-21-2010_22.00.15/mx_db02/
Protocol Type = LUN
Storage System Name = franapfs0001
Requested Mount Point = franapfs0001:/vol/vol2imx/.snapshot/exchsnap__framsexm10_02-21-2010_22.00.15/mx_db02/mec-eu02
Assigned Mount Point = C:\Program Files\NetApp\SnapManager for Exchange\SnapMgrMountPoint\MPDisk001\

Error code :

----------

EventSource: SnapDrive

EventID: 178

Failed to connect the LUN in Snapshot copy (exchsnap__framsexm10_02-21-2010_22.00.15) as drive C:\Program Files\NetApp\SnapManager for Exchange\SnapMgrMountPoint\MPDisk001\ on the computer (FRAMSEXM10).

LUN Name = mec-eu02
Storage Path = /vol/vol2imx/mx_db02/
Protocol Type = LUN
Storage System Name = franapfs0001

Error code : VDS volume object for the LUN s/n 'C4iAWJV2-xA3' has not been found

----------

EventSource: SnapManager For Exchange

EventID: 245

SnapManager database integrity verification did not complete after the backup on server 'FRAMSEXM10' for storage group/database 'mec-eu02'.

---------

EventSource: SnapManaer For Exchange

EventID: 246

SnapManager database integrity verification did not complete after the backup on server 'FRAMSEXM10' for storage group/database 'mec-eu02'.

---------

EventSource: SnapManager for Exchange

EventID: 274

Exchange writer will not truncate transaction log files because SnapManager backup verification for 'mec-eu02' failed. The SnapManager backup verification failure should be resolved right away as transaction log LUN can run out of disk space.

Component name: 99453e30-3913-4bf0-8b55-a3b136713678

---------

EventSource: MSExchangeIS

EventID: 9782

Exchange VSS Writer (instance 052d09e3-50b3-4ef9-99d8-e4621882d62b:8) has completed the backup of database 'mec-eu02' with errors. The backup did not complete successfully, and no log files were truncated for this database.

---------

EventSource: ESE

EventID: 2007

Information Store (3608) Shadow copy instance 8 aborted.

For more information, click http://www.microsoft.com/contentredirect.asp.

Hopefully somebody knows what to do!

Best regards,

Philipp

bhavik · ‎2010-03-02

Hi Philip,

You might have checked this but is the VDS service ON? If not restart it, then restart the Snapdrive service and try doing the verification again.

Thanks,

Bhavik

View solution in original post

lwei · ‎2010-02-22

So, the backup was created successfully; but the verification failed? Was the LUN mounted correctly? -Wei

mrphoenix · ‎2010-02-23

Hi lwei,

As far as I can see the backup was done.

The errormessage I get via Mail states:

*****BACKUP DETAIL SUMMARY*****

Backup group set #1:

Backup SG/DB [DB-Name] Error: Failed to verify physical integrity of the databases

*****BACKUP RESULT SUMMARY*****

Backup group set #1:

Backup SG/DB [DB-Name]: Backup OK, verification failed

Backup failed.[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV3Pvdl' has not been found

So the LUN can't be mounted ... could it be that it's a simply "Access Denied" from the Windows 2008 R2 Box? The Snapmanager trys to mount the volume in the "Progam Files"-Tree and even as an Administrator you can't add something there without this nagging "Are you really sure"-Prompt...

Greetings,

Philipp

radek_kubka · ‎2010-02-23

could it be that it's a simply "Access Denied" from the Windows 2008 R2 Box?

Hi,

I don't think this is the reason, as I've seen SnapDrive successfully mounting LUNs in a directory on Win2k8 in a lab environment, but you may change verification settings, so SME & SnapDrive uses drive letter, rather than mount point.

Regards,

Radek

mrphoenix · ‎2010-02-23

Hi Radek,

I've changed to a different mount point and to a driveletter. But there's no change.

I'm stip getting this error via Mail:

****VERIFICATION RESULT SUMMARY*****

Verification SG/DB [DB-Name]: Database Verification Failed

Database verification failed.

Error: 0xC004030F, VDS volume object for the LUN s/n '%1' has not been found

The Report states the following:

SnapManager for Exchange

******VERIFICATION OVERALL JOB DESCRIPTION*******

Server: FRAMSEXM10
Verify Storage Group/Database 1: mec-eu03
There are 1 Database(s)

Backup Management Group: Standard
This is an operation which will only run database physical integrity verification.
The most 1 recent snapshots will be verified
The verification will be run on computer FRAMSEXM10.
Verification throttling is not enabled or supported.

Exchange Server [14.0.639.21]

[PowerShell Cmdlet]: get-exchangeserver -status -identity FRAMSEXM10
[PowerShell Cmdlet]: get-mailboxdatabase -Server FRAMSEXM10 -DomainController "framsdc02.magna.global" ; get-publicfolderdatabase -Server FRAMSEXM10 -DomainController "framsdc02.magna.global"
[PowerShell Cmdlet]: Get-Mailboxserver -Identity "FRAMSEXM10"
[PowerShell Cmdlet]: Get-MailboxDatabase -id "Mailbox Database 0653707672" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mit-eu01" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu03" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu01" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu02" | select-object ActivationPreference -ExpandProperty ActivationPreference
Checking dataset availability, connecting to SnapDrive...
An error occurred when querying FlexClone status for all Exchange storage system volumes.Exception of type 'System.OutOfMemoryException' was thrown.

SnapManager FRAMSEXM10 concurrent job status at 23.02.2010 15:02:39 :
=================================================================================================
SnapManager is running 0 backup integrity verification jobs.
This DeferredBackupVerification job with JobID=1029 is waiting in the queue at position 0.
=================================================================================================

There is no unverified FRP backup prior to backup [02-22-2010_22.00.13]
pExchangeInfo != NULL

DeferredBackupVerification Job of JobID=1029 starts to run on server FRAMSEXM10.

DeferredBackupVerification job [1029] has report file : C:\Program Files\NetApp\SnapManager for Exchange\Report\Backup [FRAMSEXM10]\02-23-2010_15.02.24.txt.

Starting verification from server...

***VERIFICATION JOB DETAILS

Exchange Server: FRAMSEXM10
Storage Groups/Databases:
mec-eu03
The most 1 recent snapshots will be verified.
The verification will be run on computer FRAMSEXM10.

Verification mount directory C:\mxstore\verify.
Use drive letter if available to mount snapshot Yes.

Verification throttling is not enabled or supported.
This is an operation which will only run database verification.

Backup Management Group: Standard
Database Verification Time Stamp: 02-23-2010_15.02.24

Run Command after operation: No
Job Management is enabled : YES
Verify backup on SnapVault secondary: No

***DATABASE VERIFICATION JOB STARTED AT: [02-23-2010_15.02.41]

SnapInfo Directory located at:
C:\mxstore\mec-eu03_log\sme_snapinfo\EXCH__FRAMSEXM10\SG__mec-eu03
Getting unverified FRP backup prior to backup [02-22-2010_22.00.13]
There is no unverified FRP backup prior to backup [02-22-2010_22.00.13]

The destination volume of the source Logical Disk C:\mxstore\mec-eu03_db\ FlexClone State is Success and SnapMirror State is Unknown.
Mounting Snapshot [exchsnap__framsexm10_02-22-2010_22.00.13] for LUN C:\mxstore\mec-eu03_db\ created from Computer FRAMSEXM10
[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4J1rs' has not been found
(SnapDrive Error Code: 0xc004030f)
SnapDrive failed to mount the snapshot.
[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4J1rs' has not been found

Unable to mount snapshot, abort verification.

RUNNING TRANSACTION LOG INTEGRITY VERIFICATION
The SnapInfo directory is located at:
C:\mxstore\mec-eu03_log\sme_snapinfo\EXCH__FRAMSEXM10\SG__mec-eu03\02-22-2010_22.00.13\Logs

Running Integrity Verification using CheckSgFiles API

LogPath: C:\mxstore\mec-eu03_log\sme_snapinfo\EXCH__FRAMSEXM10\SG__mec-eu03\02-22-2010_22.00.13\Logs
LogBaseName: E03

Operation completed successfully in 3.915 seconds.

[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4J1rs' has not been found

Verification failed. Error code: 0xc004030f, LUN: C:\mxstore\mec-eu03_db\
An autosupport message is sent to the storage system of LUN [C:\mxstore\mec-eu03_db\].
Failed to verify the storage group/database [mec-eu03] .
[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4J1rs' has not been found

***DATABASE VERIFICATION JOB ENDED AT: [02-23-2010_15.09.49]
[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4J1rs' has not been found

****VERIFICATION RESULT SUMMARY*****
Verification SG/DB [mec-eu03]: Database Verification Failed
Database verification failed.

Error: 0xC004030F, VDS volume object for the LUN s/n '%1' has not been found

UpdateSelectedDestinationVolumesStatus Started
UpdateSelectedDestinationVolumesStatus Started
UpdateSelectedDestinationVolumesStatus Started
UpdateSelectedDestinationVolumesStatus Started
UpdateSelectedDestinationVolumesStatus Started
UpdateSelectedDestinationVolumesStatus Started
UpdateSelectedDestinationVolumesStatus Started
UpdateSelectedDestinationVolumesStatus Started
UpdateSelectedDestinationVolumesStatus Started
Preparing email notification message...
An email notification message is sent.

Might the missing SnapMirror-Status be the failure?

Philipp

beaupre · ‎2010-02-23

It might be that SnapMirror has not completed replication before you started your verification on the SnapMirror secondary.

Instead of running a deferred verification, can you just enable verify on destination on your backup job? This will ensure that your verification doesn't start until the replication has completed.

Also, you might want to ensure that the VDS service is running on your verification host.

Thanks,
Mark

mrphoenix · ‎2010-02-23

Hi Mark,

Maybe a stupid question but how can I set up the verification of the local backup copy only?

I do the verifies from the same mailbox-server like I do the backups.

Greetings,

Philipp

beaupre · ‎2010-02-23

Are you using Exchange 2010 in a DAG?

If you are using Exchange 2010 in a DAG, you will want to set up the backup and verification on one of the DAG member servers that is not running the active copy.

If you are using Exchange 2010 in a standalone configuration, the easiest way is to disable/delete the verification job you have created and then use the SME Backup Wizard to re-create the backup job. In the SME Backup wizard, you will select the exchange server as the verification server and on the verification configuration you will not select the Verify on SnapMirror destination option.

Thanks,
Mark

mrphoenix · ‎2010-02-23

Hi Mark,

I'm using a single exchange-server right now. If there would be a DAG the option to verify the Backup on the passive node would make sense, although there might me no passive node anymore. (the DAG can be used as an "Active-Active-Cluster").
When I'm clicking through the Backup-Wizard it states "There are no SnapMirrored Volumes available" after clicking on "Verification on Destination Volumes"-Button in the "Database Verification Settings"-Dialog.

Philipp

lwei · ‎2010-02-23

Hi Philipp,

I'll add two things to check:

In the job report you posted earlier, there was an error: "Checking dataset availability, connecting to SnapDrive...
An error occurred when querying FlexClone status for all Exchange storage system volumes.Exception of type 'System.OutOfMemoryException' was thrown." I don't know if it's related but it's good to get rid of it.
I'll go back to the nested mount point issue: rather than C:\mxstore\verify, try C:\mxstore_verify.

Thanks,

Wei

mrphoenix · ‎2010-02-23

Hi Lwei,

I've switched to a dedicated DriveLetter already - without success 😞

SnapManager for Exchange

******VERIFICATION OVERALL JOB DESCRIPTION*******

Server: FRAMSEXM10
Verify Storage Group/Database 1: mit-eu01
There are 1 Database(s)

Backup Management Group: Standard
This is an operation which will only run database physical integrity verification.
The most 1 recent snapshots will be verified
The verification will be run on computer FRAMSEXM10.
Verification throttling is not enabled or supported.

Exchange Server [14.0.639.21]

[PowerShell Cmdlet]: get-exchangeserver -status -identity FRAMSEXM10

SnapManager FRAMSEXM10 concurrent job status at 23.02.2010 17:15:27 :
=================================================================================================
SnapManager is running 0 backup integrity verification jobs.
This DeferredBackupVerification job with JobID=1031 is waiting in the queue at position 0.
=================================================================================================

[PowerShell Cmdlet]: get-mailboxdatabase -Server FRAMSEXM10 -DomainController "framsdc02.magna.global" ; get-publicfolderdatabase -Server FRAMSEXM10 -DomainController "framsdc02.magna.global"
[PowerShell Cmdlet]: Get-Mailboxserver -Identity "FRAMSEXM10"
[PowerShell Cmdlet]: Get-MailboxDatabase -id "Mailbox Database 0653707672" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mit-eu01" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu03" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu01" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu02" | select-object ActivationPreference -ExpandProperty ActivationPreference
Checking dataset availability, connecting to SnapDrive...
An error occurred when querying FlexClone status for all Exchange storage system volumes.Exception of type 'System.OutOfMemoryException' was thrown.
There is no unverified FRP backup prior to backup [02-22-2010_22.00.13]
pExchangeInfo != NULL

DeferredBackupVerification Job of JobID=1031 starts to run on server FRAMSEXM10.

DeferredBackupVerification job [1031] has report file : C:\Program Files\NetApp\SnapManager for Exchange\Report\Backup [FRAMSEXM10]\02-23-2010_17.15.12.txt.

Starting verification from server...

***VERIFICATION JOB DETAILS

Exchange Server: FRAMSEXM10
Storage Groups/Databases:
mit-eu01
The most 1 recent snapshots will be verified.
The verification will be run on computer FRAMSEXM10.

Verification mount directory C:\mx-verify.
Use drive letter if available to mount snapshot Yes.

Verification throttling is not enabled or supported.
This is an operation which will only run database verification.

Backup Management Group: Standard
Database Verification Time Stamp: 02-23-2010_17.15.12

Run Command after operation: No
Job Management is enabled : YES
Verify backup on SnapVault secondary: No

***DATABASE VERIFICATION JOB STARTED AT: [02-23-2010_17.15.29]

SnapInfo Directory located at:
C:\mxstore\mit-eu01_log\sme_snapinfo\EXCH__FRAMSEXM10\SG__mit-eu01
Getting unverified FRP backup prior to backup [02-22-2010_22.00.13]
There is no unverified FRP backup prior to backup [02-22-2010_22.00.13]

The destination volume of the source Logical Disk C:\mxstore\mit-eu01_db\ FlexClone State is Success and SnapMirror State is Unknown.
Mounting Snapshot [exchsnap__framsexm10_02-22-2010_22.00.13] for LUN C:\mxstore\mit-eu01_db\ created from Computer FRAMSEXM10
[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4QRgg' has not been found
(SnapDrive Error Code: 0xc004030f)
SnapDrive failed to mount the snapshot.
[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4QRgg' has not been found

Unable to mount snapshot, abort verification.

RUNNING TRANSACTION LOG INTEGRITY VERIFICATION
The SnapInfo directory is located at:
C:\mxstore\mit-eu01_log\sme_snapinfo\EXCH__FRAMSEXM10\SG__mit-eu01\02-22-2010_22.00.13\Logs

Running Integrity Verification using CheckSgFiles API

LogPath: C:\mxstore\mit-eu01_log\sme_snapinfo\EXCH__FRAMSEXM10\SG__mit-eu01\02-22-2010_22.00.13\Logs
LogBaseName: E04

Operation completed successfully in 12.199 seconds.

[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4QRgg' has not been found

[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4QRgg' has not been found

Verification failed. Error code: 0xc004030f, LUN: C:\mxstore\mit-eu01_db\
An autosupport message is sent to the storage system of LUN [C:\mxstore\mit-eu01_db\].
Failed to verify the storage group/database [mit-eu01] .
[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4QRgg' has not been found

***DATABASE VERIFICATION JOB ENDED AT: [02-23-2010_17.22.38]
[SnapDrive Error]:VDS volume object for the LUN s/n 'C4iAWJV4QRgg' has not been found

****VERIFICATION RESULT SUMMARY*****
Verification SG/DB [mit-eu01]: Database Verification Failed
Database verification failed.

Error: 0xC004030F, VDS volume object for the LUN s/n '%1' has not been found

[PowerShell Cmdlet]: get-mailboxdatabase -Server FRAMSEXM10 -DomainController "framsdc02.magna.global" ; get-publicfolderdatabase -Server FRAMSEXM10 -DomainController "framsdc02.magna.global"
[PowerShell Cmdlet]: Get-Mailboxserver -Identity "FRAMSEXM10"
[PowerShell Cmdlet]: Get-MailboxDatabase -id "Mailbox Database 0653707672" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mit-eu01" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu03" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu01" | select-object ActivationPreference -ExpandProperty ActivationPreference
[PowerShell Cmdlet]: Get-MailboxDatabase -id "mec-eu02" | select-object ActivationPreference -ExpandProperty ActivationPreference
Checking dataset availability, connecting to SnapDrive...
Preparing email notification message...
An email notification message is sent.

Greetings,

Philipp

lwei · ‎2010-02-23

It failed to mount. Could you try to mount it manually using the SnapDrive? -Wei

mrphoenix · ‎2010-02-24

Hi Lwei,

No I can't - so it's not a problem with SME but rather one with Snapdrive!

------------------------- SnapDrive Operation Start ---------------------------------
Disk connect operation on server - FRAMSEXM10 failed for reason
VDS volume object for the LUN s/n 'C4iAWJV5E/vO' has not been found
--------------------- The complete operation list is shown as follows -------------------

********************* Server Name: FRAMSEXM10*********************

1. Checking input parameters
2. Checking access control
3. Checking policies
4. Turning on space reservation
5. Connecting to the LUN
6. VDS volume object for the LUN s/n 'C4iAWJV5E/vO' has not been found
------------------------------------------ Operation Done ---------------------------------------

Greetings,

Philipp

moez__alibhai · ‎2010-02-23

Hi,

I saw you were having 'System.OutOfMemoryException' errors, this is a windows problem, Its a problem with VSS.

There is a patch for this from Microsoft (8MB approx), which resolve this problem.

Cheers

mrphoenix · ‎2010-02-24

Hi Moez,

Do you know the MS-KB-Number?

I know about some VSS-Bugs within Windows 2003 and Windows 2008 but are there still some with Windows 2008 R2? That would be a pity...

Greetings,

Philipp

moez__alibhai · ‎2010-02-24

Hi Phillip,

Dont have it off head, i did a windows update .. it picked up a VSS update .. After that no more issues.

Let me see if i can get the KB from the customer site.

Rgds

Moez

bhavik · ‎2010-03-02

Hi Philip,

Check the event viewer logs. What error do you see in them regarding SnapDrive. What is the eventid?

Thanks,

Bhavik

mrphoenix · ‎2010-03-02

Hi Bhavik,

I only see a warning from PartMgr within the System-Log that the DiskSignature of the Snapshot that should be mounted is the same like an existing this. I would say that's expected and can be ignored 😉

After 7 Minutes I see the EventID 1116 from SnapDrive within the Application-Log that states:

"Failed to connect to LUN. VDS volume object for the LUN s/n 'C4iAWJVCMvUY' has not been found"

So I don't know what's up here and my local Netapp-Guru is dealing with more important issues at the moment... 😞

Best regards,

Philipp

bhavik · ‎2010-03-02

Hi Philip,

You might have checked this but is the VDS service ON? If not restart it, then restart the Snapdrive service and try doing the verification again.

Thanks,

Bhavik

mrphoenix · ‎2010-03-03

Hi Bhavik,

Restarting both services helped at least something.

Now I'm getting the Error-Message "RPC-Server not available" when connecting the Snapshot manually. (Source SnapDrive, EventID: 1116)

Nevertheless the mount itself failes 😞

Best regards,

Philipp

bhavik · ‎2010-03-04

Hi Philip,

About the RPC server not available, make sure the RPC service is running. If it is have a look at this thread which has a similar error : http://communities.netapp.com/message/5691.

Another possible reason for this error is there is a problem with name resolution. Did you see this too along with the RPC error message? "Failed to get Data ONTAP version running on storage system".

If you did you want to make sure the DNS entries are right. You can try adding entries to Windows local HOSTS file. If it works, make sure you update your DNS entries.

Hope this helps.

Thanks,

Bhavik