Error Code: 0xc0040877 Full database backup sets have changed since the restore operation started. [ Edited ]

[#1]

I'm running Snapmanager 5.2 on a prod server and on my verification server. I use the clone wizard to test backups from prod to verification. I keep the same issue which I would think would manifest during the clone wizard phase. 

From the clone report:

Stream-based backup restore: No
Keep Replication: No
Run DBCC CHECKDB: No
Leave database attached after DBCC: No
Restore to alternate location using writable snapshot - Database clone: Yes
Clone on available SnapMirror destination volume: No
Change clone database paths based on new database name: No
Mount Point Directory: C:\Program Files\NetApp\SnapManager for SQL Server\SnapMgrMountPoint
Create database clone: Yes
Restore database(s) even if database(s) are online: Yes
Recover database without restoring: No
Create transaction log backup before restore: No
Abort restore if transaction log backup before restore unsuccessful: N/A
Run Command before the current operation: No
Run Command after the current operation: No

During other clone operations, I've also gotten this error: Remote clone operation failed: Error code: 0xc0041085, Error Code: 0xC0041085 Further details may be found in the SnapManager report or event logs

Any link, direction or assistance would be very much appreciated. 

Thank you.

[#2]

Do you have any messages in the Application event log on either server?  Also, it would help to have the complete clone log file for further analysis.

[#3]

The only logs I'm aware of are in C:\Program Files\NetApp\SnapManager for SQL Server\Report\Restore [DATABASE]. If anyone knows of another, please let me know. 

[10:13:04.537] Retrieving ONTAP virtual disks information...
[10:13:08.647]
[10:13:08.709] *** RESTORE DATABASE TO ALTERNATE LOCATION

[10:13:08.709] Querying installed SQL Server instances on [Prod Server]...
[10:13:08.709] Allocating ODBC environment handle...
[10:13:08.709] Setting environment attribute...
[10:13:08.709] Setting SQLBrowseConnect option...
[10:13:08.709] SQLBrowseConnect...
[10:13:08.709] Parsing SQLBrowseConnect server string...
[10:13:08.709] *** SnapManager for SQL Server Report

[10:13:08.709] Restore Time Stamp: 12-01-2014_10.13.02
[10:13:08.709] Restore Host: Prod Server
[10:13:08.709] Restore as database: new database destination name(s)
[10:13:08.709] Restore to server: new SQL Server destination(s)
[10:13:08.709] Restore From Other: No
[10:13:08.709] Restore Old Host: Prod Server
[10:13:08.709] Restore Old Instance: VDISK__D&L
[10:13:08.709] Restore filegroup: All
[10:13:08.709] Restore Full Backup Set: 11-30-2014_23.00.35
[10:13:08.709] Multiple Database Sharing Single Virtual Disk:
[10:13:08.709]
[10:13:08.709] Database 1 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 2 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 3 :XXXXXXXXXX: Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 4 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 5 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 6 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 7 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 8 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 9 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 10 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: Yes
[10:13:08.709] Restore to server: Verf. Server
[10:13:08.709] Restore as database: XXXXXXXXXX_Clone
[10:13:08.709] Leave database operational: Yes
[10:13:08.709] Total Logs Applied = 1; Available Log = 11
[10:13:08.709] Log - 1 : 11-30-2014_23.00.35
[10:13:08.709] Log - 2 : 12-01-2014_00.00.36
[10:13:08.709] Log - 3 : 12-01-2014_01.00.37
[10:13:08.709] Log - 4 : 12-01-2014_02.00.41
[10:13:08.709] Log - 5 : 12-01-2014_03.00.42
[10:13:08.709] Log - 6 : 12-01-2014_04.00.38
[10:13:08.709] Log - 7 : 12-01-2014_05.00.38
[10:13:08.709] Log - 8 : 12-01-2014_06.00.32
[10:13:08.709] Log - 9 : 12-01-2014_07.00.36
[10:13:08.709] Log - 10 : 12-01-2014_08.00.34
[10:13:08.709] Log - 11 : 12-01-2014_09.00.36
[10:13:08.709]
[10:13:08.709] Database 11 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 12 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 13 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 14 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 15 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 16 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 17 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 18 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 19 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 20 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 21 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 22 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Database 23 : XXXXXXXXXX : Prod Server
[10:13:08.709] Database selected for restore: No
[10:13:08.709]
[10:13:08.709] Stream-based backup restore: No
[10:13:08.709] Keep Replication: No
[10:13:08.709] Run DBCC CHECKDB: No
[10:13:08.709] Leave database attached after DBCC: No
[10:13:08.709] Restore to alternate location using writable snapshot - Database clone: Yes
[10:13:08.709] Clone on available SnapMirror destination volume: No
[10:13:08.709] Change clone database paths based on new database name: No
[10:13:08.709] Mount Point Directory: C:\Program Files\NetApp\SnapManager for SQL Server\SnapMgrMountPoint
[10:13:08.709] Create database clone: Yes
[10:13:08.709] Restore database(s) even if database(s) are online: Yes
[10:13:08.709] Recover database without restoring: No
[10:13:08.709] Create transaction log backup before restore: No
[10:13:08.709] Abort restore if transaction log backup before restore unsuccessful: N/A
[10:13:08.709] Run Command before the current operation: No
[10:13:08.709] Run Command after the current operation: No
[10:13:08.709]
[10:13:08.709] *** PREPARING RESTORE

[10:13:08.709] Verifying the selection of backup data set...
[10:13:09.068] Starting Dataset backup version list...
[10:13:13.490] Unable to get backup list from dataset.
[10:13:13.553] [SDAPI Error]: Dataset(2930asdfhg78) is not found. Error description: There is no object named '2930asdfhg78'..

300
[10:13:13.553] Starting Dataset backup version list...
[10:13:18.319] Unable to get backup list from dataset.
[10:13:18.365] [SDAPI Error]: Dataset(2930asdfhg78) is not found. Error description: There is no object named '2930asdfhg78'..

300
[10:13:18.365] Starting Dataset backup version list...
[10:13:23.037] Unable to get backup list from dataset.
[10:13:23.100] [SDAPI Error]: Dataset(2930asdfhg78) is not found. Error description: There is no object named '2930asdfhg78'..

300
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.100] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.115] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:23.131] WARNING: Skipping backup list using dataset, backup may not get deleted. Please ensure Protection Manager is functional correctly.
[10:13:26.537] Full backups have been changed.

[10:13:26.600] Error Code: 0xc0040877
Full database backup sets have changed since the restore operation started.

[10:13:26.662]

[#4]

There seems to be an issue with Protection Manager integration, and that is the reason the clone operation is failing.  On the server you are trying to run the clone, can you open the SnapDrive MMC and see the Secondary (Archived) snapshots?  Also, please run the following from an elevated command prompt:

sdcli dfm_config list

Next, please check within Protection Manager that your relationships/source volumes/destination volumes are in a 'Conformant' state.  If they are not, please take the appropriate steps needed to bring the relationships back to a conformant state.

Finally, please attempt to browse secondary/archive backups within the SMSQL GUI and record the results (is it successful? does the MMC crash? etc.).  There was indeed an issue regarding large datasets some time ago in a previous version of SnapDrive and Protection Manager that exhibited the behavior you are seeing as well.

Please contact NetApp Support if the above recommendations/troubleshooting steps do not resolve the clone database issue you are seeing.

[#5]

you are running a very old version of SMSQL, but unless you are doing something very specific, yet you should  be able to try different ways to clone from archive.

I want to add one checkpoint, before you open a case with support:

Please ensure that databases are not located on mount points AND drive letter at the same time:

example: you can reach out to db1.mdf both by going to F:\folder1\db1.mdf and by going to G:\folder1\db1.mdf, as we don't support that.

Domenico.