SnapDrive Failing to Mount Snapshots

[Yoda_Force]

We have recently had a new NetApp installation provisioned and configured. All seemed ok, with the exception that we are unable to perform DBCC verification on any of our SQL servers. They both snapshot and snapmirror ok.

The error we are facing is as follows:

"Failed to mount backup 'snapshot name'. Reasons Failed to mount disk (Disk ID0 in Backup (ID). Server Error: FLOW-11019: Failure in AssignNewUuidToAttachedVmdk: Index: 0, Size: 0."

We have tried to manually mount the snapshots but they fail for the same reason.

We are working with NetApp already on this but they don't know what the solution is of yet, it's been escalated to engineering.

Any help would be appreciated.

[SeanLuce]

Hello Yoda_Force,

Some more information would be helpful.  Are your SQL servers virtualized?  Are you using RDMs or VMDKs?  Which protocol are you using? NFS/iSCSI/FC? Are you trying to perform DBCC on the same SQL server that owns the database? or another SQL server?  Are you trying to mount from the primary array or the SnapMirror destination?  Is this 7-mode or cDOT?  Do you have FlexClone licenses?

What versions of the following software is being used?

VSC

SnapDrive

SMSQL

MS SQL

Windows

ESXi

ONTAP

Sorry to ask so many questions, but many problems are related to interoperability and without knowing more details about your environment it is difficult to know where to begin with troubleshooting.

Thank you,

Sean Luce

Open Systems Technologies

#NetAppATeam

@seanluce

[Yoda_Force]

Hi Sean,

We are using NFS with VMDK's for our SQL environment. We are performing DBCC verification on each respective SQL server (they server and handle their own DB's). It doesn't matter whether we mount from they primary or SnapMirror destination, same results.

VSC - 4.2.2

SnapDrive - 7.2

SQL - 2008 R2

Windows - Server 2008 R2

ESXi - 5.5

ONTAP - 8.3

While I am letting NetApp support carry on with this, I thought it would be a good idea to see if anybody on here could help. There is nothing special about our environment.

It is worth mentioning though, I created a new VM running the same environment as above and all works fine, we know this is something to do with our existing VMs. They were upgraded from hardware version 7 to 10 as part of the NetApp installation process.

Dale.

[ismopuuronen]

Hi,

Do you have data aggregates assigned to your data SVMs?
vserver show -vserver <svm-name> -fields aggr-list


Br,
Ismo

[heinowalther]

Hi

Just to verify... I didn't have any aggr assigned to my VSM, so I added them like this:

nsc02::> vserver show -vserver nsc02fc -fields aggr-list
vserver aggr-list
------- ---------
nsc02fc -

nsc02::> vserver modify -vserver nsc02fc -aggr-list nlsas_nsc02_a,nlsas_nsc02_b

nsc02::> vserver show -vserver nsc02fc -fields aggr-list
vserver aggr-list
------- ---------------------------
nsc02fc nlsas_nsc02_a,nlsas_nsc02_b

Sadly it makes no difference... 😞

/Heino

[dmauro]

are you also not posting the case#?

with case number, I can try to take a look at the logs and where the case is....

Domenico.

[heinowalther]

Of cause.. my casenumer is: 2005650842

But we haven't gotten that far yet as I expect the supporter to be more SMSQL focused as I created the case as a SnapManager failing to verify...  I have suggested that the case be escalated to some SnapDrive guys instead..

/Heino

[heinowalther]

Another small update...

I can see that SD actually creates a LUN clone of the datastore...  I guess the "mounting" of this datastore failes then.

And it also makes sense when you look at the error that SD returns...

/Heino

[heinowalther]

Just a little update..

I can verify that I am able to clone both the volume, and the LUN from SysManager.

I cannot mount the clone directly from snapdrive, as it tries to mount it via iSCSI.

I haven't tried it yet, but I am 99% sure that I would be able to mount the cloned datastore inside vCenter and attach the disk to the VM...

/Heino

[heinowalther]

I have the same problem, and I also have a case open with netapp support, but no solution yet.

My setup is:

Win 2008R2

SQL 2012SP1

SD 7.1.1

SMSQL 7.2

VSC 6.0

vCenter 6.0

cDot 8.3GA

Disks mounted via VMDK in FC mounted Datastores.

This is a new installation, and we can do backups with SMSQL just fine, but we found the problem when trying to do a verify, which involves a mount of the snapshot... and this failes just as described in this case.

I think it's worth mentionening that we have other servers where this works just fine...

We are now focusing on SnapDrive as we can reproduce the problem here just by trying to mount a snapshot of a disk.

The SMSQL backups are running fine, just with no verify enabled.

And oh yeah.. I guess we cannot restore anything, should we need to 🙂  Yet I think we can handle it in other more manual ways if we needed to...

I have written a PM to Yoda_Force in order to try to link our two cases up...

/Heino

[dmauro]

>> We are working with NetApp already on this but they don't know what the solution is of yet, it's been escalated to engineering

in that case please hold on to that and we will surely find a solution for you asap.

which case# is this about?

thanks,

Domenico.

[dmauro]

I checked the latest SMSQL backup report file: Backup [MKSQL01]_05-08-2015_11.38.37.txt

I see that SMSQL and SDW do mount the clone successfully:

[11:40:27.939] [MKSQL01] *** VERIFY DATABASE AFTER BACKUP

[11:40:27.963] [MKSQL01] Collecting verification information...
[11:40:27.986] [MKSQL01] Getting database backup information from SnapInfo file...
[11:40:28.013] [MKSQL01] Running verification from local server....
[11:40:28.023] [MKSQL01] Checking SQL Server status...
[11:40:28.194] [MKSQL01] Mounting Snapshot [sqlsnap__mksql01_05-08-2015_11.38.45__daily] for [F] of Computer [MKSQL01]
[11:40:28.197] [MKSQL01] SnapManager will use directory [C:\SMSQLMnPt] to mount snapshot [sqlsnap__mksql01_05-08-2015_11.38.45__daily]
[11:40:28.197] [MKSQL01] Mount point directory [C:\SMSQLMnPt]
[11:40:28.199] [MKSQL01] Snapshot will be mounted on directory [C:\SMSQLMnPt\MPDisk001]

[11:40:28.205] [MKSQL01] Starting SDAPI mount snapshot...
[11:40:28.214] [MKSQL01] Preparing LUN 'F:\', for SDAPI operation...
[11:40:28.214] [MKSQL01] Mounting snapshot: sqlsnap__mksql01_05-08-2015_11.38.45__daily
[11:43:09.245] [MKSQL01] Snapshot mount completed successfully.
[11:43:09.246] [MKSQL01] This Snapshot is mounted as LUN/share [C:\SMSQLMnPt\MPDisk001].
[11:43:09.246] [MKSQL01] Mount Snapshot succeeded.

[11:43:09.246] [MKSQL01]
[11:43:09.469] [MKSQL01] The new mounted drive/share is available
[11:43:09.470] [MKSQL01] The new mounted drive/share is available
[11:43:09.470] [MKSQL01] The new mounted drive/share is available
[11:43:09.470] [MKSQL01] The new mounted drive/share is available
[11:43:09.470] [MKSQL01] The new mounted drive/share is available
[11:43:09.470] [MKSQL01] The new mounted drive/share is available
[11:43:09.470] [MKSQL01] The new mounted drive/share is available
[11:43:09.470] [MKSQL01] The new mounted drive/share is available
[11:43:09.471] [MKSQL01] The new mounted drive/share is available
[11:43:09.471] [MKSQL01] The new mounted drive/share is available
[11:43:09.471] [MKSQL01] The new mounted drive/share is available
[11:43:09.471] [MKSQL01] The new mounted drive/share is available
[11:43:09.472] [MKSQL01] The new mounted drive/share is available
[11:43:09.472] [MKSQL01] The new mounted drive/share is available
[11:43:09.472] [MKSQL01] The new mounted drive/share is available
[11:43:09.472] [MKSQL01] The new mounted drive/share is available
[11:43:09.472] [MKSQL01] The new mounted drive/share is available
[11:43:09.472] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available
[11:43:09.473] [MKSQL01] The new mounted drive/share is available

[11:43:09.480] [MKSQL01] Using existing connection to server [MKSQL01]...
[11:43:09.524] [MKSQL01] Checking clone database name...

[11:43:09.546] [MKSQL01] Attaching database [betadb__Clone]...
[11:43:09.546] [MKSQL01] CREATE DATABASE [betadb__Clone] ON PRIMARY (NAME = 'betadb', FILENAME = 'C:\SMSQLMnPt\MPDisk001\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\DATA\betadb.mdf') LOG ON (NAME = 'betadb_log', FILENAME = 'C:\SMSQLMnPt\MPDisk001\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\DATA\betadb_log.ldf') FOR ATTACH
[11:43:11.169] [MKSQL01] Database attached.
[11:43:11.176] [MKSQL01]
[11:43:11.176] [MKSQL01] ALTER DATABASE [betadb__Clone] SET AUTO_SHRINK OFF
[11:43:11.184] [MKSQL01]
[11:43:11.195] [MKSQL01] DBCC CHECKDB (N'betadb__Clone') WITH NO_INFOMSGS, PHYSICAL_ONLY
[11:43:23.588] [MKSQL01]
[11:43:23.590] [MKSQL01] DBCC CHECKDB completed successfully.

[11:43:23.593] [MKSQL01] Detaching database [betadb__Clone].
[11:43:23.708] [MKSQL01] Database was detached successfully.
[11:43:23.710] [MKSQL01] Checking clone database name...

perhaps it fails randomly? perhaps when there is not enough space? if so, could you povide a log file name to look at?

I don't see VSC (smvi server.log logs or vcenter logs): smvi/vsc logs may be needed as SnapDrive communicates with VSC in order to perform all operations on VMDK's.

Kind regards,

Domenico.

[heinowalther]

Hi again

I think you are looking at one of the other servers we have, which doesn't have any problems... (MKSQL)

The one we are looking at in the case is called "VMDB"

Don't know where you fond that log ? From an autosupport maybe ?

/Heino

[heinowalther]

... regarding space, there should be enough space for mounting the datastore...

I am aware that if we wanted to restore a database, we need alot of space, infact as much space as the VMDK file, which I still do not understad is nessaray when restore one one of the databases on a VMDK.... 

I actually have another case open on that specific issue 2005650842 and I still does not excatly understand why we need all that space in order to restore a single database, but maybe you can explain it bett...

As you can see in the case we got this errors, in the snapdrive logs:

05/08-13:49:26.777 PID:6092 TID:8268 clrHelpers.h@899 RestoreSMVIBackup method exception:-VMware Task "CopyDatastoreFile_Task" for entity "MKSQL01_DS01" failed with the following error: Insufficient disk space on datastore ''.

Restore of backup (sqlsnap__mksql01_05-08-2015_11.38.45__daily) on disk (F:\) failed. Reason: (VMware Task "CopyDatastoreFile_Task" for entity "MKSQL01_DS01" failed with the following error: Insufficient disk space on datastore ''.).

We solved the issue by expanding the datastore space to allow for the whole VMDK.

We do not get the same errors on the VMDB server, so I don't think the issues are linked.

/Heino

[dmauro]

I will try to explain to the best of my knowledge:

by design:

SnapDrive uses Single File Snap Restore (SFSR) for ISCSI RDM lun restore, and SFSR is quite fast.

- In VMDK case, since it’s through NFS, there are two choices: in place and out of place restores.

a. The out of place: VSC performs a file copy to do restore and time is proportional with size of disk. SMSQL (through VSC) uses file copy to restore a VMDK of a VM with disks that are located on different datastores. This also requires an amount of space double compared to the currently occupied space at time of restore.

b. In place restore for NFS: VM, Datastore or in-place disk restore will use Single File SnapRestore SFSR.

the ‘a’ restore mode is the slower one and requires double the space. It’s also called NFS- Out of Place Disk Restore – Flexclone, Mount and copy as per the table.
This occurs when the VMDK’s belonging to a VM are spread across datastores.

In order to use SFSR:
We need to restore either the whole VM or the whole Datastore.
Workaround: move all the VMDKs (including the C drive) to a single datastore, as this should do SFSR and speed up the restore process.

does that help?

Domeonico.

[heinowalther]

OK, it makes sense if we were to restore a whole VMDK file, but that's not what we want, lets make it simple.

One volume on the NetApp containing one LUN mapped to VMWare as one datastore with two VMDKs (We use FC), the two VMDKs are mapped to 😧 (for database) and L: (for logs).

The two drives 😧 and L: contain several databases, lets say LittleDB (at 1GB) and LargeDB (at 2TB)

Now I want to restore LittleDB, and from what you and the supporter on my case has explained I need to have at least the whole size of the VMDK (2TB + 1GB) avaliable in the datastore in order to restore just the LittleDB.

But I am able to do this manually and more efficient:

Just mount the snapshot of D:, copy over the MDB file and disconnect the snapshot again, the same with L: of I wanted the logs...

It doesn't make sense to me that SMSQL does this in such a way that we need 2TB of space...

So I think the supporter is wong about this, or at least I hope so 🙂   I don't want to get back to the good old days of 100% Fractional Reserve on LUNs 😉

/Heino

[aborzenkov]

---

SnapDrive uses Single File Snap Restore (SFSR) for ISCSI RDM lun restore, and SFSR is quite fast.

---

Hmm ... that's not what I experienced. In my experience SFSR was always painfully slow. Other users commented on this as well. It is possible that it has improved in recent Data ONTAP versions though.

[dmauro]

I don't think the case owner can do much about something which works by design.

All I can say is that this is by design when we don't do Single File Snap Restore in these vmdk scenario's, as explained earlier.

If you would like further explanations, I advice you to get into a chat with the Technical Marketing Engineer.

I have forwarded the thread to that group.

hopefully they get back to you asap.

Domenico.

[heinowalther]

Thanks Domenico, but the restore is one issue, the mounting of the snapshots is not related to this issue, sorry that I got you sidetracked 🙂

/Heino