Orphaned datastores in vCenter ("vmdk_sqlnap")

[alliantenergynjk]

Not sure if this is a SnapDrive problem, SMSQL problem, VSC problem, or SMSP problem.  I think it is SnapDrive...

We recently upgraded the various SnapManager products and SnapDrive to allow us to upgrade to vCenter 5.0 U1.  Here is what we currently have:

vCenter 5.0 U1

VSC 4.1

SnapDrive 6.4.2

SnapManager SQL 6.0

SnapManager SharePoint 6.0 with Agent Patch 3 which supports SMSQL 6.0

NFS volumes in vCenter

According to the Compatibility Matrix, these are all supported.

Whenever a SnapManager for SharePoint job finishes, it leaves orphanded datastores in vCenter - one on each SP SQL server.  This used to happen once every 2 or so weeks randomly for one or two servers and I just lived with it,  Now it is doing it every night in all three envrionments and I have cleanup 7 orphans every morning.  This started happening after the upgrade.

The names of the datastores are something like this:

OriginalDSName (Backup vmdk_sqlsnap__sqlservername_12-07-2012_10.21.56)

I also get these errors in SMSQL logs which Google found no hits for:

Dismounting LUN C:\Program Files\NetApp\SnapManager for SQL Server\SnapMgrMountPoint\MPDisk001\ of SnapShot [smsql snapshot]...

SnapDrive operation to dismount LUN in snapshot copy failed. SnapDrive error code: 0xc0041071

SnapDrive operation to dismount LUN in snapshot copy failed. SnapDrive error code: 0xc0041085

[SnapDrive Error]: Unable to locate a LUN to perform requested operation.

(SnapDrive Error Code: 0xc0041085)

Re-trying to force dismounting LUN...

SnapDrive operation to force dismount LUN in snapshot copy  failed. SnapDrive error code: 0xc0041085

[SnapDrive Error]: The LUN may not be connected, because its mount point cannot be found.

(SnapDrive Error Code: 0xc0041085)

Unknown Error, Error Code: 0xc0041085

Does anyone know how to prevent this?  Is there a setting I need to enable or a hotfix to install?

Nelson

[stemmer]

H Nelson,

yes, we have the same problem at one customer side.

The environment is:

   FAS 3240A

   Ontap 8.1.2 7-mode

   Snapdrive 6.4.2

   SnapManager SQL 5.2

   VSC 4.1

   NFS based VMDKs

Same first Snapdrive errror on dismounting than yours, but secondary error code different:

   [01:13:22.818]  Database was detached successfully.

   [01:13:22.824]  Dismounting LUN C:\mnt\verify\MPDisk001 of SnapShot [sqlsnap__muc1s0043_12-12-2012_00.02.29__daily]...

   [01:13:47.341]  SnapDrive operation to dismount LUN in snapshot copy failed. SnapDrive error code: 0xc0041071

   [01:14:00.923]  SnapDrive operation to dismount LUN in snapshot copy failed. SnapDrive error code: 0x80010105

   [01:14:00.927]  Re-trying to force dismounting LUN...

   [01:14:01.128]  SnapDrive operation to force dismount LUN in snapshot copy  failed. SnapDrive error code: 0xc0040221

   [01:14:01.140]  [SnapDrive Error]: The LUN may not be connected, because its mount point cannot be found. (SnapDrive Error Code: 0xc0040221)

   [01:14:01.140]  Dismounting LUN C:\mnt\verify\MPDisk002 of SnapShot [sqlsnap__muc1s0043_12-12-2012_00.02.29__daily]...

   [01:14:10.416]  SnapDrive operation to dismount LUN in snapshot copy failed. SnapDrive error code: 0xc0041071

   [01:14:21.637]  SnapDrive operation to dismount LUN in snapshot copy failed. SnapDrive error code: 0x80010105

   [01:14:21.638]  Re-trying to force dismounting LUN...

   [01:14:21.810]  SnapDrive operation to force dismount LUN in snapshot copy  failed. SnapDrive error code: 0xc0040221

   [01:14:21.810]  [SnapDrive Error]: The LUN may not be connected, because its mount point cannot be found. (SnapDrive Error Code: 0xc0040221)

Same try to force dismount.

The error is shown on every dismount in the log, but the first dismounts are in reality succesfull. That´s why the force dismount failed, because than it´s already disconnected.

But after 7 succesfull dismounts (all 7 show the snapdrive error in log) the 8 one really miss.

It looks like a timeing problem !

History:

Last week we had Snapdrive 6.3P2 and Ontap 8.1.1 and every thing is running fine.

After an NFS outage during 2 VSC virtual machine restores, we decide to upgraded to Ontap 8.1.2 on last friday.

After that update SnapManager SQL can do succesful backups but cannot do verify´s because of the Snapdrive dismount errors.

There is no 6.4.2D1 or 6.4.2P1 at the moment, but this error smells like different function giveback codes on 8.1.2

The force dismount of snapdrive logs immediatly the errors (after 1 sek.)

Have you opened a case ?

Have you also Ontap 8.1.2 running ?

Is there feedback from NetApp ?

On our Exchange VM with iSCSI connected Raw Devices the old Snapdrive 6.3P2 works fine on that 8.1.2 based NetApp.

So i think it´s a problem with vmdk based SnapManager/Snapdrive installations and Ontap 8.1.2

Greetings

Roland

Nachricht wurde geändert durch: Roland Kurz

[alliantenergynjk]

Hi Roland,

Thanks for the information in your post.  I am 99% sure we are running 8.1.2.  I am on vacation this week, so I am going off of memory but I will check when I return to the office.  We had to upgrade many things in order to support the new vCenter 5.0 U1 install.  I know we upgraded Ontap for sure, and I am pretty sure it was 8.1.2.

I opened a case with NetApp because when we upgraded SnapManager for SQL, we broke SnapManager for SharePoint - and they had to provide us a patch to make it all work.  I asked them if I could tag this second problem (orphaned vCenter datastores) onto that case and they told me to open a new one.  I was starting my vacation so I did not have time to open the new case yet.

Nelson

[alliantenergynjk]

Sorry, I was wrong.  We upgraded to 8.1.1 not 8.1.2.  I am not a storage admin (I am a Windows Server admin that just happens to be responsible for vCenter and SnapManager for SharePoint) so I am not sure on why they selected 8.1.1 instead of 8.1.2 when they upgraded or the impact this has.

Thanks

Nelson

[NET_KOHLBACH]

Hi,

I have the same problems!

Detaching database [LOHNDATEN__Clone].

Database was detached successfully.

Dismounting LUN C:\Program Files\NetApp\SnapManager for SQL Server\SnapMgrMountPoint\MPDisk001 of SnapShot [sqlsnap__w2k8-kronos_12-22-2012_20.00.37__daily]...

SnapDrive operation to dismount LUN in snapshot copy failed. SnapDrive error code: 0xc0041071

SnapDrive operation to dismount LUN in snapshot copy failed. SnapDrive error code: 0xc0041085

[SnapDrive Error]: Unable to locate a LUN to perform requested operation. (SnapDrive Error Code: 0xc0041085)

The environment is:

   FAS 2240A

   Ontap 8.1.2 7-mode

   Snapdrive 6.4.2

   SnapManager SQL 6.0

   VSC 5.1

   NFS based VMDKs

[alliantenergynjk]

Thanks for the post.  At this time I have no solution yet.  We have a case open with NetApp and the they are working on it - but due to the Christmas and New Years holidays, things have been delayed a bit.  I suspect things will pick up now and hopefully we will have a solution shortly.  I will post what I find.

Nelson

[alliantenergynjk]

A suggestion from NetApp support was to configure SMSQL to use a directory without spaces for the verification mount point.  Currently it is set to:

C:\Program Files\NetApp\SnapManager for SQL Server\SnapMgrMountPoint

This is the default set by the installation and is not something I set or configured when I installed the product.  This path worked in the past.  We changed it to a directory on the root of C: with no spaces in the name and tested in our development environment:

C:\smsql_mnt_pnt

The backup suceeded and left no orphaned disks in vCenter.  I going to make the changes in production and see if they work tonight.

Nelson

[alliantenergynjk]

Correction.  Spaces in the name is not the problem, it is the length of the mount point path that is the problem (and in this case, spaces just make the problem worse because it adds to the length).  According to NetApp support, if the the total path is longer than 160 characters, it fails.  So in production last night, we still have 4 servers leave orphans.  The location of the data files in conjunction with the DB file names and the mount point path of "C:\Program Files\NetApp\SnapManager for SQL Server\SnapMgrMountPoint" is longer than 160 characters and the new mount path I uses is also too long.  So now I am trying this:

C:\MP

If this is still too long, I will have to have the DBA move the SP data files from "D:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA" to something like "D:\Data".

Nelson

[CASTROJSEC]

Did this work?

[alliantenergynjk]

Sort of...

It fixed both QA and DEV.  However, we have seen mixed results in Production.  It seems at least one server still has such long paths that it is failing.  The names of the some of the SP databases are so long that even with a short mount-point path, it is still too long.  But then, the Friday backup worked without a problem.  So I am going to let it run a few more days and try to figure any rhyme or reason to when it fails or works.  Before it was 100% consistent.  So we are seeing improvements...but maybe that is worse in the long run because now I cannot consistentely recreate the problem...

Nelson

[alliantenergynjk]

Oh - and I was wrong about the "D:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA" path.  That is where our system databases are stored.  Our SP databases are already in a short path location: S:\MSSQL\Data.  So I guess I could shorten a bit more by making it something like "S:\DB" but I am not sure how much of a difference that will really make.  MS putting the GUID in the name of the DB files is not really helping us

SessionStateService_d78200b5804d4e19885f5e2064ed9523.mdf

Just one example...

[chqlsnetapp]

We are having the same problem. Anyone managed to get a few pointers? I'd relate it to the new version of vcenter for now.

[kevinplaisance]

What version of SnapDrive are yall running? 6.4 this issue pops up. On my servers with 6.4.1 and 6.4.2 no issues at all. I upgraded my server that had this issue from 6.4 to 6.4.2 and BOOM goes the dynamite! Issue solved.

Upgrade SnapDrive.

this link references SME, but same situation. https://communities.netapp.com/thread/21722

[systeembeheeraahg]

We are expreriencing this issue with SnapDrive 6.4.2 and SMSQL 6.0.0.1170
This was recently upgraded SnapDrive from 6.3.some version and SMSQL 5.x

NB. Changing the Mountfolder had no effect for us..

Anyone got any other ideas on how to fix this?

[CASTROJSEC]

Have you tried going back to SMSQL 5.2?  We use that version on ESXi 5 and 5.1 alonfg with VSC 4 and 4.1.  We use LUNs and the only issue I have is SnapDrive not being able to see some vmdks.  I created a work around by using RDMs.  I'm working with netApp on the SnapDrive issue.

I would consider going back to SMSQL 5.2.

[GGSSUPPORT]

Has anyone a solution for this Problem?

TY. Walter