SnapDrive, Backups, .aux LUNs

[JCASON]

We have been attempting to analyze and troubleshoot some behavior that has (we think) been happening for a while but never been noticed.

We use NetBackup to backup our hosts.  It appears that NetBackup is aware of SnapDrive, or the NetApp VSS, and one way or another our backup jobs result in a couple of interesting things happening. 

A snapshot is taken of the volume, and a flexclone gets created and mounted.  This makes sense to us but we didn't expect it.

What doesn't make sense to us it that the process seems to create LUNs within the original volume with GUID names and with the extension of .aux.  Sometimes it just creates one.  However, sometimes the system seems to go crazy and create a bunch of these LUNs.  I can't tell if these are LUN clones or actual LUNs, since we only get notified when they are taken offline and destroyed:

This host is running SnapDrive 6.0.2.2416 for reference.  This host also is using SnapManager for SQL.  Here are a couple of the many offline/destroy messages in syslog:

Wed Dec 21 00:31:57 EST [filer: lun.destroy:info]: LUN /vol/volumename/{67bed0ff-e088-4105-a6aa-3569e9760639}.aux destroyed

Wed Dec 21 00:31:57 EST [filer: lun.offline:warning]: LUN /vol/volumename/{f36f3ed4-1708-4268-97f5-b0bd7d81e021}.aux has been taken offline

Wed Dec 21 00:31:57 EST [filer: lun.destroy:info]: LUN /vol/volumename/{f36f3ed4-1708-4268-97f5-b0bd7d81e021}.aux destroyed

Wed Dec 21 00:31:58 EST [filer: lun.offline:warning]: LUN /vol/volumename/{fdeb3761-d848-4a8d-99c7-ae1f796c37ea}.aux has been taken offline

There are 76 LUN destroy messages for this one volume in 1 minute and 12 seconds!  No typo!

Does anyone know what these are for, and why we would be getting so many?  I can't seem to find any reference to .aux LUNs anywhere.  At this point it would be more useful for us to find documentation on the creation of these LUNs instead of fixing the immediate problem with this host...

[aborzenkov]

Those are LUN clones created most likely when host needs to mount snapshot (for backup). This is well known issue with any (backup) software that is using VSS. Unless software provides some means to select VSS provider, it will just use default one, which for NetApp will incidentally be NetApp snapshot … normally they are expected to be removed after backup, but sometimes they stick.

In general the only workaround is to disable NetApp VSS hardware provider before starting backup and re-enable it after that. Ugly and racy.

Have you tried to contact Symantec regarding this issue?

[matthewb]

Aborzenkov is correct - these are being created as part of a VSS-aware backup being executed on the host.  By default, Microsoft VSS selects a Hardware provider first (in this case, NetApp's hardware provider installed by Snapdrive). 

Most VSS utilities have a way of changing which provider they use.  The Microsoft VSS spec actually allows the Requester (the backup utility) to specify which provider to use, or to just use the default.   Most backup utilities just use "default" as the default option (no pun intended).  Most newer versions of NetBackup allow you to specify which provider to use - I am unsure of other backup utilities.  It should be set to use the Microsoft VSS software provider. 

SMSQL (currently) does not use VSS for its backups, it uses VDI calls to SQL, so SMSQL is not the root of those messages.

Matt Brown

NetApp Escalation Engineer - SME/SMSQL/Snapdrive