VSC Permissions Error - Failed to determine privileges for operation

opwnetapp · ‎2014-08-19

Trying to perform a Single File Restore through the vSphere Client and I get the error "VSC Permissions Error - Failed to determine privileges for operation" If I continue I see the error in Recent Tasks "fault.VmfsAlreadyMounted.summary"

I looked in server.log file and can see this record of the action:

2014-08-19 14:06:32,201 [mountBackup:4c64fe956765c337c21ac45c281cd221:] ERROR - FLOW-11010: Operation transitioning to abort due to prior failure.

2014-08-19 14:06:32,201 [mountBackup:4c64fe956765c337c21ac45c281cd221:] DEBUG - Releasing SFR shared lock on VirtualMachine:vm-56992

2014-08-19 14:06:32,201 [mountBackup:4c64fe956765c337c21ac45c281cd221:] WARN - FLOW-11011: Operation aborted

2014-08-19 14:06:32,201 [mountBackup:4c64fe956765c337c21ac45c281cd221:] ERROR - FLOW-11008: Operation failed: Operation failed after invoking mount operation

2014-08-19 14:06:32,201 [mountBackup:4c64fe956765c337c21ac45c281cd221:] DEBUG - Released lock [50182f86-d248-8fda-bdcf-abc4327f96aa] held by 4c64fe956765c337c21ac45c281cd221

2014-08-19 14:06:32,592 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Invoked operation 4c64fe956765c337c21ac45c281cd221 completed in state FAILED

2014-08-19 14:06:32,592 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Found virtual machine ad1 using id 50182f86-d248-8fda-bdcf-abc4327f96aa

2014-08-19 14:06:32,592 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Invoked operation 8d5c8f108176da8b23fd8ff09c849227 and waiting for it to complete

2014-08-19 14:06:32,592 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Invoked operation 8d5c8f108176da8b23fd8ff09c849227 completed in state COMPLETE

2014-08-19 14:06:32,592 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Found virtual machine ad1 using id 50182f86-d248-8fda-bdcf-abc4327f96aa

2014-08-19 14:06:32,592 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Invoked operation 85ec8eb25d61573a657aa75e00749f61 and waiting for it to complete

2014-08-19 14:06:32,592 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Invoked operation 85ec8eb25d61573a657aa75e00749f61 completed in state FAILED

2014-08-19 14:06:32,607 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Starting mount request

2014-08-19 14:06:32,607 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Mounting 2 NFS datastores to 1 ESX servers

2014-08-19 14:06:32,607 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Mounting 1 VMFS datastores to 1 ESX servers

2014-08-19 14:06:32,607 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] WARN - Failed to enable alua for igroup vmware02-esx4.xxx.xxx_smvidg_iscsi on storage system 10.200.200.16. Reason : Incorrect initiator group type, must be FCP

2014-08-19 14:06:32,607 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Rescanning HBAs and VMFS volumes on ESX server "vmware02-esx4.xxx.xxx"

2014-08-19 14:06:32,607 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Renaming datastore "59e49874-0990-4e89-bc0f-c02f39b7aa97" to "vmdsZarafa1 (Backup sfr_backup_Mail_20140807193001)"

2014-08-19 14:06:32,607 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Mount datastore "vmdsZarafa1 (Backup sfr_backup_Mail_20140807193001) (netfs://filer02-nfs.xxx.xxx//vol/vmdsZarafa1_mount_sfr_8161132acec448b29b7cdf2aa81e14e4/qtree)"

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Renaming datastore "cfcc08f8-a344-4f8b-b5e8-f70c1aecbf80" to "vmdsZarafa2 (Backup sfr_backup_Mail_20140807193001)"

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Mount datastore "vmdsZarafa2 (Backup sfr_backup_Mail_20140807193001) (netfs://filer01-nfs.xxx.xxx//vol/vmdsZarafa2_mount_sfr_8161132acec448b29b7cdf2aa81e14e4/qtree)"

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Rescanning HBAs and VMFS volumes on ESX server "vmware02-esx4.xxx.xxx"

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] ERROR - One or more mount requests did not succeed. Please check vSphere Client for any error messages.

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Unmount datastore "vmdsZarafa2 (Backup sfr_backup_Mail_20140807193001) (netfs://filer01-nfs.xxx.xxx//vol/vmdsZarafa2_mount_sfr_8161132acec448b29b7cdf2aa81e14e4/qtree)"

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - Unmount datastore "vmdsZarafa1 (Backup sfr_backup_Mail_20140807193001) (netfs://filer02-nfs.xxx.xxx//vol/vmdsZarafa1_mount_sfr_8161132acec448b29b7cdf2aa81e14e4/qtree)"

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] INFO - EMS/ASUP message was sent to the storage system: filer01-nfs.xxx.xxx

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] ERROR - Operation failed after invoking mount operation

2014-08-19 14:06:32,623 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] ERROR - Failed to mount backup backup_Mail_20140807193001 and attach 2 VMDKs from it.

2014-08-19 14:06:32,639 [attachFileRestoreDisks:87f77ec5f3f3fc00eb7653a9db3aba25:] ERROR - FLOW-11019: Failure in InvokeVirtualDiskAttachTask: Failed to mount backup backup_Mail_20140807193001 and attach 2 VMDKs from it.

java.lang.RuntimeException: Failed to mount backup backup_Mail_20140807193001 and attach 2 VMDKs from it.

at com.netapp.smvi.task.filerestore.InvokeVirtualDiskAttachTask.execute(InvokeVirtualDiskAttachTask.java:69)

at com.netapp.common.flow.TaskInstanceTemplate.execute(TaskInstanceTemplate.java:325)

at com.netapp.common.flow.Operation.executeCurrentStack(Operation.java:133)

at com.netapp.common.flow.Operation.execute(Operation.java:59)

at com.netapp.common.flow.Threadpool$OperationThread.run(Threadpool.java:263)

I can't see a clear reason as too why the operation is failing...

I have tried with different combinations of VMs and they all fail.

Anyone know how to resolve this?

Thanks

Bob

DOMINIC_WYSS · ‎2014-08-19

it tries to mount two nfs datastores and one VMFS datastore. as there is an ALUA failure I guess it's an iSCSI LUN.

the two NFS datastore can be mounted fine, renaming also works. but it fails on the LUN.

fault.VmfsAlreadyMounted.summary is a hint to the VMFS signature:

as the snapshot has the same signature as the source LUN (which is still online), VMware does not mount it. in vSphere 4 there was an advanced option to automatically resignature the datastore.

in vSphere 5 you can do the resignature when mounting the datastore in the GUI.

imho VSC should handle the resignature as you cannot do it in the GUI in this case. maybe a bug or old version of VSC?

which versions of VSC, SFR and vSphere do you have?

opwnetapp · ‎2014-08-20

Thanks for the reply,

That sounds about right. We have a policy that backs up all our mail servers together (2 on NFS, 1 on iSCSI)

I have tried to manually mount the snapshot of the iSCSI LUN using Flexclone and it have failed at that too. Normally it will ask at some point do I want to keep the signature or replace/update it, but now the process just fails.

What is strange is that this all just worked a couple weeks ago during another restore! (Both using VSC and manually) And nothing has changed on the system since then...

We are running:

VSC = 4.2.1

SFR = (Not sure how to find out, but in the components of VSC it says that the Backup and Recovery is version 5.1.1)

vSphere = 4.1.0

I think the answer might be related to why I am seeing a popup error saying "VSC Permissions Error"...

DOMINIC_WYSS · ‎2014-08-20

wow, still vSphere 4 in use

have a look at the advanced options on the hosts: EnableResignature=1

maybe this is enabled on some hosts and disable on other. and it worked weeks ago on another host than this one.

I think the "VSC Permissions Error" is just the default message (or bad error handling). it had enough permissions to mount and rename the other datastores,

so it will also have enough to resignature the VMFS.

opwnetapp · ‎2014-08-20

Unfortunately the EnableResignature=1 option seems only available on the 3.5 boxes...

There is a KB Article ID: 1011441 that states the same thing you suggested, it has a date of June 20th 2014, but it is also referencing a VI3 environment. So I'll keep looking...

I do remember during a SAN upgrade a while back, I had to run some funky command on the VMware box to make sure it "forgot" the old iSCSI connection so it could mount the iSCSI LUN again. (even though it was the same ID, we just swapped filer heads.) I see if I can dig that command up and if it has any use here.

Thanks

Bob

opwnetapp · ‎2014-08-20

So the command I used last time was esxcfg-volume. It fits the bill as to what I think the problem is, but it doesn't list the any LUNs so I can't use it to fix the problem.

I'm still researching, I just wanted to document what that command was for the future.

Bob

opwnetapp · ‎2014-08-20

On my third attempt with a third snapshot, I was successful (but only with the manual method)

~ # esxcfg-volume --list

VMFS3 UUID/label: 4dd57bac-9cde1501-5f67-002219987bfe/vmdsNotes

Can mount: No (the original volume is still online)

Can resignature: Yes

Extent name: naa.60a9800044334650732b445161443366:1 range: 0 - 2037503 (MB)

~ # esxcfg-volume --mount 4dd57bac-9cde1501-5f67-002219987bfe

Mounting volume 4dd57bac-9cde1501-5f67-002219987bfe

Error: Unable to mount this VMFS3 volume due to the original volume is still online

~ # esxcfg-volume --resignature 4dd57bac-9cde1501-5f67-002219987bfe

Resignaturing volume 4dd57bac-9cde1501-5f67-002219987bfe

~ # esxcfg-volume --list

~ #

I got the file restored, but so it is no longer an emergency, but I am still curious as to why I can't do this through the VSC??

DOMINIC_WYSS · ‎2014-08-20

VSC 4.2.1 is rather new (in comparison to vSphere 4.1). maybe it has a bug because Netapp didn't test it anymore with vSphere 4.1 ?

which exact Version ofvSphere 4.1 do you have? as of the compatibility matrix you need at least 4.1U2, as 4.1GA and 4.1U1 are not supported anymore.