Vvols with VSC and VASA - Duplicate vvol names

[echolaughmk]

Hello,

I have been able to provision through the VP a few NFS vvols that get created on the array, but throw an error at the end (about VP error) and never mount to the hosts. I am unable to find them in inventory or through any esxcli vvol command in order to remove them, yet when I try to recreate another vvol of the same name it throws me an error that it already exists.......even when I don't see it in the web client and I have removed it from the array (because it failed to provision). This leaves me thinking there is a process needed to remove any remnant of the failed provisioning....but I am unable to query and fully remove these volumes....is there a process/way to do this either through VSC or the Web Client or somethinng like removing through MOID?

thanks!

[asulliva]

You can try using the VASA Provider's web interface.  Browse to https://your.vasa.provider:9083/ and login using the vpserver user (you set the password during appliance deployment).

Once there, click on the "Web Based CLI Interface Link":

In the page that opens, type "cluster list" in the input box:

You should see something like this:

Assuming it's not correct, in the command text box put "cluster rediscover" then hit the execute button.  This may take a few minutes depending on the number and size of the clusters.

Afterward, do the "cluster list" again to check for accuracy.

Hope that helps!

Andrew

[echolaughmk]

Thanks asulliva for the response. I tried those commands and they execute correctly, but I still see the old volumes that failed to provision in the output after the rediscover - is that normal? Or should I see them removed from the list since their NTAP array LUNs have been deleted and they never fully provisioned in the first place?

If I try to use the web client again after the rediscover, it still complains about a duplicate name so I'm not sure if there is another way to force a refresh of the VP to query the array and remove any remants?

Thanks!

-Keith

[asulliva]

Assuming there are no other VVol containers on your environment using the same cluster/SVM, try removing the cluster then re-adding it.  To remove the cluster use the VASA Provider web cli and issue the command "cluster delete -cluster_ip=w.x.y.z" command.  To re-add the cluster, use the command "cluster add -cluster_ip=w.x.y.z -username=admin -password=secret".

Andrew

[echolaughmk]

Thanks!

I do have one vvol that worked and i put a VM on it to test with. I am guessing that I will want to delete all of that and tear it down before running those commands? I can do that as it is just testing, but wasn't sure if there might be another way to forcefully remove the bogus volumes that show up in that list command? Those are where it seems like the hanging references are and not sure if they can be purged. 

thanks!

-Keith

[echolaughmk]

Hi, 

I figured I would try to include a graphic to help represent what I am seeing. The cluster list command shows me all of the vvols - deleted and online. The ones that show deleted are the ones either failed provisioning through the plugin or that I removed from the array manually...so it looks OK there. However I think the issue might be around the storage containers because if I list these out from the plugin's perspective, it shows the ones that are giving me the naming conflict as offline, but still in the list even though their backing storage is deleted on the array. The ones that show offline are the ones that the Web Client thinks still exist and throw a name conflict with when I try to provision a new datastore:

So if I try to reuse the name for any of the containers in the list that show as offline, that is where my issue appears to be. Since they don't exist on the array through and since we did a rediscover, how come they aren't cleaning themselves up? Or is there a manual way to clean these containers up either on the VMware side or on the NTAP side so they don't report like this?

thanks!

-Keith

[asulliva · Accepted solution]

Containers are separate from the FlexVols which back them.  Containers exist on the VASA Provider, the ONTAP itself has no knowledge of them.  You can remove a container using the web cli and the command "container delete -container_name=some_container_name".

Alternatively, you could manually add FlexVols to the container, then add it back into your vSphere environment.  From the VASA Provider web CLI, you'll need to do a series of operations:

• Use "container delete_storage -container_name=CONTAINER_NAME -svm_name=SVM_NAME -flexvol_name=VOL_NAME" for each of the invalid FlexVols in the container
• Create at least one valid FlexVol
• Rediscover the cluster, "cluster rediscover"
• Use "container add_storage -container_name=CONTAINER_NAME -svm_name=SVM_NAME -flexvol_name=VOL_NAME" to add the valid FlexVol

From the vCenter Web Client, at the root for your vCenter, browse to the manage tab, then select "Storage Providers".  Highlight the "NetApp-VP", then click the disk with a green line under it to refresh the VP.

Next, you need to add the VVol datastore back in.  Select your host or cluster, right click and select Storage -> New Datastore.  In the popup, click next, then choose "VVol" for the datastore type.  After thinking for a moment, it will give you a list of available containers to mount.  Select the one you want, click next, choose the host(s) to mount it to, and click finish.

Andrew

[echolaughmk]

Thanks Andrew! We are getting there. I was able to delete the empty flexvols and containers from the list, however with one of them that went inaccessible and the VM was removed (and assumed the vvols were too) it looks that they weren't and the provider thinks that the VM that originally was create on this vvol datastore (CMvvol) is still there according to the outputs:

It seems to think that the VM that has been removed (keith-w2k12) still has vvols on the container and the VM is nowhere in vcenter inventory that I can see:

So what I found was that I could look at the inventory (snippet above) and I found the "api deletevirtualvolume -vvol_uuid" command to manually remove all of the dead vvols that the VP thought were still out there on the DOA container:

I repeated that process for the other 2 leftover vvols and then I was able to remove the 2 flexvols from the container and then delete the container itself (container delete -container_name=CMvvol). 

Once I did all of this, I refreshed the VP from the Web Client and no longer see the duplicate name issue when trying to provision a new vvol datastore 🙂

Definitely could have been easier with plugin integration to see and query and manipulate these object better rather than executing all of these commands, but in the end it worked. Thanks for all the help and I bet lots of people will get usage from this once they start to play with vvols. 

Thanks.

-Keith