ONTAP Select 2.6 : cluster create failed (Invalid/missing disk 'scsi0:0')

[dangreau]

Hi all, 

As I've stated in other post (cf https://community.netapp.com/t5/Data-ONTAP-Discussions/ONTAP-Select-Cannot-create/td-p/131724 ) I'm in a process of evaluating  Ontap Deploy 2.6.

But each time I try to deploy a simple "single node evaluation cluster " , the cluster deployment fails with same message : ".... create failed. Reason: Error during configuration of vnics:  InvalidRequestFault." 

Esxadmin.log contains some errors:

[INFO] [Thu Nov 23 05:43:31 2017] [8815] Configuring Network ...
[INFO] [Thu Nov 23 05:43:31 2017] [8815] Error encountered while creating VM
[INFO] [Thu Nov 23 05:43:31 2017] [8815] [13] InvalidArg: <scsiC:N> invalid - Invalid/missing disk 'scsi0:0'
[ERROR] [Thu Nov 23 05:43:31 2017] [8815] [51] Err: Error during configuration of vnics: InvalidRequestFault

Our Vmware environment is as follows:

• vsphere (vcenter managed) version = 6.0.0 build 4510822 
• Vcenter version : 6.0.0 3634793
• Each tested host has 2 Distributed vswitch. One with 2*40G uplinks, and one with 2*10G uplinks. Both uplinks are configured in LACP mode
• Ontap Select networks are using 2 distributed port group: one for external/management network, and one for internal (only used for HA pair) . Both distributed PG comes from same distributed vswitch (the one using 2*40G LAG)
• local datastore tested comes from 2 local SAS disks, grouped in a RAID 1 LV managed by a Smart Array P840ar.
• SAN datastore tested comes from external LUN accessed by Fiber channel
• all our hardware stack (servers, san switchs, external array, bios firmware, etc ...) is fully supported by Vmware and in the compatibility matrix. 

FYI, below  is an example of CLI commands I used for a "single node eval cluster deployment " : 

• host add --host-id ESXHost01.lan--username vsphere.local\\myuser --password mypassword --vcenter VCenter01.lan
• host configure --host-id ESXHost01.lan --location MyLocation --storage-pool SAN_Datastore_01 --eval --management-network NFSDATA_PG --data-network NFSDATA_PG --instance-type small --capacity-limit 3072
• cluster create --name ONSCluster01 --cluster-mgmt-ip A.B.C.D --netmask 255.255.255.0 --gateway A.B.C.E --ontap-image-version 9.2P1 --node-names ONSNode1 --node-hosts ESXHost01.lan --node-mgmt-ips A.B.C.F

where :

• ESXHost01.lan is the ESX host where I want to deploy Ontap Select Node n°1. This ESX is part of a Vsphere Cluster (this ESX, and the associated Vsphere Cluster, is also a production one, with other working VM on it)
• VCenter01.lan is the Vcenter managing ESXHost01 (this Vcenter is a production one, managing many ESX clusters )
• SAN_Datastore_01  is the Shared storage datastore (from an external array accessed by FiberChannel) 
• NFSDATA_PG is the Distributed PortGroup to be used (this DPG is a production working port group, with other working VM using it)
• ONSCluster01 is the the Ontap Select cluster name (the name is already registered in the .lan DNS domain) 
• ONSNode1 is the the Ontap Select cluster name (the name is already registered in the .lan DNS domain) 

What we have tested so far:

• We have tested Ontap Deploy 2.5 and 2.6 without any improvement 
• wa have tested with Local Datastore and SAN Datastore without any improvement 
• we have tested from GUI and from CLI without any improvement 

We don't know what to test anymore and are about to give up with Ontap Select (and replace it with a non Netapp solution). So if someone has a new idea it is the moment to tell us 🙂 

Thanks. 

Regards

Pierre. . 

[SeanHatfield]

Do you get any different results if you use the administrator@vsphere.local account?  It looks like the initial push of the vm template is succeeding, but subsequent api operations on the virtual machine are not.  RBAC is a shot in the dark, but worth a quick test.  Another thing you can try is the OVA version of the single node select evaluation.  That would prove that Select itself works, and the issues you are facing revolve around the deploy tool and the api calls it makes into vcenter to build the VM.

[dangreau]

Thanks @SeanHatfield for your answer. 

I'm not allowed to use "administrator@vsphere.local" account (for security concerns) but I have full right so I created other local admin account on the local SSO domain (the "vsphere.local" one)  with the same result as with a more "classical" account (an Active directory integrated one). I've added this local account to the "administrators" group (and with the full Admin role) so this user is able to do anything on the host (and vcenter) . I've manually tested to deploy a VM from a custom ovf template, with success. 

As I wanted to test a fully supported solution (before to buy it) I had not tried to deploy Ontap Select from OVF version. But as you've asked for it, I've done the test, and installed "ONTAP® Select 9.3RC1 Evaluation OVF Template" (from ontap evaluation software page). 

With this OVF template, i've successfully been able to deploy single node by using the local admin account mentionned above, on a local DAS datastore (the one tested with Ontap Deploy) and also on a SAN shared storage datastore (the one tested with Ontap Deploy). 

Here is the screenshot of the successfull local DAS storage Ontap Select deployment  just after successfull deployment (without any modification in system manager):

And here is the snapshot of the successfull deployment on the SAN Datastore just after successfull deployment (without any modification in system manager):

I also checked at VM configuration after OVF deployment and there are some differences with OnTap Deploy deployment :

• 3 vnic (all in same PG)  with OVF vs 6 vnics (all in same PG) with Ontap Deploy
• 4 vdisks with OVF vs 1 vdisk with Ontap Deploy

Conclusion

1. It is technically better than before, as I've got a successfully deployed single node cluster 🙂

2. but I can not really test further the node deployed this way as it is an unsupported solution 😞

Regards. 

Pierre. 

[dangreau]

Anyone from Netapp to try to make us work Ontap Select in our environment before we definitely give up ? 

As there is no support from Netapp (except on Community, but without any luck in our case) for Ontap Select Evaluation and as we can not buy a Ontap Select licenced before having first validated it in our environment, it seems we are stuck and that Ontap Select solution is currently not a viable option for us 😞

[SeanHatfield]

I tried to reproduce the error in my lab, but it deployed successfully on the first go.  VSphere 6.0, dvswitch only configuration.  Anything else notable about your config?  Have you reached out to your NetApp account team?

[dangreau]

We did not contact our Account team as it is stated that Ontap Select Evaluation mode is without any official support and as our team is not really OK with Ontap Select (they tells us it is not perfectly ready for production and advice us to use instead a more classical FAS box) 

I really don't know what could be wrong with our DVS and DPG configuration. Here it is:

DVS Topology and LACP configuration

Here is the DPG configuration 

Regards. 

[SeanHatfield]

Thanks for the screenshots. I’ll see if those security restrictions will replicate the issue. 

Edit: Those are all the dvswitch defaults, and match what I have here.  What are the MTU settings?

[dangreau]

We are using 9000 MTU :