Ask The Experts

ONTAP Select 2.10 multi-node cluster and vswitch setup for KVM

drzauis
2,992 Views

Hi, I am working on setting upo Deploy 2.10 and 9.5RC1 on a box with 48G of RAM and six (6) 1.5 SSDs using KVM.  Setup of everything was very smooth until I went to deploy a multi-node ONTAP cluster and I found the networking I followed works for a single-node cluster, but is insufficient for a multi-node cluster.  I have read tr-4613.pdf and I have two (2) 10 Gbps NICs configured and followed the instructions for setup one (1) vswitch as follows:

 

[root@select-3 tmp]# systemctl status openvswitch
openvswitch.service - Open vSwitch
   Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; disabled; vendor preset: disabled)
   Active: active (exited) since Tue 2018-10-16 16:20:12 PDT; 2s ago
  Process: 6530 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 6530 (code=exited, status=0/SUCCESS)

Oct 16 16:20:12 select-3 systemd[1]: Starting Open vSwitch...
Oct 16 16:20:12 select-3 systemd[1]: Started Open vSwitch.

[root@select-3 tmp]# echo "SUBSYSTEM=="net", ACTION=="add", KERNEL=="ontapn*", ATTR{tx_queue_len}="5000"" > /etc/udev/rules.d/99-ontap-txqueuelen.rules
[root@select-3 tmp]# cat /etc/udev/rules.d/99-ontap-txqueuelen.rules
SUBSYSTEM==net, ACTION==add, KERNEL==ontapn*, ATTR{tx_queue_len}=5000
[root@select-3 tmp]# echo "KERNEL=="ontapnic*", RUN+="/sbin/ip link set %k txqueuelen 5000"" > /etc/udev/rules.d/98-ontapnic.rules
[root@select-3 tmp]# cat /etc/udev/rules.d/98-ontapnic.rules
KERNEL==ontapnic*, RUN+=/sbin/ip link set %k txqueuelen 5000

[root@select-3 tmp]#

[root@select-3 ~]# ip link set ens2f0 mtu 9000 up
[root@select-3 ~]# ip link set ens2f1 mtu 9000 up
[root@select-3 ~]# cd /etc/sysconfig/network-scripts
[root@select-3 network-scripts]# cp ifcfg-ens2f0 ifcfg-ens2f0.BAK[root@select-3 network-scripts]# cp ifcfg-ens2f1 ifcfg-ens2f1.BAK[root@select-3 network-scripts]# cat ifcfg-ens2f0
DEVICE=ens2f0
TYPE=Ethernet
MTU=9000
ONBOOT=YES
[root@select-3 network-scripts]# cat ifcfg-ens2f1
DEVICE=ens2f1
TYPE=Ethernet
MTU=9000
ONBOOT=YES
[root@select-3 network-scripts]# cat ifcfg-ontap-br
DEVICE=ontap-br
DEVICETYPE=ovs
TYPE=OVSBridge
MTU=9000
ONBOOT=yes
BOOTPROTO=static
[root@select-3 network-scripts]# cat ifcfg-bond-ontap
DEVICE=bond-ontap
DEVICETYPE=ovs
TYPE=OVSBond
MTU=9000
OVS_BRIDGE=ontap-br
BOND_IFACES=bond-ontap
OVS-OPTIONS=bond_mode=balance-slb lacp=active other_config:lacp-time=fast
ONBOOT=yes

Create the bridge on the vswitch and use LACP mode:

select-2 #ovs-vsctl show

97f6ad21-52d5-46be-8c86-d97484811900   

ovs_version: "2.0.0"

select-2 #ovs-vsctl add-br ontap-br

select-2 #ifdown  ens2f0Device 'ens2f0' successfully disconnected.

select-2 #ifdown  ens2f1Device 'ens2f1' successfully disconnected.

select-2 #ovs-vsctl add-bond ontap-br bond-ontap ens2f0 ens2f1 bond_mode=balance-slb lacp=active other_config:lacp-time=fast

select-2 #ifup  ens2f0Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7)

select-2 #ifup  ens2f1Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/8)

select-2 #ovs-vsctl show

97f6ad21-52d5-46be-8c86-d97484811900   

Bridge ontap-br       

Port bond-ontap           

Interface "ens2f0"           

Interface "ens2f1"       

Port ontap-br           

Interface ontap-br               

type: internal   

ovs_version: "2.0.0"select-2

 

My questions are:

1.)  Do I need to create entries in my vswitch setup for the various vNICs and VLANs (ontapn-2 through ontapn-5) and everything will work or is there something else to do?

2.)  Is there a step-by step example I can look at of someone who has a 2-physical NIC setup using KVM?

 

Many thanks!

5 REPLIES 5

RyanUrice
2,895 Views

When you try to deploy a multi-node cluster is it giving you any kind of error? It looks like it should work from what I see here.

 

Also, take a look at this step by step demo video: https://www.youtube.com/watch?v=Jh-YCrSkfnk

drzauis
2,842 Views

Ryan,


Thanks for the reply - in looking at this, I failed to overlook something (maybe you can confirm) - it was unclear to me that the physical switch ports needed to be tagged with VLANs in use in the Select Clusters - I was thinking it was only needed in the virtual switch setup, but when spanning multi-nodes across multiple physical hosts, physical VLAN tagging makes sense.  I will work with my networking team to ensure we develop a standard and get the correct VLANs tagged on the physical switches.

 

Thanks!

RyanUrice
2,832 Views

Sounds good. If you have any additional issues feel free to open up a support case and we can look further into it. 

drzauis
2,825 Views

Ryan,

 

One follow-up question for clarification - in TR-4517 (p. 42, Figure 13) shows the attached graphic - do the non-routable/internal LIFs need to be hosted on separate physical NICs than the external LIFs?  I have only two 10Gbps NICs per physical host and I thought I could accomplish this with VLAN tagging - just wanted to confirm.

 

Thanks!p42_fig13_multinode_network.JPG

 

 

RyanUrice
2,821 Views

I think so but I am not 100% sure. Let me confirm this and I will get back to you. 

Public