topic Re: Direct NFS for multiple Esxi in ONTAP Discussions

Direct NFS for multiple Esxi

AllanHedegaard — Sat, 08 Jun 2019 17:39:09 GMT

I would like the community advise on the correct route for connecting 2 esxi servers to an AFF using sfp+ without the use of a L2 switch. Goal is to have the same NFS datastore mounted on both servers.

Direct connection from each server NIC to a port on each controller. This is to provide redundancy in case of link or controller failure.

Is one subnet possible for multiple ports, so the same mount IP can be used from different hosts, or should the esxi hostfile be used for allowing different mount IP but same DNS name?

I really cant be the first to encounter this scenario and would like to know what is the recommended path.

Thanks

Re: Direct NFS for multiple Esxi

aborzenkov — Sun, 09 Jun 2019 03:07:09 GMT

NFS high availability is based on failing over IP to different physical port. This requires L2 switch and won’t work with direct connection.

Re: Direct NFS for multiple Esxi

AllanHedegaard — Sun, 09 Jun 2019 05:59:14 GMT

Thanks for your reply. I am not sure what you exactly mean by high availability NFS?

I am just talking about exposing LIF home and failover port to each server. Esxi support beacon probing. I am running a similar setup today without switches.

For smaller setups a 10g switch is not necessarily required. Only the amount of physical ports sets the limit.

Re: Direct NFS for multiple Esxi

SpindleNinja — Mon, 10 Jun 2019 01:02:36 GMT

For direct connect, you're best off going with iSCSI.

While I think it would work in the sense you'd be able to mount the datastores, you'd have issues during failover, which is what aborzenkov was saying.

Though, random question, what version on NFS were you planning to use?

Re: Direct NFS for multiple Esxi

aborzenkov — Mon, 10 Jun 2019 05:16:52 GMT

ESXi beacon probing relies on L2 connectivity between all physical ports in NIC team that you do not have here. I am not aware of any automatic mechanism to detect LIF move and redirect traffic to another port (in direct attach case).

Re: Direct NFS for multiple Esxi

AllanHedegaard — Mon, 10 Jun 2019 05:54:25 GMT

I am not sure the actual NFS version should matter in this case, as multipathing is not supported. So I would go v. 3 for simplicity purposes.

Consider one server:

One data LIF (a1,b1) could be connected directly to the server nic. If a1 fails the lif will migrate to b1.

I would just use multiple lif pairs for multiple servers.

Re: Direct NFS for multiple Esxi

AllanHedegaard — Mon, 10 Jun 2019 06:05:39 GMT

Just had a thought. Why not create a LACP between the two pair of ports. Would that be possible?

I know VM support port aggregation. Can I span such a 'port-channel' to both Netapp controllers? I am not looking after active/active. Just failover capability.

Re: Direct NFS for multiple Esxi

aborzenkov — Mon, 10 Jun 2019 06:14:05 GMT

@AllanHedegaard wrote:

Can I span such a 'port-channel' to both Netapp controllers?

No. ifgrp can include only ports in the same controller.

Re: Direct NFS for multiple Esxi

AllanHedegaard — Mon, 10 Jun 2019 06:26:55 GMT

Then assigning a failover adapter in vmware would be the best way to go. It will not provide load balancing, but failover cability if the actual link fails. Such an event would also move the lif to the secondary port. Correct?

Re: Direct NFS for multiple Esxi

AllanHedegaard — Tue, 11 Jun 2019 17:21:30 GMT

I can confirm that this works. A lot of money saved on Nexus and not any drawbacks from my point-of-view.

Re: Direct NFS for multiple Esxi

walterr — Fri, 22 Jan 2021 14:42:25 GMT

Could you please elaborate more on how you configured this?

How is it possible that the 2 ESX Servers can connect to the same NFS-Export with the same IP, if you do not have a layer 2 switch in between. Did you configure different NFS target IPs for the ESX hosts? Can ESX connect to the same datastores to different NFS IPs?

How did you configure the LIFs on the NetApp side and how did you configure the kernel ports on ESX?

Re: Direct NFS for multiple Esxi

AllanHedegaard — Fri, 22 Jan 2021 15:12:20 GMT

Just use the hostfile on the ESXi hosts. That way you can mount the same NFS share using same hostname, but by using different IPs on the SVM. Works fine 🙂

Re: Direct NFS for multiple Esxi

walterr — Fri, 22 Jan 2021 15:32:39 GMT

Still not sure what you mean.

Following example:

lets say I have a LIF with IP1 on e0c, and another LIF with IP2 on e0e exporting the same datastore.

Host has one port from ESX1 directly connected to e0c and one port from ESX2 directly connected to e0e.

So how can you configure in ESX1 to connect to IP1 and ESX2 to connect to IP2 if this is an ESX cluster? If I connect ESX1 to IP1, doesn't it automatically connect ESX2 also to IP1?

Re: Direct NFS for multiple Esxi

AllanHedegaard — Fri, 22 Jan 2021 15:57:22 GMT

By using the hostfile on esxi, you dont mount using IP, but by using hostname of SVM.

Re: Direct NFS for multiple Esxi

walterr — Mon, 25 Jan 2021 13:45:28 GMT

OK, I understand. I will try to implement this and will let you know if this is working.

Re: Direct NFS for multiple Esxi

jcolonfzenpr — Mon, 25 Jan 2021 18:44:59 GMT

Its work, but i think it's a unsupported configuration beware maybe netapp support won't help if you have any issues 😉

Re: Direct NFS for multiple Esxi

AllanHedegaard — Mon, 25 Jan 2021 19:17:25 GMT

I have had it working fine on +5 installations for more than 18 month now. We are very happy with the low latency and cost savings on the 10gbe switch 🙂

Re: Direct NFS for multiple Esxi

walterr — Fri, 29 Jan 2021 08:18:56 GMT

How exactly did you configure the failover of the ESX kernel port? I tried this configuration now, and it is working, but I have the problem that during a NetApp storage takeover the ESX loses its connection to the IP. Obviously it is still trying to connect through the kernel port which is connected through the controller which has rebooted and is waiting for giveback. After giveback everything is working again.

Re: Direct NFS for multiple Esxi

AllanHedegaard — Fri, 29 Jan 2021 08:24:40 GMT

Assign an active and passive physical port in the ESXi to the vSwitch. This way in case the active port goes down, it will switch to the passive.

Of course this is not bulletproof, as there are different states of 'going down'. This will only cover the physical link connection.

But to be honest for me it doesn't matter - I have 6 ESXi hosts connected, and simply use the failover capability of VMware HA. Of course it has never been needed.

Looking at a risk perspective, it is more likely that a volume runs full, and stops IO.

Re: Direct NFS for multiple Esxi

walterr — Fri, 29 Jan 2021 08:38:59 GMT

The thing is that during a Storage failover the active ports on controller 1 physically only go down shortly, and after boot of controller 1 when it goes to waiting for giveback state, the physical port goes up again, but the IPs are still on controller 2. So how would the failover kernel ports on ESX recognize this?