Is there a recommended or a limit on how many NFS mounts should have on a LIF associated with a node?
Recommendation is to spread mounts across multiple nodes/LIFs in the cluster. While you can establish up to 100,000 NAS connections on a single node, there are other resource limits you might hit, such as memory, CPU or exec contexts.
TR-4067 covers it in networking considerations.
@parisi ,
Thanks!
NFS datastores (layer-2) or volumes(layer-3) are heavily used here. NFS mounts for both type seem not evenly distributed across LIFs/nodes. Each node bears 1,200-1,600 mounts.
I have also noticed there are packets discards on some ports within ifgrp group(4x10GbE or 4x40GbE). Some of ports on some nodes have reached as high as 1.x or 2.x%. 0.1% or below could be treated as acceptable, as my understanding.
Should "packets discards" be used as the measurement to determine if LIFs/node have been overly used for NFS mounts, and if there is the need to redistribute NFS mounts? are there any other measurements?
Should we increase ifgrpfrom 4x10GbE to 4x40GbE for those ports have 10GbE, and packets discards?
Packet discards aren't generally caused by "too many NFS mounts."
ONTAP won't discard packets if it's overloaded; instead it uses flow control mechanisms to tell the clients to wait until resources are freed. Packet discards are usually network related issues - bad cables, ifgrp config issues, etc.
I'd suggest opening up a support case to try to narrow down why those are happening.
What issue prompted you to look at packet discards? Where are you seeing those accumulate? Ifstat on the cluster? Packet traces?
We some time have experienced performance issues on some batch jobs running on Linux VMs, Latency is on the datastore or volume only 1-2 ms, We couldn't identify the root cause. That made me to look down the ifgrp and ports, and find packets discards.
I run "node run -node node-name ifstat port-name", and found the "discards/total frame" is higher than expected. As you point out that too many NFS mounts won't cause discards, then what if I increase the bandwidth from 4x10GbE to 4x40GbE, would that improve ? because one of HA pair has 4x40GbE and the discards rate is very low, < 0.1%.
@netappmagic
your issue is solved ?
@parisi
what should I understand by these values? (see below)
ALLCA::*> systemshell -node * sysctl sysvar.nblade | grep -i cid(system node systemshell)
Node: ALLNA1Asysvar.nblade.debug.core.cid_in_use: 226sysvar.nblade.debug.core.cid_max: 115911sysvar.nblade.debug.core.cid_reserved: 10526sysvar.nblade.debug.core.cid_allocs: 33452sysvar.nblade.debug.core.total_execs_blocked_on_per_cid_limit: 6858371sysvar.nblade.ngprocess.rewind.PerCIDRewindContextCount: 11578
Node: ALLNA1Bsysvar.nblade.debug.core.cid_in_use: 230sysvar.nblade.debug.core.cid_max: 115911sysvar.nblade.debug.core.cid_reserved: 10526sysvar.nblade.debug.core.cid_allocs: 101805sysvar.nblade.debug.core.total_execs_blocked_on_per_cid_limit: 2844707sysvar.nblade.ngprocess.rewind.PerCIDRewindContextCount: 115782 entries were acted on.
In a vacuum, not much. You'd want to monitor it over time and see if that sysvar.nblade.debug.core.total_execs_blocked_on_per_cid_limit value increases a bunch. 9.8 and later also has EMS messages if there are large increases. Again, TR-4067 covers it.
