Question about data pathways in a switched cluster

[HUX20002000]

I'm trying to understand how data is able to move between nodes. Let's say we have two NetApps — nodes 1/2 and nodes 3/4. Both NetApps are connected to a pair of cluster switches. The e1a ports on nodes 1/2 are connected to an Ethernet switch. The e1a ports on nodes 3/4 are not connected to anything. An interface is configured for NFS use and is currently active on port e1a of node1. An NFS volume lives on node1.

I can migrate the NFS interface to node 2 and clients will still be able to reach the NFS volume via port e1a on that node. That makes sense.

My question is: if I migrate the volume to an aggregate on nodes 3/4, will a client connected via node 1 -> e1a -> NFS interface still be able to reach it via the cluster switch connection between nodes 1/2 & 3/4?

In other words, is correct to say that as long as there's at least one node with one physical port connected and one interface active on it, the underlying volume can be reachable from any node in the cluster, via the cluster switch network?

(If so, obviously this is not a best practice! Also, I'm only talking Ethernet connections here. I know that Fibre Channel & iSCSI  are different kettles of fish.)

[Sanaman]

You are saying nodes 3/4 not connected any cluster switches, if so nodes 3/4 aren't in the same cluster as 1/2. Hence, volume on nodes 3/4 not reachable to lif on node 1/2.

[TMACMD · Accepted solution]

DBLADE and NBLADE

DBLADE is where the DATA lives

NBLADE is where the Network connection point is.

In ANY ONTAP cluster with more that one node you may have a path called Indirect I/O

If a VOLUME (DBLADE) is on Node 1 and the NETWORK (NBLADE) is also on Node 1, you have DIRECT I/O.

But is the VOLUME(DBLADE) is on any other node except Node 1, the Network (NBLADE) on Node 1 needs to communicate with the other nodes to get to the data. This is Indirect I/O.

So, if my data lives on node 2 (or node 3 or node 4), the client will talk to Node 1 (that is where the network connection point/ NBLADE is) and the cluster will send the request to the appropriate DBLADE (to the node holding the volume) over the Cluster network.

In a 2-node switchless cluster, this is on the point-to-point connections.

In a 4-node or larger cluster, this request goes through the cluster switches.

If you were to migrate the LIF (NBLADE) to the same node where the volume (DBLADE) is, you eliminate the indirect I/O.

With all that said, Many tests and studies have been performed by NetApp over the years. A long time ago, it was a best practice to have nearly one LIF for EACH volume! Then if you moved a volume, you could move the LIF. Testing however, showed that the latency for the indirect I/O is very minimal and the new best practice is to simply have ONE LIF per Node per SVM (sometimes one node per protocol per node per SVM, but that is for later). Indirect I/O is bound to happen. Avoid it if you can, but it is not a showstopper if you don't!

[HUX20002000]

Thank you! This was exactly the kind of detailed info I was hoping for.

Also, for anyone else who finds this thread, this was a useful page (which can be roughly summed up as "TMACMD's post, with diagrams").