topic CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable in ONTAP Discussions

CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

CCIE17603 — Thu, 05 Jun 2025 05:40:35 GMT

Hi Everyone,

I have a quick design/technical question regarding running CDOT on a FAS2240 with a single 10Gb Cluster Interconnect/Switchless Cluster Cable...

I understand this is not best practice so please don't read me the riot act. However, technically will it work, and what are the downsides knowing if a port goes bad on either side we will have two nodes that can't communicate with each other in the cluster? I'm just trying to think if you only have 4 10Gb-E ports (2 on each controller), you could use just one from each controller for the cluster interconnect/switchless cluster and one on each controller to go up to the data network. That way you could get I/O into the controllers at 10Gb Speed from from the network on both sides and you can "vol move" over the cluster interconnect, as well as have the heart-beat over the interconnect for disk/controller takeover. I'm just trying to figure out how to make CDOT on a FAS2240 a practical option, because to me wasting your 2x 10Gb ports on each controller just for cluster interconnect/switchless cluster is just that.. a waste. Again if someone can validate the design will work but the caveat is (whatever that may be) thats what I'm looking for. Thanks.

It seems that when you go through the "cluster setup" command it says you need "at least one".. in my mind that means it should be possible. See bold below (I changed the font to call it out but this is what the message says in sequence):

(cluster setup)

Welcome to the cluster setup wizard.

You can enter the following commands at any time:

"help" or "?" - if you want to have a question clarified,

"back" - if you want to change previously answered questions, and

"exit" or "quit" - if you want to quit the cluster setup wizard.

Any changes you made before quitting will be saved.

You can return to cluster setup at any time by typing "cluster setup".

To accept a default or omit a question, do not enter a value.

Do you want to create a new cluster or join an existing cluster? {create, join}:

create

Step 1 of 5: Create a Cluster

You can type "back", "exit", or "help" at any question.

List the private cluster network ports:

At least one cluster port must be specified.

The cluster network ports are the physical ports on the controller that connect

it to the private cluster network. The controller routes cluster network

traffic over these ports by using associated cluster logical interfaces (LIFS).

Examples of cluster network ports in Data ONTAP are "e0a" and "e0b".

You can type "back", "exit", or "help" at any question.

List the private cluster network ports:

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

CCIE17603 — Sat, 15 Mar 2014 18:47:26 GMT

Well, to answer a part of my own question for everyone out there.. this is supported and it will work.

See this document: https://library.netapp.com/ecm/ecm_download_file/ECMP1157168

"If you have an existing two-node cluster that uses cluster network switches, you can apply the switchless-cluster networking option and replace the switches with direct, back-to-back connections between the nodes. This is a non-disruptive operation. The procedure you use depends on whether you have two dedicated cluster-network ports on each controller (as required on most systems) or a single cluster port on each controller (a supported option on FAS22xx storage systems)"

"Each storage system must be using a single dedicated 10-GbE cluster port providing the cluster-network connection. This is a supported configuration option for FAS22xx systems only."

Now I'm just wondering the implications are if the cluster port on either controller fails..

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

scottgelb — Sat, 15 Mar 2014 19:30:00 GMT

On a 2-node cluster there is no Epsilon, so that one controller can keep running if it does storage failover of the partner... but if both controllers keep running and there is a disconnect of the 10Gb single cluster, each node should keep running independently which could be unpredictable. Would need to test it out and see if split brain (although only 2 nodes so each volume still accessible on each node). I rarely recommend 2200 for cDOT...and when we do we prefer both 10Gb interfaces then use it as a 1Gbe NAS system.

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

CCIE17603 — Sat, 15 Mar 2014 21:58:09 GMT

Thanks for your reply Scott.

I was thinking if you can set cf takeover on network interface failure (specifically the cluster interface) at either the cluster shell or the node shell then you could mitigate or reduce your risk.

i.e.) something like this: "cf.takeover.on_network_interface_failure" = on & "cf.takeover.on_network_interface_failure.policy" = any_nic & set the /etc/rc file to include the "nfo" flag.

Or an I confusing 7M and CDOT capabilities?

Lastly, I was originally thinking of this design for VMWare NFS but is there any reason it wouldn't work for iSCSI too since thats IP based? I understand in FC LIFS don't migrate hosts just use ALUA to get to an alternate path, but again iSCSI is IP based so is there any reason on takeover that wouldn't work?

Thanks!

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

scottgelb — Sat, 15 Mar 2014 22:18:08 GMT

Nfo is 7-Mode only. Whole cluster failover if any or all nics with nfo fail depending on the ifconfig

CDot is failover group based so any lif to any interface in any node. If the 10Gb drops and moves to the partner data interface no failover is needed. The lif moves and both nodes still serve data. The concern on a single cluster path is indirect access would not be possible. With iscsi and Alua lifs never move on cDot which is different from 7-Mode which doesn't use Alua for iscsi.

I wouldn't run production on the single 10Gb cluster. Just because indirect access is not available if a single path drops and hosts could lose connectivity. For 2240 I treat it like a 4x1Gb system and if not enough then to a 3220 or 8020.

Sent from my iPhone 5

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

CCIE17603 — Sat, 15 Mar 2014 23:23:34 GMT

I think i get you.. so (setting aside single cable = single point of failure etc.) what you are saying is in NFS if the data facing port goes down the LIF will migrate to the other node and serve data across the cluster interconnect (no cf failover needed) However in iSCSI if you only had a single path ( single target IP from the Host) and the data facing port on the node went down that LIF wouldn't migrate and you are dead in the water.. Correct? I guess my disconnect is couldn't you just set up another LIF on the other node to access the LUN from, and configure multiple IP targets for the LUN at VMWare. Then let the VMWare PSP (Path Selection Policy) move the traffic to the surviving node once it senses its primary path is dead and the data will be served across the cluster interconnect just like NFS?

Thanks for your time, Scott.

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

scottgelb — Sat, 15 Mar 2014 23:40:56 GMT

With iscsi lifs don't move. You would create lifs on each controller then psp would see all paths. Mpio handles the dropped path.

With nfs lifs do move if the interface goes down and the host keeps the same ip connection.

The cluster paths are important for both. If it drops then you could lose access. Although with iscsi it could stay up with mpio recognizing lifs on one side don't see a lun. With nfs the lif would need to move back. But with volumes on both controllers you would lose access on nfs somewhere for that lif.

Sent from my iPhone 5

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

CCIE17603 — Sun, 16 Mar 2014 12:26:53 GMT

Got it. So in summary. There is no reason why this design couldn't work for NFS or iSCSI with VMWare as long as you made the necessary provisions to set up your failover groups correctly for NFS and created LIFS on both nodes to access the LUN in an iSCSI set up. To bring this all home and circle back to what you were originally trying to communicate.. the real danger with this design is if you loose the single inter cluster node connection you could have unpredictable instability.

Thanks for your time, Scott and helping me think through the entire design and its implications.

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

scottgelb — Sun, 16 Mar 2014 14:11:01 GMT

Correct. But for nfs you could lose volumes on one node if the interconnect drops. Iscsi would likely keep working since each lun could be accessed with a local path.

Sent from my iPhone 5

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

CCIE17603 — Sun, 16 Mar 2014 14:20:14 GMT

Yep, I get the importance of keeping that cluster interconnect cable live for NFS. I guess to get that same resiliency out of NFS you would really need to use pNFS which I don't believe VMWare supports today (even in ver 5.5).

Re: CDOT FAS2240 (Switchless Cluster) with a single 10Gb Cluster Interconnect Cable

scottgelb — Sun, 16 Mar 2014 15:22:25 GMT

VMware doesn't support it yet and not sure if they will. They have some other cool stuff coming out that should handle it.

Even with pnfs though one node is the metadata referral server which could cause issues if the cluster goes down but if a local connection to each mount it may be more resilient.

Sent from my iPhone 5