Intercluster traffic using specific interfaces

[swordfish]

We have a source a a destination cluster. Both clusters have a single intercluster lif for each node as of now. They have existing cluster peers with some other existing clusters. We want to setup cluster peer between these two but we want to use new interfaces. We have a stretched L2 connectivity between the two and we want to utilize that. What would be the best approach to make sure that intercluster traffic between the two goes through the new lifs that we create and does not use the existing lifs. Both can reach each other on existing lifs as well.

[TMACMD]

Intercluster networks must be full mesh, meaning each node must be able to contact every other node in the peering relationship on an intercluster LIF.

If each cluster is a 2-node cluster and you have two clusters, you will have a minimum of 2 IC LIFs on the source cluster and 2 IC LIFs on the destination cluster.

You could possibly setup a new IPspace on the clusters and place InterCluster LIFs in the new IP. Remember, that the IC LIFs must all be able to ping each other. If you want isolation, you will need an IPspace, but understand you would need a new port to do that as a port (like a0a-1010 and a0a-1212) can only belong to one IPspace and only one broadcast-domain.

[swordfish]

Thank you @TMACMD 

We have a 6 node cluster at source and 6 node at destination. Currently each node has one IC lif.  Please help me understand this correct:

We should setup new ports, assign them to new IP space at source and destination.

Create additional IC lifs on those ports. 

Make sure that these lifs can reach each other.

How we can make sure that the IC traffic will take the most optimal route? At the end we will be having 2 IC lifs on each node. One will be on the same L2 subnet as the destination and one is existing and has reachability to the destination cluster lifs through WAN. We want to make sure that the IC traffic does not consume the WAN link.

Thanks

[TMACMD]

Sounds like your current setup is good thus far.

It think it might be a good idea to post more info, like

ClusterA has a peer with ClusterB

ClusterB has a peer with ClusterC

ClusterC has a peer with ClusterA

and so on. (only looking for Cluster peering, not vserver peering)

Are peering only between a/b and the cluster peer relationship is already there? or are there more clusters involved?

Just trying to understand the full scope of what you are trying to do. I think you can only have one "cluster peer" relationship between clusters. If you are looking to replace the current ports with new ports it may be that you need to add the new ports to the broadcast-domain, Assign new IPs to the new ports, remove the old IPs, remove the old InterCluster LIFs (delete them) and the remove them from the broadcast domain. Finally, update the cluster peer addresses on both sides.

[swordfish]

@TMACMD 

Cluster A is the source and has existing peer relationships with its remote locations. 6 Nodes and 6 IC lifs.(cannot delete modify these lifs/existing-peers)

Cluster B is the destination and has existing peer relationships with its remote locations. 6 Nodes and 6 IC lifs. (cannot delete modify these lifs/existing-peers)

Cluster A and B can reach each other over the WAN using the existing IC lifs. We need to setup relationship between A and B and don't want to use the existing lifs for this traffic as they both also have stretched L2 connectivity. We want to setup new ports and lifs and want to peer them together and make sure they dont talk to each other using existing lifs over the WAN.

Thanks

[TMACMD]

---

Cluster A and B can reach each other over the WAN using the existing IC lifs. We need to setup relationship between A and B and don't want to use the existing lifs for this traffic as they both also have stretched L2 connectivity. We want to setup new ports and lifs and want to peer them together and make sure they dont talk to each other using existing lifs over the WAN.

Thanks

---

You say ClusterA and ClusterB are already Peered and can reach each other (test with "cluster peer ping").

I am pretty sure you are only allowed one cluster-peer relationship. In fact, I just tried:

Error: command failed: Cannot peer with two clusters of the name "cluster2".

So, if you want to replace the links, you can do that (see my prior post). Otherwise, I may be missing something

[GidonMarcus · Accepted solution]

Hi.

I'm not a big fan of ipspace unless you covered every other possibility. And I think in this case the answer is a bit more simple.

(see here that ipspace is mainly for multi tenancy - also in the intercluster context https://docs.netapp.com/ontap-9/topic/com.netapp.doc.pow-csp/Cluster%20and%20SVM%20peering.pdf) 

As you said you currently moving to an L2 network connectivity -  I assume you don't refer just to the name of the link - and the two clusters all in the same subnet?

In that case, simply set the address with the commands: cluster peer modify -ip-addrs <local> -peer-addrs <remote> As in

https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-cm-cmpr-980%2Fcluster__peer__modify.html&cp=5_1_10_17_2

And test with: cluster ping -destination-cluster <remote> -ttl 1 (ttl specify how many routing hops it allowed travelling)

With that you can prove that you indeed go directly to all nodes (can troubleshoot also with "network traceroute" command and "network test-path" command to compare before and after). You should expect this to exit via the local subnet (see scenario 1 https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/How_does_ONTAP_9.2__select_a_route%3F)

 If there's a difference between the two clusters subnets  - and routing is needed.  I usually create a dedicated route on the cluster vserver  to the specific destination via the gateway of my choice. And either put it in lower metric than the default gateway, or not put one at all. 

https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-nmg/GUID-D14F98C7-0B26-41C1-B6C7-0E8912734CD8.html?cp=12_0_16

Something like: network route create -vserver cluster01 -destination 10.1.1.4/30 -gateway 10.2.1.1  (assuming my remote intercluster LIFs are 10.1.1.5-6). As this is the most specific route - it likely to be chosen over any existing (or one that ever creates) default route (see 3rd link).

[TMACMD]

I too am not a fan of excessive IPspaces. It certainly can overcomplicate things quickly. One scenario that ipspaces can play a role with multiple clusters though is on the InterCluster (IC) LIFs. If I have a scenario:

ClusA <---peer---> ClusB

ClusA <---peer---> ClusC

The "cluster peer ping" command may work from ClusA, but may not work from ClusB. If ClusB and ClusC are unable to communicate, the interfaces must be isolated in IPspaces:

ClusA/ipspace-srcB <---peer---> ClusB/ipspace-dstB

ClusA/ipspace-srcC <---peer---> ClusC/ipspace-dstC

This way, a cluster ping will be isolated to an ipspace that only knows about the appropriate src/dst IC LIFs.

There is a KB on the support site about this (not looking up right now, but I know it is there!)

Without it, SnapMirror gets weird and sometimes works and then it doesnt...until it does again. 

[GidonMarcus]

@TMACMD  WOW, thanks. A bit surprised by that,  as that's not the way NetApp used to talk about IPspace before... (e.g statements like "only needed by service providers", "only if you have conflicting subnets").

I think Iv'e found the KB you're talking about, but most of the content isn't public 😕

https://mysupport.netapp.com/site/article?lang=en&page=%2FAdvice_and_Troubleshooting%2FData_Protection_and_Security%2FSnapMirror%2FHow_to_create_and_c...

And an example:

https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.pow-csp%2FGUID-E88A2094-0A0C-4EA1-B927-534F5ADD0A7A.html&cp=7_1_1_3v

I myself indeed never had the source being pulled from other clusters via separate interfaces hence can't comment much (I guess that was one of the scenarios where good-old fastpath have helped with before).

I do have one more comment for the thread owner @swordfish  to reconsider; maybe stating the obvious, but how about removing the old interfaces and move both the L2 and the routed traffic to go via the new interfaces? (*just maybe make sure to route the traffic out via the local router in each site).

[TMACMD]

In any case, the primary rule:

For each cluster peer,  **all** intercluster interfaces on the source need to be able to ping **all** intercluster interfaces on the destination. The work around is to use dedicated IPspaces on the source and destination if needed if the source and/or destination belong to other cluster peers that its partner does not know about, especially, if they have no route to those IPs.

The "cluster ping" test will only ping intercluster LIFs in the same IPspace.

It is not a good idea to route (in the context of using multiple IC LIFs and multiple cluster peers) **unless** all source IPs can actually ping all destination IPs.