Filer-initiated network connections egressing on cluster_mgmt lif

[FULLSTEAM]

dc2-nc1::*> network interface show -vserver dc2-nc1
            Logical    Status     Network            Current       Current Is
Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home
----------- ---------- ---------- ------------------ ------------- ------- ----
dc2-nc1
            cluster_mgmt up/up    10.20.8.70/21      dc2-nc1-node1 a0b     true
            mgmt1        up/up    10.20.8.71/21      dc2-nc1-node1 a0b     true
            mgmt2        up/up    10.20.8.72/21      dc2-nc1-node2 a0b     true
3 entries were displayed.

dc2-nc1::*> network route show
Vserver             Destination     Gateway         Metric
------------------- --------------- --------------- ------
dc2-nc1             0.0.0.0/0       10.20.8.1       20

dc2-nc1::*> network route show-lifs

Vserver: dc2-nc1
Destination             Gateway                 Logical Interfaces
----------------------  ----------------------  ------------------------------
0.0.0.0/0               10.20.8.1               cluster_mgmt, mgmt1, mgmt2

I had a 2-node filer on 9.1.  I had jobs doing backups (system configuration backup settings show) that would PUT to a webserver, and I had ASUPs going to NTAP over our proxies.  Those connections would emerge from the node_mgmt IPs (I'm positive about this, as only .71 and .72 were on certain ACLs), and all was well.

As of going to 9.3 (finally), I'm seeing ASUPs from node 1 being denied by the proxies because they're egressing from .70, the cluster_mgmt LIF.  Node2's filer-initiated connections come out of the node LIF's IP, .72, as I would expect.

I'm not seeing any particular advice on this.  Not seeing something in the release notes for 9.2/9.3.  I found a troubleshooting-ASUP doc that makes it sound like connections are expected to emerge from the cluster_mgmt LIF, but it's unclear.

My questions are:

• Should ASUPs/uploads be initiating out of the cluster_mgmt LIF in 9.3-and-beyond?
  • If so, can you show me what changed between 9.1 and 9.3 so I can learn from my previous expectation?
  • If not, any advice on what I should tweak in the routes so the node LIF is preferred over the cluster LIF in sourcing connections?

Thanks!

[aborzenkov]

What happens if you move cluster management LIF to another node?

My guess is that simply takes first interface on network with (default) gateway (where "first" is in some internal kernel order of creation). 

[paul_stejskal]

That's odd...

I know with changes in 9.2+, we removed part of the network stack to optimize it, but that also removed IP fastpath. As far as I know it is supposed to go out the node management LIF, not cluster LIF.

I mean the simple solution would just be to add a route to your proxy out the node management LIF, or modify the routing table. Or you could modify your proxy to allow the cluster management LIF.

If you really want a detailed dive, if you have support entitlements I'd suggest opening a case. I couldn't pull up the ASUPs searching for that node name so we'll need to probably pull logs and see what is going on.

[TMACMD · Accepted solution]

Expected behavior. Management traffic is allowed to go out any node or cluster-mgt interface. Your ACLs should include every node management, cluster management and service-processors (SP) or baseboard management controllers (BMC) IP addresses. 

[FULLSTEAM]

Thanks for looking, y'all.

To aborzenkov:

• cluster_mgmt lif on node 1: traffic coming out .70 (the cluster_mgmt lif) and .72 (node2).
• cluster_mgmt lif on node 2: traffic coming out .71 (node1) and .70 (the cluster_mgmt lif).

To paul_stejskal:

We did add the cluster_mgmt lif to the proxy ACLs as a workaround, because we needed ASUPs to fly for a case (side note, I can't believe burt 1156898 is not getting fixed in 9.3).  This question was mostly to determine whether my proxy edit was a 'temp workaround for a misconfigured filer' or if this was an intentional change in ONTAP and my proxy change needed to be made permanent.

I assume y'all can talk internally and reach a consensus, but I'm going to assume here that TMAC_CTG's answer is correct vs paul_stejskal's 'huh that's weird'  (sorry!).  I wish I had a cite or I had spotted this in some kind of changelog, but, oh well, I'm happy with someone telling me it's expected.

Thanks for the replies.