topic Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO in ONTAP Discussions

New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Wed, 04 Jun 2025 18:42:54 GMT

I've just added two new nodes to our production cluster. I've added two new iSCSI LIFs on the nodes and added those LIFs to the port group. I've used SnapDrive to add connections to the two new iSCSI LIFs on the first of many Windows hosts. After doing this, I checked the Data OnTAP DSM for MPIO software to be sure it was showing 4 nodes per LUN. It is not. It is still just showing two paths. Any ideas as to what I am missing?

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

bobshouseofcards — Mon, 10 Oct 2016 20:09:01 GMT

Assuming you are running cDoT 8.3 or higher, you are likely encountering the new "Selective LUN Mapping" feature. This feature was added in cDoT 8.3 to address the multiplication of paths across large clusters that can impact server based total and per LUN path limits.

From the SAN Administration Guide...

Selective LUN Map

Selective LUN Map (SLM) reduces the number of paths from the host to the LUN. With SLM, when a new LUN map is created, the LUN is accessible only through paths on the node owning the LUN and its HA partner.

SLM enables management of a single igroup per host and also supports nondisruptive LUN move operations that do not require portset manipulation or LUN remapping.

Portsets can be used with SLM just as in previous versions of Data ONTAP to further restrict access of certain targets to certain initiators . When using SLM with portsets, LUNs will be accessible on the set of LIFs in the portset on the node that owns the LUN and on that node's HA partner.

Beginning with Data ONTAP 8.3 SLM is enabled by default on all new LUN maps. For LUNs created prior to Data ONTAP 8.3, you can manually apply SLM by using the lun mapping remove-reporting-nodes command to remove the LUN reporting nodes and restrict LUN access to the LUN owning node and its HA partner.

I ran into the same situation myself when adding nodes 5 and 6 to a cluster, moving some LUNs there, and then not seeing direct paths. Mine was FCP, but since this applies at the LUN mapping level it would be just as applicable to iSCSI as well.

The commands of interest to you are:

lun mapping show -field reporting-nodes -vserver <SVM> -path <lun-path>

lun mapping add-reporting-nodes ...

lun mapping remove-reporting-nodes ...

SLM provides for very specific path control and in conjunction with portsets can limit path proliferation especially as the number of potential ports increases and the number of allowed hosts in a SAN cluster increases with refreshed hardware and ONTAP 9.1.

Hope this helps.

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

Kudos and accepted solutions are always accepted.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Mon, 10 Oct 2016 20:20:58 GMT

Very helpful! Thank you very much Bob.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Mon, 10 Oct 2016 20:24:43 GMT

Bob, one more question: from what I can tell the -aggregate and -volume options are optional. I simply want each LUN in a given SVM to have access to two new nodes in the cluster, regardless of aggregate. Am I correct that I can leave these switches off? Also if I enter a common path name with a wildcard for the -lun option, will that work? For example:

lun mapping add-reporting-nodes -vserver <vserver> -path <Server_Name>_* -igroup <Igroup-Name>

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

bobshouseofcards — Mon, 10 Oct 2016 20:56:16 GMT

Your comand fragment is enough to identify the particular LUN/igroup combinations of interest. You'll need one of the four optional parameters to select which nodes to add to the list...

-local-nodes [ true ] adds the nodes where the LUN currently exists to the reported path map.

-destination-aggregate <aggregate name> adds the nodes where the listed aggregate lives. Typically used before a volume would be moved to a new aggregate. Could also be used with the name of an aggregate on your new nodes - nothing says the volume has to be moved there.

-destination-volume <volume name> adds the nodes where the listed volume lives. Typically used before a LUN is moved to a new volume (lun move start ...). Could also be used with the name of a volume on your new nodes - nothing says the LUN has to be moved there.

-all adds all the nodes in the cluster. This option is available at "advanced" privilege level (set -priv adv). Gets a bit scary in large clusters.

One other thought: remember for path management the only paths you really need are the ones to the HA pair where the aggregate/volume/LUN live. If one node fails, the aggregate flips over to the other node and you still have direct paths. If both nodes fail the aggregate is offline. Having non-optimized paths through other nodes in the cluster is only needed if you suspect that all the iSCSI data paths to the owning HA pair could fail at the same time. But, if your network/system design is such that you might lose all those paths at the same time, would it also take down the paths you just added for the new nodes? If so, having the additional paths does not add protection but does add overhead at both ends (small, but adds up at scale).

Hope this helps.

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

Kudos and accepted solutions are always appreciated.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Tue, 11 Oct 2016 13:43:00 GMT

Thank you very much Bob, and yes it definitely helps. The reason for this design is that we are planning to move all of our iSCSI volumes/LUNs to the two new nodes in the cluster (they are AFF nodes).

Based on your description of the four options, it sounds like my options are "destination-aggregate" or "all". I don't really understand "local-nodes" - wouldn't the nodes that currently contain the LUNs already be "reporting nodes"? Mine are. Do you need to "add" nodes that are already applied when you are making this kind of change? My concern about the "all" option is I'm concerned about the EXISTING reporting nodes - I don't want to adversely affect them somehow.

Here are a couple possible ways for me to do this based on my understanding. Does this look correct?

lun mapping add-reporting-nodes -vserver <vserver> -path <Server_Name>_* -igroup <Igroup-Name> -destination-aggregate <aggregate_on_node_3>

lun mapping add-reporting-nodes -vserver <vserver> -path <Server_Name>_* -igroup <Igroup-Name> -destination-aggregate <aggregate_on_node_4>

lun mapping add-reporting-nodes -vserver <vserver> -path <Server_Name>_* -igroup <Igroup-Name> -all

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

aborzenkov — Tue, 11 Oct 2016 13:57:38 GMT

LUN move is actually covered pretty well in documentation (see SAN Administration Guide, Modifying SLM reporting nodes). When you move LUN to new aggregate that is hosted on different nodes, you need to add these nodes to the set of reporting nodes. Using -destination-aggregate eliminates need to know these nodes' names directly - you usually know destination aggregate/volume already. As long as you have just 4 nodes in total, this is entirely equivalent to using "all". But it makes difference if you have large cluster.

After move is complete you may remove old nodes using remove-reporting-node -remote-nodes.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Tue, 11 Oct 2016 14:27:46 GMT

Thanks aborzenkov. Do you or Bob know if there's a preferred / recommended method? I'm slightly concerned about using "all" since that includes the existing reporting nodes. Do either of you know if "all" only affects nodes that aren't currently applied?

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Tue, 11 Oct 2016 14:36:23 GMT

In particular, the description of "all" in the man pages is:

------------------------------------------------

Set the LUN mapping to report on all nodes in preparation for a revert to a previous version of Data ONTAP.

------------------------------------------------

That doesn't seem right. I prefer "all" because it's simple, if indeed it is safe to use.

If I use "destination-aggregate", does it truly matter which aggregate I choose as long as it's on the applicable node? In other words, could I choose an aggregate on node 3 for all LUNs but actually move some to node 3 and some to node 4?

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Tue, 11 Oct 2016 14:43:40 GMT

One last question re: syntax:

For the LUN path, do I need to do this (<servername>* are the volumes, <servername>*.lun are the LUNs):

-path /vol/<servername>*/<servername>*.lun

or will this work:

-path /vol/<servername>*

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

bobshouseofcards — Tue, 11 Oct 2016 15:33:38 GMT

Using "all" for a 4 node cluster is not likely to be an issue. Using "all" on a 8+ node SAN cluster where each SVM might have multiple target FC/iSCSI addresses on each node might just blow a server's pathing limitation out of the water.

For that reason only I suggest getting in the habit of not using "all". At some point you may have a few more nodes and it can being to matter - better to develop good habits now. When doing a volume/lun move, before the volume move add in the reporting nodes of the destination aggregate/volume. That particular operational pattern is how SLM is conceived.

On the "local-nodes" option you need it for remove for sure as part of the move sequence. Consider an order of operations - add new, move vol, remove old. How do you identify the old? Can't be by aggregate or volume location since the data isn't at the old location anymore. So the sequence, if you will remove reporting nodes, is "add new, remove local, move vol/lun". Obviously you'd want to verify that the new paths are working before removing the local nodes else data access might be impacted.

As with you not sure how local-nodes would not be reporting nodes, but I can imagine as a security mechanism or perhaps a testing mechanism a LUN can be created and mapped but "turned off" by removing all reporting nodes. Then you might want to add in only the local nodes to turn it back on.

Similarly, when converting from 8.2 to 8.3 one might take an outage window to address path proliferation issues by removing all reporting nodes from a LUN and then adding the local nodes back in.

Hope this helps.

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

Kudos and accepted solutions are always appreciated.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Tue, 11 Oct 2016 18:16:17 GMT

Bob,

Really appreciate your comments, they are helping a lot.

I had intended to leave all four paths available, just in case for some reason I needed to move a volume/LUN back to the original nodes. Would you recommend against that?

If I do remove the original reporting nodes, and follow the sequence "add new/remove local/move volume", won't that break the existing volumes/LUNs since they will not be on reporting nodes anymore? I may not be understanding the role of a "reporting node".

Here are my thoughts on the syntax I need. Does this look correct?

lun mapping add-reporting-nodes -vserver <vserver> -path <LUN_path> -igroup <Igroup_Name> -destination-aggregate <aggregate_on_node_3>

lun mapping add-reporting-nodes -vserver <vserver> -path <LUN_path> -igroup <Igroup_Name> -destination-aggregate <aggregate_on_node_4>

Lastly: for the <LUN_path>, can I enter a * wildcard character at the end of the volume name to make it cover any volume/LUN starting with that volume name? In other words:

Can I do this: /vol/<servername>*

Instead of this: /vol/<servername>*/<servername>*.lun

(or will either work)

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

aborzenkov — Wed, 12 Oct 2016 06:35:58 GMT

@bobshouseofcards wrote:
How do you identify the old? Can't be by aggregate or volume location since the data isn't at the old location anymore. So the sequence, if you will remove reporting nodes, is "add new, remove local, move vol/lun".

Actually easier is - move LUN, add new local paths using "lun mapping add-reporting-nodes -local-nodes true", remove old (now remote) paths using "lun mapping remove-reporting-nodes -remote-nodes true". This sequence does not require you to identify source or destination at all and can be applied to any LUN anytime.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Wed, 12 Oct 2016 13:07:45 GMT

Hey aborzenkov,

Regarding this:

------------------------

From what you are saying, you can actually move the LUN to the new nodes without having added the reporting nodes yet. Is that true? What would be the consequences of doing that? i was assuming the connection to the LUN would fail without the reporting nodes being added.

Also: I don't see a "remote-nodes" option. I am on 8.3.2 so I don't know if that's a new 9.0 option or not, but it's not in my version.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

aborzenkov — Wed, 12 Oct 2016 13:17:15 GMT

@TMADOCTHOMAS wrote:
What would be the consequences of doing that?

Data would flow across cluster interconnect, so likely increased latency.

@TMADOCTHOMAS wrote:
Also: I don't see a "remote-nodes" option. I am on 8.3.2 so I don't know if that's a new 9.0 option.

It is listed in 8.3.2 documentation. I do not have live system to check. Try in advanced mode.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Wed, 12 Oct 2016 13:48:17 GMT

i see it now. I was looking at the "add-nodes" command options, not "remove-node".

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Wed, 12 Oct 2016 13:50:00 GMT

-------------------

Data would flow across cluster interconnect, so likely increased latency.

-------------------

That's why i want to add the reporting nodes in advance.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

bobshouseofcards — Wed, 12 Oct 2016 14:53:35 GMT

Always forget about that on the remove (since I rarely do myself).

Thanks for the clarification!

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

bobshouseofcards — Wed, 12 Oct 2016 15:08:17 GMT

You'd be surprised how little latency, for a single workload, that using the Cluster Interconnect during the transfer adds. The transfer process itself probably adds more simply because there are multiple data consumers accessing the data.

Unless you have significant workload using the Cluster Interconnect or you have very tight performance requirements, changing the reporting nodes before or after isn't going to make a huge difference. Remember - the node that "owns" the LUN doesn't change during the transfer - it changes at the end of the transfer. Makes sense of course since if you cancel the move the source still has to be fully up to date.

A vol move is for all intents a SnapMirror move, update-cycle until really close, "offline", final update, break, online, teard down mirror, delete the original volume, rename the new volume all wrapped into a convenient job. The key element is the "update-cycle until really close". ONTAP volume moves have a cutover time allowed for the "offline final-update break online" sequence of 45 seconds as I recall (would have to check) during which standard configuration at the client server is expected to just wait on I/O anyway and ride through the cutover time. The "update-cycle until really close" is the same as multiple SnapMirror updates to get the state of the move really close to perfect to minimize the final update time during cutover. If the cutover time exceeds the allowed time, the source is brought back to normal online state and the vol move job goes back to the "update-cycle until really close" phase.

So even if you add the paths ahead of time, you'd still be dealing with the increased load on the source during the move and the upcoming brief "offline" time when the cutover happens. The slight latency added by using the CI after the move completes but before you'd update pathing could be less than all of that.

Hope this helps.

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

Kudos and accepted solutions are always appreciated.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS — Wed, 12 Oct 2016 15:31:53 GMT

Bob,

Thanks again for your helpful comments. So to be sure I am following, is this an accurate summation?

I can move volumes from, say, node 1 to the new node 4, without changing reporting nodes.

After the move is complete, the LUNs stay connected via the LIF on node 1 and thus use an indirect path temporarily.

I then add node 3 and 4 as reporting nodes, and the Data OnTAP for MPIO software changes the connection to the new preferred path on node 4.

Here is what threw me off:

1. The Data OnTAP for MPIO software isn't showing nodes 3 and 4 as indirect paths. It's not showing them as paths at all. It only shows paths for nodes 1 and 2. So, when the LUN lands on node four, how will it switch connections to the path on node 4 when that path doesn't show up on the MPIO software? I assumed that adding reporting nodes was required for the new nodes to show up. (NOTE: I have successfully added the iSCSI connections to nodes 3 and 4 in SnapDrive's iSCSI Management pane).

2. I don't think I have a clear understanding of what a "reporting node" is in this context. Again, I was under the assumption it was required to be in place before migrating a LUN, but that is apparently not the case.