ONTAP Discussions

New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

I've just added two new nodes to our production cluster. I've added two new iSCSI LIFs on the nodes and added those LIFs to the port group. I've used SnapDrive to add connections to the two new iSCSI LIFs on the first of many Windows hosts. After doing this, I checked the Data OnTAP DSM for MPIO software to be sure it was showing 4 nodes per LUN. It is not. It is still just showing two paths. Any ideas as to what I am missing?

27 REPLIES 27

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

bobshouseofcards

Assuming you are running cDoT 8.3 or higher, you are likely encountering the new "Selective LUN Mapping" feature.  This feature was added in cDoT 8.3 to address the multiplication of paths across large clusters that can impact server based total and per LUN path limits.

 

From the SAN Administration Guide...

 

Selective LUN Map

Selective LUN Map (SLM) reduces the number of paths from the host to the LUN. With SLM, when a new LUN map is created, the LUN is accessible only through paths on the node owning the LUN and its HA partner.

SLM enables management of a single igroup per host and also supports nondisruptive LUN move operations that do not require portset manipulation or LUN remapping.

Portsets can be used with SLM just as in previous versions of Data ONTAP to further restrict access of certain targets to certain initiators . When using SLM with portsets, LUNs will be accessible on the set of LIFs in the portset on the node that owns the LUN and on that node's HA partner.

Beginning with Data ONTAP 8.3 SLM is enabled by default on all new LUN maps. For LUNs created prior to Data ONTAP 8.3, you can manually apply SLM by using the lun mapping remove-reporting-nodes command to remove the LUN reporting nodes and restrict LUN access to the LUN owning node and its HA partner.

 

 

 

I ran into the same situation myself when adding nodes 5 and 6 to a cluster, moving some LUNs there, and then not seeing direct paths.  Mine was FCP, but since this applies at the LUN mapping level it would be just as applicable to iSCSI as well.

 

The commands of interest to you are:

 

lun mapping show -field reporting-nodes -vserver <SVM> -path <lun-path>

lun mapping add-reporting-nodes ...

lun mapping remove-reporting-nodes ...

 

SLM provides for very specific path control and in conjunction with portsets can limit path proliferation especially as the number of potential ports increases and the number of allowed hosts in a SAN cluster increases with refreshed hardware and ONTAP 9.1.

 

 

Hope this helps.

 

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

 

 

Kudos and accepted solutions are always accepted.

 

View solution in original post

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

Very helpful! Thank you very much Bob.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

Bob, one more question: from what I can tell the -aggregate and -volume options are optional. I simply want each LUN in a given SVM to have access to two new nodes in the cluster, regardless of aggregate. Am I correct that I can leave these switches off? Also if I enter a common path name with a wildcard for the -lun option, will that work? For example:

 

lun mapping add-reporting-nodes -vserver <vserver> -path <Server_Name>_* -igroup <Igroup-Name>

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

bobshouseofcards

 

Your comand fragment is enough to identify the particular LUN/igroup combinations of interest.  You'll need one of the four optional parameters to select which nodes to add to the list...

 

-local-nodes [ true ]   adds the nodes where the LUN currently exists to the reported path map.

 

-destination-aggregate <aggregate name>    adds the nodes where the listed aggregate lives.  Typically used before a volume would be moved to a new aggregate.  Could also be used with the name of an aggregate on your new nodes - nothing says the volume has to be moved there.

 

-destination-volume <volume name>   adds the nodes where the listed volume lives.  Typically used before a LUN is moved to a new volume (lun move start ...).  Could also be used with the name of a volume on your new nodes - nothing says the LUN has to be moved there.

 

-all   adds all the nodes in the cluster.  This option is available at "advanced" privilege level (set -priv adv).  Gets a bit scary in large clusters.

 

 

One other thought:  remember for path management the only paths you really need are the ones to the HA pair where the aggregate/volume/LUN live.  If one node fails, the aggregate flips over to the other node and you still have direct paths.  If both nodes fail the aggregate is offline.  Having non-optimized paths through other nodes in the cluster is only needed if you suspect that all the iSCSI data paths to the owning HA pair could fail at the same time.  But, if your network/system design is such that you might lose all those paths at the same time, would it also take down the paths you just added for the new nodes?  If so, having the additional paths does not add protection but does add overhead at both ends (small, but adds up at scale).

 

 

Hope this helps.

 

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

 

 

Kudos and accepted solutions are always appreciated.

 

 

 

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

Thank you very much Bob, and yes it definitely helps. The reason for this design is that we are planning to move all of our iSCSI volumes/LUNs to the two new nodes in the cluster (they are AFF nodes).

 

Based on your description of the four options, it sounds like my options are "destination-aggregate" or "all". I don't really understand "local-nodes" - wouldn't the nodes that currently contain the LUNs already be "reporting nodes"? Mine are. Do you need to "add" nodes that are already applied when you are making this kind of change? My concern about the "all" option is I'm concerned about the EXISTING reporting nodes - I don't want to adversely affect them somehow.

 

Here are a couple possible ways for me to do this based on my understanding. Does this look correct?

 

lun mapping add-reporting-nodes -vserver <vserver> -path <Server_Name>_* -igroup <Igroup-Name> -destination-aggregate <aggregate_on_node_3>

lun mapping add-reporting-nodes -vserver <vserver> -path <Server_Name>_* -igroup <Igroup-Name> -destination-aggregate <aggregate_on_node_4>

 

lun mapping add-reporting-nodes -vserver <vserver> -path <Server_Name>_* -igroup <Igroup-Name> -all

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

aborzenkov

LUN move is actually covered pretty well in documentation (see SAN Administration Guide, Modifying SLM reporting nodes). When you move LUN to new aggregate that is hosted on different nodes, you need to add these nodes to the set of reporting nodes. Using -destination-aggregate eliminates need to know these nodes' names directly - you usually know destination aggregate/volume already. As long as you have just 4 nodes in total, this is entirely equivalent to using "all". But it makes difference if you have large cluster.

 

After move is complete you may remove old nodes using remove-reporting-node -remote-nodes.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

Thanks aborzenkov. Do you or Bob know if there's a preferred / recommended method? I'm slightly concerned about using "all" since that includes the existing reporting nodes. Do either of you know if "all" only affects nodes that aren't currently applied?

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

In particular, the description of "all" in the man pages is:

 

------------------------------------------------

Set the LUN mapping to report on all nodes in preparation for a revert to a previous version of Data ONTAP.

------------------------------------------------

 

That doesn't seem right. I prefer "all" because it's simple, if indeed it is safe to use.

 

If I use "destination-aggregate", does it truly matter which aggregate I choose as long as it's on the applicable node? In other words, could I choose an aggregate on node 3 for all LUNs but actually move some to node 3 and some to node 4?

 

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

One last question re: syntax:

 

For the LUN path, do I need to do this (<servername>* are the volumes, <servername>*.lun are the LUNs):

 

-path /vol/<servername>*/<servername>*.lun

 

or will this work:

 

-path /vol/<servername>*

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

bobshouseofcards

Using "all" for a 4 node cluster is not likely to be an issue.  Using "all" on a 8+ node SAN cluster where each SVM might have multiple target FC/iSCSI addresses on each node might just blow a server's pathing limitation out of the water.

 

For that reason only I suggest getting in the habit of not using "all".  At some point you may have a few more nodes and it can being to matter - better to develop good habits now.  When doing a volume/lun move, before the volume move add in the reporting nodes of the destination aggregate/volume.  That particular operational pattern is how SLM is conceived.

 

On the "local-nodes" option you need it for remove for sure as part of the move sequence.  Consider an order of operations - add new, move vol, remove old.  How do you identify the old?  Can't be by aggregate or volume location since the data isn't at the old location anymore.  So the sequence, if you will remove reporting nodes, is "add new, remove local, move vol/lun".  Obviously you'd want to verify that the new paths are working before removing the local nodes else data access might be impacted.

 

As with you not sure how local-nodes would not be reporting nodes, but I can imagine as a security mechanism or perhaps a testing mechanism a LUN can be created and mapped but "turned off" by removing all reporting nodes.  Then you might want to add in only the local nodes to turn it back on.

 

Similarly, when converting from 8.2 to 8.3 one might take an outage window to address path proliferation issues by removing all reporting nodes from a LUN and then adding the local nodes back in.

 

 

 

Hope this helps.

 

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

 

 

Kudos and accepted solutions are always appreciated.

 

 

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

Bob,

 

Really appreciate your comments, they are helping a lot.

 

I had intended to leave all four paths available, just in case for some reason I needed to move a volume/LUN back to the original nodes. Would you recommend against that?

 

If I do remove the original reporting nodes, and follow the sequence "add new/remove local/move volume", won't that break the existing volumes/LUNs since they will not be on reporting nodes anymore? I may not be understanding the role of a "reporting node".

 

Here are my thoughts on the syntax I need. Does this look correct?

 

lun mapping add-reporting-nodes -vserver <vserver> -path <LUN_path> -igroup <Igroup_Name> -destination-aggregate <aggregate_on_node_3>

lun mapping add-reporting-nodes -vserver <vserver> -path <LUN_path> -igroup <Igroup_Name> -destination-aggregate <aggregate_on_node_4>

 

Lastly: for the <LUN_path>, can I enter a * wildcard character at the end of the volume name to make it cover any volume/LUN starting with that volume name? In other words:

 

Can I do this: /vol/<servername>*

Instead of this: /vol/<servername>*/<servername>*.lun

 

(or will either work)

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

aborzenkov

@bobshouseofcards wrote:

How do you identify the old?  Can't be by aggregate or volume location since the data isn't at the old location anymore.  So the sequence, if you will remove reporting nodes, is "add new, remove local, move vol/lun". 


Actually easier is - move LUN, add new local paths using "lun mapping add-reporting-nodes -local-nodes true", remove old (now remote) paths using "lun mapping remove-reporting-nodes -remote-nodes true". This sequence does not require you to identify source or destination at all and can be applied to any LUN anytime.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

Hey aborzenkov,

 

Regarding this:

------------------------

Actually easier is - move LUN, add new local paths using "lun mapping add-reporting-nodes -local-nodes true", remove old (now remote) paths using "lun mapping remove-reporting-nodes -remote-nodes true". This sequence does not require you to identify source or destination at all and can be applied to any LUN anytime.

------------------------

 

From what you are saying, you can actually move the LUN to the new nodes without having added the reporting nodes yet. Is that true? What would be the consequences of doing that? i was assuming the connection to the LUN would fail without the reporting nodes being added.

 

Also: I don't see a "remote-nodes" option. I am on 8.3.2 so I don't know if that's a new 9.0 option or not, but it's not in my version.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

aborzenkov

@TMADOCTHOMAS wrote:

What would be the consequences of doing that?


Data would flow across cluster interconnect, so likely increased latency.


@TMADOCTHOMAS wrote:

Also: I don't see a "remote-nodes" option. I am on 8.3.2 so I don't know if that's a new 9.0 option.


It is listed in 8.3.2 documentation. I do not have live system to check. Try in advanced mode.

Re: New Cluster Nodes Not Showing in NetApp DSM for MPIO

TMADOCTHOMAS

i see it now. I was looking at the "add-nodes" command options, not "remove-node".

 

 

Earn Rewards for Your Review!
GPI Review Banner
All Community Forums
Public