Node limitation for SAN and NAS in CDOT

[jsachdev_tm]

Hi,

I'm working on netapp since long, but have never came across as to why the node limitation for SAN configuration in CDOT is 8 node and that of NAS configuration is 24 node.

I have just heard from many people that beyond 8 node netapp do not support SAN configuration due to performance.

But no concrete article or refrences as what will happen if I go beyond 8 node.

Can anybody provide me reason or some refrences as why 24 node is for NAS and only 8 node for SAN

Regards..

JP

[colsen]

Hello,

The NAS/SAN numbers depend on your platform.  There's a good list of instructions on the ONTAP 9 Documentation site for figuring out your limits:

http://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-cm-sanconf%2FGUID-F664A5DB-CAB3-43FB-9E0A-6E61AEAF8466.html

For our systems (mostly FAS8Ks) the limits are 24 (NAS) and 24 (SAN).  I honestly can't speak to "why" these limits exist - I recall a discussion in a class long ago about why the limits were originally imposed but maybe some wise NetApp guru can weigh in on that part of the question.

Good luck!

Chris

[jsachdev_tm]

I have also FAS80xx in environment running 8.3.2 CDOT, but there are limitations and no where I can find the reason.

Netapp Experts on the forum please let me know if you have any reason supporting those limitations.

[robinpeter]

As mentioned earlier.. Can't answer "why" (probably because of the resources on each hardware model)

But if you like to get the accurate number based on each model.. the best place to go to is hwu.netapp.com

Once you select the ontap version, hardware-model and click result.. 

You will be able to see a section "Supported Platform Mixing" Which shows a tables as below.

Hope that help..

Robin

[colsen]

After digging through some rusty memory banks, I seem to recall that it has something to do with the maximum number of target ports (LIFs) supported by ALUA.  Since you're adding at least one iSCSI target LIF per node in a cluster I'm suspecting there's a hard limit as to how many potential target ports ALUA/MPIO can be configured from the client (initiator) perspective.  

I've done some unsuccessful digging as to what that number is from a Linux/Windows perspective - if that is indeed the limitation.  I see other storage vendors list "maximum number of iSCSI ports" as well, so it sounds like this isn't a NetApp imposed limitation...

[bobshouseofcards · Accepted solution]

Hi JP -

As I recall when I asked my local NetApp SEs the same question, the limits were based on the ability of the cluster to maintain consistency of both the data and internal cluster structures.

Consider - in a LUN scenario, multiple data access requests might come in on multiple paths, perhaps through multiple nodes.  All the eventual disk access is run through the node that owns the aggregate on which the volume contain the LUN exists, but all the nodes have to ensure that the right data is written/read based on order of arrival of the requests.  A LUN looks like a physical disk and should act like it to the server(s) where the LUN is mapped.  There is an expectation of serial results - write followed by read should return the previously written data.  If the server for some reason writes data along a non-optimized path with the read immediately following along an optimized path to the cluster nodes, the cluster still has to enforce the write first then the read.

This requires some not insignificant coordination between all the nodes when serving LUNs.  I'm sure the actual limit for stable LUN access is higher than 8 nodes, but of course NetApp would back that down a little to ensure stable operation.  You'll note that the limits vary based on hardware capability as well, so processing power factors in beyond just the number of nodes.  Same reasons why smaller nodes get lower aggregate/volume size limits.

With 9.x, the SAN limits increase to 12 nodes for newer hardware and some of the larger older hardware.  This indicates continued tuning and testing of the LUN access coordination system.

For NAS, the same coordination is not required.  A NAS system has the assumption that there will potentially be multiple independent readers and writers to the same file, which is why NFS and CIFS define locking mechanisms within their protocols.  The burden of serializing operations falls to the consumers of the NAS files rather than the cluster.  Coordinating the lock requests is a much lower burden in a NAS environment than enforcing access serialization for LUNs, so the operational limits for the cluster are higher for a pure NAS cluster.

I hope this helps you.

Bob Greenwald

Senior System Engineer | cStor

NCIE SAN ONTAP, Data Protection

Kudos and accepted answers are always appreciated.

[jsachdev_tm]

Hi Bob,

Thanks for the explaination, seems like due to co-ordination that limit is set for nodes in SAN environment.

I would also request if by any chance you have any references or any article shared by netapp on same, Then please send me the link/doc.

Thanks again for the solution.

Regards..

JP

[bobshouseofcards]

Hi JP -

The HWU has the per node and per ONTAP version limits that apply.  Beyond that documentation I'm not sure there is anything else official and public.  Certainly, I don't have any specific documentation or articles that support the SAN node count limit.  The explanation is my best recollection of conversations with assorted NetApp SEs and other resources since I started working with Clustered Data Ontap - mostly because I really like to understand these details and grab whatever tidbits on internal operations I can whenever I can.

I hope this helps you.

Bob Greenwald

Senior Systems Engineer | cStor

NCIE SAN ONTAP, Data Protection

Kudos and accepted answers are always appreciated.