Re: Multistore and VIFs

nsitps1976 · ‎2011-03-18

Is it possible to aggregate all Ethernet interfaces on a controller into a static multimode VIF and have this VIF be partitioned across multiple vFilers?

I thought I read somewhere that you could only assign physical interfaces to vFilers? – I want all vFilers to be able to utilise the entire aggregate bandwidth of a VIF which includes 4 Ethernet interfaces, I do not want to limit each vFiler to one Ethernet interface.

scottgelb · ‎2011-03-18

Yes, you can do this. A vFiler needs a minimum of 1 IP address which can be assigned to a physcal interface, a vif or a vlan, or an alias to any of those. You could create a single vif, then either use vlans or aliases to assign IPs to each vFiler. If you are using IPspaces other than the default-ipspace, then you would need VLANs since an alias has to be in the same IPspace as the interface that is being aliased. But multiple vFilers in the same ipspace can create aliases on the same physical interface.

nsitps1976 · ‎2011-03-19

Great, thanks.

Does this mean you can also have multiple IPs per vFiler for iSCSI / NFS load balancing?

Also, in your experience are there any drawbacks or things to be aware of when using multistore? Multiple customers will be hosted on the system running multistore, each in an iSCSI SAN environment, using VMware SRM and Snapmirror replicating to a system which does not utilise mulistore.

Any advice you can give would be greatly appreciated as I am a little worried that the introduction of multistore may cause some issues, by that time it may be too late!

tjlee · ‎2011-03-19

Does this mean you can also have multiple IPs per vFiler for iSCSI / NFS load balancing?

Yes, you can have multiple IP addresses per vFiler... In the example below, it's used for traffic segregation and not load balancing as both the "iSCSI" (VLAN-1) and "NFS" (VLAN-10) networks are on the same "multitrunk" VIF.

Also, in your experience are there any drawbacks or things to be aware of when using multistore? Multiple customers will be hosted on the system running multistore, each in an iSCSI SAN environment, using VMware SRM and Snapmirror replicating to a system which does not utilise mulistore.

Any advice you can give would be greatly appreciated as I am a little worried that the introduction of multistore may cause some issues, by that time it may be too late!

With regards to advice, here's a couple of gotchas that you'll want to avoid:

- Set the "vfiler limit" as soon as possible, since the default is quite low and modifying this setting requires a reboot of the controller.

- Be careful with using IPspaces as this feature is extremely powerful, so you'll really want to understand it thoroughly before you decide if and how you want to implement it. For the most part I'd stick with the "default-ipspace"

Cheers, Tony

scottgelb · ‎2011-03-19

ONTAP reserves about 400k of memory for each vFiler set in the limit. The default 11 (really 10 since it includes vfiler0) meets most customers requirements. Even though 400k is a small footprint we prefer often don't increase it unless we know they will use more than the 10 since it will slightly decrease memory (although probably would never be noticed). I debate whether to set it to max sometimes on every system then after the debate we leave alone or set it slightly over the estimated number that will be used. Unfortunatley ONTAP doesn't dynamically allocate the memory reservation so requires a reset....but if no CIFS, this can be done like an ndu upgrade with a cf takeover/giveback. We probably won't see this changed until convergance of vFilers into c-mode vServers.

For IPspaces, it can get really confusing but some whiteboarding then the use cases help. In our Insight class, Roger and I give a few use cases when to use and not use IPspaces. The easiest use case is if they need a spearate routing table.

When to use them

A de-militarized zone (DMZ)
Multiple Windows, NIS or LDAP Domains in the enterprise organization
A requirement for the same IP address within different vFiler units on the same system
A separate routing table is needed

When not to use them

All vFiler units joined to the same domain
Flat IP network with no security needs
No VLANs or other virtual interfaces in use
All vFilers can share a routing table
Hosts and Servers need access to multiple vFilers from non-routable network

Another consideration are routes needed and having to add routes in the /etc/rc file of vfiler0. When MultiStore is licensed, routed is disabled. Attached is a really good architecture slide from our class (I think Roger's boss created it for the preso) that really does a nice job showing the different levels of separation (although some argue that VLANs aren't separation which is off topic )

The other main gotcha we run into is management. With no GUI yet realize that all the management is command line either by vfiler context or vfiler run from vfiler0 or non-interactive ssh into the vfiler directly. Also some commands/functions (growing volumes for example) are only available to the vfiler0 admin.

scottgelb · ‎2011-03-19

I forgot to mention that if you use IPspaces, make sure to create the same IPspace with an interface added on BOTH controllers of the cluster or failover won't work for the network even if an ipspace is only on one node, it i needed on the other side for failover. And since you are using vifs and vlans/aliases, make sure you have the partner interface set on the ifconfig (except alias ifconfig statements which don't support that since they use the parent ifconfig) on each controller.

Then test cf takeover/giveback before going into production. Make sure all of your IPs failover and giveback on both nodes (if any issues, fix in /etc/rc of vfiler0) then you know you won't have any issues when you go into production.

nsitps1976 · ‎2011-03-20

Thanks again for all this great info. A few other questions if I can:

1 - So all vfiler config is done via the CLI? – What about the VSC 2.0, does this work with vFilers? Can storage etc be monitored from any GUI tools?

2 – Can a volume from within a vfiler be replicated via snapmirror to a none mutistore system?

3 - Why create two VIFs and bind them together rather than one VIF with four physical interfaces and four IP addresses to allow ip hash load balancing?

4 - Can you point me at some decent documentation / training material? All I can find ONTAP 7.3 Stroage Managment Guide

scottgelb · ‎2011-03-20

Yes.. VSC 2.0 supports vFilers... just make sure in each vfiler (vfiler context or vfiler run options) to enable httpd.admin.enable in each vfiler.

Yes.. You can mirror from a vFiler to a system without multistore and vice-versa. VSC initiates mirrors from a vfiler to the target as well.

Network..your option...if you really wanted to load balance to two different switch fabrics this would work..if all to the same switch fabric, a 4 port vif keeps it simple which is easier to manage

The vFiler.pdf doc in the ontap guides has a lot of good info on multistore. I also have a simulator lab based class we give or can be used at your own pace. Send me an email and I can send you some of the basic labs to try out.

henrypan1 · ‎2011-03-20

Good Morning Scott,

Would you be kind enough to also email me (Henry.pan@ironmountain.com) your good staff on Multistore and VIFs as well?

Thanks & Good w/e

Henry

nsitps1976 · ‎2011-03-20

I also have the DOT 7.3 Mutistore Management Guide

scottgelb · ‎2011-03-20

Sent to both of you..2 PDFs. Also try to get to the EMEA Insight conference this fall and we should have an updated class ready to go.

nsitps1976 · ‎2011-03-22

I have another question please:

I want to ensure that snapmirror replication traffic for each vfiler uses the default gateway for the ipspace in which the vfiler resides – I do not want all customer (vfilers) volume replication to go via the default gateway of vfiler0. Is this possible?

scottgelb · ‎2011-03-22

Setup the relationship with the vFiler as the source of the mirror and as long as the interface in the vFiler is reachable by the target mirror, all good.. Set snapmirror.access and mirror direct from the vFiler.

If using vfiler dr we have to use vfiler0.. Otherwise you can mirror from the vFiler... And snapmanager products also work from the vfiler.. From the prior httpd.admin.enable on in each vfiler for smvi to work.

Typos Sent on Blackberry Wireless

scottgelb · ‎2011-03-19

Yes on multiple interfaces. But if you want load balancing you might want to have 2 active/acive ifgrps (vifs) then load balance the IPs across both vifs. Then decide whether you will use vlans for further separation of traffic or aliases on the same physical links.

For SRM, the NetApp SRA adapter is certified to work with MultiStore and each vFiler is treated like a storage array.

nsitps1976 · ‎2011-03-22

Can volumes etc be added to a vfiler once the vfiler is up and running in production – On P37 of the DOT 7.3 Mutistore Mgmt Guide it states “When resources are being moved, all network connections to the vFiler units are terminated” – This is a worry as I cannot disrupt service if I want to add extra storage etc?????

scottgelb · ‎2011-03-22

This only applies during a vfiler migrate or data motion when the vfiler is moving between 2 different controllers. You can absolutely add volumes and IP addresses to vfilers (and remove them too).. then if vfiler dr is running, just update with dr resync to get the new volumes (you have to activate the dr vfiler to add ips, then resync).

nsitps1976 · ‎2011-03-22

Nice one

D_BEREZENKO · ‎2012-05-07

About security. If every vFilers manages its VLANs, this situation creates security hole in case of sharing a few physical Ethernet interfaces with a few vFilers. How NetApp handle it?

scottgelb · ‎2012-05-07

Ipspaces keep interfaces in their own Routing space. Even sharing an interface you can have each vlan in its own Ipspace separating traffic at both vlan and routing table.

Sent from my iPhone 4S

D_BEREZENKO · ‎2012-05-07

Sorry I do not understand what you are meaning. Maybe it is because of my poor English.

Because "you can have each vlan" it is not enough. I mean that one organization which manages one vFiler can configure the same VLAN(s) as second vFiler and potentially will have opportunity to steal sensitive data. And so on... The security of whole system will be equal to the most insecure vFiler. For example a vFiler with the worst security can be hacked and through it all others vFiler can be hacked too (including vFiler0).

How to avoid security compromising vFilers in case they can configure VLANs for themselves?

aborzenkov · ‎2012-05-07

I mean that one organization which manages one vFiler can configure the same VLAN(s) as second vFiler and potentially will have opportunity to steal sensitive data.

vFilers are managed through vfiler0. There is no way administrator who has access only to vfiler != vfiler0 can add additional interfaces to this vfiler. It has to be done through vfiler0.

And yes, you have to trust whoever manages vfiler0. Just as you have to trust your cloud provider, your hosting provider and dozens of other providers whose services you use everyday.