Which NIC should I use for SnapMirror?

[chris_davis]

I'm currently setting up a SnapMirror relationship and I noticed that my transfer was going extremely slow.  Turns out the e0M interface is only 100mb.

My setup:

I have a FAS2240-4 with two controllers running 8.1.4P1 7-Mode

Both controllers use this setup:

e0a and e0c are used for my iSCSI VIF

e0b and e0d are used for my NFS VIF

Both VIFs are configured to use lacp

e0m is my management nic

Right now, DNS points to the management IP for each server.

I also have Exchange data stored on this filer and SnapDrive on my Exchange servers configured to point to the management interface.  I'm working with support to figure out why SME won't interface with SnapMirror but that's outside the scope of this discussion.

What are my options for getting my SnapMirror on a gigabit interface?  Is it a good idea to piggyback on one of my VIFs.

[JGPSHNTAP]

Im not a fan of e0m, and never was... It changes on the 8000 series, but i've been in shops with 200+ filers and we never used e0m..

We usually send snapmirror traffic over 10g vif as well.. e0m is useless 🙂

[chris_davis]

How do you manage your filer without using the management interface?

My nfs and my iscsi vifs are both vlanned so the traffic for each is isolated from the rest of the network.

Is my only option to use one of the nfs and iscsi vifs and vlan for my snapmirror traffic?

Can I add another ethernet nic to the controllers and use a port off of it?

[vims]

I have that e0M interfaced disabled.

then  group e1a and e1b using ifgrp (could be e0a and e0b up to you) 

created vifs and tag them for management traffic iscsi, nfs and so on.

[aborzenkov]

Do not be confused. You do not need to use e0M - any interface can be used to manage filer.

You can add mezzanine card with 2 x 10G. You can't add more 1GbE to 2240.

[JGPSHNTAP]

Exactly

e0m confuses people... In my opinion, at 100mbit, it's useless.. and in both the environments where i've been (large) we never uses it..

[aborzenkov]

The only reason to use e0M is when you absolutely need separate physical management network and do not want to waste 1GbE (or 10G) for it.

[JGPSHNTAP]

You should be able to get  a riser card in that box and use it for your transfers.

[chris_davis]

This is where the documentation doesn't transfer to the real world.

Here's a little more information on my setup.  Perhaps you guys can tell me how you would configure it so I'm getting the most out of these systems.

e0a and e0c connected to switch1 (2960-S).  lacp is enabled and the port groups are configured for my iscsi vlan (exchange and file servers)

e0b and e0d connected to switch2 (2960-S).  lacp is enabled and the port groups are configured for my nfs vlan (vmware)

e0m on both controllers are either connected to switch3 or switch4

switch1 and switch2 are disparate and do not have any connectivity to the public network aside from one management port.

switch3 and switch4 are my internal facing switches for my vm traffic, management, etc.

My second filer, my snapmirror destination filer, is configured the same way.

Should I talk to my network team about stacking switches 1 and 2?  I could then use a 4-port lacp for e0a-e0d and use vlan tagging in my rc file.  It wouldn't take much to add my management vlan to that switch.

Then configure the hosts file on each controller to point to either the nfs or the iscsi vif on my second filer?

Or would it be the same to just leave it as it is, and configure the hosts file for each filer to use the nfs or iscsi for all communication, including snapmirror transfers, leaving e0m for management.  That would be the easiest and least disruptive.

[JGPSHNTAP]

Documentation vs real-world... yeah.. that's always a treat.

I see no issue with vlan tagging and lacp.. But it's up to you.. I fyou want to keep it all segrated, get a riser card

[chris_davis]

To expand from my earlier question and get slightly off topic ....  what is the preferred network configuration that you guys use?

Would you guys mind posting your /etc/rc file and your cisco configs for the ports your filers are connected to?

I would like to resolve this without purchasing additional hardware if at all possible.

Here's mine and maybe you guys can let me know what to change:

> rdfile /etc/rc

hostname FILER1

ifgrp create lacp VIF_ISCSI -b ip e0a e0c

ifgrp create lacp VIF_NFS -b ip e0b e0d

ifconfig VIF_ISCSI `hostname`-VIF_ISCSI netmask 255.255.255.0 partner VIF_ISCSI mtusize 1500

ifconfig VIF_NFS `hostname`-VIF_NFS netmask 255.255.255.0 partner VIF_NFS mtusize 1500

ifconfig e0M `hostname`-e0M netmask 255.255.255.0 partner 10.10.10.91 mtusize 1500

route add default 10.10.10.1 1

routed on

options dns.enable on

options dns.domainname domain.com

options nis.enable off

savecore

> rdfile /etc/hosts

127.0.0.1 localhost localhost-stack

127.0.10.1 localhost-10 localhost-bsd

127.0.20.1 localhost-20 localhost-sk

172.30.2.30     FILER1-VIF_ISCSI

172.30.2.31     FILER2-VIF_ISCSI

172.30.3.30     FILER1 FILER1-VIF_NFS

172.30.3.31     FILER2 FILER2-VIF_NFS

10.10.10.90     FILER1-e0M

10.10.10.91     FILER2-e0M

10.127.11.35   mailhost

FILER3 is my SnapMirror destination:

FILER1*> getXXbyYY gethostbyname_r FILER3

name: filer3.domain.com

aliases:

IPv4 addresses: 10.10.10.122

FILER3*> getXXbyYY gethostbyname_r FILER1

name: filer1.domain.com

aliases:

IPv4 addresses: 10.10.10.90

And here are the Cisco port configs:

interface Port-channel5

description Virtual interface for Etherchannel for FILER1 VIF_ISCSI

switchport access vlan 6

switchport mode access

switchport nonegotiate

flowcontrol receive on

spanning-tree portfast

spanning-tree guard loop

!

interface Port-channel6

description Virtual interface for Etherchannel for FILER1 VIF_NFS

switchport access vlan 7

switchport mode access

switchport nonegotiate

flowcontrol receive on

spanning-tree portfast

spanning-tree guard loop

!

interface GigabitEthernet0/27

description FILER1 E0A VIF_ISCSI

switchport access vlan 6

switchport mode access

switchport nonegotiate

flowcontrol receive on

spanning-tree portfast

spanning-tree guard loop

channel-protocol lacp

channel-group 5 mode active

!

interface GigabitEthernet0/28

description FILER1 E0C VIF_ISCSI

switchport access vlan 6

switchport mode access

switchport nonegotiate

flowcontrol receive on

spanning-tree portfast

spanning-tree guard loop

channel-protocol lacp

channel-group 5 mode active

!

interface GigabitEthernet0/29

description FILER1 E0B VIF_NFS

switchport access vlan 7

switchport mode access

switchport nonegotiate

flowcontrol receive on

spanning-tree portfast

spanning-tree guard loop

channel-protocol lacp

channel-group 6 mode active

!

interface GigabitEthernet0/30

description FILER1 E0D VIF_NFS

switchport access vlan 7

switchport mode access

switchport nonegotiate

flowcontrol receive on

spanning-tree portfast

spanning-tree guard loop

channel-protocol lacp

channel-group 6 mode active

[vims · Accepted solution]

rc file is below, there is nothing special in hosts file.

I have no access to CISCO configuration, managed bu network team.

/etc/rc

#Generated  July 2012

hostname van-netapp-opr1

# Log on to Host and run an "ifconfig -a" to find your interface names.

# To create the dynamic multi-mode ifgrp for an LACP connection. Node 1 will have dmvif1

ifgrp create lacp dmvif1 e2a e2b

# To Create the VLans interfaces for dmvif1

vlan create dmvif1 216 629 630 632

# Internal Server network

ifconfig dmvif1-216 10.8.131.8 netmask 255.255.240.0 mtusize 1500 partner dmvif2-216

# iSCSI Network - Jumbo frames enabled for this network

ifconfig dmvif1-629 10.219.240.27 netmask 255.255.255.128 mtusize 9000 partner dmvif2-629

# NFS Network - Jumbo Frames enabled for this network

ifconfig dmvif1-630 10.219.240.132 netmask 255.255.255.128 mtusize 9000 partner dmvif2-630

# SAM Backup Network - Jumbo Frames enabled for this network

ifconfig dmvif1-632 10.219.241.134 netmask 255.255.255.128 mtusize 9000 partner dmvif2-632

# Config for Vfiler van-swebpr-opr1

ifconfig dmvif1-216 alias 10.8.131.62 netmask 255.255.240.0

ifconfig dmvif1-632 alias 10.219.241.137 netmask 255.255.255.128

# Config for Vfiler van-swebqa-opr1

ifconfig dmvif1-216 alias 10.8.131.63 netmask 255.255.240.0

ifconfig dmvif1-632 alias 10.219.241.138 netmask 255.255.255.128

# Config for vfiler van-ky-opr1

ifconfig dmvif1-216 alias 10.8.131.71 netmask 255.255.240.0

# Config for vfiler jackjackopr1

ifconfig dmvif1-216 alias 10.8.131.65 netmask 255.255.240.0

ifconfig dmvif1-630 alias 10.219.240.163 netmask 255.255.255.128

# Config for vfiler bane2

ifconfig dmvif1-216 alias 10.8.131.125 netmask 255.255.240.0

ifconfig dmvif1-629 alias 10.219.240.69 netmask 255.255.255.128

route add default 10.8.143.254 1

routed on

route add net 10.219.192.0/21 10.219.241.129 1

route add net 10.219.252.0/22 10.219.241.129 1

options dns.domainname coastcapital.local

options dns.enable on

options nis.enable off

savecore

[chris_davis]

Since you use vfilers, your setup is going to be a little different than mine.

You don't have any issues with pushing all that data through 2 ports?  Is NetApp smart enough to monitor the bandwidth and switch ports if needed or do you run the risk of having iSCSI, NFS, backup, and your management all over one port?  Would there be any benefit of adding a 3rd interface to the mix?

I'm thinking of putting my iSCSI, NFS, and management on one VIF that's comprised of 3 ports (e0a, b, and c)

Then utilizing e0d and going straight to e0d of my backup filer (completely bypassing the switch) for SnapMirror transfers.

* As demonstrated here: http://keepingitclassless.net/2012/04/a-quick-and-dirty-netapp-snapmirror/

This will require changing my switch's lacp groups and ports to trunk mode and tagging my netapp packets with the necessary vlan data like you're doing in your /etc/rc file.  I never thought about enabling jumbo frames ... probably a good idea to configure my switches and vmware servers to utilize those.

I'm not looking forward to shutting everything down again (netapp filers and all of my 100+ VMs) and pulling an all-nighter to get this changed over but my snapmirror transfers are taking DAYS to update.  6 days for the baseline Exchange transfer PER DATABASE so that was a month-long process.  And now that all that extra SnapMirror traffic (vmware, exchange, file server, data network traffic) is going over a 100 mb link I'm getting a lot of SnapMirror errors and timeouts.  It has become a very unstable system.

For your snapmirror.conf file ... are you utilizing compression for your transfers?

[chris_davis]

Just to put a close to the ticket ... I grouped all four of my interfaces (e0a-e0d) on each filer into one big dynamic multimode lacp vif and configured lacp on the switches.

Filer A is my production filer

Filer B is my backup (snapmirror destination) filer

Controller A (iSCSI) on Filer A and Filer B are connected to switch A

Controller B (NFS) on Filer A and Filer B are connected to switch B

VLAN 246 (1.1.1.0/24) is my data network

VLAN 006 (1.1.2.0/24) is for iSCSI

VLAN 007 (1.1.3.0/24) is for NFS

VLAN 010 (1.1.6.0/24) is for management

Those aren't real IPs.

/etc/rc

NETAPP01=> rdfile /etc/rc

hostname NETAPP01

ifgrp create lacp dmvif01 e0a e0b e0c e0d

vlan create dmvif01 6 7 10 246

ifconfig dmvif01-246 1.1.1.90 netmask 255.255.255.0 mtusize 1500 -wins partner dmvif01-246

ifconfig dmvif01-6 1.1.2.30 netmask 255.255.255.0 mtusize 1500 -wins partner dmvif01-6

ifconfig dmvif01-7 1.1.3.30 netmask 255.255.255.0 mtusize 1500 -wins partner dmvif01-7

ifconfig dmvif01-10 1.1.6.30 netmask 255.255.255.0 mtusize 1500 -wins partner dmvif01-10

route add default 1.1.1.1 1

routed on

options dns.enable on

options dns.domainname domain.com

options nis.enable off

savecore

rdfile /etc/hosts

127.0.0.1	localhost localhost-stack
127.0.10.1	localhost-10 localhost-bsd
127.0.20.1	localhost-20 localhost-sk
1.1.2.30	NETAPP01-dmvif01-6
1.1.2.31	NETAPP02-dmvif01-6
1.1.2.40	NETAPP03-dmvif01-6
1.1.2.41	NETAPP04-dmvif01-6
1.1.3.30	NETAPP01-dmvif01-7
1.1.3.31	NETAPP02-dmvif01-7
1.1.3.40	NETAPP03-dmvif01-7
1.1.3.41	NETAPP04-dmvif01-7
1.1.6.30	NETAPP01 NETAPP01-dmvif01-10
1.1.6.31	NETAPP02 NETAPP02-dmvif01-10
1.1.6.40	NETAPP03 NETAPP01-dmvif01-10
1.1.6.41	NETAPP04 NETAPP02-dmvif01-10
1.1.1.90	NETAPP01-dmvif01-246
1.1.1.91	NETAPP02-dmvif01-246
1.1.1.122	NETAPP03-dmvif01-246
1.1.1.123	NETAPP04-dmvif01-246
1.1.1.102	mailhost

Switch Config

interface Port-channel2

description NETAPP01 LACP DMVIF01 E0A E0B E0C E0D

switchport mode trunk

!

interface GigabitEthernet0/35

description NETAPP01 LACP DMVIF01 E0A

switchport mode trunk

spanning-tree portfast trunk

channel-protocol lacp

channel-group 2 mode active

!

interface GigabitEthernet0/36

description NETAPP01 LACP DMVIF01 E0B

switchport mode trunk

spanning-tree portfast trunk

channel-protocol lacp

channel-group 2 mode active

!

interface GigabitEthernet0/37

description NETAPP01 LACP DMVIF01 E0C

switchport mode trunk

spanning-tree portfast trunk

channel-protocol lacp

channel-group 2 mode active

!

interface GigabitEthernet0/38

description NETAPP01 LACP DMVIF01 E0D

switchport mode trunk

spanning-tree portfast trunk

channel-protocol lacp

channel-group 2 mode active