Re: slowly network connection

discretixit · ‎2009-03-17

hi

i have netapp FAS 3020 and cisco 4560 they connect via 3 cable each 1 GB on etherchannel (no lacp) and when i copy a 10 G file it's take ~10 min ,the destination is win server 2003 also connect to 4560 with 1G,what the problem ?and what is the best way to connect NepApp to ciscp switch?

danbrown1 · ‎2009-03-17

You've got a problem there someplace - 10GB in 10 minutes is, by my calculation, 10240MB/600seconds == 17MB/second, which is < 15% of what should be possible with a 1 gigabit ethernet. See: http://en.wikipedia.org/wiki/List_of_device_bandwidths

There's a lot of places you could start looking for the cause. I'd start by making sure the network settings are all koser. Check duplex and speed settings on all ports, etc. Maybe something is set for fast ethernet (100Mbps?)

NetApp Network best practices here: https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb8454

discretixit · ‎2009-03-17

hi

thanks,i checked the setting in server and i have Auto-negotiate 1000 Mbps

danielpr · ‎2009-03-17

1Gig interface is a shared medium on the ethernet layer. I think the traffic should be high in this case thats reason for the delay.

Thanks

Daniel

discretixit · ‎2009-03-18

hi

i didn't understand could you please explain ?

danielpr · ‎2009-03-18

You can check the wiki here for the shared medium details. http://en.wikipedia.org/wiki/Ethernet#CSMA.2FCD_shared_medium_Ethernet

The collisions reduce throughput often, in your case you may be trying to use the same interface for all the Data/Management traffic.

You can have a dedicated interface for the high through put.

Please check the windows machine if the speed is 1.0Gbps for connected interface status.

Thanks

Daniel

danbrown1 · ‎2009-03-18

Have you checked the settings on the switch for both the NetApp and the Windows server?

If you haven't already, you may want to draw a picture of the setup (even if it is as simple as it sounds) and then start checking each connection, each link, each port, each setting, and working to eliminate possible problem areas. Be objective, don't make assumptions. If the settings are all set correctly, start looking deeper - are there errors? Collisions? Dropped Packets? Other bottlenecks? Is the switch overloaded? Are CPU's overloaded? Etc.

I recently helped a site who having trouble reaching their RLM ports on their filers - everything appeared to be setup correctly on the NetApp, routing, ip-addresses, etc. Turned out that a network administrator had set all of the ports on their Cisco switches to locked at 1000Gbps, full duplex. RLM ports are only 100Mbps (fast ethernet) ports, and so couldn't talk to the switch at 1000Mbps. My point here is you can't assume things will work, just because the should. The fix was simple, but it required looking in the right place to figure out what needed to happen. It took a process of elimination to identify that right place to look.

discretixit · ‎2009-03-18

hi

i was check cpu --->no problem (peak 46%)

the cisco port set to auto speed & duplex and Port Fast set to disabled

chriskranz · ‎2009-03-18

You say the link aggregate (VIF) is over 3 ports? I saw an issue similar to this the other day where a VIF created with 3 ports going into a Cisco switch caused a problem. If this was changed to 1, 2 or 4 ports, there was no issue at all, and everything worked as expected.

Try simply unplugging one of the ports (so you have 2 active) and running the tests again.

discretixit · ‎2009-03-18

hi

I unplugging one of the ports and try to copy the file --- ? the same problem ...

chriskranz · ‎2009-03-18

If you run "ifstat -a" on the filer, you'll get a print out of all the stats on the various interfaces. Amongst other things, this can show you errors and also show if you have a biased to any of the interfaces (due to the load balancing algorithms, or possible errors with that).

Have a look through the output, or possibly post it here and I can cast my eyes over it for you.

The other thing to try is reduce the VIF to a single port. This will almost totally rule out any issues with the teaming used on either side (NetApp or Cisco).

Do you know if the MTU has been changed at all (default is 1500, but often is changed to 9000).

discretixit · ‎2009-03-18

ifstat -a

-- interface e0a (187 days, 9 hours, 24 minutes, 15 seconds) --

-- interface e0b (187 days, 9 hours, 24 minutes, 15 seconds) --

-- interface e0c (187 days, 9 hours, 24 minutes, 15 seconds) --

-- interface e0d (187 days, 9 hours, 24 minutes, 15 seconds) --

-- interface lo (187 days, 9 hours, 23 minutes, 49 seconds) --

RECEIVE
Packets:         22488k | Bytes:           45935m | Errors:              0
Queue full:          0
TRANSMIT
Packets:         22488k | Bytes:           45935m | Errors:              0
Collisions:          0

-- interface vh (187 days, 9 hours, 23 minutes, 49 seconds) --

RECEIVE
Packets:             0 | Bytes:               0 | Errors:              0
Queue full:          0
TRANSMIT
Packets:             0 | Bytes:               0 | Errors:              0
Collisions:          0

-- interface vif0 (187 days, 9 hours, 23 minutes, 33 seconds) --

chriskranz · ‎2009-03-18

That all looks mostly okay. You have no errors or collisions, very few discared and most of the traffic seems to be perfectly fine. The only thing I noticed is that there are a few (although not hundreds) of "Up to downs" showing the the interface has gone down a certain number of times. This may be from your testing, but could also show the ports are flapping (being set down and up by the switch).

I know you have already given some details on this, but can you double check the VIF mode on the filer, and what the link aggregation is setup on the Cisco side please? Can you also double check the ports that are in the VIF, and that they are into the correct ports on the Cisco? I know this is basic stuff, but it is often useful to take a step back and completely rule this out.

discretixit · ‎2009-03-18

hi

i check the port on cisco they connect to correct place.

vif status vif0
default: transmit 'IP Load balancing', VIF Type 'multi_mode', fail 'log'
vif0: 3 links, transmit 'Round-Robin Load balancing', VIF Type 'multi_mode' fail 'default'
VIF Status Up Addr_set
up:

also i add screenshot on cisco etherchannel conf (2323.bmp)

thanks

chriskranz · ‎2009-03-18

You have the VIF configured with round-robin load balancing? I would also recommend to use either IP or MAC based hashing. Can you try change this back to the default (Source based IP Hash) and see if you still get the same issues please? I think this may be conflicting with the Cisco configuration.

discretixit · ‎2009-03-18

hi

i change the vif from rr to ip :

vif status vif0
default: transmit 'IP Load balancing', VIF Type 'multi_mode', fail 'log'
vif0: 3 links, transmit 'IP Load balancing', VIF Type 'multi_mode' fail 'default'
VIF Status Up Addr_set
up:

and the same problem (i copy file from server to NetApp with ~17 MB/s)

any ideas ?

danielpr · ‎2009-03-19

Can you please share your /etc/rc file and ifconfig -a output ?

Thanks

Daniel

discretixit · ‎2009-03-19

hi

ifconfig -a
e0a: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        ether ******* (auto-1000t-fd-up) flowcontrol full
        trunked vif0
e0b: flags=4848043<UP,BROADCAST,RUNNING,MULTICAST,NOWINS> mtu 1500
        inet *.*.*.* netmask 0xffffff00 broadcast *.*.0.255
        ether ****** (auto-1000t-fd-up) flowcontrol full
e0c: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        ether ***** (auto-1000t-fd-up) flowcontrol full
        trunked vif0
e0d: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        ether **** (auto-1000t-fd-up) flowcontrol full
        trunked vif0
lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188
        inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1
vif0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet ***** netmask 0xffff0000 broadcast *.*.255.255
        ether *** (Enabled virtual interface)

rc output

#Regenerated by registry Tue Mar 10 16:01:30 IDT 2009
#Auto-generated by setup Thu Jan 19 08:10:19 IST 2006
hostname
vif create multi vif0 -b ip e0d e0c e0a
#Disabled# ifconfig e0a `hostname`-e0a mediatype auto flowcontrol full
ifconfig e0b `hostname`-e0b netmask 255.255.255.0 mediatype auto flowcontrol full -wins
ifconfig vif0 `hostname`-vif0 netmask 255.255.0.0 mtusize 1500 wins
ifconfig e0a mediatype auto
ifconfig e0c mediatype auto
ifconfig e0d mediatype auto
route add default 172.16.1.1 1
routed on
options dns.domainname domain.com
options dns.enable on
options nis.domainname domain
options nis.enable on
savecore
timezone Israel

danielpr · ‎2009-03-19

Cohen,

Thanks for sharing the information, the problem is looking different here. You are very sure about the MTU size and the auto speed settings in all the

three medium but still you are facing the problem. We need to debug more here, Can you please share "netdiag" and "sysconfig -v"output.

Thanks

Daniel

discretixit · ‎2009-03-19

hi

thanks again

danielpr · ‎2009-03-19

Hi,

The output clearlysays there is the problem with the other side network components (Switch & System) basically the 244

is very low in CIFS. You need to check the iconfiguration of the interface "from host: **** " . There is no issue with the

Netapp Storage System side.

>Average size of CIFS TCP packets received from host: **** is 244.
>This is less than the MTU (1500 bytes) of the interface involved in
>the data transfer.
>The maximum segment size being used by TCP for this host is: 1460.
>Low average size of packets received by this system might be
>because of a misconfigured client system, or a poorly written
>client application.
>Press enter to continue

Thanks

Daniel