Read performance is very poor in nfs share

SagarDH · ‎2017-01-04

Hi,

I have Netapp 3220 Box and It has 7 mode 8.1.

When i am trying to rsync by mounting in linux box or taking backing using netback mounted on linux as a nfs share. It is taking 5 to 6 days to complete 1.4TB size disk. I dont know why this much of the time it is taking time.

We have 10G uplink for network data connectivity.

I might have done wrong configuration. Please advice to fix this issue.

Below is the share

/vol/home1 -sec=sys,rw,root=10.193.180.0/24:10.193.182.0/24:10.193.184.0/22,nosuid

and Below is the snapshot of the performance. and Please let me know any thing i need to provide info

Read performance using DD

[root@vncin56 ~]# dd if=/home1/home1.img of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.888529 seconds, 1.2 GB/s

Write performance using DD

[root@vncin56 ~]# dd if=/dev/zero of=/home1/home1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 8.089 seconds, 133 MB/s

configured with Parity DP and we have 24 SATA 1.62TB disks and 1 disks as spare.

Thanks

parisi · ‎2017-01-04

Have a look at XCP.

http://xcp.netapp.com

tmac · ‎2017-01-05

Sounds like your network may be misconfigured.

If you are using 10-gig, check all this:

1. on the netapp -> ifstat -a (command in 7-mode or a node shell command in cdot)

-> look for any errors. Errors could be bad cable, bad transceiver, etc.

-> Flow control: make sure FLOW CONTROL is OFF. This is indicated near the end of the output

2. Jumbo Frames. Are you using them? are htey configured all the way through?

SagarDH · ‎2017-01-05

Hi TMAC,

Jumbop frames not enabled.

When try to make none for flow control

ifconfig my10g flowcontrol none
ifconfig: (my10g) : SIOCSFLOWCONTROL - Operation not supported by device

bheema> Thu Jan 5 20:36:04 PST [bheema:net.ifconfig.ioctlError:info]: ioctl error encountered while configuring interface 'my10g' using ioctl 'SIOCSFLOWCONTROL' (error string='Operation not supported by device', error code=19).

and For device access /Management also i am using same my10G interface and not e0M

my10g interface in using for nfs shares.

ifstat -a

-- interface e0a (64 days, 22 hours, 11 minutes, 57 seconds) --

-- interface e0b (64 days, 22 hours, 11 minutes, 57 seconds) --

-- interface e1a (64 days, 22 hours, 11 minutes, 57 seconds) --

-- interface e1b (64 days, 22 hours, 11 minutes, 57 seconds) --

-- interface c0a (64 days, 22 hours, 11 minutes, 57 seconds) --

-- interface c0b (64 days, 22 hours, 11 minutes, 57 seconds) --

-- interface e0M (64 days, 22 hours, 11 minutes, 57 seconds) --

-- interface e0P (64 days, 22 hours, 11 minutes, 57 seconds) --

-- interface lo (64 days, 22 hours, 11 minutes, 52 seconds) --

RECEIVE
Packets:          9323 | Bytes:             943k | Errors:              0
Queue full:          0
TRANSMIT
Packets:          9323 | Bytes:             943k | Errors:              0
Collisions:          0

-- interface vh (64 days, 22 hours, 11 minutes, 52 seconds) --

RECEIVE
Packets:             0 | Bytes:               0 | Errors:              0
Queue full:          0
TRANSMIT
Packets:             0 | Bytes:               0 | Errors:              0
Collisions:          0

-- interface losk (64 days, 22 hours, 11 minutes, 52 seconds) --

RECEIVE
Packets:             0 | Bytes:               0 | Errors:              0
Queue full:          0
TRANSMIT
Packets:          1281k | Bytes:            1237m | Errors:              0
Collisions:          0

-- interface my10g (64 days, 22 hours, 11 minutes, 46 seconds) --

tmac · ‎2017-01-06

Looks like e1a / e1b are the interfaces for you my10g interface.

Please send this info also:

ifgrp show -instance

net port show -port e1a|e1b -instance

If you want to turn of FlowControl on the 10G interface, you need to do in the physical interface, not the group:

net port modify -node xx -port e1a|e1b -flowcontrol-admin none (where xx is your node name)

--> this is mildly disruptive to set. This will disable at the netapp, you really should make sure it is disabled at the switch as well.

Looks like a bad cable. are you using fiber or copper?

You are getting lots of CRC errors on the receive (aka writes).

Bad transciever? bad cable? time to troubleshoot.

maybe even open a case

SagarDH · ‎2017-01-08

Hi Tmac,

ifgrp show command not found

Interface group(trunk) my10g
        e1b                 e1a
Pkts In   Pkts Out Pkts In   Pkts Out
14125m    6150m     783m      759m
Sun Jan 8 21:15:45 PST [bheema:netif.linkUp:info]: Ethernet e1a: Link up.
5390      5276      1         1
6267      5398      4         2
13446     10525     5         1
5258      5102      16        2
4825      4681      18        2
5327      5197      11        1
5212      5089      7         2
5368      5252      3         2
6776      5924      6         1
8480      7262      12        2
4951      4820      21        2
4988      4855      10        1
4946      4825      5         2
5546      5348      8         2
5382      5251      6         1
5053      4911      13        2
4949      4796      17        2
5009      4875      12        1
4940      4815      10        2
5396      5211      7         2
6109      5897      2         1

Checked cables we are using copper cable it is working.

Moved my10g (ifgrp) to e1b and e1a but same issue.

Disabled flowcontrol in both the interfaces e1a and e1b. Still not able to fix.

Attached my cabling diagram is that correct cabling ? WE have only one controller and one Disk shelf.

tmac · ‎2017-01-09

Please send:

ifstat -a (make sure to include the e1a/e1b/my10g)

lets take a look at that.

SagarDH · ‎2017-01-09

Hi Tmac,

complete output.

bheema> ifstat -a

-- interface e0a (68 days, 4 hours, 59 minutes, 53 seconds) --

-- interface e0b (68 days, 4 hours, 59 minutes, 53 seconds) --

-- interface e1a (68 days, 4 hours, 59 minutes, 53 seconds) --

-- interface e1b (68 days, 4 hours, 59 minutes, 53 seconds) --

-- interface c0a (68 days, 4 hours, 59 minutes, 53 seconds) --

-- interface c0b (68 days, 4 hours, 59 minutes, 53 seconds) --

-- interface e0M (68 days, 4 hours, 59 minutes, 53 seconds) --

-- interface e0P (68 days, 4 hours, 59 minutes, 53 seconds) --

-- interface lo (68 days, 4 hours, 59 minutes, 48 seconds) --

RECEIVE
Packets:         20195 | Bytes:            1664k | Errors:              0
Queue full:          0
TRANSMIT
Packets:         20195 | Bytes:            1664k | Errors:              0
Collisions:          0

-- interface vh (68 days, 4 hours, 59 minutes, 48 seconds) --

RECEIVE
Packets:             0 | Bytes:               0 | Errors:              0
Queue full:          0
TRANSMIT
Packets:             0 | Bytes:               0 | Errors:              0
Collisions:          0

-- interface losk (68 days, 4 hours, 59 minutes, 48 seconds) --

RECEIVE
Packets:             0 | Bytes:               0 | Errors:              0
Queue full:          0
TRANSMIT
Packets:          1347k | Bytes:            1299m | Errors:              0
Collisions:          0

-- interface my10g (68 days, 4 hours, 59 minutes, 42 seconds) --

Thanks,

tmac · ‎2017-01-09

Try different cables or different ports...something.

You are seeing CRC errors on both e1a/e1b

This almost always indicats bad hardware someplace.

The cable, the port on the switch, the port on the netapp.

Time to bust out troubleshooting skills 101 and find the problem.

Are the cables Cisco or at least NetApp Certified?

may want to get a case opened to help out.