NFS performance on 10Gb link

[BLACKRABBIT]

Hello all!

I have an interesting task: write to array a dump of network traffic from 10Gbit/s channel.

Topology is quiet simple: network tap, capture server and NetApp array.

NetApp array directly connected to the server via another 10Gbps port. Also I am using NFSv3.

Now I tested this solution with one FAS6250 in cluster mode together with 20HDD of SAS drives or 20HDD of SATA drives.

Performance is the same (SATA/SAS) - near 450MB per second. This is 2 times slower than I expected... 😃

We planned to use HA pair of FAS8040 in the future and expected 3000MB speed (2x10Gb links).

Now we have a lot of questions. Is it possible? May be we missed some configuration? Could you help me with performance tuning or it is impossible?

On the capture server (NFS client) side I have mount options:

type nfs (rw,bg,hard,intr,rsize=1048576,wsize=1048576,vers=3,proto=tcp,timeo=600,retrans=2)

Writing has 32 threads inside.

Also I tuned the sysctl.conf parameters:

net.core.rmem_default=524288

net.core.wmem_default=524288

net.core.rmem_max=16777216

net.core.wmem_max=16777216

net.ipv4.tcp_rmem=4096 524288 16777216

net.ipv4.tcp_wmem=4096 524288 16777216

net.ipv4.ipfrag_high_thresh=524288

net.ipv4.ipfrag_low_thresh=393216

net.ipv4.tcp_timestamps=0

net.ipv4.tcp_window_scaling=1

net.core.optmem_max=524287

net.core.netdev_max_backlog=2500

sunrpc.tcp_slot_table_entries=128

sunrpc.udp_slot_table_entries=128

I will wait your feedback.

Thank you in advance!

[shane_bradley]

We'd really need to see a perfstat or at the very least a statit

[BLACKRABBIT]

Of course! This is perfstat of the best result that we have: [link deleted]

Write speed is near 900MB via two 10Gb links. Aggregate -max-write-alloc-blocks was set to 256K.

[shane_bradley]

So this is 100% writes? IF so i wouldnt expect an 6250 to have much more in it.

We would get about ~1.3GB/s max write onto our FAS6280's the best i've seen was a FAS80x0 with SSD doing about 3.5GB/s (but that was disk bound on 20 data spindles.)

[BLACKRABBIT]

Thank you for the answer!

Yes. My result for 100% write.

Your results only for one node? Can I expect double result for the HA-pair?

[shane_bradley]

Yeah thats one node.