Snapmirror transfer rate over 10GbE link

[BoJlk]

Hello everyone,

I can't figure out the issue why my snapmirror over 10Gbps network not exceeding 370Mbps.

I've created a snapmirror between two clusters FAS2750 and FAS2552

Both have SAS disks, they're conncted to the same Juniper 10GbE switch.

FAS2750 have LACP configured for two ports, FAS2552 have one dedicated port for replication.

1. disabled Flow-Control on Juniper and Netapp

Juniper = set interfaces xe-0/1/6 ether-options no-flow-control

ONTAP = network port modify -node <node> -port e0d -flowcontrol-admin none

2. increased TCP windows size (both clusters)

options snapmirror.window_size (8,388,608 bytes for 10GbE network)

https://library.netapp.com/ecmdocs/ECMM1278318/html/onlinebk/protecting/task/t_oc_prot_sm-adjust-tcp-window-size.html

3. Throttle disabled (both clusters)

replication.throttle.enable off -
replication.throttle.incoming.max_kbs unlimited -
replication.throttle.outgoing.max_kbs unlimited -

Juniper interface transfer rate 387315168 bps rate

monitor interface xe-0/1/6

ONTAP interface transfer rate 38.9 MB

statistics show -object nic_common -instance e0d -counter rx_bytes -node <node> 

node run -node <node> sysstat -c 10 -x shows

Source 30% CPU and 80% Disk Utilization

Target 90% CPU and 90% Disk Utilization

i'm lost and made too many changes to find the root cause, any advices ?

[mjdalton1]

Just a thought, Are you using MTU 9000 end to end?

High CPU on the 2750 looks like it's rebuilding packets.

# from a desktop
ping -l (buffer size ie 9000) -f x.x.x.x (destination) Test across entire network

[BoJlk]

Nope, using MTU 1500 end to end.

Juniper interface shows 1514 and NetApp broadcast domain set to 1500 mtu.

Protection policy is set to use Network Compression.

[paul_stejskal · Accepted solution]

Those systems will not max out a 10Gb link. SnapMirror is not designed to be a super high transfer of data, but more in the background. With BRE throttling it will be even slower so it doesn't use all CPU resources from frontend ops.

Was your system sold to be able to do 10Gb SnapMirror throughput or sized for that? What is the expected speed?

Also, if your disk usage is 80-90% busy, more than likely you don't have the headroom on your disks to move data very fast.

If it really is a concern, please open a case and we can help direct things, also check network for things like packet loss.