So in my most recent scenario, and after some additional testing, here's what I can see. Mind you, I'm still waiting on the client's network team to validate the results from the latest transfer.And keep in mind that this testing is over a 10 Gbps WAN link, not local LAN. Latency is around 40 ms. Last test was done with MTU 9000 from end-to-end (a test earlier in the day with MTU 1500 yielded approximately the same transfer rates / times, give or take a few minutes, and granted, the bump from 1500 MTU to 9000 MTU is going to be leveraged far more efficiently by things that take better advantage of the larger overhead, like iSCSI, so I don't expect a drastic difference with 1500 MTU vs. 9000 MTU SnapMirror traffic).
Topology: Site A Cluster --> 5K --> 7K --> 7K --> 5K -- Site B Cluster
Using InterCluster SnapMirror over intercluster LIFs (one per node) on a 2-port 10 Gb ifgrp (data ports) VPC using LACP
Separate VLAN per site (so routed VLAN, not extended via OTV)
As a test earlier, I set up a SnapMirror for a 1 TB volume from Site A (Prod) to Site B (DR).
If I'm calculating things correctly (and again, if someone can validate if this is not correct, please call it out),
Theoretical limit of a 10 Gbps links is 1.25 GBps
Using the calculation of 10,000 bits / 8 = 1,250 bits
Transferred 1 TB of data from Site A to Site B in 1 hour
Using the calculation of 1 TB = 8796093022210 bits
Using the calculation of 1 hour = 3,600 seconds
So then (if that's all correct), 8796093022178 bits / 3600 seconds = 2443359173 bits
Convert 2443359173 bits to MB = 291.27 MBps = 2.275 Gbps = 0.284 GBps
So at 0.284 GBps, and a theoretical line rate for 10 Gbps of 1.250 GBps, that translates to 22.72% efficiency. Granted, this could be expected, due to additional configuration issues in the data path, or something else I haven't come across yet.
(I'm sure there's a quicker way to calculate that math, this is just how I walked through it in my head and on paper to get a better understanding)
Main thing for me here is that I'm trying to gauge how long some other, larger transfers are going to take (say, 10-15 TB or so) and to determine if the process is working as expected according to client expectations (again, not knowing if there might be some network-related issues they need to examine to improve efficiency, if if they run something like iPerf, they get the same results). The client has never had to push this much data across the WAN before, so it's essentially their first real test of high throughput on the links (they actually have two, redundant 10 Gbps links, but it's more so for that - redundancy - and not performance). Question is - can the Production cluster at Site A really PUSH total expected line rate with the transfers (most likely not), so are these results desirable? Here, in this scenario, I would think so. Will I see any any queuing or drops? I don't think I will, but we'll see.
So to the OPs point (and maybe to answer your question as well), he was seeing only about 2 MBps over a 10 Gb link, and if that's a local LAN link, the efficiency should be much higher. In this case, I achieved 291.27 MBps over a 10 Gb WAN link at a far lower (expected) efficiency rate.
