are you using jumbo frames everywhere or just on the snapmirror interfaces? make sure every switch between the source and destination filer is configured correctly for jumbo frames.
which size are the jumbo frames? if the switches and Ontap are configured to 9000 and you are using vlans, then either set the switches bigger or Ontap smaller.
check your routing table (route -s). if you have e0M on the same subnet as another interface, then sometimes the default gateway routing goes over it (and that's 100Mbit!). then you need to redesign your network config...
just to be sure I would temporarely set the MTU to standard 1500 (just on source and destination filers).
we had a customer with jumbo frames problems on 10GbE and it was an issue on the switches. they just couldn't handle jumbos correctly with 10GbE.
another thing may be flowcontrol, we always disable it on 10GbE: ifconfig eXa flowcontrol none. I've seen it at some customers that Netapp Professional Services also disables it on all 10GbE interfaces.
I echo the above... I've also seen instances where a snapmirror appears "stuck" When everything checks out and the configurations are correct, I've had to abort the snapmirror and start it over and everything worked fine.
Question - can you post an output of the following:
- Traceroute from source to destination IC LIF
- routing-group show
- routing-group route show
I'm just curious
So I've been battling this same issue for a client for a large part of the day today.
FAS8040 at source, FAS3220 at destination. Both sites using 10 Gb interface groups for data with respective VLAN tagging. Using the same data interface group vs. a dedicated port (client doesn't have the proper SFPs for the other UTA2 ports to dedicate - maybe in the future). The client's network admin was only seeing about 1 Gb utilization over the 10 Gb link between sites. So I was like..yeah...great...I've seen this one before.
I ran into two issues:
1) Routing group configuration
2) Jumbo Frames configuration
When the node intercluster LIFs got created, it created the respective routing groups (but not routes). The VLAN isn't spanned, so I needed to create the proper routes. Did that. Can ping fine. Yay! Still had issues. So I ran a traceroute from the IC LIF on Node A at souce site to the IC LIF on Node B at the destination site. Even with the routing group (and route) in place, the first hop at each site was to the node / cluster management VLAN on the respective node (which was interesting to me since management is on 1) a different physical port and 2) a different subnet.
The route for inter-cluster was using a destination of 0.0.0.0/0 (but with the proper gateway - the SVI of the SnapMirror VLAN). Decided to delete the route and create a new one. So, I went old skool 7-Mode static route on it and set the proper destination (the subnet of the destination site) and the same gateway (the local SnapMirror VLAN SVI). Now the trace comes back clean and the first hop is the SnapMirror VLAN SVI. w00t!
Still won't work.
Turns out that there WASN'T Jumbo Frames all the way from source to destination. The topology has 5Ks at each site then two 7Ks in between the sites over to 10 Gb links. I believe the network admin forgot to set the per-VLAN MTU size for the SnapMirror VLAN on the 7Ks to 9000. I'll get him to fix that tomorrow.
So I went back and set the MTU to 1500 on the VLAN interfaces on each node. Aborted / restarted the SnapMirror relationships (they seemed hung). Now it works like a charm
Hope to have some metrics to compare against in the AM to validate.
Another thing I could think to check would be the ingress/egress of the ports. Perhaps some QoS policy issues at the network layer (probably unlikely but worth looking in to - I've seen cases where someone would set QoS at the switch level where bandwidth was getting choked to death for Ethernet traffic).