ONTAP Discussions

Maximum SnapMirror Speed on a 6080

gjmiddleton
8,624 Views

Hello! 

I have seen a lot of documentation on throttling SM, but does anyone know what a 6080 is capable pushing when it comes to SM?  I know it depends on how many disks make up the RGs in the aggregate for which the SM source resides, but given the largest aggregate size using 600GB SAS drives, on a relatively quiet system with 10GbE, could it actually push 6,000-8,000 Mbps?

Is there a guide that talks about the theoretical throughput maximums for each filer model?  Again, I realize there are a lot of "it depends" scenarios that we could run through, but what is a 6080 capable of?     

Thanks,

Jason

14 REPLIES 14

fjohn
8,574 Views

You'll need to increase the window size for snapmirror for one.  I believe the formula is in the DPG.  Then, it's a matter of bandwith.  In a recent engagement, with multiple snapmirror sessions (multiple volumes in multiple protection manager jobs) running simultaniously across multiple 1Ge NICs running though Riverbed compression across dual 600Mbs WAN connections with 60ms latency over a distance of 2500 miles, I was able to get pretty close to that range with 3170s.  The limitation was bandwith.  The filers were not getting pounded.

J

johnlockie
8,574 Views

I have seen 2040's eat at a 1Gb LAN network like it was nothing while seeding in our labs.  So I have no doubts that a 6000 series could run at 6x that....

Like the post above says you are going to see more limitation on the wire before anywhere else - especially in the 6000 series.  I also like to point out we were running multiple jobs too.

Note that Riverbed can produce results up to 70% on SnapMirror traffic (I believe this is compression, but regardless for every 1Mb you get another 700Kb).  I have seen this in my networks....quite amazing actually.

So if you think about it, if you saw 10Gb in your WAN, potentially the NetApp is actually pushing ~17Gb.....and so on for larger circuits.  In fact, you can see this in kB/out in sysstat.....monitor the WAN interface of your router, while running sysstat with Riverbed 1050 between you will be amazed.

thomas_glodde
8,574 Views

Hi,

not sure if it helps you but you can also multipath snapmirror transfers, afaik up to 4 links. They can be used simultaniously for one sm transfer!

But using a FAS6080 with plenty of SAS disks, yep, im sure you can almost max that link out.

Kind regards

Thomas

johnlockie
8,574 Views

Are you referring to VIFs?

gjmiddleton
8,574 Views

No, but you bring up another good way to word my question.  If I create a VIF with two 10Gb NICs, can a 6080 push a SM at 20Gb to a destination that can also handle 20Gb transfers? 

Bottom line, what is the maximum Ethernet BW that a 6080 can push?  VIFs or no VIFs.  Whatever config is required to push the maximum is what I am after.  I do understand that rg and aggr sizes matter, but laying that aside, what can a 6080 push?

Thanks again,
Jason

gjmiddleton
8,574 Views

Am I understanding you correctly in that I could max out two 10Gb links?

Jason

johnlockie
8,574 Views

Do you mind if I ask you, why are you needing to know?

I do not know a single person who has a dedicated 20Gb WAN link for snapmirror.  In fact, I do not even know anyone with that period, let alone such a thing dedicated for replication.


That said, sure - "in theory" maybe a 6000 series can rip the wire at 20Gb running just snapmirror replication.  But even if it could I think this is a poor design choice.

If you have this kind of network strength (and cash) at your disposal you probably need to be looking at MetroCluster designs.

gjmiddleton
8,574 Views

Hi John,


The speed of a WAN link is irrelevant info with regard to answering my question, not to mention that I never mentioned WAN in my previous posts :). 

I am simply interested in knowing what a 6080 is capable of pushing with regard to SM over an Ethernet link. 

You are correct in that the customer does not have a 20Gb WAN link dedicated to SM, but they do have a 10Gb WAN link that is currently dedicated to SM.  Hence the need to optimize SM seeds at greater than 10Gb on the LAN side in order to get more data over the WAN in a smaller window. 

Thanks for your help!

Jason

johnlockie
8,574 Views

Fine.

No question in a LAN you can push 10Gb easily running SnapMirror - LAN or WAN actually (I think we have all already stated that).

Note - the reason I asked about the VIF thing is because that is NetApp's way of saying LACP.  You will need to configure load balancing properly on the VIF pair and even then there is no guarantee snapmirror will use anything greater than the 10Gb.  NetApp uses source MAC and/or source IP I believe (or a hash of both) to determine which physical segment to send layer 2 traffic on.

I would refer you to the SnapMirror Guide, because I am sure they touch on this (possibly if NetApp has a white paper on VIF behavior look there).  Think: LAYER 2.  Off the top of my head I know that there is a round robin option in NetApp, and I would probably set that in a lab and see what happens.

The point is, SnapMirror is filer to filer communication.  This means little to no variance in network source and destination frames/packets (l2 or l3).  So your VIF is going to be more like a failover than it is an actual etherchannel aggregation.  I would almost bet my lunch that if we configured a 20Gb VIF and pushed snapmirror you will not see it  crest above 10Gb.  This is because of layer 2 limitations.

Creating a VIF is the only way I know to extend 10Gb beyond 10 in to 20, 30, or 40Gb.

I would recommend without question if the client is looking to deploy Riverbeds (and has not already) that you place the riverbeds in path in the LAN assuming you seed the SnapMirror in a LAN. Note - if the customer already has Riverbed deployed and it truly is a 10Gb WAN connection then there is no doubt you should seed this over the WAN.  The point is, make sure that the Riverbed appliances are between these heads when you seed.

fjohn
7,073 Views

It needs to be a multimode VIF, and SnapMirror compression will help as well.

JohnFul

johnlockie
7,073 Views

I am talking about multimode VIF's.....not single mode.

I am willing to bet in multimode you will not see much greater than 10Gb with SnapMIrror.

NetApp default is IP load balance on multimode VIF's, but you can change this to MAC and source/destination.  But the nature of SnapMirror is the issue.  It's not like you have a thousand clients connecting that it can "load balance" accross the physical links on the virtual interface....


Just saying.  I would love to find this out for sure, unfortunately my labs are not rich enough to afford me to play with 10Gb links!!!!

shane_bradley
7,073 Views

The behaviour should be the same with 1GbE ifgrps or 10GbE ifgrps, would be interesting to test if you had the time.

shane_bradley
7,073 Views

Yeap, Multimode vifs would most likely not help. by default that are balanced via IP (you can also set them to Round robin, or Mac based)

Given you snap mirror relation ship is most likely going to be a one to one adress relationship you wont blow out more than the total of a single link. You could work around this possibly with multiple IP's and multiple snapmirror sessions configured.

As for how fast you can get a FAS6080 to go, single 6080 node simple read, straight from cache (i.e. the best possible read load) the highest i've ever been able to get a FAS6080 running 7.3  was around 1600MB/s  that was with a few hundred linux workstations all requesting the same data set at the same time.  Under "normal" loads we generally dont see more than 1000MB/s, and thats screaming its guts out.

FrostByte
7,073 Views

For what it's worth, we are maxing out at 450Mb for snapmirror over 10Gb links on our snapmirrors happening in the building.

Source is 6080 with 120 15k SAS drives with 512 GB PAM pushing to two locations at once (partner head and a third head).  Partner has 80 15k FC drives and the third head is a 3140 vFiler on top of a third party array.  Source array is relatively underused pusing some 8,000-12,000 during day to day activities.

Network consists of two Nexus 5010s that are stacked.  The VIFs on the filers are single mode.  Switches have almost no traffic on them as they were purchased for the snapmirrors and are not yet in use for other systems.

We have set MTU to 9000, adjusted window size, disabled/enable delayed_acks.  An SE (not ours) suggested that we were bumping into a system throttle and that it may need to be adjusted to get higher throughput... something done from debug mode.

Public