topic Re: ESX setup using fas3140 and cisco 3750E switches in VMware Solutions Discussions

ESX setup using fas3140 and cisco 3750E switches

ian_campbell — Thu, 05 Jun 2025 07:18:42 GMT

Hi,

We're looking at configuring our ESX vsphere cluster to take advantage of 10G ethernet throughput on a FAS3140 dual controller filer using 2 x Cisco 3750E switches.

It is my understanding that the 10G uplink ports on the 3750E are designed as interconnects or uplinks rather than directly attaching to 10G etherport ports on the netapp or ESX hosts.

I came across this post below involving a similar setup.

http://episteme.arstechnica.com/eve/forums/a/tpc/f/469092836/m/521000722931

One of the comments states:

"That is part of why I can't figure out why you'd use 3750 (even E) for 10G. The 10G is for uplinks on those. The stacking connector barely supports the bandwith of a couple of full (48 port) gigabit line rate switches, I can't remember if it is 36G or 72G- and that maybe non-e vs E speed. Stacking is for manageability and edge ports not aggregation and high availability of 10G nodes."

I know that cross stack LACP is now supported so that should be fine however is this a valid option to be using these 10G ports for this purpose. My idea is that each 3140 controller would have a 10G connection to each of the 2 switches and then I'd port channel 4 GB ports on each ESX host across the 2 switches.

Can anyone advise if this is a good configuration keeping in mind my primary goals are:

1. Throughput

2. Redundancy

3. Layer 3 routing

Sorry if I'm posting this in the wrong place.

Thanks,

Ian

Re: ESX setup using fas3140 and cisco 3750E switches

BrendonHiggins — Fri, 12 Feb 2010 10:02:49 GMT

Hi, welcome to the comminty and you have found the right place.

What version of DoT do you have on your filers as this will limit you 10 gb options? The switch ports are indead uplinks so you will need different switches if you want to build a 10 Gb system. There may be a kludge available but every time there is any type issue support will say your design is "non standard" at best, so it is just not worth going down that road.

http://www.cisco.com/en/US/products/ps7077/prod_models_home.html

Hope this helps

Bren

Re: ESX setup using fas3140 and cisco 3750E switches

BrendonHiggins — Fri, 12 Feb 2010 10:07:47 GMT

Also flagged this thread to a couple of network chaps who have much greater understanding than mine.

Bren

Re: ESX setup using fas3140 and cisco 3750E switches

treyl — Fri, 12 Feb 2010 14:11:41 GMT

Ian

My name is Trey. I am a Consulting SE at NetApp focused on Ethernet Storage Technologies. I am also a former Cisco employee so can answer some of the architecture specifics of the 3750E. One of the changes or introductions with the E version of the 3750 was the enhancement to the stacking architecture Cisco calls Stackwise to Stackwise+. The Stackwise interconnect in the older 3750 (non E series) had a 32 Gigabit per second bandwidth. Now that 32Gbps path was actually two counter-rotating 16Gbps paths yielding a bi-directional total.

Check out this link for details in differences between Stackwise and Stackwise+ on the 3750E platform.

http://www1.cisco.com/en/US/prod/collateral/switches/ps5718/ps5023/prod_white_paper09186a00801b096a.pdf

There are several enhancements of the Stackwise+ technology and the Catalyst 3750E. I find that many customers haven't gotten the update on what differences actually existed between the 3750 and the E-series so sometimes folks referring to the 3750E will quote specs from the 3750. Here are a few differences.

Catalyst 3750G

Port ASIC controls ring, There is no switch fabric

3750 has external TCAMs

3750 only runs StackWise mode 32Gbps (Bi-Directional 16Gbps)

1:1.6 Oversubscription - Blocking Architecture

Source Strip of Frame from Stacking Ring

Catalyst 3750E

Switch fabric allows local switching

128Gbps Stacking Bandwidth (Bi-Directional 64Gbps)

Non-Blocking Switch Architecture

Destination Strip - Spacial Reuse from Stacking Ring

So, those differences produce the following architecture limits in the 3750 that are not present in the 3750E.

3750 does not have a local switch fabric preventing it from being able to locally switch, it must send a frame on the ring to determine its destination. The sources of this are centered around the TCAM subsystem which the 3750 uses versus the 3750E having TCAMs in local port ASICs. TCAMs are basically the memory in a switch that is used to perform lookups to execute L2 switching or L3 routing. < Trying not to write a book here.....

Read this for an explanation of resource limits which can be reached on the 3750 (not the E series)

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00801e7bb9.shtml

Now, with regard to use of 3750E with a 10Gbps workload on ports which are considered uplinks. You are correct in that Cisco is positioning the 3750E 10G ports as uplinks. The changes in the E architecture enable the switch to be used in different, more data center oriented workloads. I would tell you that it is always preferred that we deploy Nexus 5000s for data center Ethernet Storage connectivity. However, realities are that many have 3750s which they are using in data centers. The E Series introduction of 10Gbps and architecture changes enable it to support 10Gbps successfully and can tell you that we have several customers who are using E series switches for 10Gbps connectivity. They are running fine and happy 3750E with NetApp customers. The link to provided to the episteme.arstechnica.com site was/is a configuration issue versus architecture of the switch. The topic on running LACP is odd to me because when I research the releases presently available for the 3750E I see no restriction documented for LACP. There is even a section regarding EtherChannel Configuration Guidelines which don't depict this limitation. However, not having access to the 3750E immediately I can state that if you can't configure LACP the simply use Static Etherchannels. I find that there is so much confusion in general about this because so many people use so many different terms. In a nutshell, it is our suggestion that you always run LACP because of the benefits of that protocol, right up until the point that you can't run it. When you can't simply run static etherchannels. The configuration document linked to for the 3750E above actually states this as well.

LACP in Cisco command lingo is "channel-group mode active" or "channel-group mode passive". When you are configuring LACP to a NetApp always use "active".

Static Etherchannels in Cisco command lingo is "channel-group mode on"

I detail this entire concept here along with all the associated NetApp commands:

http://communities.netapp.com/blogs/ethernetstorageguy/2009/04/04/multimode-vif-survival-guide

With regard to configuration best practices in general the same best practices that exist for Gigabit Ethernet deployments with NetApp apply to 10Gbps. There aren't typically any differences that we would call out.

I have documented a few recomendations for mixing 10G with 1G configs here:

http://communities.netapp.com/blogs/ethernetstorageguy/2009/09/13/8021q-vlan-trunking-with-mixed-jumbo-standard-ethernet-frames-using-second-level-vifs

The concept of Multi-Chassis Port Aggregation is documented here as there is a different technology for just about every type of Cisco switch you wish to leverage this functionality with:

http://communities.netapp.com/blogs/ethernetstorageguy/2009/09/23/virtual-port-channels-vpc-cross-stack-etherchannels-multichassis-etherchannels-mec--what-does-it-all-mean-and-can-my-netapp-controller-use-them

Now our best practices for VMware vSphere have recently been updated and the formal release of our best practices guide is scheduled for February 18th. It has been available on lulu.com for a few weeks now ( TR-3749 ) < for those of you who want it in book form. Next week you can get it online via PDF from our Technical Library in TR-3749 we document our Networking best practices as they relate to vSphere with NetApp for NFS to include 10Gbps and Jumbo Frames.

Finally, I would suggest a couple of things regarding cabling connectivity as it is a hot topic because of cost. If the 3750E is near (within 5 meters) of the NetApp then you are a candidate for using twinax cabling. Twinax cabling is available in 1, 3 and 5 meter lengths as is far less expensive than the optical 10G interfaces you would typically be required to deploy. When using twinax cabling the cost of 3 1Gbps ports is the equivilent of a single 10Gbps port. Just doing some rough math. A twinax cable includes the SFP+ interface at both ends of the cable for about $260 list price. When using optical 10G you must purchase a 10G optical SFP both both sides of the connection which are typically around $1900 list each. When specifically talking about NetApp 10G interfaces we sell a 10G card which supports twinax cabling and actually sell Cisco branded twinax cables. I have detailed the different card options here: NetApp 10G Options

You will see a X1107 10G card which supports twinax and should be used if you are within the distance requirements. That same X1107 card can have a optical SFP inserted to support longer distances. Now other NetApp 10G cards don't have a modular interface selection type so the cost of that optic is often bundled into the total price of the card, so using twinax saves you money.

Now, if you have followed along to this point you are likely going to say "Trey, twinax is great but it don't work in a Catalyst 3750E, they have X2 slots for 10G". Nice catch which is why Cisco released the OneX converter, another GEM of a solution which converts an X2 slot into a SFP+ interface and supports the use of twinax cabling. You will notice on the link I provide below there is also a compatibility matrix for this and every other 10G optic Cisco sells. Go to the link and look up your switch (3750E is on it).

OneX Converter

http://www.cisco.com/en/US/prod/collateral/modules/ps5455/data_sheet_c78-547521.html

Hope this helps a little,

Trey

Re: ESX setup using fas3140 and cisco 3750E switches

ian_campbell — Mon, 15 Feb 2010 00:49:02 GMT

Thanks Bren and special thanks to Trey! I've used support forums extensively over the years but this response is easily the most comprehensive and useful I've ever had.

I think the 3750E's will be a good way for us to get started with 10G ethernet as we can't afford a Nexus series switch at this point. Thanks Trey also for the info on Twinax - it looks like that will be a huge cost saving as well. I'm still going over all of your posts on your ethernet storage guy blog - there's a lot of great info there.

Thanks again and no doubt I'll have some follow up questions along the way.

Ian

Re: ESX setup using fas3140 and cisco 3750E switches

ian_campbell — Tue, 22 Jun 2010 02:05:44 GMT

Well, we bought some Cisco 3750E's along with a Netapp 3140 and some twinax cables and onex converter modules for running 10GB to the filer. Below are some results from running iometer tests on our 15k SAS disks and SATA disks.

We have 11 disks in each aggregate. I ran the tests on a single Windows 2003 R2 VM running on a HP DL380 G5 X5460 with 24GB RAM running Vsphere ESX 4.0 Update 1. There are no other VM's on this ESX host and nothing else accessing the 3140 storage during the test.

I was surprised to see very little difference between having jumbo frames enabled or not. If anyone has any comments on these stats or any stats from a similar setup it would be much appreciated.

	3140 SAS Jumbo				3140 SAS No Jumbo
Access Specification Name	Iops	MBps	Average Response Time	Access Specification Name	Iops	MBps	Average Response Time
Max Throughput-100%Read	3635.42	113.6	16.49	Max Throughput-100%Read	3418.26	106.82	17.18
RealLife-60%Rand-65%Read	4587.14	35.83	11.12	RealLife-60%Rand-65%Read	4531.22	35.4	11.21
Max Throughput-50%Read	4769.49	149.04	11.55	Max Throughput-50%Read	5463.92	170.74	10.9
Random-8k-70%Read	5148.85	40.22	9.49	Random-8k-70%Read	4993.75	39.01	9.86

	3140 SATA Jumbo				3140 SATA No Jumbo
Access Specification Name	IOps	MBps	Average Response Time	Access Specification Name	IOps	MBps	Average Response Time
Max Throughput-100%Read	3633.52	113.54	16.5	Max Throughput-100%Read	3457.64	108.05	17.17
RealLife-60%Rand-65%Read	1718.11	13.42	30.72	RealLife-60%Rand-65%Read	1679.44	13.12	31.38
Max Throughput-50%Read	4864.33	152.01	11.53	Max Throughput-50%Read	5476.14	171.12	10.9
Random-8k-70%Read	2537.18	19.82	19.39	Random-8k-70%Read	2432.67	19	20.3

Re: ESX setup using fas3140 and cisco 3750E switches

fujitsusiemens01 — Tue, 22 Jun 2010 11:53:22 GMT

Hi Trey,

Great stuff, thanks for your info!

Unfortunately, every time I try this link:

http://blogs.netapp.com/ethernet/2010/01/netapp-10gbps-ethernet-storage-options.html

I just get a blank screen!?

Can you help?

Thanks,

Mark

Re: ESX setup using fas3140 and cisco 3750E switches

jeddgo123 — Tue, 02 Nov 2010 17:25:09 GMT

Hi Ian

just saw your post. thanks so much for the info. It is surprising jumbo frames is not what it's cracked up to be

Re: ESX setup using fas3140 and cisco 3750E switches

vmsjaak13 — Tue, 02 Nov 2010 20:48:02 GMT

Hi,

What I usually find is that using Jumbo frames saves CPU cycles !

Regards,

Niek