nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

A700s QSFP+ connection via EX4550-32F switch doesn't work

CloudShareian

We are using a EX4550-32F x 2 switches stack with a QSFP+ Juniper modules in PIC 1 on both switches for connecting an A700s Netapp to the Data network. For the sake of the question I will address only one of the connections we currently have to the switches.

We are using the following setup as advised by Netapp:

Node B is currently connected with only e0e to the 2nd EX4550-32F QSFP+ module, with a QSFP+-40G-CU3M cable (recognized by the Juniper switch) but shown on the Netapp as "CISCO-JPC", and configured on et-1/1/0 with access mode. A LIF is configured with an IP on e0e directly.

The current status is that network port show command shows healthy status on e0e for Node B, right duplex, mtu etc.

Same is on the et-1/1/0 interface on Juniper.

But if I run the ifstat e0e command on Node B I see CRC errors accumulating on e0e (I don't see any errors on et-1/1/0 interface on the Juniper).

My question is - what can be the reason for the CRC errors (that are probably the reason for the fact we don't have any connectivity on this LIF that uses this port) and what additional things should we check?
Our vendor states that all the hdw is compatible.

Thank you.

Find more posts tagged with

AFF

Accepted answers

TMACMD

Yeah, my guy says this is on the switch end. I use those all the time on NetApp too Cisco switches without incident. To other switch manufacturers, not so much. Even Arista, it can be hit or miss depending on switch model and os revision. Using optics levels the paying field. Use vendor supported optics at either end and use a neutral fibre to connect them.

All comments

TMACMD

Please show the rest of the ifstat output!

I’ve never had any good luck using twinax between any Netapp and a juniper switch. Pretty much always used optical transceivers on both ends.

you could try:

turn off auto negotiation on both ends

forcing flow control to none on both ends

forcing speed to 40000 on both ends

forcing duplex to full on both ends

then physically remove the cable, zero the stats (ifstat -z -a) and plug the cable back in

if it doesn’t work, get APPROVED/SUPPORTED optics for both ends.

twinax is great but doesn’t always work. It does require some bit of supportability in the code on both ends.

I’ve seen customers get special twinax cables with each end programmed differently (like Cisco at one end and Intel at the other).

CloudShareian

Thank for your for you comment! 🙏
We actually already ordered optical transceivers and cables to try that from our vendor because my theory was exactly that the cables we using are no good.

We will try your suggestion as soon as those arrive this week and hopefully it will solve our issue.

CloudShareian

Full 'ifstat e0e' for the port in question:

TMACMD

It’s not the the cables are no good it’s that one or both ends (Netapp and/or switch) don’t have sufficient programming to support that cable.

just curios, are you able to send a picture of the cable ends (with markings)? If not, no worries. The optics are the best bet in this case anyway

CloudShareian

Here are the cables used (both ends are the same):

TMACMD

CloudShareian

Thank you for your help!
I will post our results when we have them.

CloudShareian

We replaced the QSFP+-40G-CU3M (37-1317-03) twinax cables with:
EX4550-EM-2QSFP (711-039086) + QSFP+-40G-eSR4 (740-045627) (Juniper side),
QSFP, OPT, TRANSCEIVER, 40GbE, SR (332-00389) (NetApp side),
CABLE,OPT,OM4,MPO/MPO,TYPE B,2M (112-00326) (MPO cable)

On all 4 ports on both A700s Nodes: e0e, e0j / e0e, e0j
All ports show link and all ports recognize 40GbE connection.

We managed to configure an IP on 2 of the ports and got a stable ping.

On all 4 ports e0e, e0j / e0e, e0j on both nodes we see constantly generated CRC errors
I think that we may need to use another type of transceivers and cable again despite the fact it looks like the things we use are compatible.

andris

X65402 (332-00389) is also a 40GBASE-eSR4 QSFP transceiver and is supported on AFF A700s onboard ports.

Maybe it's time to start suspecting the EX4550-EM-2QSFP module?

TMACMD

A few things to try. Are they currently connected as individual ports or a port-channel?

Are the CRCs on the NetApp side (system node run -node * ifstat e0e. then repeat with e0j) or on the switch side or both? Are you seeing just CRCs or anything populated in the Xon/Xoff/Pause fields? (if those are populated -> somewhere you have flo-control enabled...turn it off!)

First, pull the cables an wipe the ends with a clean/lint-free cloth. I know some places take the caps of the ends and they get dirty. Not saying that is the case, but clean the ends. Try hitting all the optics with a puff of air. NOT COMPRESSED AIR. just those little "puffers".

You can clean ONTAP stats by rebooting (clears all stats) or: system node run -node * -ifstat -z -a (zero all interfaces, at least for ifstat)

Try setting BOTH sides to: 40G, full duplex, no-flow control

Some switches do not auto-negotiate nicely. Even Cisco, occaionally I need to force speed/duplex

CloudShareian

Thank you, we will try to replace the modules and you are correct about the X65402. The funny thing that some vendors, as FINISAR we use are marked as QSFP and SR not QSSFP+ and SR4. It's just an observation and probably a big, red herring

CloudShareian

e0e and e0j of node-A are configured as port channel, e0e and e0j on node-B are configured as individuals and those are the ports I have a steady ping on.
the CRC errors are on all the ports, there are no errors in xon/xoff/pause fields, we see the CRC errors only on RECEIVE side of the Netapp the switch show no errors not in the port statistics and not in the logs
We will defiantly try to clean them, hopefully today or tomorrow, thank you for the tip
I did zeroed all the interfaces every time we tried reconnecting different cables etc and the errors coming back instantly
I will try to set everything to static and see if it changes anything, thank you

CloudShareian

Strangely, I'm getting this when trying to set the speed of the port to 40GbE:

How is that possible? What am I doing wrong?