new cna adapter not working qlogic 1143

[zenmatrix83]

We just installed a x1143a cna adapter with the assistance of a contracter, and the links aren't coming up . I'll add some information below, as we are unsure where the issue is currently.

running system controller config show-errors -verbose shows

sysconfig: slot 1 OK: X2069: PMC PM8072; PCI-E quad-port SAS (PM8072)
sysconfig: slot 2 OK: X1143A: QLogic ISP 8324; Dual-Port 16Gb FC/10GbE PCIe CNA
sysconfig: No shelf configuration errors detected.
sysconfig: There are no configuration errors.
Configuration Info and Errors for Node:

running  system hardware unified-connect show shows

Current Current Pending Pending Admin
Node Adapter Mode Type Mode Type Status
------------ ------- ------- --------- ------- --------- -----------
san-nap01-03 2a cna target - - online
san-nap01-03 2b cna target - - online

and running network fcp adapter show -node san-nap01-03 -instance

Node: san-nap01-03
Adapter: 2a
Description: Fibre Channel Target Adapter 2a (QLogic CNA 8324 (8362), rev. 2, CNA_10G)
Physical Protocol: ethernet
Maximum Speed: 10
Administrative Status: up
Operational Status: link not connected
Extended Status: ADAPTER UP
Host Port Address: 0
Firmware Revision: 8.7.0
Data Link Rate (Gbit): 0
Fabric Established: false
Fabric Name: -
Connection Established: loop
Is Connection Established: false
Mediatype: ptp
Configured Speed: auto
Data Protocols Supported: fcp
Adapter WWNN: 50:0a:09:80:80:81:ed:70
Adapter WWPN: 50:0a:09:81:80:81:ed:70
Switch Port: :
Form Factor Of Transceiver: SFP
Vendor Name Of Transceiver: FINISAR CORP.
Part Number Of Transceiver: FTLX8574D3BCL-QL
Revision Of Transceiver: A
Serial Number Of Transceiver: AZK0AHV
FC Capabilities Of Transceiver: 10 (Gbit/sec)
Vendor OUI Of Transceiver: 00:90:65
Wavelength In Nanometers: 850
Date Code Of Transceiver: 18:05:08
Validity Of Transceiver: true
Connector Used: LC
Encoding Used: 64B66B
Is Internally Calibrated: true
Received Optical Power: 0.0 (uWatts)
Is Received Power In Range: false
SFP Transmitted Optical Power: 0.0 (uWatts)
Is Xmit Power In Range: false

[CHRISMAKI]

Looks like you have a 10Gbps Ethernet SFP+ in there, do you have it connected to an Ethernet switch or FC switch?

[zenmatrix83]

Oh sorry they are connected to ethernet switches, pretty sure they are nexus 7700 series I don't have the exact model on me.

[CHRISMAKI]

I'm going to assume you've tried the obvious (to me) fixes of flipping the fibres in case of a TX/RX reversal as well as tried a new cable?

Assuming the above, can we get the following from the cluster:

net int show

net port show

And then the following from the Cisco:

show run int <relevant interface>

[zenmatrix83]

Yep, we flipped the pairs our selves, and our network team went back out and checked again. We don't have any lifs on the ports yet as they are showing as down but here is what network port show sees(minus me playing with the ip ranges just in case). I'm pending the cisco output, but we can see light being sent through the cable as well, it feels like something isn't enabled somewhere. From my research at least we are using the correct cable type, correct sfp modules, and everything else so I'm at a loss on what we are missing.

---

node: san-nap01-03
Speed(Mbps) Health
Port IPspace Broadcast Domain Link MTU Admin/Oper Status
--------- ------------ ---------------- ---- ---- ----------- --------
a0a Default - up 1500 auto/- healthy
a0a-298 Default 111.11.11.0/23 up 1500 auto/- healthy
a0a-3000 Default 11.1.1.0/21 up 1500 auto/- healthy
a0a-527 Default 111.11.11.0/24 up 1500 auto/- healthy
e0M Default 111.11.11.0/24 up 1500 auto/1000 healthy
e0a Cluster Cluster up 9000 1000/10000 healthy
e0b Cluster Cluster up 9000 1000/10000 healthy
e0c Default - up 1500 1000/1000 healthy
e0d Default - up 1500 1000/1000 healthy
e2a Default - down 1500 auto/- -
e2b Default - down 1500 auto/- -

[CHRISMAKI]

I should have asked for this the first time, but how about:

net port show -node san-nap01-03 -port e2* -instance

To take the switch out of the equation, have you tried connecting e2a and e2b back-to-back?

[zenmatrix83]

The output is below. Once thing we found was is the switch is setup for lacp, which we wanted, but they tech didn't setup the ifgrp. I spent most the morning tracing the cables and making sure that the correct ports where cabled to the correct ports on the switch but still no luck. I forgot about your idea of connecting the directly and maybe thats the next thing we can try.

san-nap01::> net port show -node san-nap01-03 -port e2* -instance
(network port show)

Node: san-nap01-03
Port: e2a
Link: down
MTU: 1500
Auto-Negotiation Administrative: true
Auto-Negotiation Operational: false
Duplex Mode Administrative: auto
Duplex Mode Operational: half
Speed Administrative: auto
Speed Operational: -
Flow Control Administrative: full
Flow Control Operational: -
MAC Address: 34:80:0d:67:17:70
Port Type: physical
Interface Group Parent Node: san-nap01-03
Interface Group Parent Port: a0b
Distribution Function: -
Create Policy: -
Parent VLAN Node: -
Parent VLAN Port: -
VLAN Tag: -
Remote Device ID: -
IPspace Name: Default
Broadcast Domain: -
MTU Administrative: 1500
Port Health Status: -
Ignore Port Health Status: false
Port Health Degraded Reasons: -

Node: san-nap01-03
Port: e2b
Link: down
MTU: 1500
Auto-Negotiation Administrative: true
Auto-Negotiation Operational: false
Duplex Mode Administrative: auto
Duplex Mode Operational: half
Speed Administrative: auto
Speed Operational: -
Flow Control Administrative: full
Flow Control Operational: -
MAC Address: 34:80:0d:67:17:78
Port Type: physical
Interface Group Parent Node: san-nap01-03
Interface Group Parent Port: a0b
Distribution Function: -
Create Policy: -
Parent VLAN Node: -
Parent VLAN Port: -
VLAN Tag: -
Remote Device ID: -
IPspace Name: Default
Broadcast Domain: -
MTU Administrative: 1500
Port Health Status: -
Ignore Port Health Status: fals

[CHRISMAKI · Accepted solution]

I see in your original post a0b did not exist, but now the two ports, e2[a,b] are both in a0b. I don't think I'd try to the back-to-back connection while they're still in an ifgrp however, so you may wish to tear that down prior to that test.

Everything I see on the NetApp side looks good, at this point I'd check/try:

• Replacing Fibre
  • Making sure it's multi-mode, assuming you're using SR SFP+s in a DC
• Verifying SFP+ compatibility (SR vs. LR)
• Verifying switch configuration

Just for completeness of information, maybe dump the output of ifgrp show -node <nodename> -ifgrp a0b -instance though at this point I'm pretty sure your side is fine, time to look at the physical stuff and the Cisco config.

[zenmatrix83]

It turned out it was the first thing you mentioned, and something we've tried a few times, the polarity was reversed in the cables. We spent time with a  light meter  validating light was coming out of the cards and the cables and switched them again for the 4th or 5th time and the links came up. 

[CHRISMAKI]

Hey, I'm glad you got it working, I'm sure that was frustrating for you.

Remember, always blame the network! 😉