VIA Interconnect is down (link 0 down, link 1 down)

[AllianceITSupport]

Hi,

I'm trying to set up a used filer.  It contains two controllers in a single chassis and runs ONTAP 7.3.  They both have IP's and disks assigned and are both reachable on the network.  

When I type 'cf status' it responds with 'Cluster disabled.  VIA Interconnect is down (link 0 down, link 1 down)'

According to this article, two controllers in the same chassis do not require cabling to be HA.  

Why does it say VIA Interconnect is down?

[georgevj]

Which model of FAS?

Was this system previously working in HA configuration and then converted to standalone? 

What does the following command say?

priv set advanced; cf monitor all

[AllianceITSupport]

I don't know what model it is, I don't see a model number on the chassis.  How can I check?

This was bought used, I'm not sure if it was working before but it is licensed.

The output of the command is:

cf: Current monitor status (26May2016 14:23:46):
partner 'unknown', VIA Interconnect is down (link 0 down, link 1 down)
state UP, time 82706734, event CHECK_FSM, elem SChkNoTkOver (14)
mirrorConsistencyRequired TRUE
takeoverByPartner 0x2041 <NVRAM_DOWN,CLUSTER_DISABLE,TAKEOVER_ON_PANIC>
mirrorEnabled TRUE, lowMemory FALSE, memio UNINIT, killPackets TRUE
degraded TRUE, reservePolicy ALWAYS_AFTER_TAKEOVER, resetDisks TRUE
hw_assist status:
hw_assist Inactive on NETAPP-BOTTOM: NETAPP-BOTTOM not monitoring alerts from partner(partner)
hw_assist Inactive on partner: partner not monitoring alerts from partne r NETAPP-BOTTOM
timeouts:
fast 1000, slow 2500, mailbox 10000, connect 5000
operator 600000, firmware 15000 (recvd 22536), dumpcore 60000
booting 300000 (recvd 0)
transit timer enabled TRUE, transit 600000 (last 29055)
mailbox disks:
Disk 4c.00.0 is a local mailbox disk
Disk 4c.00.3 is a local mailbox disk
No partner disks attached!
primary state:
version 2, senderSysid 151754054
cluster_time 1464118431, hbt 204185, node_status TAKEOVER_DISABLED
info 0x2041 <NVRAM_DOWN,CLUSTER_DISABLE,TAKEOVER_ON_PANIC>
flags 0x0 <>
channel CHANNEL_MAILBOX, abs_time 1464272624, sk_time 82704734
channel_status 0
channel CHANNEL_IC, abs_time 0, sk_time 0
channel_status 5
channel CHANNEL_NETWORK, abs_time 0, sk_time 0
channel_status -1
backup state:
version 2, senderSysid 0
cluster_time 0, hbt 0, node_status UNKNOWN
info 0x0 <>
flags 0x0 <>
channel CHANNEL_MAILBOX, abs_time 0, sk_time 0
channel_status 1
Channel Read Ctx:
version 2, senderSysid 0
cluster_time 0, hbt 0, node_status UNKNOWN
info 0x0 <>
flags 0x0 <>
channel CHANNEL_IC, abs_time 0, sk_time 0
channel_status 3
Channel Read Ctx:
version 2, senderSysid 0
cluster_time 0, hbt 0, node_status UNKNOWN
info 0x0 <>
flags 0x0 <>
channel CHANNEL_NETWORK, abs_time 0, sk_time 0
channel_status -1
Channel Read Ctx:
version 2, senderSysid 0
cluster_time 0, hbt 0, node_status UNKNOWN
info 0x0 <>
flags 0x0 <>
takeoverState FT_NONE, takeoverString 'No takeover information'
givebackState FT_NONE, givebackString 'No giveback information'
givebackRetries 0, givebackRequested FALSE
autoGivebackEnabled FALSE, autoGivebackWasDone FALSE, autoGivebackCifsStopping F ALSE
autoGivebackLastVetoCheck 0, autoGivebackAttemptsExceeded FALSE
Maximum primary disk mailbox io times: normal = 575, transition = 0
Maximum backup disk mailbox io times: normal = 0, transition = 0
Num times logs unsynced : 0
Total system uptime: 82706855 msec

[RPHELANIN]

half way down is says "No partner disks attached!" which means HA is not configured correctly... send the output of "sysconifg -a"

[AllianceITSupport]

NetApp Release 7.3.7P2: Sun Apr 21 03:12:49 PDT 2013
System ID: 0151754054 (NETAPP-BOTTOM); partner ID: <unknown> ()
System Serial Number: 200000200137 (NETAPP-BOTTOM)
System Rev: B1
System Storage Configuration: Multi-Path HA
System ACP Connectivity: NA
slot 0: System Board 2.4 GHz (System Board XV A5)
Model Name: FAS3140
Part Number: 110-00120
Revision: A5
Serial Number: 759110
BIOS version: 4.4.0
Loader version: 1.8
Agent FW version: 3
Processors: 2
Processor ID: 0x40f13
Microcode Version: 0x0
Memory Size: 4096 MB
Memory Attributes: Bank Interleaving
Hoisting
Chipkill ECC
CMOS RAM Status: OK
Controller: B
Remote LAN Module Status: Online
Part Number: 110-00057
Revision: G0
Serial Number: 946770
Firmware Version: 4.1
Mgmt MAC Address: 00:A0:98:27:9F:0E
Ethernet Link: up
Using DHCP: no
IPv4 configuration:
IP Address: 10.1.6.51
Netmask: 255.255.255.0
Gateway: 10.1.6.5
IPv6 configuration: Disabled
slot 0: Internal BGE 10/100 Ethernet Controller
e0M MAC Address: 00:a0:98:27:d5:c2 (auto-100tx-fd-up)
Device Type: BCM5703C_B0
memory mapped I/O base 0x00000000c0e00000, size 0x10000
slot 0: BGE 10/100/1000 Ethernet Controller
e0a MAC Address: 00:a0:98:27:d5:c0 (auto-unknown-cfg_down)
e0b MAC Address: 00:a0:98:27:d5:c1 (auto-unknown-cfg_down)
Device Type: BCM5715C
memory mapped I/O base 0x00000000c0400000, size 0x10000
memory mapped I/O base 0x00000000c0410000, size 0x10000
memory mapped I/O base 0x00000000c0420000, size 0x10000
memory mapped I/O base 0x00000000c0430000, size 0x10000
slot 0: NVRAM (NetApp NVRAM VII)
Revision: A5
Serial Number: 759110
Memory Size: 512 MB
Battery Status: OK (3960 mV)
Charger Status: OFF
Running Firmware: 4 (4.8.940)
LIDs: 0x0005[0x0000], 0x0007[0x0000]
Cluster Interconnect Port 1: copper
Cluster Interconnect Port 2: copper
memory mapped I/O base 0xc2000000, size 0x100000
prefetchable memory base 0xc1800000, size 0x800000
prefetchable memory base 0x3100000000, size 0x20000000
slot 0: Fibre Channel Target Host Adapter 0a
(Dual-channel, QLogic 2432(2462) rev. 2, 64-bit, <OFFLINED BY US ER/SYSTEM>)
Firmware rev: 4.5.2
Host Port Addr: 000000
Cacheline size: 16
SRAM parity: Yes
FC Nodename: 50:0a:09:80:89:8b:95:46 (500a0980898b9546)
FC Portname: 50:0a:09:83:89:8b:95:46 (500a0983898b9546)
Connection: No link
SFP Vendor Name: FINISAR CORP.
SFP Vendor P/N: FTLF8524P2BNV
SFP Vendor Rev: A
SFP Serial No.: PHL02YM
SFP Connector: LC
SFP Capabilities: 1, 2, 4 Gbit/Sec
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0c00000, size 0x4000
slot 0: Fibre Channel Target Host Adapter 0b
(Dual-channel, QLogic 2432(2462) rev. 2, 64-bit, <OFFLINED BY US ER/SYSTEM>)
Firmware rev: 4.5.2
Host Port Addr: 000000
Cacheline size: 16
SRAM parity: Yes
FC Nodename: 50:0a:09:80:89:8b:95:46 (500a0980898b9546)
FC Portname: 50:0a:09:84:89:8b:95:46 (500a0984898b9546)
Connection: No link
SFP Vendor Name: FINISAR CORP.
SFP Vendor P/N: FTLF8524P2BNV
SFP Vendor Rev: A
SFP Serial No.: PHL02UN
SFP Connector: LC
SFP Capabilities: 1, 2, 4 Gbit/Sec
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0c04000, size 0x4000
slot 0: Fibre Channel Target Host Adapter 0c
(Dual-channel, QLogic 2432(2462) rev. 2, 64-bit, <OFFLINED BY US ER/SYSTEM>)
Firmware rev: 4.5.2
Host Port Addr: 000000
Cacheline size: 16
SRAM parity: Yes
FC Nodename: 50:0a:09:80:89:8b:95:46 (500a0980898b9546)
FC Portname: 50:0a:09:81:89:8b:95:46 (500a0981898b9546)
Connection: No link
SFP Vendor Name: FINISAR CORP.
SFP Vendor P/N: FTLF8524P2BNV
SFP Vendor Rev: A
SFP Serial No.: PHL02PD
SFP Connector: LC
SFP Capabilities: 1, 2, 4 Gbit/Sec
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0a00000, size 0x4000
slot 0: Fibre Channel Target Host Adapter 0d
(Dual-channel, QLogic 2432(2462) rev. 2, 64-bit, <OFFLINED BY US ER/SYSTEM>)
Firmware rev: 4.5.2
Host Port Addr: 000000
Cacheline size: 16
SRAM parity: Yes
FC Nodename: 50:0a:09:80:89:8b:95:46 (500a0980898b9546)
FC Portname: 50:0a:09:82:89:8b:95:46 (500a0982898b9546)
Connection: No link
SFP Vendor Name: FINISAR CORP.
SFP Vendor P/N: FTLF8524P2BNV
SFP Vendor Rev: A
SFP Serial No.: PHK6TTC
SFP Connector: LC
SFP Capabilities: 1, 2, 4 Gbit/Sec
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0a04000, size 0x4000
slot 0: ATA/IDE Adapter 0e (0x00000000000001f0)
0e.0 STEC NACF1GM1U-B11 01/17/07 977MB 512B/ sect (STI1MQZ410190165003)
I/O base 0x0000000000001018, size 0x8
I/O base 0x0000000000001024, size 0x4
I/O base 0x0000000000001010, size 0x8
I/O base 0x0000000000001020, size 0x4
I/O base 0x0000000000001000, size 0x10
slot 1: FCVI Host Adapter 1a (QLogic 2432(2462) rev. 3, unknown, <DOWN>)
Physical link: DOWN
FC Node Name: 21:00:00:24:ff:28:4c:aa
Firmware rev: 5.0.2 Serial No: RFD1011A48226
Host Port Id: 0xffffffff
Cacheline size: 16 FC Packet size: 2048
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHA1BFV
SFF Capabilities: 1, 2 or 4 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1400000, size 0x4000
slot 1: FCVI Host Adapter 1b (QLogic 2432(2462) rev. 3, unknown, <DOWN>)
Physical link: DOWN
FC Node Name: 21:00:00:24:ff:28:4c:ab
Firmware rev: 5.0.2 Serial No: RFD1011A48226
Host Port Id: 0xffffffff
Cacheline size: 16 FC Packet size: 2048
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHA1BG2
SFF Capabilities: 1, 2 or 4 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1404000, size 0x4000
slot 2: FC Host Adapter 2a (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1017B71658
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:609114
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC5FYM
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1000000, size 0x4000
slot 2: FC Host Adapter 2b (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1017B71658
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:609115
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC5GAD
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1004000, size 0x4000
slot 2: FC Host Adapter 2c (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1017B71658
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:609116
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC5F2H
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1200000, size 0x4000
slot 2: FC Host Adapter 2d (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1017B71658
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:609117
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC5FY3
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1204000, size 0x4000
slot 3: FC Host Adapter 3a (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1020B77119
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:60b7c8
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHD8B5F
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0000000, size 0x4000
slot 3: FC Host Adapter 3b (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1020B77119
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:60b7c9
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHD8AGM
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0004000, size 0x4000
slot 3: FC Host Adapter 3c (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1020B77119
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:60b7ca
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHD7DMC
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0200000, size 0x4000
slot 3: FC Host Adapter 3d (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1020B77119
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:60b7cb
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHD7ZNR
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0204000, size 0x4000
slot 4: SAS Host Adapter 4a (PMC-Sierra PM8001 rev. C, SAS, <OFFLINE (ha rd)>)
MFG Part Number: USI 6086-000010-00 rev. MP V4.1
Part number: 111-00341+B0
Serial number: ST078A7A0ZC
Date Code: 20100710
Firmware rev: 01.12.06.00
Base WWN: 5:00e004:00010a3:c8
Phy State: [0] Disabled
[1] Disabled
[2] Disabled
[3] Disabled
QSFP Vendor: not available
QSFP Part Number: not available
QSFP Type: not available
QSFP Serial Number: not available
slot 4: SAS Host Adapter 4b (PMC-Sierra PM8001 rev. C, SAS, <OFFLINE (ha rd)>)
MFG Part Number: USI 6086-000010-00 rev. MP V4.1
Part number: 111-00341+B0
Serial number: ST078A7A0ZC
Date Code: 20100710
Firmware rev: 01.12.06.00
Base WWN: 5:00e004:00010a3:cc
Phy State: [4] Disabled
[5] Disabled
[6] Disabled
[7] Disabled
QSFP Vendor: not available
QSFP Part Number: not available
QSFP Type: not available
QSFP Serial Number: not available
memory mapped I/O base 0x00000000c0800000, size 0x10000
memory mapped I/O base 0x00000000c0810000, size 0x10000
memory mapped I/O base 0x00000000c0820000, size 0x10000
memory mapped I/O base 0x00000000c0830000, size 0x10000
slot 4: SAS Host Adapter 4c (PMC-Sierra PM8001 rev. C, SAS, <UP>)
MFG Part Number: USI 6086-000010-00 rev. MP V4.1
Part number: 111-00341+B0
Serial number: ST078A7A0ZC
Date Code: 20100710
Firmware rev: 01.12.06.00
Base WWN: 5:00e004:00010a3:c0
Phy State: [0] Enabled, 3.0 Gb/s
[1] Enabled, 3.0 Gb/s
[2] Enabled, 3.0 Gb/s
[3] Enabled, 3.0 Gb/s
QSFP Vendor: Molex Inc.
QSFP Part Number: 112-00177+A0
QSFP Type: Passive Copper 2m ID:01
QSFP Serial Number: 123220584
00.0 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01463027)
00.3 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01206430)
00.4 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01172380)
00.5 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0624407)
00.6 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01374948)
00.7 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01493371)
00.8 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0625887)
00.9 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01209675)
00.10: NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0622577)
00.11: NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01746907)
Shelf 0: IOM3 Firmware rev. IOM3 A: 0200 IOM3 B: 0200
slot 4: SAS Host Adapter 4d (PMC-Sierra PM8001 rev. C, SAS, <UP>)
MFG Part Number: USI 6086-000010-00 rev. MP V4.1
Part number: 111-00341+B0
Serial number: ST078A7A0ZC
Date Code: 20100710
Firmware rev: 01.12.06.00
Base WWN: 5:00e004:00010a3:c4
Phy State: [4] Enabled, 3.0 Gb/s
[5] Enabled, 3.0 Gb/s
[6] Enabled, 3.0 Gb/s
[7] Enabled, 3.0 Gb/s
QSFP Vendor: Molex Inc.
QSFP Part Number: 112-00177+A0
QSFP Type: Passive Copper 2m ID:00
QSFP Serial Number: 123220596
00.0 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01463027)
00.3 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01206430)
00.4 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01172380)
00.5 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0624407)
00.6 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01374948)
00.7 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01493371)
00.8 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0625887)
00.9 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01209675)
00.10: NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0622577)
00.11: NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01746907)
Shelf 0: IOM3 Firmware rev. IOM3 A: 0200 IOM3 B: 0200
memory mapped I/O base 0x00000000c0600000, size 0x10000
memory mapped I/O base 0x00000000c0610000, size 0x10000
memory mapped I/O base 0x00000000c0620000, size 0x10000
memory mapped I/O base 0x00000000c0630000, size 0x10000

[RPHELANIN]

that node can only see its own disks, can you connect to the other controller and run "cf status" and "sysconfig -a"

[AllianceITSupport]

Sure

cf status:

Cluster disabled.
VIA Interconnect is down (link 0 down, link 1 down).

sysconfig -a:

NetApp Release 7.3.7P2: Sun Apr 21 03:12:49 PDT 2013
System ID: 0151734812 (NETAPP-TOP); partner ID: <unknown> ()
System Serial Number: 200000077451 (NETAPP-TOP)
System Rev: B1
System Storage Configuration: Multi-Path HA
System ACP Connectivity: NA
slot 0: System Board 2.4 GHz (System Board XV A5)
Model Name: FAS3140
Part Number: 110-00120
Revision: A5
Serial Number: 739868
BIOS version: 4.4.0
Loader version: 1.8
Agent FW version: 3
Processors: 2
Processor ID: 0x40f13
Microcode Version: 0x0
Memory Size: 4096 MB
Memory Attributes: Bank Interleaving
Hoisting
Chipkill ECC
CMOS RAM Status: OK
Controller: A
Remote LAN Module Status: Online
Part Number: 110-00057
Revision: G0
Serial Number: 946576
Firmware Version: 4.1
Mgmt MAC Address: 00:A0:98:27:9F:A5
Ethernet Link: up
Using DHCP: no
IPv4 configuration:
IP Address: 10.1.6.53
Netmask: 255.255.255.0
Gateway: 10.1.6.5
IPv6 configuration: Disabled
slot 0: Internal BGE 10/100 Ethernet Controller
e0M MAC Address: 00:a0:98:13:0e:ea (auto-100tx-fd-up)
Device Type: BCM5703C_B0
memory mapped I/O base 0x00000000c0e00000, size 0x10000
slot 0: BGE 10/100/1000 Ethernet Controller
e0a MAC Address: 00:a0:98:13:0e:e8 (auto-unknown-cfg_down)
e0b MAC Address: 00:a0:98:13:0e:e9 (auto-unknown-cfg_down)
Device Type: BCM5715C
memory mapped I/O base 0x00000000c0400000, size 0x10000
memory mapped I/O base 0x00000000c0410000, size 0x10000
memory mapped I/O base 0x00000000c0420000, size 0x10000
memory mapped I/O base 0x00000000c0430000, size 0x10000
slot 0: NVRAM (NetApp NVRAM VII)
Revision: A5
Serial Number: 739868
Memory Size: 512 MB
Battery Status: OK (3978 mV)
Charger Status: OFF
Running Firmware: 4 (4.8.940)
LIDs: 0x0004[0x0000], 0x0006[0x0000]
Cluster Interconnect Port 1: copper
Cluster Interconnect Port 2: copper
memory mapped I/O base 0xc2000000, size 0x100000
prefetchable memory base 0xc1800000, size 0x800000
prefetchable memory base 0x3100000000, size 0x20000000
slot 0: Fibre Channel Target Host Adapter 0a
(Dual-channel, QLogic 2432(2462) rev. 2, 64-bit, <OFFLINED BY US ER/SYSTEM>)
Firmware rev: 4.5.2
Host Port Addr: 000000
Cacheline size: 16
SRAM parity: Yes
FC Nodename: 50:0a:09:80:89:db:4a:1c (500a098089db4a1c)
FC Portname: 50:0a:09:83:89:db:4a:1c (500a098389db4a1c)
Connection: No link
SFP Vendor Name: FINISAR CORP.
SFP Vendor P/N: FTLF8524P2BNV
SFP Vendor Rev: A
SFP Serial No.: PHL02TJ
SFP Connector: LC
SFP Capabilities: 1, 2, 4 Gbit/Sec
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0c00000, size 0x4000
slot 0: Fibre Channel Target Host Adapter 0b
(Dual-channel, QLogic 2432(2462) rev. 2, 64-bit, <OFFLINED BY US ER/SYSTEM>)
Firmware rev: 4.5.2
Host Port Addr: 000000
Cacheline size: 16
SRAM parity: Yes
FC Nodename: 50:0a:09:80:89:db:4a:1c (500a098089db4a1c)
FC Portname: 50:0a:09:84:89:db:4a:1c (500a098489db4a1c)
Connection: No link
SFP Vendor Name: FINISAR CORP.
SFP Vendor P/N: FTLF8524P2BNV
SFP Vendor Rev: A
SFP Serial No.: PHL0JS6
SFP Connector: LC
SFP Capabilities: 1, 2, 4 Gbit/Sec
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0c04000, size 0x4000
slot 0: Fibre Channel Target Host Adapter 0c
(Dual-channel, QLogic 2432(2462) rev. 2, 64-bit, <OFFLINED BY US ER/SYSTEM>)
Firmware rev: 4.5.2
Host Port Addr: 000000
Cacheline size: 16
SRAM parity: Yes
FC Nodename: 50:0a:09:80:89:db:4a:1c (500a098089db4a1c)
FC Portname: 50:0a:09:81:89:db:4a:1c (500a098189db4a1c)
Connection: No link
SFP Vendor Name: FINISAR CORP.
SFP Vendor P/N: FTLF8524P2BNV
SFP Vendor Rev: A
SFP Serial No.: PHL0YL9
SFP Connector: LC
SFP Capabilities: 1, 2, 4 Gbit/Sec
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0a00000, size 0x4000
slot 0: Fibre Channel Target Host Adapter 0d
(Dual-channel, QLogic 2432(2462) rev. 2, 64-bit, <OFFLINED BY US ER/SYSTEM>)
Firmware rev: 4.5.2
Host Port Addr: 000000
Cacheline size: 16
SRAM parity: Yes
FC Nodename: 50:0a:09:80:89:db:4a:1c (500a098089db4a1c)
FC Portname: 50:0a:09:82:89:db:4a:1c (500a098289db4a1c)
Connection: No link
SFP Vendor Name: FINISAR CORP.
SFP Vendor P/N: FTLF8524P2BNV
SFP Vendor Rev: A
SFP Serial No.: PHL0YSG
SFP Connector: LC
SFP Capabilities: 1, 2, 4 Gbit/Sec
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0a04000, size 0x4000
slot 0: ATA/IDE Adapter 0e (0x00000000000001f0)
0e.0 STEC NACF1GM1U-B11 01/17/07 977MB 512B/ sect (STI1MQK110190165046)
I/O base 0x0000000000001018, size 0x8
I/O base 0x0000000000001024, size 0x4
I/O base 0x0000000000001010, size 0x8
I/O base 0x0000000000001020, size 0x4
I/O base 0x0000000000001000, size 0x10
slot 1: FCVI Host Adapter 1a (QLogic 2432(2462) rev. 3, unknown, <DOWN>)
Physical link: DOWN
FC Node Name: 21:00:00:24:ff:28:4c:72
Firmware rev: 5.0.2 Serial No: RFD1012A50835
Host Port Id: 0xffffffff
Cacheline size: 16 FC Packet size: 2048
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC4JYF
SFF Capabilities: 1, 2 or 4 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1400000, size 0x4000
slot 1: FCVI Host Adapter 1b (QLogic 2432(2462) rev. 3, unknown, <DOWN>)
Physical link: DOWN
FC Node Name: 21:00:00:24:ff:28:4c:73
Firmware rev: 5.0.2 Serial No: RFD1012A50835
Host Port Id: 0xffffffff
Cacheline size: 16 FC Packet size: 2048
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC4KLD
SFF Capabilities: 1, 2 or 4 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1404000, size 0x4000
slot 2: FC Host Adapter 2a (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1017B71701
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:6097a0
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC4DZ1
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1000000, size 0x4000
slot 2: FC Host Adapter 2b (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1017B71701
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:6097a1
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC42DE
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1004000, size 0x4000
slot 2: FC Host Adapter 2c (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1017B71701
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:6097a2
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC4JP0
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1200000, size 0x4000
slot 2: FC Host Adapter 2d (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1017B71701
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:6097a3
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHC402Y
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c1204000, size 0x4000
slot 3: FC Host Adapter 3a (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1020B77463
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:60b630
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHD78V3
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0000000, size 0x4000
slot 3: FC Host Adapter 3b (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1020B77463
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:60b631
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHD77NU
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0004000, size 0x4000
slot 3: FC Host Adapter 3c (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1020B77463
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:60b632
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHD7X1N
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0200000, size 0x4000
slot 3: FC Host Adapter 3d (QLogic 2432 rev. 3, L-port, <OFFLINE (hard)> )
Board name: QLE2464
Serial Number: LFD1020B77463
Firmware rev: 5.4.100
Host Loop Id: 0
FC Node Name: 2:100:0024ff:60b633
SFF Vendor: FINISAR CORP.
SFF Part Number: FTLF8524E2KNL
SFF Serial Number: PHD72FW
SFF Capabilities: 1, 2 or 4 Gbit
Link Data Rate: 1 Gbit
I/O base 0x0000000000000000, size 0x0
memory mapped I/O base 0x00000000c0204000, size 0x4000
slot 4: SAS Host Adapter 4a (PMC-Sierra PM8001 rev. C, SAS, <OFFLINE (ha rd)>)
MFG Part Number: USI 6086-000010-00 rev. MP V4.1
Part number: 111-00341+B0
Serial number: ST078A7A0YW
Date Code: 20100710
Firmware rev: 01.12.06.00
Base WWN: 5:00e004:00010a3:18
Phy State: [0] Disabled
[1] Disabled
[2] Disabled
[3] Disabled
QSFP Vendor: not available
QSFP Part Number: not available
QSFP Type: not available
QSFP Serial Number: not available
slot 4: SAS Host Adapter 4b (PMC-Sierra PM8001 rev. C, SAS, <OFFLINE (ha rd)>)
MFG Part Number: USI 6086-000010-00 rev. MP V4.1
Part number: 111-00341+B0
Serial number: ST078A7A0YW
Date Code: 20100710
Firmware rev: 01.12.06.00
Base WWN: 5:00e004:00010a3:1c
Phy State: [4] Disabled
[5] Disabled
[6] Disabled
[7] Disabled
QSFP Vendor: not available
QSFP Part Number: not available
QSFP Type: not available
QSFP Serial Number: not available
memory mapped I/O base 0x00000000c0800000, size 0x10000
memory mapped I/O base 0x00000000c0810000, size 0x10000
memory mapped I/O base 0x00000000c0820000, size 0x10000
memory mapped I/O base 0x00000000c0830000, size 0x10000
slot 4: SAS Host Adapter 4c (PMC-Sierra PM8001 rev. C, SAS, <UP>)
MFG Part Number: USI 6086-000010-00 rev. MP V4.1
Part number: 111-00341+B0
Serial number: ST078A7A0YW
Date Code: 20100710
Firmware rev: 01.12.06.00
Base WWN: 5:00e004:00010a3:10
Phy State: [0] Enabled, 3.0 Gb/s
[1] Enabled, 3.0 Gb/s
[2] Enabled, 3.0 Gb/s
[3] Enabled, 3.0 Gb/s
QSFP Vendor: Molex Inc.
QSFP Part Number: 112-00177+A0
QSFP Type: Passive Copper 2m ID:01
QSFP Serial Number: 123220581
00.0 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01463027)
00.3 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01206430)
00.4 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01172380)
00.5 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0624407)
00.6 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01374948)
00.7 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01493371)
00.8 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0625887)
00.9 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01209675)
00.10: NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0622577)
00.11: NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01746907)
Shelf 0: IOM3 Firmware rev. IOM3 A: 0200 IOM3 B: 0200
slot 4: SAS Host Adapter 4d (PMC-Sierra PM8001 rev. C, SAS, <UP>)
MFG Part Number: USI 6086-000010-00 rev. MP V4.1
Part number: 111-00341+B0
Serial number: ST078A7A0YW
Date Code: 20100710
Firmware rev: 01.12.06.00
Base WWN: 5:00e004:00010a3:14
Phy State: [4] Enabled, 3.0 Gb/s
[5] Enabled, 3.0 Gb/s
[6] Enabled, 3.0 Gb/s
[7] Enabled, 3.0 Gb/s
QSFP Vendor: Molex Inc.
QSFP Part Number: 112-00177+A0
QSFP Type: Passive Copper 2m ID:00
QSFP Serial Number: 123320246
00.0 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01463027)
00.3 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01206430)
00.4 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01172380)
00.5 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0624407)
00.6 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01374948)
00.7 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01493371)
00.8 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0625887)
00.9 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01209675)
00.10: NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA UR0622577)
00.11: NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01746907)
Shelf 0: IOM3 Firmware rev. IOM3 A: 0200 IOM3 B: 0200
memory mapped I/O base 0x00000000c0600000, size 0x10000
memory mapped I/O base 0x00000000c0610000, size 0x10000
memory mapped I/O base 0x00000000c0620000, size 0x10000
memory mapped I/O base 0x00000000c0630000, size 0x10000

[RPHELANIN]

what is output of "options cf"

[AllianceITSupport]

cf.giveback.auto.cifs.terminate.minutes 5
cf.giveback.auto.enable off
cf.giveback.auto.terminate.bigjobs on
cf.giveback.check.partner on
cf.hw_assist.enable on
cf.hw_assist.partner.address
cf.hw_assist.partner.port 4444
cf.quickloop.enable false (same value required in local+partner)
cf.takeover.change_fsid on
cf.takeover.detection.seconds 15
cf.takeover.on_disk_shelf_miscompare off
cf.takeover.on_failure on
cf.takeover.on_network_interface_failure off
cf.takeover.on_network_interface_failure.policy all_nics (same value in local+partner recommended)
cf.takeover.on_panic on
cf.takeover.on_short_uptime on
cf.takeover.use_mcrc_file off (value might be overwritten in takeover)

[RPHELANIN]

Run “cf enable” and send me the output

[AllianceITSupport]

The second one says clustering is enabled.

The first one says:

NETAPP-BOTTOM*> Thu May 26 16:39:27 GMT [NETAPP-BOTTOM: cf.misc.operatorEnable:w arning]: Cluster monitor: operator initiated enabling of cluster
Thu May 26 16:39:28 GMT [NETAPP-BOTTOM: cf.fsm.takeoverOfPartnerDisabled:notice] : Cluster monitor: takeover of partner disabled (partner mailbox disks not acces sible or invalid)
Thu May 26 16:39:28 GMT [NETAPP-BOTTOM: cf.fsm.takeoverByPartnerDisabled:notice] : Cluster monitor: takeover of NETAPP-BOTTOM by partner disabled (interconnect e rror)
Thu May 26 16:39:32 GMT [NETAPP-BOTTOM: cf.fsm.partnerNotResponding:notice]: Clu ster monitor: partner not responding
Thu May 26 16:39:42 GMT [NETAPP-BOTTOM: ha.takeoverImpDegraded:warning]: Takeover of the partner node is impossible due to lack of connectivity to the partner mailbox disks.

[georgevj]

Here is some interesting info:

from NETAPP-BOTTOM

NetApp Release 7.3.7P2
Model Name: FAS3140
System Storage Configuration: Multi-Path HA

This is FAS3140, without any internal disks.

slot 4: SAS Host Adapter 4a (PMC-Sierra PM8001 rev. C, SAS, <OFFLINE (ha rd)>)
slot 4: SAS Host Adapter 4b (PMC-Sierra PM8001 rev. C, SAS, <OFFLINE (ha rd)>)
slot 4: SAS Host Adapter 4c (PMC-Sierra PM8001 rev. C, SAS, <UP>)
slot 4: SAS Host Adapter 4d (PMC-Sierra PM8001 rev. C, SAS, <UP>)

00.0 : NETAPP X306_WMANT02TSSM NA08 1695.4GB 512B/sect (WD-WMA Y01463027)

You have a SAS HBA inserted into slot4 of each controller. two of the ports on that HBA are connected to an external disk shelf, with 12 HDDs
If you search for the serial number of one disk, you can see that each controller is seeing the disk twice, which means that the system is configured for
multipath HA, which is obvious from this sysconfig -a output :

System Storage Configuration: Multi-Path HA

following disks are mailbox disks on NETAPP-BOTTOM, but it cannot see NETAPP-TOP's mailbox disks!

mailbox disks:
Disk 4c.00.0 is a local mailbox disk
Disk 4c.00.3 is a local mailbox disk
No partner disks attached!

You have not provided "cf monitor all" output of NETAPP-TOP, if you do, we can confirm whether the mailbox disks are missing or corrupted.

If you think there is no important data on these drives, you could reinitialize both controllers and start fresh. but be careful about licenses.
Otherwise you may try re-creating the mailbox disks, with a 50-50 chances of getting it right.
Please let us know your thoughts.

[AllianceITSupport]

'cf monitor all' doesn't wokrk, but 'cf monitor' shows:

current time: 26May2016 17:18:10
UP 1+02:05:06, partner 'unknown', cluster monitor enabled
VIA Interconnect is down (link 0 down, link 1 down), takeover capability off-line (partner mailbox disks not accessible or invalid)
takeover by partner off-line (interconnect error)
partner may be down, last partner update UNKNOWN (26May2016 15:10:10)

I just reinitialized all of these disks, so I'm not sure if that will help.  My plan is to split the disks between the two controllers, have them all on one aggregate (on each), and set up the controllers for takeover.  If there are any problems with this config, please let me know.

[georgevj]

Oh. forgot to mention:

the command is , 

priv set advanced; cf monitor all

Its okay to configure it the way you want, but please keep these things in mind.

You have only 12 disks as per the outputs posted earlier.

each controller should have at least 3 disks assigned, for the root aggregate(s). 

Typically, 6 disks should be assigned for each controller. but if you need all the disks in one aggregate, remaining 6 can be assigned to only one controller, out of which one would be a hotspare. So, you will have 5 disks in the data aggregate. Again, this configuration would cause the storage controller to be put in an active/passive configuration, meaning only the controller that owns the data aggregate can serve data, the other one will be a hot-standby.

[AllianceITSupport]

There are actually only 10 disks, I don't know why it says 12.

I will wipe the disks, but can you explain how I should configure them aftwards to achieve that configuration?  I'm fine with the active/passive configuration, I want to maximize disk space.  Should I assign all but three disks to one controller, and give three to the other?

[georgevj]

Yes, you are right. there are only 10 disks. I counted it wrong. The disk numbers 4c.00.1 and 4c.00.2 are missing.(Hopefully those were the failing controller's mailbox disks!).

Regarding the new configuration, the idea is to keep minimum drives on standby controller, and assign rest of the disks to the active one. If you decide to go for raid4 for root aggregate, you may even live with 2 disks for each root aggregate and rest of the disks for data+1 hotspare. Its too late for me to sleep now, will check the recomendations for fas3xxx series and will post the steps tomorrow.

Meanwhile, expecting suggestions from the community as well .. 🙂

[AllianceITSupport]

Thanks for your help!

[AllianceITSupport]

Hi,

So I reinitialized all of the disks on both controllers, went through basic setup, applied the licenses and still a no go.  cf status reports:

partner may be down, takeover disabled because of reason (partner mailbox disks not accessible or invalid)
NETAPP-BOTTOM has disabled takeover by partner (interconnect error)
VIA Interconnect is down (link 0 down, link 1 down).

Any other suggestions?

Thanks

[georgevj]

1. Halt both controllers
2. Boot controller1 and press CTRL+C to get the boot menu
3. Choose option number 5 from the boot menu to get into maintenance mode
4. At maintenance mode command line, type the following commands:

Release all reservations on the disk drives:

*> storage release disks

Destroy the HA Pair mailboxes:

*> mailbox destroy local
*> mailbox destroy partner

      Repeat these steps for the other node, and then reboot both nodes

[AllianceITSupport]

Unfortunately it still gives the same errors.

Could this have something to do with cabling?  Also, the second controller was taken from another chassis, not sure if that could have something to do with it.

[georgevj]

Thats it. 

FAS 3140 comes in standalone and HA configurations.

If this controller was originally standalone, with a blank in slotB, it will not support HA configuration. That's because the way the backplane and its firmware is configured at the factory. This cannot be changed in the field, meaning the entire chassis should be replaced. You will have to get in touch with NetApp support to see what can be done.