Solved: FAS3220 Dual Chassis HA interconnect error

unitech89e9 · ‎2014-08-16

Hi everybody

I start to create a FAS3220 Dual Chassis HA enviroment,

but we don't have optional addition sas adapter,

so I decide to connect the controller to disk shelves by single path,

the topology just like that

but the cluster failover is not working,

when I use the cf status on each node,

the system show：

partner may be down, takeover disabled because of reason (partner mailbox disks not accessible or invalid)

FAS3220 has disabled takeover by partner (interconnect error)

RDMA Interconnect is down (Link 0 down).

When I use the cf monitor on each node,

the system show：

current time: 15Aug2014 04:19:51

UP 02:00:49, partner 'unknown', cluster monitor enabled

RDMA Interconnect is down (Link 0 down), takeover capability off-line (partner mailbox disks not accessible or invalid)

takeover by partner off-line (interconnect error)

partner may be down, last partner update UNKNOWN (15Aug2014 04:18:37)

When I use the cf hw_assist status

the system show：

Local Node(FAS3220) Status:

Inactive: FAS3220not monitoring alerts from partner(partner)

Interconnect is down between nodes.

Partner Node(partner) Status:

Can not read partner state

Interconnect is down between nodes

so I open the Oncommand System Manager to see the network interface

I see the the c0a and c0b is show the status like this

but I don't know where is my mistake?

what will make the interconnect port(c0a and c0b) be down?

Please help me!

Thanks very much

aborzenkov · ‎2014-08-18

Please boot into maintenance mode and use “ha-config show” to check current HA settings. Looks like your chassis is set to HA, which means it will disable external ports and try internal backplane connection.

View solution in original post

DOMINIC_WYSS · ‎2014-08-16

it looks to me that you accidentally miscabled the interconnects. don't cross them out! c0a goes to c0a on the other head, not to c0b!

you get these link states (one interconnect up, the other down) when you crossed them out.

btw: if I remember correctly, it's now allowed to do ONE transition between IOM3 and IOM6 per stack (it was only allowed on FAS2000 series until recently).

so you can cable both shelf types into one stack and have it multipathed.

unitech89e9 · ‎2014-08-16

Hi Dominic：

I cable the c0a on the other c0a,

and c0b to the other c0b,

is that wrong?

DOMINIC_WYSS · ‎2014-08-17

I cable the c0a on the other c0a,

and c0b to the other c0b,

is that wrong?

no, that is correct. and also your schema is correct. but double check if you have not accidentally crossed the ports.

and check if you have the right cables! there are two types of these copper cables with the same SFP+ interface. I think the Ciscos do not work on the interconnect ports.

what's the output of these ports in sysconfig -a?

shane_bradley · ‎2014-08-17

Have you checked the disks are all visible from both nodes? could indicate that they cant see partner disks.

unitech89e9 · ‎2014-08-17

Hi Shane：

I can sure that, every node can see all the disk!

unitech89e9 · ‎2014-08-17

N6220-SAS> sysconfig -a

Data ONTAP Release 8.1.4P4 7-Mode: Sat Jun 28 01:52:51 PDT 2014

System ID: 2018635377 (N6220-SAS); partner ID: 2018730260 ()

System Serial Number: 2858C2513F009011 (N6220-SAS)

System Rev: B0

System Storage Configuration: Single-Path HA

System ACP Connectivity: NA

slot 0: System Board 2.3 GHz (System Board XVI B0)

Model Name: N6220

Machine Type: IBM-2858-C25

Part Number: 111-01333

Revision: B0

Serial Number: 9448935534

BIOS version: 5.2.1

Loader version: 3.4

Processors: 4

Processor ID: 0x1067a

Microcode Version: 0xa0b

Processor type: Intel(R) Xeon(R) CPU L5410 @ 2.33GHz

Memory Size: 12288 MB

Memory Attributes: Bank Interleaving

Hoisting

Rank Interleaving

Chipkill ECC

NVMEM Size: 1600 MB of Main Memory Used

CMOS RAM Status: OK

Service Processor Status: Online

Firmware Version: 1.3.3

Mgmt MAC Address: 00:A0:98:5A:FC:02

Ethernet Link: down

Using DHCP: yes

IPv4 configuration:

IP Address: unknown

Netmask: unknown

Gateway: unknown

IPv6 configuration: Disabled

slot 0: Dual 10G Ethernet Controller T320E-SFP/KR

Device Type: CT-FE-3

Version Number: T3-SRAM1.1.0-BR1016-02-01-FW7.7.209-DR04

Serial Number: jb04050693

c0a MAC Address: 00:a0:98:5a:fb:fe (auto-unknown-cfg_down)

c0b MAC Address: 00:a0:98:5a:fb:ff (auto-unknown-enabling)

memory mapped I/O base 0xde801000, size 0x1000

memory mapped I/O base 0xde000000, size 0x800000

memory mapped I/O base 0xde800000, size 0x1000

slot 0: Internal BGE 10/100 Ethernet Controller

e0M MAC Address: 00:a0:98:5a:fc:00 (auto-unknown-down)

e0P MAC Address: 00:a0:98:5a:fc:01 (auto-100tx-fd-up)

Device Type: BCM5721

memory mapped I/O base 0xdf800000, size 0x10000

memory mapped I/O base 0xdf900000, size 0x10000

slot 0: Dual 10/100/1000 Ethernet Controller G20

e0a MAC Address: 00:a0:98:5a:fb:fc (auto-1000t-fd-up)

e0b MAC Address: 00:a0:98:5a:fb:fd (auto-unknown-down)

Device Type: Rev 6

memory mapped I/O base 0x00000000df720000, size 0x20000

memory mapped I/O base 0x00000000df700000, size 0x20000

I/O base 0x0000000000003000, size 0x20

memory mapped I/O base 0x00000000df760000, size 0x20000

memory mapped I/O base 0x00000000df740000, size 0x20000

I/O base 0x0000000000003020, size 0x20

slot 0: SAS Host Adapter 0a (PMC-Sierra PM8001 rev. C, SAS, <UP>)

Firmware rev: 01.11.07.00

Base WWN: 5:00a098:00364e4:08

Phy State: [0] Enabled, 6.0 Gb/s

[1] Enabled, 6.0 Gb/s

[2] Enabled, 6.0 Gb/s

[3] Enabled, 6.0 Gb/s

QSFP Vendor: Molex Inc.

QSFP Part Number: 112-00177+A0

QSFP Type: Passive Copper 2m ID:00

QSFP Serial Number: 408720176

02.0 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J4DDSP)

02.1 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J4GWPP)

02.2 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J4PE1P)

02.3 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J4LDNP)

02.4 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J4K6KP)

02.5 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J4689P)

02.6 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J4GWRP)

02.7 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J467YP)

02.8 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J466VP)

02.9 : NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J5053P)

02.10: NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J4WVNP)

02.11: NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J5044P)

02.12: NETAPP X306_HMRKP02TSSA NA00 1695.4GB 512B/sect (P6J504GP)

03.0 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1277461)

03.1 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1272876)

03.2 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1274858)

03.3 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1274741)

03.4 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1274793)

03.5 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1273338)

03.6 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1271576)

03.7 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1272160)

03.8 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1273637)

03.9 : NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1274446)

03.10: NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1272310)

03.11: NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1277264)

03.12: NETAPP X306_WKOJN02TSSA NA01 1695.4GB 512B/sect (WD-WCC1P1275875)

Shelf 2: IOM6 Firmware rev. IOM6 A: 0162 IOM6 B: 0162

Shelf 3: IOM6 Firmware rev. IOM6 A: 0162 IOM6 B: 0162

slot 0: SAS Host Adapter 0b (PMC-Sierra PM8001 rev. C, SAS, <UP>)

Firmware rev: 01.11.07.00

Base WWN: 5:00a098:00364e4:0c

Phy State: [4] Enabled, 6.0 Gb/s

[5] Enabled, 6.0 Gb/s

[6] Enabled, 6.0 Gb/s

[7] Enabled, 6.0 Gb/s

QSFP Vendor: Molex Inc.

QSFP Part Number: 112-00177+A0

QSFP Type: Passive Copper 2m ID:00

QSFP Serial Number: 408720154

00.0 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMJHWF)

00.1 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGNZ2NF)

00.2 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGG3YAM)

00.3 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGNZ67F)

00.4 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGJ7V4R)

00.5 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGNYTGF)

00.6 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGNZ2JF)

00.7 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMRH8M)

00.8 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMU9NM)

00.9 : NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMW1PM)

00.10: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMT8XM)

00.11: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMSGYM)

00.12: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGKNH0M)

00.13: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGK33SR)

00.14: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMRPDM)

00.15: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGL17AM)

00.16: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGEX0XR)

00.17: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMSYMM)

00.18: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGEJHVM)

00.19: NETAPP X422_HCOBE600A10 NA00 560.0GB 520B/sect (KWGMJ4UF)

01.0 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44O0A0OCFUZ6)

01.1 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44O0A0NJFUZ6)

01.2 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44O0A0MRFUZ6)

01.3 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44N0A058FUZ6)

01.4 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44O0A0MBFUZ6)

01.5 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44N0A055FUZ6)

01.6 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44O0A0LMFUZ6)

01.7 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44O0A0L4FUZ6)

01.8 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44O0A0KOFUZ6)

01.9 : NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44O0A0N7FUZ6)

01.10: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44N0A05BFUZ6)

01.11: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44N0A06YFUZ6)

01.12: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44M0A05RFUZ6)

01.13: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44M0A0ELFUZ6)

01.14: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44N0A0AXFUZ6)

01.15: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44N0A05JFUZ6)

01.16: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44M0A0EBFUZ6)

01.17: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44N0A04ZFUZ6)

01.18: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44N0A09JFUZ6)

01.19: NETAPP X422_TAL13600A10 NA01 560.0GB 520B/sect (44M0A0E3FUZ6)

Shelf 0: IOM6 Firmware rev. IOM6 A: 0162 IOM6 B: 0162

Shelf 1: IOM6 Firmware rev. IOM6 A: 0162 IOM6 B: 0162

memory mapped I/O base 0xdf530000, size 0x10000

memory mapped I/O base 0xdf520000, size 0x10000

memory mapped I/O base 0x00000000df510000, size 0x10000

memory mapped I/O base 0x00000000df500000, size 0x10000

slot 0: FC Host Adapter 0c (QLogic 2432 rev. 2, N-port, <UP>)

Firmware rev: 5.4.100

Host Port Id: 0x10300

FC Node Name: 5:00a:098003:64e400

FC Port Name: 5:00a:098003:64e400

SFP Vendor: FINISAR CORP.

SFP Part Number: FTLF8524P2BNV

SFP Serial Number: PQR1V3L

SFP Capabilities: 1, 2 or 4 Gbit

Link Data Rate: 4 Gbit

Switch Port: IBM_2498_B24:3

I/O base 0x0000000000002000, size 0x100

memory mapped I/O base 0xdf600000, size 0x4000

slot 0: FC Host Adapter 0d (QLogic 2432 rev. 2, N-port, <UP>)

Firmware rev: 5.4.100

Host Port Id: 0x10300

FC Node Name: 5:00a:098103:64e400

FC Port Name: 5:00a:098103:64e400

SFP Vendor: FINISAR CORP.

SFP Part Number: FTLF8524P2BNV

SFP Serial Number: PQS0QLZ

SFP Capabilities: 1, 2 or 4 Gbit

Link Data Rate: 4 Gbit

Switch Port: IBM_2498_B24:3

I/O base 0x0000000000002400, size 0x100

memory mapped I/O base 0xdf604000, size 0x4000

slot 0: Intel ICH USB EHCI Adapter u0a (0xdfa01400)

boot0 Micron Technology 0x655, class 0/0, rev 2.00/11.10, addr 2 1936MB 512B/sect (2DF0022700016909)

memory-mapped I/O base 0xdfa01400, size 0x400

slot 1: FC Host Adapter 1a (QLogic 2532 rev. 2, L-port, <OFFLINE (hard)>)

Board name: QLE2564

Serial Number: RFD1404U23108

Firmware rev: 5.4.130

Host Loop Id: 0

FC Node Name: 2:000:0024ff:731854

FC Port Name: 2:100:0024ff:731854

SFP Vendor: AVAGO

SFP Part Number: AFBR-57D9AMZ-QL

SFP Serial Number: AD13393043V

SFP Capabilities: 2, 4 or 8 Gbit

Link Data Rate: N/A

I/O base 0x00000000ffffff00, size 0x100

memory mapped I/O base 0xdee00000, size 0x4000

memory mapped I/O base 0xdec00000, size 0x100000

slot 1: FC Host Adapter 1b (QLogic 2532 rev. 2, L-port, <OFFLINE (hard)>)

Board name: QLE2564

Serial Number: RFD1404U23108

Firmware rev: 5.4.130

Host Loop Id: 0

FC Node Name: 2:000:0024ff:731855

FC Port Name: 2:100:0024ff:731855

SFP Vendor: AVAGO

SFP Part Number: AFBR-57D9AMZ-QL

SFP Serial Number: AD13393043X

SFP Capabilities: 2, 4 or 8 Gbit

Link Data Rate: N/A

I/O base 0x00000000ffffff00, size 0x100

memory mapped I/O base 0xdee04000, size 0x4000

memory mapped I/O base 0xded00000, size 0x100000

slot 1: FC Host Adapter 1c (QLogic 2532 rev. 2, L-port, <OFFLINE (hard)>)

Board name: QLE2564

Serial Number: RFD1404U23108

Firmware rev: 5.4.130

Host Loop Id: 0

FC Node Name: 2:000:0024ff:731856

FC Port Name: 2:100:0024ff:731856

SFP Vendor: AVAGO

SFP Part Number: AFBR-57D9AMZ-QL

SFP Serial Number: AD13393043Y

SFP Capabilities: 2, 4 or 8 Gbit

Link Data Rate: N/A

I/O base 0x00000000ffffff00, size 0x100

memory mapped I/O base 0xdf100000, size 0x4000

memory mapped I/O base 0xdef00000, size 0x100000

slot 1: FC Host Adapter 1d (QLogic 2532 rev. 2, L-port, <OFFLINE (hard)>)

Board name: QLE2564

Serial Number: RFD1404U23108

Firmware rev: 5.4.130

Host Loop Id: 0

FC Node Name: 2:000:0024ff:731857

FC Port Name: 2:100:0024ff:731857

SFP Vendor: AVAGO

SFP Part Number: AFBR-57D9AMZ-QL

SFP Serial Number: AD13393043U

SFP Capabilities: 2, 4 or 8 Gbit

Link Data Rate: N/A

I/O base 0x00000000ffffff00, size 0x100

memory mapped I/O base 0xdf104000, size 0x4000

memory mapped I/O base 0xdf000000, size 0x100000

slot 2: FCVI Host Adapter 2a (QLogic 2532(2562) rev. 2, unknown, <DOWN>)

Physical link: DOWN

FC Node Name: 21:00:00:24:ff:58:4f:58

Firmware rev: 5.3.140 Serial No: BFD1408T29623

Host Port Id: 0xffffffff

Cacheline size: 8 FC Packet size: 2048

SFP Vendor: FINISAR CORP.

SFP Part Number: FTLF8528P3BCV-QL

SFP Serial Number: PR11JG5

SFP Capabilities: 2, 4 or 8 Gbit

I/O base 0x00000000ffffff00, size 0x100

memory mapped I/O base 0xdf200000, size 0x4000

slot 2: FCVI Host Adapter 2b (QLogic 2532(2562) rev. 2, unknown, <DOWN>)

Physical link: DOWN

FC Node Name: 21:00:00:24:ff:58:4f:59

Firmware rev: 5.3.140 Serial No: BFD1408T29623

Host Port Id: 0xffffffff

Cacheline size: 8 FC Packet size: 2048

SFP Vendor: FINISAR CORP.

SFP Part Number: FTLF8528P3BCV-QL

SFP Serial Number: PR11JFW

SFP Capabilities: 2, 4 or 8 Gbit

I/O base 0x00000000ffffff00, size 0x100

memory mapped I/O base 0xdf204000, size 0x4000

unitech89e9 · ‎2014-08-17

rdfile /etc/log/messages
Sun Aug 17 00:19:34 GMT [N6220-SAS:callhome.management.log:info]: Call home for MANAGEMENT_LOG

Sun Aug 17 00:19:34 GMT [N6220-SAS:callhome.weekly:info]: Call home for WEEKLY_LOG

Sun Aug 17 00:19:35 GMT [N6220-SAS:kern.log.rotate:notice]: System N6220-SAS (ID 2018635377) is running 8.1.4P4

Sun Aug 17 00:56:11 GMT [N6220-SAS:asup.post.drop:error]: AutoSupport message (HA Group Notification from N6220-SAS (PERFORMANCE DATA) INFO) for host (0) was not posted to IBM. The system will drop the message.

Sun Aug 17 01:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 1:00am up 1 day, 22:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 01:00:03 GMT [N6220-SAS:raid.rg.scrub.start:notice]: /aggr0/plex0/rg2: starting scrub

Sun Aug 17 01:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 01:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 2801 minutes

Sun Aug 17 01:00:04 GMT [N6220-SAS:raid.rg.scrub.start:notice]: /aggr0/plex0/rg1: starting scrub

Sun Aug 17 01:00:05 GMT [N6220-SAS:raid.rg.scrub.start:notice]: /aggr0/plex0/rg0: starting scrub

Sun Aug 17 01:56:16 GMT [N6220-SAS:asup.post.drop:error]: AutoSupport message (HA Group Notification from N6220-SAS (MANAGEMENT_LOG) INFO) for host (0) was not posted to IBM. The system will drop the message.

Sun Aug 17 02:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 2:00am up 1 day, 23:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 02:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 02:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 2861 minutes

Sun Aug 17 02:56:26 GMT [N6220-SAS:asup.post.drop:error]: AutoSupport message (HA Group Notification from N6220-SAS (WEEKLY_LOG) INFO) for host (0) was not posted to IBM. The system will drop the message.

Sun Aug 17 03:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 3:00am up 2 days, 40 mins, 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 03:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 03:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 2921 minutes

Sun Aug 17 04:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 4:00am up 2 days, 1:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 04:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 04:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 2981 minutes

Sun Aug 17 04:35:21 GMT [N6220-SAS:raid.rg.scrub.done:notice]: /aggr0/plex0/rg2: scrub completed in 3:35:17.00

Sun Aug 17 04:35:21 GMT [N6220-SAS:raid.rg.scrub.summary.pi:notice]: Scrub found 0 parity inconsistencies in /aggr0/plex0/rg2.

Sun Aug 17 04:35:21 GMT [N6220-SAS:raid.rg.scrub.summary.cksum:notice]: Scrub found 0 checksum errors in /aggr0/plex0/rg2.

Sun Aug 17 04:35:21 GMT [N6220-SAS:raid.rg.scrub.summary.media:notice]: Scrub found 0 media errors in /aggr0/plex0/rg2.

Sun Aug 17 04:35:21 GMT [N6220-SAS:raid.rg.scrub.summary.lw:notice]: Scrub found 0 RAID write signature inconsistencies in /aggr0/plex0/rg2.

Sun Aug 17 05:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 5:00am up 2 days, 2:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 05:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 05:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3041 minutes

Sun Aug 17 05:39:58 GMT [N6220-SAS:raid.rg.scrub.done:notice]: /aggr0/plex0/rg1: scrub completed in 4:39:53.00

Sun Aug 17 05:39:58 GMT [N6220-SAS:raid.rg.scrub.summary.pi:notice]: Scrub found 0 parity inconsistencies in /aggr0/plex0/rg1.

Sun Aug 17 05:39:58 GMT [N6220-SAS:raid.rg.scrub.summary.cksum:notice]: Scrub found 0 checksum errors in /aggr0/plex0/rg1.

Sun Aug 17 05:39:58 GMT [N6220-SAS:raid.rg.scrub.summary.media:notice]: Scrub found 0 media errors in /aggr0/plex0/rg1.

Sun Aug 17 05:39:58 GMT [N6220-SAS:raid.rg.scrub.summary.lw:notice]: Scrub found 0 RAID write signature inconsistencies in /aggr0/plex0/rg1.

Sun Aug 17 06:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 6:00am up 2 days, 3:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 06:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 06:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3101 minutes

Sun Aug 17 06:44:41 GMT [N6220-SAS:raid.rg.scrub.done:notice]: /aggr0/plex0/rg0: scrub completed in 5:44:36.00

Sun Aug 17 06:44:41 GMT [N6220-SAS:raid.rg.scrub.summary.pi:notice]: Scrub found 0 parity inconsistencies in /aggr0/plex0/rg0.

Sun Aug 17 06:44:41 GMT [N6220-SAS:raid.rg.scrub.summary.cksum:notice]: Scrub found 0 checksum errors in /aggr0/plex0/rg0.

Sun Aug 17 06:44:41 GMT [N6220-SAS:raid.rg.scrub.summary.media:notice]: Scrub found 0 media errors in /aggr0/plex0/rg0.

Sun Aug 17 06:44:41 GMT [N6220-SAS:raid.rg.scrub.summary.lw:notice]: Scrub found 0 RAID write signature inconsistencies in /aggr0/plex0/rg0.

Sun Aug 17 07:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 7:00am up 2 days, 4:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 07:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 07:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3161 minutes

Sun Aug 17 08:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 8:00am up 2 days, 5:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 08:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 08:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3221 minutes

Sun Aug 17 09:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 9:00am up 2 days, 6:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 09:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 09:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3281 minutes

Sun Aug 17 10:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 10:00am up 2 days, 7:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 10:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 10:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3341 minutes

Sun Aug 17 11:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 11:00am up 2 days, 8:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 11:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 11:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3401 minutes

Sun Aug 17 12:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 12:00pm up 2 days, 9:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 12:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 12:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3461 minutes

Sun Aug 17 13:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 1:00pm up 2 days, 10:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 13:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 13:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3521 minutes

Sun Aug 17 14:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 2:00pm up 2 days, 11:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 14:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 14:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3581 minutes

Sun Aug 17 15:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 3:00pm up 2 days, 12:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 15:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 15:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3641 minutes

Sun Aug 17 16:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 4:00pm up 2 days, 13:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 16:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 16:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3701 minutes

Sun Aug 17 17:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 5:00pm up 2 days, 14:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 17:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 17:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3761 minutes

Sun Aug 17 18:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 6:00pm up 2 days, 15:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 18:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 18:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3821 minutes

Sun Aug 17 19:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 7:00pm up 2 days, 16:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 19:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 19:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3881 minutes

Sun Aug 17 20:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 8:00pm up 2 days, 17:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 20:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 20:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 3941 minutes

Sun Aug 17 21:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 9:00pm up 2 days, 18:40 0 NFS ops, 196 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 21:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled due to reason : partner mailbox disks not accessible or invalid.

Sun Aug 17 21:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 4001 minutes

Sun Aug 17 21:46:17 GMT [console_login_mgr:info]: root logged in from console
Sun Aug 17 21:46:25 GMT [N6220-SAS:cf.misc.operatorDisable:warning]: Failover monitor: operator initiated disabling of takeover

Sun Aug 17 21:46:26 GMT [N6220-SAS:cf.fsm.takeoverOfPartnerDisabled:notice]: Failover monitor: takeover of partner disabled (Controller Failover takeover disabled).

Sun Aug 17 21:46:26 GMT [N6220-SAS:cf.fsm.takeoverByPartnerDisabled:notice]: Failover monitor: takeover of N6220-SAS by partner disabled (Controller Failover takeover disabled).

Sun Aug 17 21:58:16 GMT [rc:notice]: The system was down for 662 seconds
Sun Aug 17 21:48:53 GMT [N6220-SAS:fci.adapter.link.online:info]: Fibre Channel adapter 0c link online.

Sun Aug 17 21:48:53 GMT [N6220-SAS:fci.adapter.link.online:info]: Fibre Channel adapter 0d link online.

Sun Aug 17 21:48:54 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0c reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004110 WWPN: 5005076803044110, switch:port IBM_2498_B24:0.

Sun Aug 17 21:48:54 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0c reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004111 WWPN: 5005076803044111, switch:port IBM_2498_B24:1.

Sun Aug 17 21:48:54 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0d reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004111 WWPN: 5005076803084111, switch:port IBM_2498_B24:1.

Sun Aug 17 21:48:54 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0d reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004110 WWPN: 5005076803084110, switch:port IBM_2498_B24:0.

Sun Aug 17 21:48:59 GMT [N6220-SAS:fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 1a.

Sun Aug 17 21:48:59 GMT [N6220-SAS:fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 1b.

Sun Aug 17 21:48:59 GMT [N6220-SAS:fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 1d.

Sun Aug 17 21:48:59 GMT [N6220-SAS:fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 1c.

Sun Aug 17 21:49:00 GMT [N6220-SAS:netif.linkDown:info]: Ethernet Wrench Port: Link down, check cable.

Sun Aug 17 21:49:05 GMT [N6220-SAS:diskown.isEnabled:info]: software ownership has been enabled for this system

Sun Aug 17 21:49:05 GMT [N6220-SAS:dcs.framework.enabled:info]: The DCS framework is enabled on this node.

Sun Aug 17 21:49:07 GMT [N6220-SAS:wafl.memory.status:info]: 7092MB of memory is currently available for the WAFL file system.

Sun Aug 17 21:49:09 GMT [N6220-SAS:shelf.config.spha:info]: System is using single path HA attached storage only.

Sun Aug 17 21:49:09 GMT [N6220-SAS:netif.linkUp:info]: Ethernet e0P: Link up.

Sun Aug 17 21:49:10 GMT [N6220-SAS:ses.multipath.ReqError:CRITICAL]: SAS-Shelf24 detected without a multipath configuration.

Sun Aug 17 21:49:11 GMT [N6220-SAS:mgr.boot.reason_ok:notice]: System rebooted after a reboot command.

Sun Aug 17 21:49:11 GMT [N6220-SAS:callhome.reboot.reboot:info]: Call home for REBOOT (reboot command)

Sun Aug 17 21:49:12 GMT [N6220-SAS:netif.linkDown:info]: Ethernet e0b: Link down, check cable.

Sun Aug 17 21:49:12 GMT [N6220-SAS:netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Sun Aug 17 21:49:14 GMT [N6220-SAS:netif.linkUp:info]: Ethernet e0P: Link up.

Sun Aug 17 21:49:14 GMT [N6220-SAS:netif.linkDown:info]: Ethernet e0M: Link down, check cable.

Sun Aug 17 21:50:27 GMT [N6220-SAS:fci.link.break:error]: Link break detected on Fibre Channel adapter 0c.

Sun Aug 17 21:50:58 GMT [N6220-SAS:fci.link.error:error]: Could not recover link on Fibre Channel adapter 0c after 30 seconds. Offlining the adapter.

Sun Aug 17 21:51:41 GMT [N6220-SAS:fci.adapter.link.online:info]: Fibre Channel adapter 0c link online.

Sun Aug 17 21:51:41 GMT [N6220-SAS:fci.adapter.online:info]: Fibre Channel adapter 0c is now online.

Sun Aug 17 21:51:41 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0c reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004110 WWPN: 5005076803044110, switch:port IBM_2498_B24:0.

Sun Aug 17 21:51:41 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0c reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004111 WWPN: 5005076803044111, switch:port IBM_2498_B24:1.

Sun Aug 17 21:57:59 GMT [N6220-SAS:fci.adapter.link.online:info]: Fibre Channel adapter 0c link online.

Sun Aug 17 21:57:59 GMT [N6220-SAS:fci.adapter.link.online:info]: Fibre Channel adapter 0d link online.

Sun Aug 17 21:57:59 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0c reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004110 WWPN: 5005076803044110, switch:port IBM_2498_B24:0.

Sun Aug 17 21:57:59 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0c reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004111 WWPN: 5005076803044111, switch:port IBM_2498_B24:1.

Sun Aug 17 21:57:59 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0d reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004110 WWPN: 5005076803084110, switch:port IBM_2498_B24:0.

Sun Aug 17 21:57:59 GMT [N6220-SAS:mlm.array.portEmpty.switch:info]: Fibre Channel host bus adapter 0d reports no LUNs presented by storage array IBM_2145_?, target port WWNN: 5005076803004111 WWPN: 5005076803084111, switch:port IBM_2498_B24:1.

Sun Aug 17 21:58:05 GMT [N6220-SAS:fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 1a.

Sun Aug 17 21:58:05 GMT [N6220-SAS:fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 1b.

Sun Aug 17 21:58:05 GMT [N6220-SAS:fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 1c.

Sun Aug 17 21:58:05 GMT [N6220-SAS:fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 1d.

Sun Aug 17 21:58:05 GMT [N6220-SAS:netif.linkDown:info]: Ethernet Wrench Port: Link down, check cable.

Sun Aug 17 21:58:11 GMT [N6220-SAS:diskown.isEnabled:info]: software ownership has been enabled for this system

Sun Aug 17 21:58:11 GMT [N6220-SAS:dcs.framework.enabled:info]: The DCS framework is enabled on this node.

Sun Aug 17 21:58:11 GMT [N6220-SAS:config.noPartnerDisks:CRITICAL]: No disks were detected for the partner; this node will be unable to takeover correctly

Sun Aug 17 21:58:11 GMT [N6220-SAS:callhome.dsk.config:warning]: Call home for DISK CONFIGURATION ERROR

Sun Aug 17 21:58:13 GMT [N6220-SAS:wafl.memory.status:info]: 7070MB of memory is currently available for the WAFL file system.

Sun Aug 17 21:58:13 GMT [N6220-SAS:fmmb.instStat.change:info]: no mailbox instance on local side.

Sun Aug 17 21:58:13 GMT [N6220-SAS:fmmb.instStat.change:info]: no mailbox instance on partner side.

Sun Aug 17 21:58:13 GMT [N6220-SAS:cf.fm.noMBDisksOrIc:warning]: Could not find the local mailbox disks. Could not determine the firmware state of the partner through the HA interconnect.

Sun Aug 17 21:58:31 GMT [N6220-SAS:tar.csum.match:info]: Stored checksum matches, not extracting /mroot_late.tgz.

Sun Aug 17 21:58:31 GMT [N6220-SAS:tar.csum.match:info]: Stored checksum matches, not extracting /platform/pmroot_late.tgz.

Sun Aug 17 21:58:32 GMT [N6220-SAS:sp.notConfigured:warning]: The system's Service Processor (SP) is not configured. Use the 'sp setup' command to configure it.

Sun Aug 17 21:58:32 GMT [N6220-SAS:sp.network.link.down:warning]: Service Processor (SP) network port link down due to cable or network errors.

Sun Aug 17 21:58:32 GMT [N6220-SAS:cf.hwassist.DefaultPrtnrPort:notice]: The system has chose the default hardware-assist partner port number 4444. If you want to use a different port number, change it using the command 'options cf.hw_assist.partner.port'.

Sun Aug 17 21:58:32 GMT [N6220-SAS:cf.hwassist.DefaultEnabled:info]: Hardware-assisted takeover feature is enabled on this system. The partner can take over this node faster during certain failures.

Sun Aug 17 21:58:32 GMT [N6220-SAS:cf.hwassist.Default.NodemgmtAddr:notice]: The system automatically chose 192.168.1.139 as the local hardware-assist IP address.

Sun Aug 17 21:58:36 GMT [N6220-SAS:nbt.nbns.socketError:error]: NBT: Cannot send broadcast message on NBNS socket. Error 0x32: Network is down.

Sun Aug 17 21:58:49 GMT [N6220-SAS:nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.

Sun Aug 17 21:59:00 GMT [N6220-SAS:monitor.globalStatus.critical:CRITICAL]: Controller failover partner unknown. Controller failover not possible.

Sun Aug 17 21:59:10 GMT [console_login_mgr:info]: root logged in from console
Sun Aug 17 21:59:26 GMT [N6220-SAS:callhome.performance.snap:info]: Call home for PERFORMANCE SNAPSHOT

Sun Aug 17 22:00:00 GMT [N6220-SAS:kern.uptime.filer:info]: 10:00pm up 2 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Sun Aug 17 22:00:04 GMT [N6220-SAS:sp.network.link.down:warning]: Service Processor (SP) network port link down due to cable or network errors.

Sun Aug 17 22:00:04 GMT [N6220-SAS:cf.takeover.disabled:warning]: Controller Failover is licensed but takeover of partner is disabled.

Sun Aug 17 22:00:04 GMT [N6220-SAS:cf.ic.hourlyBackplaneLinkDownTime:error]: Backplane Interconnect link has been down for 2 minutes

Sun Aug 17 22:00:35 GMT [N6220-SAS:cf.misc.operatorEnable:notice]: Failover monitor: operator initiated enabling of failover.

Sun Aug 17 22:00:35 GMT [N6220-SAS:cf.fsm.takeoverOfPartnerDisabled:notice]: Failover monitor: takeover of partner disabled (partner mailbox disks not accessible or invalid).

Sun Aug 17 22:00:35 GMT [N6220-SAS:cf.fsm.takeoverByPartnerDisabled:notice]: Failover monitor: takeover of N6220-SAS by partner disabled (interconnect error).

Sun Aug 17 22:00:39 GMT [N6220-SAS:cf.fsm.partnerNotResponding:notice]: Failover monitor: partner not responding

Sun Aug 17 22:00:49 GMT [N6220-SAS:ha.takeoverImpDegraded:warning]: Takeover of the partner node is impossible due to lack of connectivity to the partner mailbox disks.

Sun Aug 17 22:01:04 GMT [N6220-SAS:netif.linkUp:info]: Ethernet e0a: Link up.

Sun Aug 17 22:03:11 GMT [N6220-SAS:wafl.quota.sec.change:notice]: security style for /vol/vol0/ changed from unix to mixed

Sun Aug 17 22:04:00 GMT [N6220-SAS:wafl.quota.sec.change:notice]: security style for /vol/vol0/ changed from mixed to ntfs

Sun Aug 17 22:06:20 GMT [N6220-SAS:netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Sun Aug 17 22:06:24 GMT [N6220-SAS:netif.linkUp:info]: Ethernet e0a: Link up.

Sun Aug 17 22:08:51 GMT [telnet_0:info]: root logged in from host: 192.168.0.100
Sun Aug 17 22:09:30 GMT [N6220-SAS:netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Sun Aug 17 22:09:43 GMT [N6220-SAS:netif.linkUp:info]: Ethernet e0a: Link up.

Sun Aug 17 22:09:50 GMT [telnet_0:info]: root logged in from host: 192.168.0.100
Sun Aug 17 22:10:48 GMT [N6220-SAS:callhome.partner.down:CRITICAL]: Call home for PARTNER DOWN, TAKEOVER IMPOSSIBLE

Sun Aug 17 22:11:16 GMT [N6220-SAS:netif.linkDown:info]: Ethernet e0a: Link down, check cable.

Sun Aug 17 22:25:08 GMT [N6220-SAS:netif.linkUp:info]: Ethernet e0a: Link up.

Sun Aug 17 22:25:24 GMT [telnet_0:info]: root logged in from host: 192.168.0.100
N6220-SAS*>

DOMINIC_WYSS · ‎2014-08-17

slot 0: Dual 10G Ethernet Controller T320E-SFP/KR

Device Type: CT-FE-3

Version Number: T3-SRAM1.1.0-BR1016-02-01-FW7.7.209-DR04

Serial Number: jb04050693

c0a MAC Address: 00:a0:98:5a:fb:fe (auto-unknown-cfg_down)

c0b MAC Address: 00:a0:98:5a:fb:ff (auto-unknown-enabling)

I think you have a physical connection issue on the ports, they are not coming up.

which cables are you using?

aborzenkov · ‎2014-08-18

Please boot into maintenance mode and use “ha-config show” to check current HA settings. Looks like your chassis is set to HA, which means it will disable external ports and try internal backplane connection.

DOMINIC_WYSS · ‎2014-08-18

good idea about the ha-config!

but it has to be set to ha, even if it's a single-chassis.

https://library.netapp.com/ecmdocs/ECMM1249823/html/haconfig/GUID-F4091101-4D6B-4577-B337-CBA633143BAC.html

maybe one of the chassis is set to non-ha and does not respond to the interconnect requests?

shane_bradley · ‎2014-08-19

ah of course !

didnt even think about that