fault:CRITICAL]: Fault reported on disk storage shelf attached to channel 0a

KyleKJ20 · ‎2017-04-13

Hoping someone can help

Description : [monitor.shelf.fault:CRITICAL]: Fault reported on disk storage shelf attached to channel 0a.

Wed Apr 12 16:49:15 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to enclosure services device 0a.15 (0x0100000f).

Wed Apr 12 16:49:28 PDT [CAT1papnas001b: ses.status.electronicsError:CRITICAL]: DS14-Mk2-AT shelf 1 on channel 0b environmental monitoring error for SES electronics 2: failed This module is on the rear side of the shelf, at the bottom center.

Wed Apr 12 16:49:28 PDT [CAT1papnas001b: ses.status.ATFCXError:CRITICAL]: DS14-Mk2-AT shelf 1 on channel 0b error for AT-FCX Bridge electronics SES controller module B: critical status; CEMI Micro Controller Fatal Error. .

Wed Apr 12 16:49:28 PDT [CAT1papnas001b: monitor.shelf.fault:CRITICAL]: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power supplies, disks, and temperature sensors.

Wed Apr 12 16:49:41 PDT [CAT1papnas001b: asup.post.sent:notice]: Cluster Notification message posted to IBM: Cluster Notification from CAT1papnas001b (SHELF_FAULT) WARNING
Wed Apr 12 16:50:00 PDT [CAT1papnas001b: monitor.globalStatus.critical:CRITICAL]: Disk shelf fault.

Wed Apr 12 16:50:56 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to enclosure services device 0a.14 (0x0100000e).
Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.16 (0x01000010).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.17 (0x01000011).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.18 (0x01000012).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.19 (0x01000013).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.20 (0x01000014).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.21 (0x01000015).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.22 (0x01000016).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.23 (0x01000017).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.24 (0x01000018).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.25 (0x01000019).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.26 (0x0100001a).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.27 (0x0100001b).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.28 (0x0100001c).

Wed Apr 12 16:51:16 PDT [CAT1papnas001b: fci.device.login.failure:error]: Fibre Channel adapter 0a could not login to disk device 0a.29 (0x0100001d).

Wed Apr 12 17:00:00 PDT [CAT1papnas001b: kern.uptime.filer:info]:   5:00pm up 569 days, 4:36 0 NFS ops, 95575007 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Wed Apr 12 17:00:00 PDT [CAT1papnas001b: monitor.shelf.fault:CRITICAL]: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power supplies, disks, and temperature sensors.

Wed Apr 12 17:18:55 PDT [CAT1papnas001b: nbt.WINS.registrationTimeout:info]: NBT: No WINS server are responding. The filer will continue to try to register with WINS.

Wed Apr 12 17:54:19 PDT [CAT1papnas001b: auth.dc.trace.DCConnection.errorMsg:error]: AUTH: Domain Controller error: NetLogon error 0xc0000022: - Filer's security information differs from domain controller \\CAACPWPADS013.

Wed Apr 12 17:54:26 PDT [CAT1papnas001b: asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Alert from CAT1papnas001b

===== STORAGE =====

Slot:            0a
Description:     Fibre Channel Host Adapter 0a
FC Packet Size: 2048
Link Data Rate: 2 Gbit
SRAM Parity:     Yes
External GBIC:   No
State:           Enabled
In Use:          No
Redundant:       Yes

asulliva · ‎2017-04-13

Looks like a failed SES module on your shelf. Open a support case, they will walk you through the process of troubleshooting and/or replacing it (or work with you to coordinate on-site service).

Andrew

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

andris · ‎2017-04-14

I noticed that this is an IBM N-series variant - you'd need to work with IBM Support.

That said, take a look at this public report for Bug 245162. It has some guidance (e.g. reseat the module)

"CEMI Micro controller Fatal Error" leads to loss of access to AT-FCX module

http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=245162