Fas 3240 Faul Report

[syedkhan]

Hello,

      We have 2 contrillers namely netappctrl1 and netappctrl2 both are FAS3240 connected to DS4246 and DS2243 in HA Pair.  I am seeing an error.

    Fault reported on disk storage shelf attached to channel 4a. Please check fans, power supplies, disks, and temperature sensors

To investigate more I ran the below command and found this 

netappctrl1> environment status shelf 4a
Environment for channel 4a
Number of shelves monitored: 2 enabled: yes
Environmental failure on shelves on this channel? yes

Channel: 4a
Shelf: 10
SES device path: local access: 0b.10.99
Module type: IOM6; monitoring is active
Shelf status: non-critical condition
SES Configuration, shelf 10:
logical identifier=0x500a09800006b35c
vendor identification=NETAPP
product identification=DS2246
product revision level=0111
Vendor-specific information:
Product Serial Number: 6000012730
Status reads attempted: 27567; failed: 0
Control writes attempted: 120; failed: 0
Shelf bays with disk devices installed:
16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
with error: none
Power Supply installed element list: 1, 2; with error: none
Power Supply information by element:
[1] Serial number: XXT103302661 Part number: 114-00065+A0
Type: 9C
Firmware version: 020F Swaps: 0
[2] Serial number: XXT103302660 Part number: 114-00065+A0
Type: 9C
Firmware version: 020F Swaps: 0
Voltage Sensor installed element list: 1, 2, 3, 4; with error: none
Shelf voltages by element:
[1] 12.30 Volts Normal voltage range
[2] 5.07 Volts Normal voltage range
[3] 12.18 Volts Normal voltage range
[4] 5.07 Volts Normal voltage range
Current Sensor installed element list: 1, 2, 3, 4; with error: none
Shelf currents by element:
[1] 0 mA Normal current range
[2] 0 mA Normal current range
[3] 0 mA Normal current range
[4] 4410 mA Normal current range
Cooling Unit installed element list: 1, 2, 3, 4; with error: none
Cooling Units by element:
[1] 2970 RPM
[2] 3000 RPM
[3] 3000 RPM
[4] 3000 RPM
Temperature Sensor installed element list: 1, 2, 3, 4, 5, 6, 7, 8; with error: 1
Shelf temperatures by element:
[1] 2 C (35 F) (ambient) Undertemperature warning!
[2] 11 C (51 F) Normal temperature range
[3] 11 C (51 F) Normal temperature range
[4] 28 C (82 F) Normal temperature range
[5] 13 C (55 F) Normal temperature range
[6] 32 C (89 F) Normal temperature range
[7] 12 C (53 F) Normal temperature range
[8] 15 C (59 F) Normal temperature range
Temperature thresholds by element:
[1] High critical: 42 C (107 F); high warning 40 C (104 F)
Low critical: 0C (32 F); low warning 5 C (41 F)
[2] High critical: 55 C (131 F); high warning 50 C (122 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[3] High critical: 55 C (131 F); high warning 50 C (122 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[4] High critical: 70 C (158 F); high warning 65 C (149 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[5] High critical: 55 C (131 F); high warning 50 C (122 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[6] High critical: 70 C (158 F); high warning 65 C (149 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[7] High critical: 60 C (140 F); high warning 55 C (131 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[8] High critical: 60 C (140 F); high warning 55 C (131 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
ES Electronics installed element list: 1, 2; with error: none
ES Electronics reporting element: 1
ES Electronics information by element:
[1] Serial number: 8000150130 Part number: 111-00690+A2
CPLD version: 23 Swaps: 0
[2] Serial number: 8000145197 Part number: 111-00690+A2
CPLD version: 23 Swaps: 0
SAS connector attached element list: 1, 2, 3, 4; with error: none
SAS cable information by element:
[1] Vendor: Molex Inc.
Type: QSFP copper 2m ID: 01 Swaps: 0
Serial number: 122720047 Part number: 112-00177+A0
[2] Vendor: Molex Inc.
Type: QSFP copper 5m ID: 00 Swaps: 0
Serial number: 127920282 Part number: 112-00178+A0
[3] Vendor: Molex Inc.
Type: QSFP copper 2m ID: 00 Swaps: 0
Serial number: 122720048 Part number: 112-00177+A0
[4] Vendor: Molex Inc.
Type: QSFP copper 5m ID: 01 Swaps: 0
Serial number: 127920290 Part number: 112-00178+A0
ACP installed element list: 1, 2; with error: none
ACP information by element:
[1] MAC address: 00:A0:98:14:BF:C2
[2] MAC address: 00:A0:98:14:C3:60
SAS Expander Module installed element list: 1, 2; with error: none
SAS Expander master module: 1

Channel: 4a
Shelf: 11
SES device path: local access: 0b.11.99
Module type: IOM6; monitoring is active
Shelf status: normal condition
SES Configuration, shelf 11:
logical identifier=0x500a09800006b7e1
vendor identification=NETAPP
product identification=DS2246
product revision level=0173
Vendor-specific information:
Product Serial Number: 6000012546
Status reads attempted: 27567; failed: 0
Control writes attempted: 89; failed: 0
Shelf bays with disk devices installed:
14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
with error: none
Power Supply installed element list: 1, 2; with error: none
Power Supply information by element:
[1] Serial number: XXT103302737 Part number: 114-00065+A0
Type: 9C
Firmware version: 020F Swaps: 0
[2] Serial number: XXT103302730 Part number: 114-00065+A0
Type: 9C
Firmware version: 020F Swaps: 0
Voltage Sensor installed element list: 1, 2, 3, 4; with error: none
Shelf voltages by element:
[1] 12.18 Volts Normal voltage range
[2] 5.07 Volts Normal voltage range
[3] 12.18 Volts Normal voltage range
[4] 5.07 Volts Normal voltage range
Current Sensor installed element list: 1, 2, 3, 4; with error: none
Shelf currents by element:
[1] 0 mA Normal current range
[2] 0 mA Normal current range
[3] 0 mA Normal current range
[4] 0 mA Normal current range
Cooling Unit installed element list: 1, 2, 3, 4; with error: none
Cooling Units by element:
[1] 3000 RPM
[2] 2970 RPM
[3] 3000 RPM
[4] 2970 RPM
Temperature Sensor installed element list: 1, 2, 3, 4, 5, 6, 7, 8; with error: none
Shelf temperatures by element:
[1] 6 C (42 F) (ambient) Normal temperature range
[2] 12 C (53 F) Normal temperature range
[3] 11 C (51 F) Normal temperature range
[4] 31 C (87 F) Normal temperature range
[5] 15 C (59 F) Normal temperature range
[6] 32 C (89 F) Normal temperature range
[7] 13 C (55 F) Normal temperature range
[8] 16 C (60 F) Normal temperature range
Temperature thresholds by element:
[1] High critical: 42 C (107 F); high warning 40 C (104 F)
Low critical: 0C (32 F); low warning 5 C (41 F)
[2] High critical: 55 C (131 F); high warning 50 C (122 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[3] High critical: 55 C (131 F); high warning 50 C (122 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[4] High critical: 70 C (158 F); high warning 65 C (149 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[5] High critical: 55 C (131 F); high warning 50 C (122 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[6] High critical: 70 C (158 F); high warning 65 C (149 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[7] High critical: 60 C (140 F); high warning 55 C (131 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
[8] High critical: 60 C (140 F); high warning 55 C (131 F)
Low critical: 5C (41 F); low warning 10 C (50 F)
ES Electronics installed element list: 1, 2; with error: none
ES Electronics reporting element: 1
ES Electronics information by element:
[1] Serial number: 7902480619 Part number: 111-00190+B0
CPLD version: 23 Swaps: 0
[2] Serial number: 8000148151 Part number: 111-00690+A2
CPLD version: 23 Swaps: 0
SAS connector attached element list: 1, 2, 3, 4; with error: none
SAS cable information by element:
[1] Vendor: Molex Inc.
Type: QSFP copper 5m ID: 01 Swaps: 0
Serial number: 127920282 Part number: 112-00178+A0
[2] Vendor: Molex Inc.
Type: QSFP copper 5m ID: 01 Swaps: 0
Serial number: 127920298 Part number: 112-00178+A0
[3] Vendor: Molex Inc.
Type: QSFP copper 5m ID: 00 Swaps: 0
Serial number: 127920290 Part number: 112-00178+A0
[4] Vendor: Molex Inc.
Type: QSFP copper 5m ID: 00 Swaps: 0
Serial number: 127920293 Part number: 112-00178+A0
ACP installed element list: 1, 2; with error: none
ACP information by element:
[1] MAC address: 00:A0:98:75:9F:DD
[2] MAC address: 00:A0:98:14:C0:64
SAS Expander Module installed element list: 1, 2; with error: none
SAS Expander master module: 2

Shelf mapping (shelf-assigned addresses) for channel 4a:
Shelf 10: XXX XXX XXX XXX XXX XXX XXX 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
Shelf 11: XXX XXX XXX XXX XXX XXX XXX XXX XXX 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

[SpindleNinja]

emperature Sensor installed element list: 1, 2, 3, 4, 5, 6, 7, 8; with error: 1
Shelf temperatures by element:
[1] 2 C (35 F) (ambient) Undertemperature warning!

could be failed.    Open a support case. 

[syedkhan]

Yes true this is due to temprature but its not clear that the sensor is a problem. If you see the Log carefully 

emperature Sensor installed element list: 1, 2, 3, 4, 5, 6, 7, 8; with error: 1
Shelf temperatures by element:
[1] 2 C (35 F) (ambient) Undertemperature warning!

Temperature thresholds by element:
[1] High critical: 42 C (107 F); high warning 40 C (104 F)
Low critical: 0C (32 F); low warning 5 C (41 F)

Low warninig comes when the temp is dropped more than 5 degrees .

The point is how can I remove this or increase the threshold

[andris]

ONTAP version?

IOM6 FW version?

What is the ambient temp at that shelf?

[syedkhan]

ONTAP version?

NetApp Release 8.1RC3 7-Mode: Wed Feb 15 19:28:21 PST 2012

IOM6 FW version?

Shelf 0: IOM3 Firmware rev. IOM3 A: 0132 IOM3 B: 0132
Shelf 1: IOM3 Firmware rev. IOM3 A: 0132 IOM3 B: 0132

Shelf 10: IOM6 Firmware rev. IOM6 A: 0111 IOM6 B: 0111
Shelf 11: IOM6 Firmware rev. IOM6 A: 0173 IOM6 B: 0111

What is the ambient temp at that shelf ?? 

Shelf temperatures by element:
[1] 6 C (42 F) (ambient) Normal temperature range
[2] 12 C (53 F) Normal temperature range
[3] 12 C (53 F) Normal temperature range
[4] 32 C (89 F) Normal temperature range
[5] 16 C (60 F) Normal temperature range
[6] 33 C (91 F) Normal temperature range
[7] 14 C (57 F) Normal temperature range
[8] 17 C (62 F) Normal temperature range
Temperature thresholds by element:
[1] High critical: 42 C (107 F); high warning 40 C (104 F)
Low critical: 0C (32 F); low warning 5 C (41 F)

What if the temprature is below mentioned in the thresholds, will it cause a problem to the disk shelves or its just a warning.

[andris]

I meant "if you take a thermometer and stand by the shelf, what is the air temperature"?

Your location seems to be on the cold side - is that the reality?

Being too cold is not as critical as too hot, but condensation can become an issue if it actually is approaching 2C.

But what is evident is that the software and firmware is in dire need of updates (ONTAP, SP, shelf, disk).

I suggest you start with moving to the best 8.1.x release we have - 8.1.4P10.

Once you are stable there, then move on to 8.2.5P3 - the latest and best for 7-Mode.

Note: You'll have to ensure that your environment (hosts, SnapManager, etc.) versions are compatible with ONTAP 8.2.5, if you go there.

You can independently upgrade the SP, shelf and disk firmware, or just pick up what is bundled in ONTAP, for now.

[paul_stejskal]

I agree it is cold. I have seen it before where one sensor could be broken, but you have 6 sensors on the cool side. It might not hurt to verify if the heat in the location the filer is at is working decently by the filer. It's almost like it's by an open door and it's winter out.

[syedkhan]

Will it create any problem or leave it as it is

[paul_stejskal]

I would get that taken care of ASAP.