2013-02-06 12:47 AM
Hi, we have FAS3020c and this morning at around quartert to 6 DFM reported that the partner node is dead. I ran into the office and checked the lights on the harddisks on the shelves. No amber lights! So I thought it is not a harddisk problem. I tried a cf giveback on the takeover node and this reported two disks missing on the partner node. I checked again and all lights were green. I forced a giveback and noticed that the complete aggr0 was broken.
vol status -f showed no broken disks
I turned off the disk shelve and then 3 FC disks were reported as broken. How likely is it that 3 disks fail at the same time? I mean there wasn't even notification from DFM that one of the disks had failed.
Could this be a firmware bug?
Any Ideas how to recover some of the data of the broken aggregate.
2013-02-06 03:15 AM
Ok, I rebootet the partner node, reconnected all shelve cables and did a cf giveback -f.
I unfailed all 3 failed disks and brought the aggregate online again.
So far it is running and I can migrate all VMs to a new datastore on a different storage. I have no clue what happened on the FAS but I will move all data off it and not use it anymore.
2013-02-06 04:16 AM
In the messages log these two errors are displayed before the filer switched to it's parter node and declared itself as dead:
Wed Feb 6 05:43:46 CET [DS-SAN-01: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Bad response entry" on adapter 0a.
Wed Feb 6 05:43:46 CET [DS-SAN-01: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Revisited response entry" on adapter 0a.
I can't find any good information on these two error. Any clues?
2013-02-06 07:09 AM
Gah.. Sorry I don't have a better answer for you. The only two things I can think of are:
1) Bad hardware
2) Driver or software bug
It really sounds like #1 to me. You may be able to verify by booting to the diags on the controller and funning a full diag against the FC adapters. Below is the guide for the diag tools:
Hope that helps.