topic 3 FC disks failed at the same time?und in ONTAP Hardware

3 FC disks failed at the same time?und

oweinmann — Thu, 05 Jun 2025 06:10:56 GMT

Hi, we have FAS3020c and this morning at around quartert to 6 DFM reported that the partner node is dead. I ran into the office and checked the lights on the harddisks on the shelves. No amber lights! So I thought it is not a harddisk problem. I tried a cf giveback on the takeover node and this reported two disks missing on the partner node. I checked again and all lights were green. I forced a giveback and noticed that the complete aggr0 was broken.

vol status -f showed no broken disks

I turned off the disk shelve and then 3 FC disks were reported as broken. How likely is it that 3 disks fail at the same time? I mean there wasn't even notification from DFM that one of the disks had failed.

Could this be a firmware bug?

Any Ideas how to recover some of the data of the broken aggregate.

Best Regards,

Oliver

Re: 3 FC disks failed at the same time?und

oweinmann — Wed, 06 Feb 2013 11:15:10 GMT

Ok, I rebootet the partner node, reconnected all shelve cables and did a cf giveback -f.

I unfailed all 3 failed disks and brought the aggregate online again.

So far it is running and I can migrate all VMs to a new datastore on a different storage. I have no clue what happened on the FAS but I will move all data off it and not use it anymore.

Re: 3 FC disks failed at the same time?und

oweinmann — Wed, 06 Feb 2013 12:16:50 GMT

In the messages log these two errors are displayed before the filer switched to it's parter node and declared itself as dead:

Wed Feb 6 05:43:46 CET [DS-SAN-01: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Bad response entry" on adapter 0a.

Wed Feb 6 05:43:46 CET [DS-SAN-01: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Revisited response entry" on adapter 0a.

I can't find any good information on these two error. Any clues?

Re: 3 FC disks failed at the same time?und

bsti — Wed, 06 Feb 2013 14:34:00 GMT

I've not run into that particular issue. Sounds like potential bad hardware. Have you opened a case with support on this?

Re: 3 FC disks failed at the same time?und

oweinmann — Wed, 06 Feb 2013 14:43:03 GMT

Hi,

the system is out of warranty, so I don't think support will give me any hints on this issue.

Re: 3 FC disks failed at the same time?und

bsti — Wed, 06 Feb 2013 15:09:49 GMT

Gah.. Sorry I don't have a better answer for you. The only two things I can think of are:

1) Bad hardware

2) Driver or software bug

It really sounds like #1 to me. You may be able to verify by booting to the diags on the controller and funning a full diag against the FC adapters. Below is the guide for the diag tools:

https://library.netapp.com/ecm/ecm_get_file/ECMP1112531

Hope that helps.