2011-10-20 01:54 PM
This week all of a sudden we started to see disconnected mounts from the NAS. Looking at the logs I see
Thu Oct 20 16:00:00 GMT [raon-netapp1: kern.uptime.filer:info]: 4:00pm up 1 day, 16 mins, 366015 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 DAFS ops, 0 FCP ops, 0 iSCSI ops
Thu Oct 20 16:00:00 GMT [raon-netapp1: statd:ALERT]: Cluster is licensed but takeover of partner is disabled
Thu Oct 20 16:32:43 GMT [raon-netapp1: cf.fsm.firmwareStatus:info]: Cluster monitor: partner halted
Thu Oct 20 16:32:44 GMT [raon-netapp1: cf.nm.nicViError:info]: Interconnect nic 0 has error on VI #4 SEND_DESC_ERROR 2
Thu Oct 20 16:32:44 GMT [raon-netapp1: cf.nm.nicTransitionDown:warning]: Interconnect link 0 is DOWN
Thu Oct 20 16:32:44 GMT [raon-netapp1: cf.rv.notConnected:error]: Connection for cfo_rv failed
Thu Oct 20 16:32:50 GMT [raon-netapp1: cf.fsm.partnerNotResponding:notice]: Cluster monitor: partner not responding
The network module LEDs are all off and the network activity LEDs are not flashing.
If I power cycle the appliance or re-seat the network module, it starts to work ok for a day or so.
After re-seating the module ....
Thu Oct 20 19:31:56 GMT [raon-netapp1: cf.nm.nicTransitionUp:info]: Interconnect link 0 is UP
Thu Oct 20 19:31:57 GMT [raon-netapp1: cf.fsm.firmwareStatus:info]: Cluster monitor: partner kernel loaded
Thu Oct 20 19:32:33 GMT [raon-netapp1: cf.fsm.partnerOk:notice]: Cluster monitor: partner ok
Thu Oct 20 19:32:33 GMT [raon-netapp1: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of raon-netapp2 disabled (partner booting)
Thu Oct 20 19:32:35 GMT [raon-netapp1: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of raon-netapp2 enabled
Thu Oct 20 19:32:35 GMT [raon-netapp1: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of raon-netapp1 by raon-netapp2 enabled
Thu Oct 20 19:33:17 GMT [raon-netapp1: monitor.globalStatus.ok:info]: The system's global status is normal.
Does anyone know what is going on here. Any help will be appreciated.
2011-10-20 02:01 PM
The cluster interconnect... which is implemented over Gigabit Ethernet but internally inside the DS14mk2 shelf. Since reseating fixes it but it comes back, sounds like a hardware issue. Either in the controller or shelf (shelf is passive so more likely the 270). If you have support it would be worth an RMA on the controller.
2011-10-27 01:13 PM
I am out of the office until 10/31/2011.
I will be out of the office visiting customers with limited access to phone
or email. I will be returning to the office on 10/31/2011.
If you have any questions or need immediate assist, please contact the
Sushil U. Nanavati
IBM Technical Sales Specialist, Storage and System z East IMT
Richard Swain - IBM N series Technical Resource Southeast
Thnak you and have a great day.
Note: This is an automated response to your message ""FAS270 network
sent on 10/27/2011 15:53:26.
This is the only notification you will receive while this person is away.