NetApp FAS8700 service-processor connection refused

[ppadmgeo1]

We recently had NetApp install a two-node FAS8700 cluster, but when we checked - we can only connect to 1 out of the service-processors. We can connect to the management lifs of both nodes.

Any help appreciated troubleshooting this...

Both showing online and with correct IPs, but I can only connect to the first one 10.7.190.14 - the second one says connection refused:

pnl0004scpr1655::> system service-processor show -instance

Node: pnl0004scpr1655-01
Type of Device: BMC
Status: online
Is Network Configured: true
Public IP Address: 10.7.190.14
MAC Address: d0:39:ea:13:0c:e1
Firmware Version: 13.2
Part Number: NA
Serial Number: NA
Device Revision: 13.2
Is Firmware Autoupdate Enabled: true

Node: pnl0004scpr1655-02
Type of Device: BMC
Status: online
Is Network Configured: true
Public IP Address: 10.7.190.15
MAC Address: d0:39:ea:13:16:36
Firmware Version: 13.2
Part Number: NA
Serial Number: NA
Device Revision: 13.2
Is Firmware Autoupdate Enabled: true
2 entries were displayed.

I came across this command, but it is not accepted:

pnl0004scpr1655::> system service-processor ssh show

Error: show failed: This command is not supported in this platform.

This is the detailed network setup which seems correct:

pnl0004scpr1655::> system service-processor network show -instance

Node: pnl0004scpr1655-01
Address Family: IPv4
Interface Enabled: true
Type of Device: BMC
Status: online
Link Status: up
DHCP Status: none
IP Address: 10.7.190.14
MAC Address: d0:39:ea:13:0c:e1
Netmask: 255.255.255.0
Prefix Length of Subnet Mask: -
Router Assigned IP Address: -
Link Local IP Address: -
Gateway IP Address: 10.7.190.1
Time Last Updated: 2020:08:03.22:49:35
Subnet Name: -
Enable IPv6 Router Assigned Address: -
SP Network Setup Status: succeeded
SP Network Setup Failure Reason: -

Node: pnl0004scpr1655-01
Address Family: IPv6
Interface Enabled: false
Type of Device: BMC
Status: online
Link Status: disabled
DHCP Status: none
IP Address: -
MAC Address: d0:39:ea:13:0c:e1
Netmask: -
Prefix Length of Subnet Mask: 0
Router Assigned IP Address: -
Link Local IP Address: -
Gateway IP Address: -
Time Last Updated: 2020:08:03.22:49:35
Subnet Name: -
Enable IPv6 Router Assigned Address: -
SP Network Setup Status: not-setup
SP Network Setup Failure Reason: -

Node: pnl0004scpr1655-02
Address Family: IPv4
Interface Enabled: true
Type of Device: BMC
Status: online
Link Status: up
DHCP Status: none
IP Address: 10.7.190.15
MAC Address: d0:39:ea:13:16:36
Netmask: 255.255.255.0
Prefix Length of Subnet Mask: -
Router Assigned IP Address: -
Link Local IP Address: -
Gateway IP Address: 10.7.190.1
Time Last Updated: 2020:08:03.22:42:47
Subnet Name: -
Enable IPv6 Router Assigned Address: -
SP Network Setup Status: succeeded
SP Network Setup Failure Reason: -

Node: pnl0004scpr1655-02
Address Family: IPv6
Interface Enabled: false
Type of Device: BMC
Status: online
Link Status: disabled
DHCP Status: none
IP Address: -
MAC Address: d0:39:ea:13:16:36
Netmask: -
Prefix Length of Subnet Mask: 0
Router Assigned IP Address: -
Link Local IP Address: -
Gateway IP Address: -
Time Last Updated: 2020:08:03.22:42:47
Subnet Name: -
Enable IPv6 Router Assigned Address: -
SP Network Setup Status: not-setup
SP Network Setup Failure Reason: -
4 entries were displayed.

[hmoubara]

Hello,

Can you try to console to the BMC/SP and check if you are getting any error messages. It could be that the config file is corrupted and might need to reconfigure it. Please share the output once connected directly through console.

Thanks 

[ppadmgeo1]

sorry, the nodes are on a remote site and I had difficulty with our black boxes which is where the console ports are connected.

I connected to the console for both nodes and there are no messages displayed - happily displayed the login prompt

login: admin
Password:
******************************************************
* This is a serial console session. Output from this *
* session is mirrored on the SP console session. *
******************************************************
pnl0003scpr1605::>

[amans]

Hello,

You could be possibly hitting https://mysupport.netapp.com/site/bugs-online/product/ONTAP/BURT/1328905

Once you do serial console and if you get to see something like below, then suggest to open a support ticket.

sh: can't kill pid 25487: No such process
network: Secure Port is enabled for [ssh] with 22
Unable to load host key "/etc/ssh/ssh_host_rsa_key": invalid format
Unable to load host key "/etc/ssh/ssh_host_rsa_key": invalid format
Unable to load host key: /etc/ssh/ssh_host_rsa_key
Unable to load host key "/etc/ssh/ssh_host_dsa_key": invalid format
Unable to load host key "/etc/ssh/ssh_host_dsa_key": invalid format
Unable to load host key: /etc/ssh/ssh_host_dsa_key

Regards,

Aman

[ppadmgeo1]

Aman,

I've not seen that output,b ut have a case raised anyway - waiting for analysis it says

[amans]

Hi,

Can u let me know the case number just to see the progress.

Regards,

Aman

[ppadmgeo1]

Thanks,

Case # 2008409097

[amans]

Looks like internal workaround from the BUG I shared earlier was implemented and issue was resloved.

Regards,

Aman

[sritam212]

May i know what is the workaround for this which got implemented as i have exactly the same issue that i can't login to one of the BMC node.

[ppadmgeo1]

There is a patch, but that prevents this condition from happening. Once you are affected by this bug you need to contact support and what they will do is sp wipeclean (in diag mode)

[amans]

Hello,

I would suggest to open a support ticket,  so that support team can share the workaround.

Regards,

Aman

[sritam212]

Yes thats correct and opened a case also. the wipe clean of config is going to happen today mostly from diag mode. but after that will i need to get any patch .. so that it won't happen in future.