Filer repeatedly attempting admin login to switch

[EmconIT]

Hello, I am working with an ONTAP 8.3 system that is attempting to login as admin to a Cisco Nexus 5596 switch that is listed as a management-network switch in cshm config.  The filer's IP is making 3 attempts every 15 minutes via the node management LIF,  which is on e0i and e0m.  Switch logs show these messages repeatedly:

%AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user admin from <FILER_MGMT_IP>  - sshd[17590]
%DAEMON-3-SYSTEM_MSG: error: PAM: Authentication failure for admin from <FILER_MGMT_IP> - sshd[17587]

The CSHM config and device discovery show 4 switches, 2 cluster and 2 management.  SNMP file version is SNMPv2c, and as far as I can tell SNMPv3 is not enabled although for some reason I am seeing activity for snmpv3 in vmstat_m.  I don't know if that observation is relevant or not.

I conducted a packet trace and sure enough there is an SSH transaction with the filer IP as client, but I can find no correllation to that in the ASUP logs anywhere.  There is nothing in messages, mgwd, auditlog, notifyd, or any other timestamped log file.  No reference to an ssh session, or the switch IP, nothing.

I have scoured the ONTAP documentation, knowledgebase and google for any kind of breadcrumb to follow, but I still don't even have a hint at what could be causing this.  I am completely stumped!

Unfortunately due to security policy I am not able to provide any logs, and I am aware that I may not be able to get a solid answer but any insights or suggestions would be greatly appreciated.

Thanks

Switch info:

Model: Cisco Nexus 5596 (NX5596)

Software: NX-OS 5.2(1)N1(1a)

[AlexDawson]

Hi there!

These logins are being generated due to chsm log collection being enabled for the switch.

Disabling log collection should stop these login attempts - https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-cm-cmpr-940%2FTOC__system__cluster-switch__log.html&cp=4_3_25_0_6

I agree that the documentation doesn't make it clear it will use ssh to attempt to collect the logs. I had to use internal documentation to confirm this. I have submitted a comment to our documentation team about this.

Hope this helps!

[EmconIT]

Thanks Alex, I will give this a shot.

Just one question, though.  I don't see the system cluster-switch log commands in the 8.3 man pages, will it still work?  Do I need to change privilege level?

[AlexDawson]

Oops, sorry, I missed the 8.3 qualifier.

It looks like it is doing this to check the RCF version, which is displayed in the ssh login banner. 

It can apparently be disabled by running "system cluster-switch modify -device <device name> -is-monitoring-enabled-admin false"

[EmconIT]

Hi Alex,

We already tried that command but only on one of the affected switches, I will run it again for the other and see if that helps.

Also, something I am wondering about from the packet trace:

Client: Protocol (SSH-2.0-OpenSSH_5.4p1 FreeBSD-20100308)

Could this possibly be some activity in systemshell that is not visible from ONTAP?

[AlexDawson · Accepted solution]

You would need to run it for each switch.

If you disable cdp on the controller, and then use "system cluster-switch delete", it should stop trying to log in at all. 

Yes, it uses the openSSH client to view the banner for these switches - it doesn't actually use any passwords to attempt to connect - literally just opens a session, reads the banner and closes it again. Yes, it isn't visible to ONTAP and it isn't terribly well documented (ie, I had to read the source code to find this out...)