SSH reset or multiple sessions

bbjholcomb · ‎2015-01-13

We are encountering SSH connection refused on our Red Hat servers, nothing consistent. Happens at different days and times, multiple Red Hat servers (about 70 of them), occurring on two different cDOT systems. It happens on different commands, vol show, vol snap show. We tried staggering the number of concurrent times this script is running, no help. We are running Red Hat 6.4 with SSH V2 5.3. We haven't had a chance to try a new version of SSH. We are running cDOT Ontap 8.2.1. We have never encountered this problem when we do SSH command on the cluster management LIF, this problem is occurring on the VSM management. We moved the VSM LIF management to a different node from the cluster management.

I found that if I retry command multiple times while sleeping between each command it works, a couple of times up to 10 times before it works. I don't believe we are encountering 64 concurrent SSH sessions or 10 per second but I can't prove it.

We are working with NetApp support, we also found a few restrictions. From NetApp:

The Data ONTAP 8.2 release family supports OpenSSH client version 5.4p1 and OpenSSH server version 5.4p1.

    Only the SSH v2 protocol is supported; SSH v1 is not supported.
    Data ONTAP supports a maximum of 64 concurrent SSH sessions per node.

    If the cluster management LIF resides on the node, it shares this limit with the node management LIF.

    If the rate of in-coming connections is higher than 10 per second, the service is temporarily disabled for 60 seconds.

manistorage · ‎2017-04-11

hi,

Time Node Severity Event

------------------- ---------------- ------------- ---------------------------

ERROR xinetd.hit.cps.limit: Number of incoming network connections exceeded the configured limit of 10 connections per second for the service ssh. This service will be stopped and restarted after 60 seconds.

While attempting to use some script to obtain information of storage using ssh protocol, the following error is reported:

xinetd.hit.cps.limit: Number of incoming network connections exceeded the configured limit of 10 connections per second for the service ssh. This service will be stopped and restarted after 60 seconds.

This message occurs when the rate of incoming network connections for a service exceeds the maximum allowed rate. The service is temporarily suspended and restarted after a configured wait time. Many such connection attempts can potentially disallow other users from logging in to the storage system, causing a Denial of Service (DOS) attack.

Engineering Team had stated that this limitation is partly in place to avoid DOS attacks, and there are currently no plans to change the limits or allow for them to be configured.

There is nothing we can do on this. Our Engineering Team is working on this. There is no Fix or workaround for this at the moment.

Regards,

Mani

RomVDP · ‎2017-10-03

While I fully agree that there is a need to protect against DDoS, is there a way to get the offending clients' IP?

Or from increase of log verbosity to get it in the syslogs?

Thanks,

Rom;)