Exceeded the maximum allowed ssh session in ontap 8.0.2P1

jerome_barrelet · ‎2011-09-06

Hi,

We use a Linux Redhat to manage our filers (FAS 3240, FAS 3160)

Since the upgrade to Ontap 8.0.2P1 we have every week this message when we want to connect in SSH to our filer.

In the Linux Console : ssh_exchange_identification: Connection closed by remote host

In the Putty :

If I go to the system in RLM I see this message in console of the Filer:

gdc00181> Mon AugMon Aug 29 12:50:22 CEST [openssh.session.limit:in 29 12:fo]: Exceeded the maximum allowed ssh sessions: 24.

50:22 CEST [openssh.session.limit:info]: Exceeded the maximum allowed ssh sessions: 24.

The only workaroud found is to reboot the filer

I want to know if somebody have noticed the same problems

Thanks in advance

Jerome

novemberico · ‎2011-09-21

Hello,

You're not the only one - ssh is the only way we manage our filers, so this puts effectively puts us down until netapp can fix it. This happened after a recent upgrade from 6070->6280s to 8.0.2.p2 (7mode)

I'll reply again if we fix it without a reboot.

scottgelb · ‎2011-09-21

Looks like an 8.0.2P3 fix for this.

scottgelb · ‎2011-09-21

which isn't out yet but I keep checking every day since it will be worth the ssh fix.

novemberico · ‎2011-09-21

Do you know where I can view the details of this bug?

scottgelb · ‎2011-09-21

I had a customer hit it but was an internal burt I couldn't view. Support will know though on your existing case.

Typos Sent on Blackberry Wireless

novemberico · ‎2011-09-21

So, no fix or workaround - apparently OnTap is completely opaque (even to the diaguser systemshell on FreeBSD); killing the sshd process inside of ontap (which is separate from the one running under inetd in systemshell) is impossible. We just went through a hellishly obstructive (paperwork-wise) upgrade of our main netapp filers, and are now going to have to do it again in a few weeks when 8.0.2p3 comes out. This is extremely bad for us, because ssh is the only way we manage the filers. All of our scripts will have to be rewritten temporarily and insecure protocols turned on for us to do anything with these filers.

To anyone else that experiences this issue and finds this thread - don't waste your time with the diaguser for the entire day trying to find a way to fix it like I did! (Or just wait for 8x to become stable if you haven't upgraded yet.)

jerome_barrelet · ‎2011-09-21

we open a case for that and we had an answer of the support:

it's a bug, here is the burt :

http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=484047

http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=514876

it's corrected in version : ONTAP 8.0.2P2D4

I will try this version and let you know.

novemberico · ‎2011-09-21

Thanks for those links - support couldn't provide them. we experienced activity similar to the one in 48407:

<foobar ~> sudo tcpdump -i eth1 'host toaster1'

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes

00:17:50.276938 IP foobar.40047 > toaster1.ssh: Flags [S] [...]

00:17:50.277795 IP toaster1.ssh > foobar.40047: Flags [S.] [...]

00:17:50.277813 IP foobar.40047 > toaster1.ssh: Flags [.] [...]

00:17:50.278732 IP toaster1.ssh > foobar.40047: Flags [F.] [...]

00:17:50.279027 IP foobar.40047 > toaster1.ssh: Flags [F.] [...]

00:17:50.279842 IP toaster1.ssh > foobar.40047: Flags [.] [...]

Since this is a minor-rev upgrade, we would be able to do it without taking down the cluster. Let us know how it goes and if you continue seeing symptoms;

scottgelb · ‎2011-09-22

This is different than the bug where all 24 ssh sessions are locked and can’t be deleted…or might be in this fix but I don’t see it in the fix list.

novemberico · ‎2011-09-22

Ah, I thought they were one in the same, since we're seeing both the [openssh.session.limit:info] error in the RLM (that's how I found this thread) and the premature FIN (as per the packet trace above.) I hope they're not being tracked and fixed as two separate bugs - that would really be a kick in the nuts if we had to upgrade to 8.0.2p4 a month or so after upgrading to 8.0.2p3.

jerome_barrelet · ‎2011-09-30

10 days ago we upgraded to Ontap ONTAP 8.0.2P2D4

Since 10 days we didn' t noticed this problem again.

For us it's solved.

Jerome

mjschneider · ‎2011-10-03

Found this thread while searching for more info on bug id 514876.

I dont know if the problem your having is the same we've had for over a year now, but I was able to fix this by doing the following (pasted from our internal wiki):

I used SecureCRT for this fix, putty has similar options and should work but was not tested.

Configure SSH session to filer to run a 'Remote Command' unpon login. I simply used '?'.
Connect to filer, enter credentials and the session will fail/close.
Remove the 'remote command' settings from your session options.
Connect to filer and it should now work.

Hope this helps some of you. I never saw messages regarding the 24 sessions, but we arn't on 8.0.2x yet. But i've had this particular issue since 7.3.3, currently on 8.0.1P5 (also have seen this on 4 separate v3140s. Never experienced on our v3240.)

sbrooks15 · ‎2011-10-04

mjschneider --> Thank you, that worked like a charm, and I used putty!

mjschneider · ‎2011-10-04

Great! Glad that worked... The first time i tried that the "Remote Command" i used contained many expletives

itcambridge · ‎2012-02-15

This little workaround saved me a drive into the datacenter. Thanks!!