Too Many HTML Connections Message

[waynes]

When trying to access a controller through System Manager, I am getting the message "Too many HTML connections".  Has anybody else seen this?  Is there a way to determine who or what connections are connected to the filer through http or https?

[DSCHREIFELS]

I have also seen this issue on both FAS3140 and FAS3210. there are some soultions. 

1. increase the max connection limit from the default 512 to 1023 by running the following commnad: options httpd.admin.max_connections 1023

    If you want to see the http connections type the following: httpstat -a (if the Peak connection count is what is set on your httpd.admin.max_connections the only way to clear the connections is to do a take over and give back on the filer head).

2. last resort is to do a take over and a give back wich will clear the peak connection counter.  watch the peak connection counter using httpstat -a over the next few days to make sure that the connection count doesn't grow to the max of the httpd.admin.max_connections limit.

Doing the take over and give back worked for me my peak connection counter is at 19 and hasn't gone above that for the past week.

Hope this helps

[MARTINLEGGATT]

We are seeing the same error message when attempting to connect with Oncommand (Version 2.0) to a Filer running Ontap 8.02P3:

"The maximum number of Administrative HTTP connections specified by httpd.admin.max_connections has been exceeded"

We increased the httpd.admin.max_connections to 1023 from 512 on advice from Netapp support and we now get an "Uknown" error when connecting from Oncommand.

When running an httpstat -s you can see the open HTTP connections on the Filer increasing over time until they reach < 900 and then the open counter resets and begins to increment again:

TOASTER1> date

Mon Feb 13 08:39:42 GMT 2012

TOASTER1> httpstat -s

          Connection

       Open       Peak      Waits

        814       1037          0

TOASTER1> date

Mon Feb 13 08:47:21 GMT 2012

TOASTER1> httpstat -s

          Connection

       Open       Peak      Waits

        928       1037          0

when the Open connections reach the peak and reset we also see an "NFS Down" error in DFM, although NFS is actually running on the Filer.

An "httpstat -a" shows the Errors (44450) and BadRequests (4234) counters incrementing also.

When the Filer is removed from DFM the "httpstat -a" open counter stops incrementing. We have also tried a failover and giveback of the head and the Filer could communicate with Oncommand and DFM for a short time but then the "NFS down errors" re-appeared and the httpstat open connections began incrementing again.

I've sent Netapp some packet captures from the Filer with the issue and a healthy Filer filtered on the DFM IP address as that is what appears to be flooding the Filer with invalid HTTP requests.

Netapp are analysing the packet captures and have open a Bug for this as it only seems to of started since we upgraded to 8.02P3.

[philiparnason]

Yep, I get the same exact thing.  I'm running 8.1RC2 and still an issue.  Whenever we want to use system manager or even Netapp Insight we shut down OnCommand for a day or so and then it starts working.  Wish this would get fixed.

Philip

[MARTINLEGGATT]

Hi Phil,

Netapp have a Bug (565589) for this. What I find odd is we have multiple Filers with similar configuration and versions but we don't see it on any of those.

Regards

Martin

[philiparnason]

Do you have a URL for that bug?  565589 doesn't appear to be a valid bug ID where I looked.  Is it internal only?

Philip

[MARTINLEGGATT]

Hi Phil,

I just reconfirmed that bug number from the Netapp ticket and here's the URL for the bug, it doesn't show any details yet.

http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=565589

I'm not sure if this is because it's internal only or just recently added.  I've put it on my bug watch list so I will see any public updates posted.

Martin

[thommy_lan]

Anyone gt more information about this issue?

I have it on one filer and got big problems with protection manager then the http connections hits 1023.

[MARTINLEGGATT]

Hi,

I'm still providing Netapp support with details of this via a Support case.  From running the DFM apitest tool suggested by Netapp it looks as though the Filer intermittently fails to respond to HTTP requests or takes excessively long.

From looking at packet captures from the Filer, DFM polls the Filer forms an TCP connection and then the Filer fails to close the connection correctly which I assume is the reason that we see the excessive number of HTTP connections on the Filer?

[thommy_lan]

We have the problem on a secondary filer and the system can run fine for some time after a reboot.

But if run httpstat -s 1 and check the stats we can see when the filer starts getting the problem with close_wait connections we are filling the total of 1023 connections in a couple of minutes.

This problem is a big issue for us, since when the connections are full our DFM stops working and we got problems with the backups.

Got a case for this late 2011 and the solution from Netapp was to restart the filer, but that's not a solution for me, and the problems has come back now again.

If you get any got information please sent it to this forum

[abuchmann]

Same here. It's definitly not a solution for us too as we have to reboot our secondary once a week. I have now also opened a case to increase the pressure...

[chris_hague]

Also had this issue since october 2011, SMVI stops working, system manager wont work, high connections shown in httpstat.

Had numerous support tickets open with various "workarounds" and "fixed in latest version of ontap" none of which have managed to stop the issue for more than a week (usually it will die after a day, an hour or so after the netapp case has been archived!!).

Been fobbed off with 526771 and 521513 during that time also.

Think we need to keep the pressure on.

[philiparnason]

Turn off AUTO-DISCOVERY on the ONCOMMAND DFM server.  That has shown success for us so far (fingers crossed).

Philip

[lee_harrison]

One of my Controllers has just had this problem - I upped the max connections to 1000 and it just went up to that amount but I still could not login via Systems Manager. I have raised a case with NetApp but has anyone found a work-a-round ... some way of manually clearing down connections? I have also disabled auto discovery on the DFM server.

I am using ONTAP 8.0.2P4 7-Mode

[philiparnason]

Unfortunately I haven't found one and I've spent a long while on this problem.  My previous recommendation about AUTO-DISCOVERY didn't seem to work, we are now trying to use HTTPS as the transport instead of HTTP (stay tuned regarding that).  I opened a case as well, and their response was to reboot the filer (yeah right).

Philip

[lee_harrison]

Thanks for the quick reply ... I'll also share any updates.

[chris_hague]

We've always used HTTPS and have the same issues so dont hold much hope on that one..

Also, when we perform a mount \ restore from netapp tab on a VM it works for NFS, but FC takes out whichever host you are mapping it to and the vmware management services have to be restarted to get it back into the VC.

Still passing various bits of information to netapp and awaiting some kind of resolution....

[lee_harrison]

When I came in the next morning the connections had dropped to 5 and I could then access via Systems Manager. I have updated NetApp case but not heard anything yet? I smell a bug-et?

[koglmeier]

Ran into this problem today and found this thread during search.  Filer was maxed out at 512 connections but rather than increasing the max connections, I wanted to try to find a way to kill all those existing connections.  It may have just been dumb luck, but I may have stumbled on to something.  I first modified the httpd.admin.max_connections option and set it down to 1 (which is the minimum).  Then I adjusted the httpd.timeout options down to 30 seconds (again the minimum value).  After a minute or so, I set them back to the default values of 512 connections and 300 seconds.  The open sessions dropped from 512 to 0 and has been bouncing around but has stayed below 10.  Feedback is welcome, let me know if it works for anyone.

Running 8.0.2 P3 7-Mode on FAS3170

[abuchmann]

The issue appeared again in our environment and sadly, your suggested workaround did not help.

Last time i had the error it also solved itself after a while. Maybe it was just a lucky coincidence? 🙂

Running 7.3.5.1 P5 7-Mode on FAS3160

[MARTINLEGGATT]

I am not sure why but DFM began monitoring again and when I check "httpstat -s" the connections aren't going over 10.

I am sure this issue will return though.

Martin