Data Infrastructure Management Software Discussions

Highlighted

Too Many HTML Connections Message

When trying to access a controller through System Manager, I am getting the message "Too many HTML connections".  Has anybody else seen this?  Is there a way to determine who or what connections are connected to the filer through http or https?

61 REPLIES 61
Highlighted

Re: Too Many HTML Connections Message

I have also seen this issue on both FAS3140 and FAS3210. there are some soultions. 

1. increase the max connection limit from the default 512 to 1023 by running the following commnad: options httpd.admin.max_connections 1023

    If you want to see the http connections type the following: httpstat -a (if the Peak connection count is what is set on your httpd.admin.max_connections the only way to clear the connections is to do a take over and give back on the filer head).

2. last resort is to do a take over and a give back wich will clear the peak connection counter.  watch the peak connection counter using httpstat -a over the next few days to make sure that the connection count doesn't grow to the max of the httpd.admin.max_connections limit.

Doing the take over and give back worked for me my peak connection counter is at 19 and hasn't gone above that for the past week.

Hope this helps

Highlighted

Re: Too Many HTML Connections Message

We are seeing the same error message when attempting to connect with Oncommand (Version 2.0) to a Filer running Ontap 8.02P3:

"The maximum number of Administrative HTTP connections specified by httpd.admin.max_connections has been exceeded"

We increased the httpd.admin.max_connections to 1023 from 512 on advice from Netapp support and we now get an "Uknown" error when connecting from Oncommand.

When running an httpstat -s you can see the open HTTP connections on the Filer increasing over time until they reach < 900 and then the open counter resets and begins to increment again:

TOASTER1> date

Mon Feb 13 08:39:42 GMT 2012

TOASTER1> httpstat -s

          Connection

       Open       Peak      Waits

        814       1037          0

TOASTER1> date

Mon Feb 13 08:47:21 GMT 2012

TOASTER1> httpstat -s

          Connection

       Open       Peak      Waits

        928       1037          0

when the Open connections reach the peak and reset we also see an "NFS Down" error in DFM, although NFS is actually running on the Filer.

An "httpstat -a" shows the Errors (44450) and BadRequests (4234) counters incrementing also.

When the Filer is removed from DFM the "httpstat -a" open counter stops incrementing. We have also tried a failover and giveback of the head and the Filer could communicate with Oncommand and DFM for a short time but then the "NFS down errors" re-appeared and the httpstat open connections began incrementing again.

I've sent Netapp some packet captures from the Filer with the issue and a healthy Filer filtered on the DFM IP address as that is what appears to be flooding the Filer with invalid HTTP requests.

Netapp are analysing the packet captures and have open a Bug for this as it only seems to of started since we upgraded to 8.02P3.

Highlighted

Re: Too Many HTML Connections Message

Yep, I get the same exact thing.  I'm running 8.1RC2 and still an issue.  Whenever we want to use system manager or even Netapp Insight we shut down OnCommand for a day or so and then it starts working.  Wish this would get fixed.

Philip

Highlighted

Re: Too Many HTML Connections Message

Hi Phil,

Netapp have a Bug (565589) for this. What I find odd is we have multiple Filers with similar configuration and versions but we don't see it on any of those.

Regards

Martin

Highlighted

Re: Too Many HTML Connections Message

Do you have a URL for that bug?  565589 doesn't appear to be a valid bug ID where I looked.  Is it internal only?

Philip

Highlighted

Re: Too Many HTML Connections Message

Hi Phil,

I just reconfirmed that bug number from the Netapp ticket and here's the URL for the bug, it doesn't show any details yet.

http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=565589

I'm not sure if this is because it's internal only or just recently added.  I've put it on my bug watch list so I will see any public updates posted.

Martin

Highlighted

Re: Too Many HTML Connections Message

Anyone gt more information about this issue?

I have it on one filer and got big problems with protection manager then the http connections hits 1023.

Highlighted

Re: Too Many HTML Connections Message

Hi,

I'm still providing Netapp support with details of this via a Support case.  From running the DFM apitest tool suggested by Netapp it looks as though the Filer intermittently fails to respond to HTTP requests or takes excessively long.

From looking at packet captures from the Filer, DFM polls the Filer forms an TCP connection and then the Filer fails to close the connection correctly which I assume is the reason that we see the excessive number of HTTP connections on the Filer?

Highlighted

Re: Too Many HTML Connections Message

We have the problem on a secondary filer and the system can run fine for some time after a reboot.

But if run httpstat -s 1 and check the stats we can see when the filer starts getting the problem with close_wait connections we are filling the total of 1023 connections in a couple of minutes.

This problem is a big issue for us, since when the connections are full our DFM stops working and we got problems with the backups.

Got a case for this late 2011 and the solution from Netapp was to restart the filer, but that's not a solution for me, and the problems has come back now again.

If you get any got information please sent it to this forum

Highlighted

Re: Too Many HTML Connections Message

Same here. It's definitly not a solution for us too as we have to reboot our secondary once a week. I have now also opened a case to increase the pressure...

Highlighted

Re: Too Many HTML Connections Message

Also had this issue since october 2011, SMVI stops working, system manager wont work, high connections shown in httpstat.

Had numerous support tickets open with various "workarounds" and "fixed in latest version of ontap" none of which have managed to stop the issue for more than a week (usually it will die after a day, an hour or so after the netapp case has been archived!!).

Been fobbed off with 526771 and 521513 during that time also.

Think we need to keep the pressure on.

Highlighted

Re: Too Many HTML Connections Message

Turn off AUTO-DISCOVERY on the ONCOMMAND DFM server.  That has shown success for us so far (fingers crossed).

Philip

Highlighted

Re: Too Many HTML Connections Message

One of my Controllers has just had this problem - I upped the max connections to 1000 and it just went up to that amount but I still could not login via Systems Manager. I have raised a case with NetApp but has anyone found a work-a-round ... some way of manually clearing down connections? I have also disabled auto discovery on the DFM server.

I am using ONTAP 8.0.2P4 7-Mode

Highlighted

Re: Too Many HTML Connections Message

Unfortunately I haven't found one and I've spent a long while on this problem.  My previous recommendation about AUTO-DISCOVERY didn't seem to work, we are now trying to use HTTPS as the transport instead of HTTP (stay tuned regarding that).  I opened a case as well, and their response was to reboot the filer (yeah right).

Philip

Highlighted

Re: Too Many HTML Connections Message

Thanks for the quick reply ... I'll also share any updates.

Check out the KB!
Knowledge Base
All Community Forums