API call (quota-report-iter) stopped working

[bpape]

We have a script that calls the 'quota-report-iter' API every 15 minutes and has been doing so for a year. No changes have been made to the script or the filer (or the network).

On Thu, 21 May 2015 06:45:02 -0700, the filer stopped responding properly to the API requests (and I have not been able to make any API calls since that time).

I'm looking for a place to start troubleshooting this issue.

Here's a traceback from the python script that's doing the query:
# ./get_na_quota.sh
Traceback (most recent call last):
File "./NetApp-quotalist.py", line 190, in
get_quota_info()
File "./NetApp-quotalist.py", line 126, in get_quota_info
print_tree_init(s,None,0)
File "./NetApp-quotalist.py", line 86, in print_tree_init
n=nas.invoke("quota-report-iter", "max-records", req_size)
File "./NetApp-API/NaServer.py", line 542, in invoke
return self.invoke_elem(xi)
File "./NetApp-API/NaServer.py", line 490, in invoke_elem
response = connection.getresponse()
File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
response.begin()
File "/usr/lib64/python2.6/httplib.py", line 391, in begin
version
, status, reason = self._read_status()
File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
line = self.fp.readline()
File "/usr/lib64/python2.6/socket.py", line 433, in readline
data = recv(1)

[bpape]

The problem is that the mgmt lif on node n1 is not answering properly (it's immediately closing the connection).  Node n2 is working properly.

From script:
filer='1.2.3.38'
response = s.set_server_type("FILER")
response = s.set_transport_type('HTTP')

From API:
self.port = 80

na-hpc1::> net int show
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
na-hpc1
cluster_mgmt up/up 1.2.3.40/24 na-hpc1-n1 a0a true
na-hpc1-n1
clus1 up/up 169.254.32.23/16 na-hpc1-n1 e0c true
clus2 up/up 169.254.162.242/16 na-hpc1-n1 e0e true
mgmt1 up/up 1.2.3.38/24 na-hpc1-n1 e0M true
na-hpc1-n2
clus1 up/up 169.254.230.109/16 na-hpc1-n2 e0c true
clus2 up/up 169.254.96.120/16 na-hpc1-n2 e0e true
mgmt1 up/up 1.2.3.39/24 na-hpc1-n2 e0M true

Connecting to mgmt lif on node n1:
# telnet 1.2.3.38 80
Trying 1.2.3.38...
Connected to 1.2.3.38.
Escape character is '^]'.
Connection closed by foreign host.

Connecting to mgmt lif on node n2:
# telnet 1.2.3.39 80
Trying 1.2.3.39...
Connected to 1.2.3.39.
Escape character is '^]'.
[...]