Harvest is giving error of API request rejected => System busy: 7 requests on table

[wdiane]

Why is this error occurring?

{"level":"info","Poller":"netapp-1","collector":"ZapiPerf:NFSv3Node","error":"API request rejected => System busy: 7 requests on table \"perf_object_get_instances\" have been pending for 1 seconds. The last completed call took 25 seconds. errNum=\"13001\" statusCode=\"0\"","task":"data","caller":"collector/collector.go:365","time":"2023-09-12T16:23:41-07:00","message":"API rejected, entering standby mode"}
{"level":"info","Poller":"netapp-1","collector":"ZapiPerf:NFSv3","error":"API request rejected => System busy: 7 requests on table \"perf_object_get_instances\" have been pending for 1 seconds. The last completed call took 25 seconds. errNum=\"13001\" statusCode=\"0\"","task":"data","caller":"collector/collector.go:365","time":"2023-09-12T16:23:41-07:00","message":"API rejected, entering standby mode"

[dawnr]

That error is coming from the cluster. Make sure the cluster is on a version where bug 1469076 is fixed. If it is, you will need to look at what's going on with CMD.

[thomas99]

Engineering reviewed the case and that bug isn't the problem in this instance.  Is there something else other than sync-coring CMD we can do to figure out what is going on here?

[dawnr · Accepted solution]

The only thing we (in OnCommand) would recommend would be to try restarting the CMD process on all nodes, and to make sure you're not running in a mixed version or incomplete cluster join.  Aside from that, if this is the case I think it is, Diane is following up with engineering to evaluate what bug is affecting CMD if it's not that specific one.

But in general, there's nothing Harvest or any of the OnCommand products can do when the cluster is responding with system busy. That must be resolved on the cluster first before doing any further troubleshooting with APIs. At worst, it's a complete blocker, at best, it makes responses intermittent.