Re: NetApp-Harvest poller issue

KZAWILINSKI · ‎2016-08-26

I have installed Graphana today using SDK 5.5. Something seems to be missing though, not sure what exactly. I get Graphana to display all graphs but only the test ones so I know it is using NetApp-Harvest to some extent. I think that the poller is failing to do something, not sure what though. I am running it against a sim 8.3.1 using password authentication for now. The poller seems to be logging in correctly as it does get the version information back and so on but it gets stuck in a loop on the first LUN poll it seems (there are no LUNs on the system - would that be an issue?)

some snitppets:

[2016-08-26 01:28:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:28:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:28:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:28:01] [WARNING] [lun] data-list update failed.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 96.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 97.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 164.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 165.
[2016-08-26 01:29:00] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:29:00] [WARNING] [lun] data-list update failed.
[2016-08-26 01:29:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:29:01] [WARNING] [lun] data-list update failed.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 96.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 97.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 96.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 97.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 96.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 97.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 96.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 97.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 96.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 97.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 96.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 97.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 96.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 97.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 164.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 165.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 164.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 165.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 164.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 165.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 164.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 165.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 164.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 165.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 164.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 165.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 164.
Use of uninitialized value in numeric lt (<) at /opt/netapp-harvest/plugin/cdot-volume line 165.
[2016-08-26 01:30:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:30:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:31:00] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:31:00] [WARNING] [lun] data-list update failed.
[2016-08-26 01:32:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:32:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:33:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:33:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:34:00] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:34:00] [WARNING] [lun] data-list update failed.
[2016-08-26 01:35:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:35:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:36:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:36:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:37:00] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:37:00] [WARNING] [lun] data-list update failed.
[2016-08-26 01:38:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:38:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:39:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:39:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:40:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:40:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:41:01] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:41:01] [WARNING] [lun] data-list update failed.
[2016-08-26 01:42:00] [WARNING] [lun] update of data cache failed with reason: For lun object, no instances were found to match the given query.
[2016-08-26 01:42:00] [WARNING] [lun] data-list update failed.

dymusi@ubuntu:~$ /opt/netapp-harvest/netapp-worker -poller zawik-cl
[2016-08-26 01:43:55] [NORMAL ] WORKER STARTED [Version: 1.2.2] [Conf: netapp-harvest.conf] [Poller: zawik-cl]
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:43:55] [WARNING] Started in foreground mode; messages to STDERR are redirected to the logfile and are not visible on the console.
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:43:55] [NORMAL ] [main] Poller will monitor a [FILER] at [192.168.147.30:443]
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:43:55] [NORMAL ] [main] Poller will use [password] authentication with username [netapp-harvest] and password [**********]
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:05] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:15] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:15] [NORMAL ] [main] Collection of system info from [192.168.147.30] running [NetApp Release 8.3.1] successful.
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:15] [NORMAL ] [main] Using best-fit collection template: [cdot-8.3.0.conf]
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:15] [NORMAL ] [main] Using graphite_root [netapp.perf.lab.zawik-cl]
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:15] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.lab.zawik-cl]
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:15] [NORMAL ] [smb2:vserver] Collection of object not enabled; skipping
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:15] [NORMAL ] [smb2:node] Collection of object not enabled; skipping
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor
[2016-08-26 01:44:15] [NORMAL ] [main] Startup complete. Polling for new data every [60] seconds.
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor

madden · ‎2016-08-29

Hi @KZAWILINSKI

Errors like these:

[2016-08-26 01:44:15] [NORMAL ] [main] Using best-fit collection template: [cdot-8.3.0.conf]
print() on closed filehandle LOGFILE at /opt/netapp-harvest/netapp-worker line 2754.
Write to logfile failed: Bad file descriptor

Make me think your disk is full. Can you check?

For luns I think there may be an issue if you have LUNs defined but none of them are online/mapped (like might happen in a lab, or on a DR site with lots of mirrored LUNs but none that are online/mapped). I put in a fix (in v1.3 that will release soon on Toolchest) but you can patch v1.2.2 yourself:

1) Open /opt/netapp-harvest/netapp-worker in a text editor

2) Replace lines 1141 -1151 with:

			elsif ($obj eq 'lun') ## Workaround bug 764178 and 789091 whereby LUN names can be truncated or not returned at all via perf APIs
		{
			$in = NaElement->new('lun-get-iter');
			$in->child_add_string("max-records",$batch_size);
			#Get only specific attributes we use
			my $desiredAttributes = new NaElement('desired-attributes');
			$in->child_add($desiredAttributes);
			my $lunInfo = new NaElement('lun-info');
			$desiredAttributes->child_add($lunInfo);
			$lunInfo->child_add_string('path','');
			$lunInfo->child_add_string('uuid','');
			#Get only online and mapped LUNs (only these have CM stats instances)
			my $query = new NaElement('query');
			$in->child_add($query);
			my $queryLunInfo = new NaElement('lun-info');
			$query->child_add($queryLunInfo);
			$queryLunInfo->child_add_string('online','true');
			$queryLunInfo->child_add_string('mapped','true');
		}

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!