Discovery failure of netapp cluster (ontap 9.5P3) on arxscan tool

AN2595128 · ‎2020-06-23

We are using Arxscan Arxview tool in our environment for monitoring and reporting.

We have recently noticed that the discovery of one of the netapp cluster MICMO1xxxxx is failing with these errors on Arxview -

2020-06-16 07:49:39 56433 [debug] Querying instance-info for obj: processor
2020-06-16 07:49:40 56433 [debug] instance-info: qtree
2020-06-16 07:49:40 56433 [debug] Querying instance-info for obj: qtree
2020-06-16 07:50:31 56433 [debug] Query for instance-info for obj: qtree failed: perf-object-instance-list-info-iter failed. Reason: Input is not proper UTF-8, indicate encoding ! Bytes: 0xE9 0x5F 0x30 0x31
2020-06-16 07:50:31 56433 [warning] RETRYING command for: instance-info for obj: qtree (1):
2020-06-16 07:51:23 56433 [debug] Query for instance-info for obj: qtree failed: perf-object-instance-list-info-iter failed. Reason: Input is not proper UTF-8, indicate encoding ! Bytes: 0xE9 0x5F 0x30 0x31
2020-06-16 07:51:23 56433 [warning] RETRYING command for: instance-info for obj: qtree (2):
2020-06-16 07:52:15 56433 [debug] Query for instance-info for obj: qtree failed: perf-object-instance-list-info-iter failed. Reason: Input is not proper UTF-8, indicate encoding ! Bytes: 0xE9 0x5F 0x30 0x31
2020-06-16 07:52:15 56433 [warning] RETRYING command for: instance-info for obj: qtree (3):
2020-06-16 07:53:09 56433 [debug] Query for instance-info for obj: qtree failed: perf-object-instance-list-info-iter failed. Reason: Input is not proper UTF-8, indicate encoding ! Bytes: 0xE9 0x5F 0x30 0x31
2020-06-16 07:53:09 56433 [warning] ERROR initializing API: Error querying counter: perf-object-instance-list-info-iter failed. Reason: Input is not proper UTF-8, indicate encoding ! Bytes: 0xE9 0x5F 0x30 0x31
2020-06-16 07:53:09 56433 [debug] Sleeping...
2020-06-16 07:53:09 56433 [debug] Sleeping...

require your support to find out the root cause and a fix of this issue

Ontapforrum · ‎2020-06-24

Hi,

To be honest I would suggest reaching out to 'Arxscan' for assistance. NetApp has a free monitoring tool Active-IQ Monitoring GUI tool and we can provide help around that. I looked up for this tool, and it states NetApp as partner but there is not much information around it. In any case, it's worth reaching out to Arxscan for assistance.

In general number of factors could come into picture:
1) Is the API user created?
2) Is the API user given "ONTAPI" rights?
3) Is API access to Cluster allowed through firewall?

Thanks!

AN2595128 · ‎2020-06-24

hi,

this was working fine till last week, so I believe these things are already in place

1) Is the API user created? Yes
2) Is the API user given "ONTAPI" rights? Yes
3) Is API access to Cluster allowed through firewall? Yes

Ontapforrum · ‎2020-06-24

Only hint I see in that error is 'ERROR initializing API-Reason: Input is not proper UTF-8". Most likely these queries were running against a volume which has a language NOT set to 'UTF-8', and as far as I understand ONTAP API doesn't support extended ASCII characters. If my understanding is correct wrt to your issue, may be it's worth checking the 'volume's language' or something to do with non-utf8 format.

AN2595128 · ‎2020-06-26

this is the response I got from arxscan support

AN2595128 · ‎2020-07-08

Hi All,

please let us know in case you have any further suggestion on this

Ontapforrum · ‎2020-07-08

Just had a look at the email screenshot from Arxscan: Sounds confusing and contradicting by look of it ?

It says - Asking NetApp API engineering to investigate why "...not sent as 'UTF-8' and just above that it says '...in any cases where the record at the end of a page happens to have non-ascii characters, that response breaks XML parsing. They answered themselves. In any case, don't think it's to do with "Application", it's the 'data' in non-ascii format. Any names with non-ASCII characters on volume that has non-utf8 format set, will reject it. In my previous email, I had requested to verify the 'vol' language settings:

Could you give this output:
::> volume show -fields language

AN2595128 · ‎2020-07-08

MICMO1P1NAP01::> volume show -fields language
vserver volume language
-------- -------- --------
ADMINSVM svm_root en_US
ADMINSVM test -
ATKFS401 ATKFS401_root C.UTF-8
ATKFS401 CIFS_FS_APPS_AATKP101 C
ATKFS401 CIFS_FS_SHARE_AATKP111 C
ATKFS401 CIFS_FS_SITE_AATKP110 C
ATKFS401 CIFS_FS_USER01_AATKP105 C
AULFS401 AULFS401_root C.UTF-8
AULFS401 CIFS_FS_APPS_AAULP101 C
AULFS401 CIFS_FS_SHARE_AAULP111 C
AULFS401 CIFS_FS_SITE_AAULP110 C
AULFS401 CIFS_FS_USER01_AAULP105 C
BLSFS401 BLSFS401_root C.UTF-8
BLSFS401 CIFS_FS_APPS_BBLSP101 C
BLSFS401 CIFS_FS_SHARE_BBLSP111 C
BLSFS401 CIFS_FS_SITE_BBLSP110 C
BLSFS401 CIFS_FS_USER01_BBLSP105 C
BREFS401 BREFS401_root C.UTF-8
BREFS401 CIFS_FS_APPS_ABREP101 C
BREFS401 CIFS_FS_SHARE_ABREP111 C
BREFS401 CIFS_FS_SITE_ABREP110 C
BREFS401 CIFS_FS_USER01_ABREP105 C
BZTFS401 BZTFS401_root C.UTF-8
BZTFS401 CIFS_FS_APPS_ABZT101 C
BZTFS401 CIFS_FS_SHARE_ABZT111 C
BZTFS401 CIFS_FS_SITE_ABZT110 C
BZTFS401 CIFS_FS_USER01_ABZT105 C
CARFS401 CARFS401_root C.UTF-8
CARFS401 CIFS_FS_APPS_ACAR101 C
CARFS401 CIFS_FS_SHARE_ACAR111 C
CARFS401 CIFS_FS_SITE_ACAR110 C
CARFS401 CIFS_FS_USER01_ACAR105 C
CEKFS401 CIFS_FS_APPS_ACEKP101 C
CEKFS401 CIFS_FS_SHARE_ACEKP111 C
CEKFS401 CIFS_FS_SITE_ACEKP110 C
CEKFS401 CIFS_FS_USER01_ACEKP105 C
CEKFS401 CKEFS401_root C.UTF-8
CFAFS401 CFAFS401_root C.UTF-8
CFAFS401 CIFS_FS_APPS_ACFAP101 C
CFAFS401 CIFS_FS_SHARE_ACFAP111 C
CFAFS401 CIFS_FS_SITE_ACFAP110 C
CFAFS401 CIFS_FS_USER01_ACFAP105 C
CMLFS401 CIFS_FS_APPS_ACML101 C
CMLFS401 CIFS_FS_SHARE_ACML111 C
CMLFS401 CIFS_FS_SITE_ACML110 C
CMLFS401 CIFS_FS_USER01_ACML105 C
CMLFS401 CMLFS401_root C.UTF-8
COMFS401 CIFS_FS_APPS_BCOM101 C
COMFS401 CIFS_FS_SHARE_BCOM111 C
COMFS401 CIFS_FS_SITE_BCOM110 C
COMFS401 CIFS_FS_USER01_BCOM105 C
COMFS401 COMFS401_root C.UTF-8
CTXFS401 CIFS_FS_APPS_BCTX101 C
CTXFS401 CIFS_FS_SHARE_BCTX111 C
CTXFS401 CIFS_FS_SITE_BCTX110 C
CTXFS401 CIFS_FS_USER01_BCTX105 C
CTXFS401 CTXFS401_root C.UTF-8
DCRFS401 CIFS_FS_APPS_ADCRP101 C
DCRFS401 CIFS_FS_SHARE_ADCRP111 C
DCRFS401 CIFS_FS_SITE_ADCRP110 C
DCRFS401 CIFS_FS_USER01_ADCRP105 C
DCRFS401 DCRFS401_root C.UTF-8
EMAFS401 CIFS_FS_APPS_AEMAP101 C
EMAFS401 CIFS_FS_SHARE_AEMAP111 C
EMAFS401 CIFS_FS_SITE_AEMAP110 C
EMAFS401 CIFS_FS_USER01_AEMAP105 C
EMAFS401 EMAFS401_root C.UTF-8
FGIFS401 CIFS_FS_APPS_AFGIP101 C
FGIFS401 CIFS_FS_SHARE_AFGIP111 C
FGIFS401 CIFS_FS_SITE_AFGIP110 C
FGIFS401 CIFS_FS_USER01_AFGIP105 C
FGIFS401 FGIFS401_root C.UTF-8
GRVFS401 CIFS_FS_APPS_BGRV101 C
GRVFS401 CIFS_FS_SHARE_BGRV111 C
GRVFS401 CIFS_FS_SITE_BGRV110 C
GRVFS401 CIFS_FS_USER01_BGRV105 C
GRVFS401 GRVFS401_root C.UTF-8
KIVFS401 CIFS_FS_APPS_AKIVP101 C
KIVFS401 CIFS_FS_SHARE_AKIVP111 C
KIVFS401 CIFS_FS_SITE_AKIVP110 C
KIVFS401 CIFS_FS_USER01_AKIVP105 C
KIVFS401 KIVFS401_root C.UTF-8
LADFS401 CIFS_FS_APPS_BLAD101 C
LADFS401 CIFS_FS_SHARE_BLAD111 C
LADFS401 CIFS_FS_SITE_BLAD110 C
LADFS401 CIFS_FS_USER01_BLAD105 C
LADFS401 LADFS401_root C.UTF-8
LSBFS401 CIFS_FS_APPS_BLSBP101 C
LSBFS401 CIFS_FS_SHARE_BLSBP111 C
LSBFS401 CIFS_FS_SITE_BLSBP110 C
LSBFS401 CIFS_FS_USER01_BLSBP105 C
LSBFS401 LSBFS401_root C.UTF-8
LYOFS401 CIFS_FS_SHARE_ALYOP111 C
LYOFS401 CIFS_FS_SITE_ALYOP110 C
LYOFS401 CIFS_FS_USER01_ALYOP105 C
LYOFS401 LYOFS401_root C.UTF-8
MIAFS401_NEW CIFS_FS_APPS_AMIA101 C
MIAFS401_NEW CIFS_FS_SHARE_AMIA111 C
MIAFS401_NEW CIFS_FS_SITE_AMIA110 C
MIAFS401_NEW CIFS_FS_USER01_AMIA105 C
MIAFS401_NEW MIAFS401_root C.UTF-8
MIAFS403_DR CIFS_FS_APPS_BMIA301 C
MIAFS403_DR CIFS_FS_BALTICS_BMIA314 C
MIAFS403_DR CIFS_FS_EUR_BMIA312 C
MIAFS403_DR CIFS_FS_FR_BMIA313 C
MIAFS403_DR CIFS_FS_IBM_BMIA314 C
MIAFS403_DR CIFS_FS_SHARE_BMIA311 C
MIAFS403_DR CIFS_FS_SITE_BMIA310 C
MIAFS403_DR CIFS_FS_USER01_BMIA305 C
MIAFS403_DR CIFS_FS_USMT01_BMIA306 C
MIAFS403_DR MIAFS403_root C.UTF-8
MICMO1P1NAP01 MDV_CRS_64f995e5236311e7be7000a098f388c6_A en_US
MICMO1P1NAP01 MDV_CRS_64f995e5236311e7be7000a098f388c6_B en_US
MICMO1P1NAP01-01 vol0 -
MICMO1P1NAP01-02 vol0 -
MLAFS401 CIFS_FS_APPS_BMLA101 C
MLAFS401 CIFS_FS_SHARE_BMLA111 C
MLAFS401 CIFS_FS_SITE_BMLA110 C
MLAFS401 CIFS_FS_USER01_BMLA105 C
MLAFS401 MLAFS401_root C.UTF-8
MOWFS401 CIFS_FS_APPS_AMOWP101 C
MOWFS401 CIFS_FS_SHARE_AMOWP111 C
MOWFS401 CIFS_FS_SITE_AMOWP110 C
MOWFS401 CIFS_FS_USER01_AMOWP105 C
MOWFS401 MOWFS401_root C.UTF-8
MPLNAS04_NEW MPLNAS04_root C.UTF-8
MPLNAS04_NEW V_MPLNAS04_01_APP C.UTF-8
MPLNAS04_NEW V_MPLNAS04_02_APP C.UTF-8
MPLNAS05 MPLNAS05_root C.UTF-8
MPLNAS05 V_MPLNAS05_01_APP C.UTF-8
MPLNAS10 MPLNAS10_root C.UTF-8
MPLNAS10 V_MPLNAS10_01_APP C.UTF-8
MPLNAS10 V_MPLNAS10_02_APP C.UTF-8
MPLNAS12 MPLNAS12_root en_US
MPLNAS12 V_MPLNAS12_01_APP en_US
MPLNAS15 MPLNAS15_root en_US
MPLNAS15 V_MPLNAS15_01_APP en_US
MPLNAS19 MPLNAS19_root C.UTF-8
MPLNAS19 V_MPLNAS19_01_APP C.UTF-8
MPLNASE037RNEC MPLNASE037RNEC_root C.UTF-8
MPLNASE037RNEC V_MPLNASE037RNEC_01_APP C.UTF-8
PRAFS401 CIFS_FS_APPS_BPRAP101 C
PRAFS401 CIFS_FS_SHARE_BPRAP111 C
PRAFS401 CIFS_FS_SITE_BPRAP110 C
PRAFS401 CIFS_FS_USER01_BPRAP105 C
PRAFS401 PRAFS401_root C.UTF-8
ROUFS401 CIFS_FS_APPS_BROUP101 C
ROUFS401 CIFS_FS_SHARE_BROUP111 C
ROUFS401 CIFS_FS_SITE_BROUP110 C
ROUFS401 CIFS_FS_USER01_BROUP105 C
ROUFS401 ROUFS401_root C.UTF-8
VACFS401 CIFS_FS_APPS_AVAC101 C
VACFS401 CIFS_FS_SHARE_AVAC111 C
VACFS401 CIFS_FS_SITE_AVAC110 C
VACFS401 CIFS_FS_USER01_AVAC105 C
VACFS401 VACFS401_root C.UTF-8
156 entries were displayed.

MICMO1P1NAP01::>

Ontapforrum · ‎2020-07-08

I have put those volumes in the excel attached. Looks like most of the CIFS Volumes are set to 'C POSIX', instead of 'C.UTF-8'. Strangely, root volumes are C.UTF8, so are they explicitly set to 'C' during creation.

According to cDOT: You can specify the language for a volume when creating a volume and it can be different from the language of an SVM. If you do not specify the language for a volume then it inherits the language setting of its SVM. After the volume is created, you cannot modify the language of a volume. Therefore, you must be aware of the available language options.

I guess, you could modify the SVM Language, however that will only allow 'NEW' volumes to inherit the language settings. There isn't much information around why it cannot be done, but I believe you need to just weigh-off options, what is more priority. As I had mentioned, there is free monitoring tool from NetApp, which you can for health/performance related stuff, however if you would like to change the volume then as I see, you will have to create a new Volume with correct language setting and they copy the data over. Is it worth it?

List of language options:
https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-ivg%2FGUID-8420C0E3-4208-4CD0-AAC6-2197C10FE9F7.html