Sporadic error "The specified volume <volume> is offline" while getting snapshot list and volume list from NetApp

[YEGOR_FADEEV]

Hello!

I'm using NetApp simulator version "NetApp Release 8.1X47 7-Mode"

I need to monitor volumes as well as related snapshots and qtrees by using Data ONTAP APIs.

The general flow is

1) retrieve volume list;

2) iterate through each volume;

3) retrieve snapshot list for each volume;

4) retrieve qtree list for each volume;

Here I have a code snippet according to general flow

volume-list-info-iter-start

volume-list-info-iter-next

     snapshot-list-info

     snapshot-list-info

     ....

     qtree-list

     qtree-list

     ...

When I poll NetApp with timeout in 3 minutes I get error "volume is offline" sometime regardless the fact that volume is online..

I enabled Audit logging in order to see what happen on server. I noticed that error log outputs some 'xml' after snapshot-list-info command (see below) when error is occured.

Wed Dec 12 16:03:48 PST [api_mpool_06:debug]: root@[#.#.10.156]:API:http in:<?xml version='1.0' encoding='UTF-8'?><!DOCTYPE netapp SYSTEM 'file:/etc/netapp_filer.dtd'><netapp xmlns="http://www.netapp.com/filer/admin" version="1.7" vfiler="vfiler8"><snapshot-list-info><target-name>test4_q_1</target-name><target-type>volume</target-type></snapshot-list-info></netapp>

Wed Dec 12 16:03:48 PST [api_mpool_06:debug]: NULL:API:in:API snapshot-list-info forwarded to vfiler vfiler8 for execution from filer vfiler0.

Wed Dec 12 16:03:48 PST [api_mpool_09:debug]: root@[#.#.10.156]:API:http in:<?xml version='1.0' encoding='UTF-8'?><!DOCTYPE netapp SYSTEM 'file:/etc/netapp_filer.dtd'><netapp xmlns="http://www.netapp.com/filer/admin" version="1.7" vfiler="vfiler8"><snapshot-list-info><target-name>test4_1</target-name><target-type>volume</target-type></snapshot-list-info></netapp>

Wed Dec 12 16:03:48 PST [api_mpool_09:debug]: NULL:API:in:API snapshot-list-info forwarded to vfiler vfiler8 for execution from filer vfiler0.

Wed Dec 12 16:03:56 PST [api_mpool_00:debug]: root@[#.#.32.165]:API:http in:<?xml version='1.0' encoding='utf-8' ?> <!DOCTYPE netapp SYSTEM 'file:/etc/netapp_filer.dtd'> <netapp version="1.0" xmlns="http://www.netapp.com/filer/admin"><perf-object-get-instances><objectname>aggregate</objectname><counters><counter>cp_read_blocks</counter><counter>cp...

Wed Dec 12 16:03:56 PST [api_mpool_08:debug]: root@[#.#.32.165]:API:http in:<?xml version='1.0' encoding='utf-8' ?> <!DOCTYPE netapp SYSTEM 'file:/etc/netapp_filer.dtd'> <netapp version="1.0" xmlns="http://www.netapp.com/filer/admin"><perf-object-get-instances><objectname>fcp</objectname><counters><counter>fcp_ops</counter><counter>fcp_read_data</...

Wed Dec 12 16:03:56 PST [api_mpool_05:debug]: root@[#.#.32.165]:API:http in:<?xml version='1.0' encoding='utf-8' ?> <!DOCTYPE netapp SYSTEM 'file:/etc/netapp_filer.dtd'> <netapp version="1.0" xmlns="http://www.netapp.com/filer/admin"><perf-object-get-instances><objectname>nfsv3</objectname><counters><counter>nfsv3_avg_op_latency_base</counter><cou...

 

Could you explain me why does OnTAP server return error and what does mean log data from audit log ?

Regards,

Yegor...