ERROR netapp.manage.NaAPIFailedException: (errno=16)

[chicagometallic]

Error when trying to delete snapshot, the snapshot is never deleted

I can manually delete the snapshot

SnapCreator debug log

########## Running Snapshot copy Delete on Primary ##########

[2013-08-07 17:05:36,963] INFO: STORAGE-02090: Listing Snapshot copies on volume [wy1_vsphere_nfs_dev_linux].

[2013-08-07 17:05:36,963] INFO: STORAGE-02091: Listing Snapshot copies on volume [wy1_vsphere_nfs_dev_linux] finished successfully.

[2013-08-07 17:05:37,884] INFO: STORAGE-02013: Removing Snapshot copy [sc-hourly_20130803110100] on volume [wy1_vsphere_nfs_dev_linux].

[2013-08-07 17:05:37,884] ERROR: com.netapp.snapcreator.storage.executor.ZapiExecutorException: netapp.manage.NaAPIFailedException:  (errno=16)

    at com.netapp.snapcreator.storage.executor.ZapiExecutorImpl.run(ZapiExecutorImpl.java:62)

    at com.netapp.snapcreator.storage.api.ontap.Ontap7ModeApi.snapshotDelete(Ontap7ModeApi.java:228)

    at com.netapp.snapcreator.storage.StorageCoreImpl.snapshotDelete(StorageCoreImpl.java:831)

    at com.netapp.snapcreator.workflow.task.ZAPITask.deleteSnapshot(ZAPITask.java:908)

    at com.netapp.snapcreator.workflow.task.SnapDelete.execute(SnapDelete.java:99)

    at com.netapp.snapcreator.workflow.impl.SCTaskCallable.call(SCTaskCallable.java:48)

    at com.netapp.snapcreator.workflow.impl.SCTaskCallable.call(SCTaskCallable.java:20)

    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

    at java.util.concurrent.FutureTask.run(Unknown Source)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

    at java.lang.Thread.run(Unknown Source)

Caused by: netapp.manage.NaAPIFailedException:  (errno=16)

    at netapp.manage.NaServer.invokeElem(NaServer.java:671)

    at com.netapp.snapcreator.storage.executor.ZapiExecutorImpl.run(ZapiExecutorImpl.java:53)

    ... 11 more

[2013-08-07 17:05:37,884] ERROR: STORAGE-02015: Removing Snapshot copy [sc-hourly_20130803110100] on volume [wy1_vsphere_nfs_dev_linux] failed with werror [netapp.manage.NaAPIFailedException:  (errno=16)].

########## Snap Creator Framework 4.0p1 failed ##########

[2013-08-07 17:05:37,884] INFO: Pre Exit commands are not defined. Skipping !

[2013-08-07 17:05:38,072] INFO: STORAGE-05016: Creating OM Event [script:critical-event] on [beoncommand].

[2013-08-07 17:05:38,072] INFO: STORAGE-05017: OM Event [script:critical-event] on [beoncommand] created successfully.

Netapp Filer log

BEONCOMMAND [bevcenter5.cmceu.chicago-metallic.ad:9091]: SNAPCREATOR Snap Creator Framework 4.0p1: (0) Backup Started: INFO: Snap Creator Framework 4.0p1 Backup for sc ACTION: backup POLICY: hourly Plugin: vibe Supported Volumes: wy1_vsphere_nfs_dev_linux, Started

BEONCOMMAND [bevcenter5.cmceu.chicago-metallic.ad:9091]: SNAPCREATOR Snap Creator Framework 4.0p1: (0) Backup Completed: INFO: Snap Creator Framework 4.0p1 Backup for sc ACTION: backup POLICY: hourly Plugin: vibe Supported Volumes: wy1_vsphere_nfs_dev_linux, finished

BEONCOMMAND [bevcenter5.cmceu.chicago-metallic.ad:9091]: SNAPCREATOR Snap Creator Framework 4.0p1: (0) Snapmirror Backup of bebrfiler1:snapmirror_wy1_vsphere_nfs_dev_linux Started: INFO: Snap Creator Framework 4.0p1 Snapmirror Backup of bebrfiler1:snapmirror_wy1_vsphere_nfs_dev_linux for sc ACTION: backup POLICY: hourly Plugin: vibe Supported Volumes: wy1_vsphere_nfs_dev_linux, Started

[ktenzer]

SC is getting error from ONTAP. What could be happening is that you are using snapmirror and what happens in this case is snapshots get locked for a little while. Usually folks add sleep NTAP_POST_DATA_TRANSFER_CMD01=sleep 60. If that isnt case there may be some other issue, I would open an NGS support case as there could be something else non-SC related happening.

Regards,

Keith

[chicagometallic]

Hi Keith,

I can't find any information on NTAP_POST_DATA_TRANSFER_CMD01

So I have tried to add POST_NTAP_DATA_TRANSFER_CMD01=sleep 60 to the snapcreator cinfig file.

But I get the following error message:

[2013-08-08 09:32:11,056] INFO: [bevcenter5.cmceu.chicago-metallic.ad:9091 (4.0.0.1)] Executing Post data transfer command [sleep 60] on bevcenter5.cmceu.chicago-metallic.ad

[2013-08-08 09:32:11,056] ERROR: [bevcenter5.cmceu.chicago-metallic.ad:9091 (4.0.0.1)] [agt-00011] Command [sleep 60] is not allowed

or i'm i doing something wrong? probably 🙂

[ktenzer]

Sorry didnt know you had agent, you should read about commands in AIG

POST_NTAP_DATA_TRANSFER_CMD01=SERVER: sleep 60

That should work

Keith

[chicagometallic]

AIG = ?

The only explanation I can find on the command is:

     POST_NTAP_DATA_TRANSFER_CMD<##>

     This is a post data transfer command that runs after SnapVault or SnapMirror transfer, where ## is a number from 01-99.

The SERVER: sleep 60 command

     SERVER = just SERVER or the Netapp Filer name, ...?

Just using SERVER

########## POST DATA TRANSFER COMMANDS ##########

[2013-08-08 13:34:30,709] INFO: Executing Post data transfer command  sleep 60 on server

[2013-08-08 13:34:30,709] ERROR: commands failed with following error

[2013-08-08 13:34:30,709] ERROR: Cannot run program "sleep": CreateProcess error=2, The system cannot find the file specified

########## Snap Creator Framework 4.0p1 failed ##########

Using netapp filer name

########## POST DATA TRANSFER COMMANDS ##########

[2013-08-08 13:43:20,134] INFO: [bevcenter5.cmceu.chicago-metallic.ad:9091 (4.0.0.1)] Executing Post data transfer command [BEWYFILER1: sleep 60] on bevcenter5.cmceu.chicago-metallic.ad

[2013-08-08 13:43:20,134] ERROR: [bevcenter5.cmceu.chicago-metallic.ad:9091 (4.0.0.1)] [agt-00011] Command [BEWYFILER1: sleep 60] is not allowed

########## Snap Creator Framework 4.0p1 failed ##########

SnapCreator server = Windows 2008 R2

SnapCreator framework= 4.0p1

Agent server= Windows 2008 R2 (vCenter 5.1U1)

NetApp filers: NetApp Release 8.1 7-Mode

Grz,

Geert

[ktenzer]

AIG is the admin guide

I guess this isnt documented there...my apologies

Anyway for all pre / post cmds default behavior is send cmds to SC Agent. If you want to run them on server you need to specify SERVER: or you can specify AGENT: for agent. There is also CLONETARGET: if you are using a second agent for cloning.

You are also using windows so I think you need POST_NTAP_DATA_TRANSFER_CMD01=SERVER: cmd /c sleep 60 OR POST_NTAP_DATA_TRANSFER_CMD01=SERVER: "cmd /c sleep 60" OR POST_NTAP_DATA_TRANSFER_CMD01=SERVER: "sleep 60"

One or all of those should work

Keith

[ktenzer]

Ok nevermind obviously windows doesnt have a sleep command...sorry I am a unix guy.

Point is you need to try and have SC wait and you do this by sleeping. I see nothing really that useful on windows so most people say to use ping

ping localhost -n 1 -w 60000 > nul

This should ping for 60 seconds

POST_NTAP_DATA_TRANSFER_CMD01=SERVER: ping localhost -n 1 -w 60000 > nul

Hope this helps, if not I am afraid someone else who knows windows better will have to jump in

Keith

[spinks]

Depending on the version of Windows the TIMEOUT command should work for you.

All we're doing is sending the command to introduce a pause.

Open a command line on your Windows system and type:

TIMEOUT /T 60

You should get a countdown on screen that reads : Waiting for 60 seconds, press a key to continue ...

If this works add it into your config file:

POST_NTAP_DATA_TRANSFER_CMD01=SERVER: "TIMEOUT /T 60"

Don't forget the space after the ":" in SERVER:

If it doesn't work you may need to specify that this is a command line call by using:

POST_NTAP_DATA_TRANSFER_CMD01=SERVER: cmd.exe /c "TIMEOUT /T 60"

Hope this helps,

John

[chicagometallic]

Little update (going crazy here)

Some information I found:

     http://communities.quest.com/thread/17401

     http://stackoverflow.com/questions/9251656/reading-inputstream-from-a-batch-files-process-skips-to-next-line

The last 2 work

SERVER: timeout 30

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 14:47:36,455] INFO: Executing Post data transfer command  timeout 30 on server

    [2013-08-08 14:47:36,455] INFO: stdout:

    [2013-08-08 14:47:36,455] ERROR: stderr:ERROR: Input redirection is not supported, exiting the process immediately.

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: "TIMEOUT /T 60"

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 14:52:04,629] INFO: Executing Post data transfer command  "TIMEOUT /T 60" on server

    [2013-08-08 14:52:04,645] ERROR: commands failed with following error

    [2013-08-08 14:52:04,645] ERROR: Cannot run program ""TIMEOUT": CreateProcess error=2, The system cannot find the file specified

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: C:\Windows\System32\timeout /T 5

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:01:24,851] INFO: Executing Post data transfer command  C:\Windows\System32\timeout /T 5 on server

    [2013-08-08 15:01:24,851] INFO: stdout:

    [2013-08-08 15:01:24,851] ERROR: stderr:ERROR: Input redirection is not supported, exiting the process immediately.

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: C:\Windows\System32\timeout /T 5 1>nul

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:05:42,869] INFO: Executing Post data transfer command  C:\Windows\System32\timeout /T 5 1>nul on server

    [2013-08-08 15:05:42,869] INFO: stdout:

    [2013-08-08 15:05:42,869] ERROR: stderr:ERROR: Invalid syntax. Default option is not allowed more than '1' time(s).

    Type "TIMEOUT /?" for usage.

    ########## Snap Creator Framework 4.0p1 failed ##########

c:\temp\test.cmd = C:\Windows\System32\timeout /T 5  1>nul

SERVER: c:\temp\test.cmd

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:13:07,137] INFO: Executing Post data transfer command  c:\temp\test.cmd on server

    [2013-08-08 15:13:07,137] INFO: stdout:

    C:\Windows\system32>C:\Windows\System32\timeout /T 5  1>nul

    [2013-08-08 15:13:07,137] ERROR: stderr:ERROR: Input redirection is not supported, exiting the process immediately.

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: ping -n 5 127.0.0.1 1>nul

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:25:30,454] INFO: Executing Post data transfer command  ping -n 5 127.0.0.1 1>nul on server

    [2013-08-08 15:25:30,454] INFO: stdout:Bad parameter 1>nul.

    [2013-08-08 15:25:30,454] ERROR: stderr:

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: ping -n 5 127.0.0.1

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:20:03,624] INFO: Executing Post data transfer command  ping -n 5 127.0.0.1 on server

    [2013-08-08 15:20:03,624] INFO: stdout:

    Pinging 127.0.0.1 with 32 bytes of data&colon;

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Ping statistics for 127.0.0.1:

        Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

        Minimum = 0ms, Maximum = 0ms, Average = 0ms

    [2013-08-08 15:20:03,624] INFO: Post data transfer completed successfully

    ########## POST DATA TRANSFER COMMANDS FINISHED SUCCESSFULLY ##########   

c:\temp\test.cmd = ping -n 5 127.0.0.1 1>nul

SERVER: c:\temp\test.cmd

   ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:28:38,877] INFO: Executing Post data transfer command  c:\temp\test.cmd on server

    [2013-08-08 15:28:38,877] INFO: stdout:

    C:\Windows\system32>ping -n 5 127.0.0.1  1>nul

    [2013-08-08 15:28:38,877] INFO: Post data transfer completed successfully

    ########## POST DATA TRANSFER COMMANDS FINISHED SUCCESSFULLY ##########