Data Backup and Recovery

ERROR netapp.manage.NaAPIFailedException: (errno=16)

chicagometallic
5,267 Views

Error when trying to delete snapshot, the snapshot is never deleted

I can manually delete the snapshot

SnapCreator debug log

########## Running Snapshot copy Delete on Primary ##########

[2013-08-07 17:05:36,963] INFO: STORAGE-02090: Listing Snapshot copies on volume [wy1_vsphere_nfs_dev_linux].

[2013-08-07 17:05:36,963] INFO: STORAGE-02091: Listing Snapshot copies on volume [wy1_vsphere_nfs_dev_linux] finished successfully.

[2013-08-07 17:05:37,884] INFO: STORAGE-02013: Removing Snapshot copy [sc-hourly_20130803110100] on volume [wy1_vsphere_nfs_dev_linux].

[2013-08-07 17:05:37,884] ERROR: com.netapp.snapcreator.storage.executor.ZapiExecutorException: netapp.manage.NaAPIFailedException:  (errno=16)

    at com.netapp.snapcreator.storage.executor.ZapiExecutorImpl.run(ZapiExecutorImpl.java:62)

    at com.netapp.snapcreator.storage.api.ontap.Ontap7ModeApi.snapshotDelete(Ontap7ModeApi.java:228)

    at com.netapp.snapcreator.storage.StorageCoreImpl.snapshotDelete(StorageCoreImpl.java:831)

    at com.netapp.snapcreator.workflow.task.ZAPITask.deleteSnapshot(ZAPITask.java:908)

    at com.netapp.snapcreator.workflow.task.SnapDelete.execute(SnapDelete.java:99)

    at com.netapp.snapcreator.workflow.impl.SCTaskCallable.call(SCTaskCallable.java:48)

    at com.netapp.snapcreator.workflow.impl.SCTaskCallable.call(SCTaskCallable.java:20)

    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

    at java.util.concurrent.FutureTask.run(Unknown Source)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

    at java.lang.Thread.run(Unknown Source)

Caused by: netapp.manage.NaAPIFailedException:  (errno=16)

    at netapp.manage.NaServer.invokeElem(NaServer.java:671)

    at com.netapp.snapcreator.storage.executor.ZapiExecutorImpl.run(ZapiExecutorImpl.java:53)

    ... 11 more

[2013-08-07 17:05:37,884] ERROR: STORAGE-02015: Removing Snapshot copy [sc-hourly_20130803110100] on volume [wy1_vsphere_nfs_dev_linux] failed with werror [netapp.manage.NaAPIFailedException:  (errno=16)].

########## Snap Creator Framework 4.0p1 failed ##########

[2013-08-07 17:05:37,884] INFO: Pre Exit commands are not defined. Skipping !

[2013-08-07 17:05:38,072] INFO: STORAGE-05016: Creating OM Event [script:critical-event] on [beoncommand].

[2013-08-07 17:05:38,072] INFO: STORAGE-05017: OM Event [script:critical-event] on [beoncommand] created successfully.

Netapp Filer log

BEONCOMMAND [bevcenter5.cmceu.chicago-metallic.ad:9091]: SNAPCREATOR Snap Creator Framework 4.0p1: (0) Backup Started: INFO: Snap Creator Framework 4.0p1 Backup for sc ACTION: backup POLICY: hourly Plugin: vibe Supported Volumes: wy1_vsphere_nfs_dev_linux, Started

BEONCOMMAND [bevcenter5.cmceu.chicago-metallic.ad:9091]: SNAPCREATOR Snap Creator Framework 4.0p1: (0) Backup Completed: INFO: Snap Creator Framework 4.0p1 Backup for sc ACTION: backup POLICY: hourly Plugin: vibe Supported Volumes: wy1_vsphere_nfs_dev_linux, finished

BEONCOMMAND [bevcenter5.cmceu.chicago-metallic.ad:9091]: SNAPCREATOR Snap Creator Framework 4.0p1: (0) Snapmirror Backup of bebrfiler1:snapmirror_wy1_vsphere_nfs_dev_linux Started: INFO: Snap Creator Framework 4.0p1 Snapmirror Backup of bebrfiler1:snapmirror_wy1_vsphere_nfs_dev_linux for sc ACTION: backup POLICY: hourly Plugin: vibe Supported Volumes: wy1_vsphere_nfs_dev_linux, Started

8 REPLIES 8

ktenzer
5,268 Views

SC is getting error from ONTAP. What could be happening is that you are using snapmirror and what happens in this case is snapshots get locked for a little while. Usually folks add sleep NTAP_POST_DATA_TRANSFER_CMD01=sleep 60. If that isnt case there may be some other issue, I would open an NGS support case as there could be something else non-SC related happening.

Regards,

Keith

chicagometallic
5,268 Views

Hi Keith,

I can't find any information on NTAP_POST_DATA_TRANSFER_CMD01

So I have tried to add POST_NTAP_DATA_TRANSFER_CMD01=sleep 60 to the snapcreator cinfig file.

But I get the following error message:

[2013-08-08 09:32:11,056] INFO: [bevcenter5.cmceu.chicago-metallic.ad:9091 (4.0.0.1)] Executing Post data transfer command [sleep 60] on bevcenter5.cmceu.chicago-metallic.ad

[2013-08-08 09:32:11,056] ERROR: [bevcenter5.cmceu.chicago-metallic.ad:9091 (4.0.0.1)] [agt-00011] Command [sleep 60] is not allowed

or i'm i doing something wrong? probably 🙂

ktenzer
5,268 Views

Sorry didnt know you had agent, you should read about commands in AIG

POST_NTAP_DATA_TRANSFER_CMD01=SERVER: sleep 60

That should work

Keith

chicagometallic
5,268 Views

AIG = ?

The only explanation I can find on the command is:

     POST_NTAP_DATA_TRANSFER_CMD<##>

     This is a post data transfer command that runs after SnapVault or SnapMirror transfer, where ## is a number from 01-99.

The SERVER: sleep 60 command

     SERVER = just SERVER or the Netapp Filer name, ...?

Just using SERVER

########## POST DATA TRANSFER COMMANDS ##########

[2013-08-08 13:34:30,709] INFO: Executing Post data transfer command  sleep 60 on server

[2013-08-08 13:34:30,709] ERROR: commands failed with following error

[2013-08-08 13:34:30,709] ERROR: Cannot run program "sleep": CreateProcess error=2, The system cannot find the file specified

########## Snap Creator Framework 4.0p1 failed ##########

Using netapp filer name

########## POST DATA TRANSFER COMMANDS ##########

[2013-08-08 13:43:20,134] INFO: [bevcenter5.cmceu.chicago-metallic.ad:9091 (4.0.0.1)] Executing Post data transfer command [BEWYFILER1: sleep 60] on bevcenter5.cmceu.chicago-metallic.ad

[2013-08-08 13:43:20,134] ERROR: [bevcenter5.cmceu.chicago-metallic.ad:9091 (4.0.0.1)] [agt-00011] Command [BEWYFILER1: sleep 60] is not allowed

########## Snap Creator Framework 4.0p1 failed ##########

SnapCreator server = Windows 2008 R2

SnapCreator framework= 4.0p1

Agent server= Windows 2008 R2 (vCenter 5.1U1)

NetApp filers: NetApp Release 8.1 7-Mode

Grz,

Geert

ktenzer
5,268 Views

AIG is the admin guide

I guess this isnt documented there...my apologies

Anyway for all pre / post cmds default behavior is send cmds to SC Agent. If you want to run them on server you need to specify SERVER: or you can specify AGENT: for agent. There is also CLONETARGET: if you are using a second agent for cloning.

You are also using windows so I think you need POST_NTAP_DATA_TRANSFER_CMD01=SERVER: cmd /c sleep 60 OR POST_NTAP_DATA_TRANSFER_CMD01=SERVER: "cmd /c sleep 60" OR POST_NTAP_DATA_TRANSFER_CMD01=SERVER: "sleep 60"

One or all of those should work

Keith

ktenzer
5,268 Views

Ok nevermind obviously windows doesnt have a sleep command...sorry I am a unix guy.

Point is you need to try and have SC wait and you do this by sleeping. I see nothing really that useful on windows so most people say to use ping

ping localhost -n 1 -w 60000 > nul

This should ping for 60 seconds

POST_NTAP_DATA_TRANSFER_CMD01=SERVER: ping localhost -n 1 -w 60000 > nul

Hope this helps, if not I am afraid someone else who knows windows better will have to jump in

Keith

spinks
5,268 Views

Depending on the version of Windows the TIMEOUT command should work for you.

All we're doing is sending the command to introduce a pause.

Open a command line on your Windows system and type:

TIMEOUT /T 60

You should get a countdown on screen that reads : Waiting for 60 seconds, press a key to continue ...

If this works add it into your config file:

POST_NTAP_DATA_TRANSFER_CMD01=SERVER: "TIMEOUT /T 60"

Don't forget the space after the ":" in SERVER:

If it doesn't work you may need to specify that this is a command line call by using:

POST_NTAP_DATA_TRANSFER_CMD01=SERVER: cmd.exe /c "TIMEOUT /T 60"

Hope this helps,

John

chicagometallic
5,268 Views

Little update (going crazy here)

Some information I found:

     http://communities.quest.com/thread/17401

     http://stackoverflow.com/questions/9251656/reading-inputstream-from-a-batch-files-process-skips-to-next-line

The last 2 work

SERVER: timeout 30

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 14:47:36,455] INFO: Executing Post data transfer command  timeout 30 on server

    [2013-08-08 14:47:36,455] INFO: stdout:

    [2013-08-08 14:47:36,455] ERROR: stderr:ERROR: Input redirection is not supported, exiting the process immediately.

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: "TIMEOUT /T 60"

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 14:52:04,629] INFO: Executing Post data transfer command  "TIMEOUT /T 60" on server

    [2013-08-08 14:52:04,645] ERROR: commands failed with following error

    [2013-08-08 14:52:04,645] ERROR: Cannot run program ""TIMEOUT": CreateProcess error=2, The system cannot find the file specified

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: C:\Windows\System32\timeout /T 5

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:01:24,851] INFO: Executing Post data transfer command  C:\Windows\System32\timeout /T 5 on server

    [2013-08-08 15:01:24,851] INFO: stdout:

    [2013-08-08 15:01:24,851] ERROR: stderr:ERROR: Input redirection is not supported, exiting the process immediately.

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: C:\Windows\System32\timeout /T 5 1>nul

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:05:42,869] INFO: Executing Post data transfer command  C:\Windows\System32\timeout /T 5 1>nul on server

    [2013-08-08 15:05:42,869] INFO: stdout:

    [2013-08-08 15:05:42,869] ERROR: stderr:ERROR: Invalid syntax. Default option is not allowed more than '1' time(s).

    Type "TIMEOUT /?" for usage.

    ########## Snap Creator Framework 4.0p1 failed ##########

c:\temp\test.cmd = C:\Windows\System32\timeout /T 5  1>nul

SERVER: c:\temp\test.cmd

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:13:07,137] INFO: Executing Post data transfer command  c:\temp\test.cmd on server

    [2013-08-08 15:13:07,137] INFO: stdout:

    C:\Windows\system32>C:\Windows\System32\timeout /T 5  1>nul

    [2013-08-08 15:13:07,137] ERROR: stderr:ERROR: Input redirection is not supported, exiting the process immediately.

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: ping -n 5 127.0.0.1 1>nul

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:25:30,454] INFO: Executing Post data transfer command  ping -n 5 127.0.0.1 1>nul on server

    [2013-08-08 15:25:30,454] INFO: stdout:Bad parameter 1>nul.

    [2013-08-08 15:25:30,454] ERROR: stderr:

    ########## Snap Creator Framework 4.0p1 failed ##########

SERVER: ping -n 5 127.0.0.1

    ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:20:03,624] INFO: Executing Post data transfer command  ping -n 5 127.0.0.1 on server

    [2013-08-08 15:20:03,624] INFO: stdout:

    Pinging 127.0.0.1 with 32 bytes of data&colon;

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Reply from 127.0.0.1: bytes=32 time<1ms TTL=128

    Ping statistics for 127.0.0.1:

        Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

        Minimum = 0ms, Maximum = 0ms, Average = 0ms

    [2013-08-08 15:20:03,624] INFO: Post data transfer completed successfully

    ########## POST DATA TRANSFER COMMANDS FINISHED SUCCESSFULLY ##########   

c:\temp\test.cmd = ping -n 5 127.0.0.1 1>nul

SERVER: c:\temp\test.cmd

   ########## POST DATA TRANSFER COMMANDS ##########

    [2013-08-08 15:28:38,877] INFO: Executing Post data transfer command  c:\temp\test.cmd on server

    [2013-08-08 15:28:38,877] INFO: stdout:

    C:\Windows\system32>ping -n 5 127.0.0.1  1>nul

    [2013-08-08 15:28:38,877] INFO: Post data transfer completed successfully

    ########## POST DATA TRANSFER COMMANDS FINISHED SUCCESSFULLY ##########

Public