agt-00005 error when running agent as a service for SMSQL job

[michael9]

SnapCreator 3.4P2 is running both on the SC server and for the agent on the SQL server itself.

When we start the SC Agent manually (snapcreator --start-agent --debug)  The backup job works fine.

When the SC Agent has been started a service the Application Quiesce phase immediately fails with the agt-00005 error. 

Any idea what may be causing this?

[ktenzer · Accepted solution]

Are you trying to use the SMSQL plugin? I am assuming you are so adding info about a known issue with SC's SMSQL plugin

There is a known issue with the scAgent running as a service and running powershell cmds. So there is a conflict we cant yet resolve in running powershell commands through a plugin that is running as a service. This is how the SC SMSQL plugin calls SMSQL through powershell.

The workarounds are:

1. run scServer locally so dont use scAgent (I know that defeats purpose of the solution to manage SnapManagers remotely)

2. run the scAgent in command mode. You can use task manager in windows to setup task to start scAgent when system boots, etc, and it can run it in background similar to service. Re-install scAgent and leave "install as service checkbox" unchecked.

We are working on this issue but for now have not found a solution

Let me know if this helps and if the workarounds are viable for you?

Regards,

Keith

[michael9]

Hi Keith-

Thanks for the quick response; we are indeed using the SMSQL plugin.  We tried adding -noprofile and -noninteractive as options to the Powershell command that SnapCreator is passing on but it didn't seem to help.  One thing we did notice is that we get the agt-00005 error when running the scAgent service with an admin level account but simply get a timeout error when running it as local system.

As far as the workarounds option 1 essentially defeats the reason we wanted to run the SMSQL jobs via SC and option 2 isn't a viable option in this environment due to security restrictions.  For now we'll fall back to traditional scheduling of SMSQL jobs via the SQL Server Agent but will continue experimenting with possible solutions.

Regards,

Mike

[ktenzer]

Hi Mike,

We found this issue late in the QA cycle and we tried a lot of different combinations of powershell commands and we tested using admin user and it would hang. Basically, powershell when executed through another process running as a service simply hangs. We are looking at new methods or ways to invoke SMSQL but unfortunately don't have anything yet. I will bring this up w/engineering and make sure it gets more attention

I do have another idea for workaround, I am guessing it will also hang unless you can call SMSQL without powershell but worth a shot. In SC you can turn off snapshot creation, NTAP_SNAPSHOT_DISABLE=Y and NTAP_SNAPSHOT_NODELETE=Y.

You could then just have a CMD like

PRE_NTAP_CMD01=powershell or bat script which calls powershell or even non-powershell call using SMSQL cli.

SC would simply execute CMD PRE_NTAP_CMD01 through agent. You would need to allow CMD in the scAgent/config/agent.conf file, ie: "command: powershell".

Regards,

Keith

[michael9]

Keith-

Have you tried the following syntax:

C:\system32\WindowsPowerShell\v1.0\powershell.exe -psconsolefile "C:\Program Files\NetApp\SnapManager for SQL Server\smsqlShell.psc1" -command &{new-backup -svr 'SERVER' -d 'SERVER', '5', 'master', 'model', 'msdb', 'VCDB', 'VUMDB' -RetainBackups 7 -lb -bksif -RetainSnapofSnapInfo 8 -trlog -mgmt daily}

(Use the &{    cmdlet }  )

I wasn't able to test that but it may help.

[ktenzer]

I will send this to engineering, thanks for the feedback!

If you get a chance can you try this? If we get confirmation that it works, that would really help!

I know powershell itself isn't the problem because our plugin interface allows you to create SC plugins in powershell and that works through agent as service so it is something about the syntax or way we are making the call to the SnapManager.

Regards,

Keith

[mrettl]

Hi!

I have the same issue on a Win2008R2 x64 machine with MySQL 5.5!

It is a demo-machine, so I have MySQL, scServer and scAgent running on the same machine.

When I start scAgent as a service, I also receive these errors:

########## Application quiesce ##########

[Thu Jan 12 16:26:06 2012] [snapcreator.vie.demo:9090(3.4.0.1)] ERROR: [agt-00005] Another quiesce operation or watchdog is already running on this config

[Thu Jan 12 16:26:06 2012] [snapcreator.vie.demo:9090(3.4.0.1)] ERROR: [scf-00053] Application quiesce for plugin mysql failed with exit code 100, Exiting!

When I stop the service and run the agent with --start-agent from the CLI, it works fine.

Then I have added a scheduled task to run when the system starts, issuing "snapcreator.exe" with the "--start-agent" as argument in the task. This also fails. I have then created a simple batch-script to start the agent, configured this as the startup-action and it works.

Regards,

Matthias

[erpsourcing]

Hi Matthias

I have also a Win2008 R2 x64 machine and created a scheduled task for Snap Creator Agent.

It works perfectly in the background, here are my parameters:

Program/Script:

"C:\Program Files (x86)\Netapp\NetApp_Snap_Creator_Framework\scAgent3.4p2\snapcreator.exe"

Arguments:

--start-agent 9090

Security Options:

Run whether user is logged on or not

Settings:
Uncheck "Stop the task if it runs longer than"

Regards

Reto

[charly]

Hi Matthias,

We have fixed this problem in 3.5. We had some issue with watchdog lock file path.

You can just search for '.lck' file and delete the one created by snapcreator (*<configname>*.lck).

regards

Charly