Data Backup and Recovery
Data Backup and Recovery
Hi Everyone
Seems I have a stuck task in SnapCreator GUI. A Job started a week ago was never reported completed or failed and therefore is still listed in progress.
Trying to kill it in GUI only resulted in error, apparently a not existing or already completed job cannot be aborted or corrected in status.
We have monitoring in place that reports such Jobs and creates nasty alerts.
Any advice what to do here? Tried to find a CLI Option to try cancel from there, couldn't find any.
Thanks in advance
Stefan
Here's what the GUI.log writes regarding that error:
2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Can't stop workflow, workflow not found with id:1521
2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Can't stop workflow, workflow not found with id:1521
2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - com.netapp.snapcreator.workflow.exception.WorkFlowException: Can't stop workflow, workflow not found with id:1521
at com.netapp.snapcreator.workflow.impl.WorkflowEngineImpl.stopWorkflow(WorkflowEngineImpl.java:883)
at com.netapp.snapcreator.engine.EngineImpl.stopWorkflow(EngineImpl.java:1883)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)
at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)
at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186)
at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - com.netapp.snapcreator.workflow.exception.WorkFlowException: Can't stop workflow, workflow not found with id:1521
at com.netapp.snapcreator.workflow.impl.WorkflowEngineImpl.stopWorkflow(WorkflowEngineImpl.java:883)
at com.netapp.snapcreator.engine.EngineImpl.stopWorkflow(EngineImpl.java:1883)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)
at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)
at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186)
at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Couldn't stop workflow
2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Couldn't stop workflow
2014-04-24 08:04:16,575 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Job with job id: 30993 could not be cancelled.
2014-04-24 08:04:16,575 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Job with job id: 30993 could not be cancelled.
2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Can't stop workflow, workflow not found with id:1521
2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Can't stop workflow, workflow not found with id:1521
2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - com.netapp.snapcreator.workflow.exception.WorkFlowException: Can't stop workflow, workflow not found with id:1521
at com.netapp.snapcreator.workflow.impl.WorkflowEngineImpl.stopWorkflow(WorkflowEngineImpl.java:883)
at com.netapp.snapcreator.engine.EngineImpl.stopWorkflow(EngineImpl.java:1883)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)
at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)
at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186)
at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - com.netapp.snapcreator.workflow.exception.WorkFlowException: Can't stop workflow, workflow not found with id:1521
at com.netapp.snapcreator.workflow.impl.WorkflowEngineImpl.stopWorkflow(WorkflowEngineImpl.java:883)
at com.netapp.snapcreator.engine.EngineImpl.stopWorkflow(EngineImpl.java:1883)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)
at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)
at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186)
at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Couldn't stop workflow
2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Couldn't stop workflow
2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Job with job id: 30993 could not be cancelled.
2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Job with job id: 30993 could not be cancelled.
Stefan,
Can you confirm what version of Snap Creator you are running?
I pinged the dev team to see if this is something we've seen before.
Thanks,
John
Hi John
We are running 4.0p1 on that Machine
BR
Stefan
Gesendet: Donnerstag, 24. April 2014 15:11
An: Vollrath, Stefan
Betreff: - Re: Workflow cannot be stopped
<https://communities.netapp.com/index.jspa>
Re: Workflow cannot be stopped
created by John Spinks<https://communities.netapp.com/people/spinks> in Snap Creator Framework - View the full discussion<https://communities.netapp.com/message/129339#129339>
Stefan,
Can you confirm what version of Snap Creator you are running?
I pinged the dev team to see if this is something we've seen before.
Thanks,
John
Reply to this message by replying to this email -or- go to the message on NetApp Community<https://communities.netapp.com/message/129339#129339>
Start a new discussion in Snap Creator Framework by email<mailto:discussions-community-productsandsolutions-datastoragesoftware-dataprotectionsoftware-backupandrestore-snapcreatorframework@communities.netapp.com> or at NetApp Community<https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=4147>
There *may* be a fix for this in Snap Creator 4.1, but that fix would only prevent this from happening in the future.
(The description on the fixed defect sounds similar to this issue)
Basically to resolve you will need to manually alter the database row.
You can open a support case and the support team can walk you through these steps.
You can also email me at spinks -at- netapp (dot) com and I can email you the rough steps if you would rather try this yourself.
Thanks,
John
As a workaround the Job-Status can be reset directly in the Database.
Doesn't fix the problem itself but helps with a cleaner status-overview.
Stop SnapCreator and execute the following (path may need to be adjusted to local environment):
export CLASSPATH=/usr/local/scServer4.0p1/engine/lib/derby.jar/usr/local/scServer4.0p1/engine/lib/derbytools.jar:.
cd /usr/local/scServer4.0p1/engine/
java org.apache.derby.tools.ij
CONNECT 'jdbc:derby:snapcreator';
SET SCHEMA SNAPCREATOR;
SELECT JOBID, PROFILE, CONFIG, JOBSTATUS, STARTTIME, ENDTIME FROM JOB WHERE JOBSTATUS = 1;
UPDATE JOB SET JOBSTATUS = -1 WHERE JOBID = <jobid from previous command>;
EXIT;