Data Backup and Recovery

Workflow cannot be stopped

SVOLLRAT1
4,543 Views

Hi Everyone

Seems I have a stuck task in SnapCreator GUI. A Job started a week ago was never reported completed or failed and therefore is still listed in progress.

Trying to kill it in GUI only resulted in error, apparently a not existing or already completed job cannot be aborted or corrected in status.

We have monitoring in place that reports such Jobs and creates nasty alerts.

Any advice what to do here? Tried to find a CLI Option to try cancel from there, couldn't find any.

Thanks in advance

Stefan

Here's what the GUI.log writes regarding that error:

2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Can't stop workflow, workflow not found with id:1521

2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Can't stop workflow, workflow not found with id:1521

2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - com.netapp.snapcreator.workflow.exception.WorkFlowException: Can't stop workflow, workflow not found with id:1521

    at com.netapp.snapcreator.workflow.impl.WorkflowEngineImpl.stopWorkflow(WorkflowEngineImpl.java:883)

    at com.netapp.snapcreator.engine.EngineImpl.stopWorkflow(EngineImpl.java:1883)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:616)

    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)

    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)

    at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60)

    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

    at java.util.concurrent.FutureTask.run(FutureTask.java:166)

    at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)

    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)

    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)

    at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)

    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)

    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186)

    at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)

    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)

    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)

    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

    at org.mortbay.jetty.Server.handle(Server.java:326)

    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)

    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)

    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)

    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - com.netapp.snapcreator.workflow.exception.WorkFlowException: Can't stop workflow, workflow not found with id:1521

    at com.netapp.snapcreator.workflow.impl.WorkflowEngineImpl.stopWorkflow(WorkflowEngineImpl.java:883)

    at com.netapp.snapcreator.engine.EngineImpl.stopWorkflow(EngineImpl.java:1883)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:616)

    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)

    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)

    at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60)

    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

    at java.util.concurrent.FutureTask.run(FutureTask.java:166)

    at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)

    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)

    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)

    at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)

    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)

    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186)

    at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)

    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)

    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)

    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

    at org.mortbay.jetty.Server.handle(Server.java:326)

    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)

    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)

    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)

    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Couldn't stop workflow

2014-04-24 08:04:16,574 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Couldn't stop workflow

2014-04-24 08:04:16,575 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Job with job id: 30993 could not be cancelled.

2014-04-24 08:04:16,575 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Job with job id: 30993 could not be cancelled.

2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Can't stop workflow, workflow not found with id:1521

2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Can't stop workflow, workflow not found with id:1521

2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - com.netapp.snapcreator.workflow.exception.WorkFlowException: Can't stop workflow, workflow not found with id:1521

    at com.netapp.snapcreator.workflow.impl.WorkflowEngineImpl.stopWorkflow(WorkflowEngineImpl.java:883)

    at com.netapp.snapcreator.engine.EngineImpl.stopWorkflow(EngineImpl.java:1883)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:616)

    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)

    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)

    at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60)

    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

    at java.util.concurrent.FutureTask.run(FutureTask.java:166)

    at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)

    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)

    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)

    at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)

    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)

    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186)

    at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)

    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)

    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)

    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

    at org.mortbay.jetty.Server.handle(Server.java:326)

    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)

    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)

    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)

    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - com.netapp.snapcreator.workflow.exception.WorkFlowException: Can't stop workflow, workflow not found with id:1521

    at com.netapp.snapcreator.workflow.impl.WorkflowEngineImpl.stopWorkflow(WorkflowEngineImpl.java:883)

    at com.netapp.snapcreator.engine.EngineImpl.stopWorkflow(EngineImpl.java:1883)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:616)

    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)

    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)

    at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60)

    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75)

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

    at java.util.concurrent.FutureTask.run(FutureTask.java:166)

    at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)

    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)

    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)

    at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)

    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)

    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186)

    at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)

    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)

    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)

    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

    at org.mortbay.jetty.Server.handle(Server.java:326)

    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)

    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)

    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)

    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Couldn't stop workflow

2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Couldn't stop workflow

2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Job with job id: 30993 could not be cancelled.

2014-04-24 08:04:32,913 [744701262@qtp-1833350875-1031] com.netapp.snapcreator.ui.service.SnapCreatorGUILogger ERROR - Job with job id: 30993 could not be cancelled.

4 REPLIES 4

spinks
4,542 Views

Stefan,

Can you confirm what version of Snap Creator you are running?

I pinged the dev team to see if this is something we've seen before.

Thanks,

John

SVOLLRAT1
4,542 Views

Hi John

We are running 4.0p1 on that Machine

BR

Stefan

Von: John Spinks

Gesendet: Donnerstag, 24. April 2014 15:11

An: Vollrath, Stefan

Betreff: - Re: Workflow cannot be stopped

<https://communities.netapp.com/index.jspa>

Re: Workflow cannot be stopped

created by John Spinks<https://communities.netapp.com/people/spinks> in Snap Creator Framework - View the full discussion<https://communities.netapp.com/message/129339#129339>

Stefan,

Can you confirm what version of Snap Creator you are running?

I pinged the dev team to see if this is something we've seen before.

Thanks,

John

Reply to this message by replying to this email -or- go to the message on NetApp Community<https://communities.netapp.com/message/129339#129339>

Start a new discussion in Snap Creator Framework by email<mailto:discussions-community-productsandsolutions-datastoragesoftware-dataprotectionsoftware-backupandrestore-snapcreatorframework@communities.netapp.com> or at NetApp Community<https://communities.netapp.com/choose-container.jspa?contentType=1&containerType=14&container=4147>

spinks
4,542 Views

There *may* be a fix for this in Snap Creator 4.1, but that fix would only prevent this from happening in the future.

(The description on the fixed defect sounds similar to this issue)

Basically to resolve you will need to manually alter the database row.

You can open a support case and the support team can walk you through these steps.

You can also email me at spinks -at- netapp (dot) com and I can email you the rough steps if you would rather try this yourself.

Thanks,

John

SVOLLRAT1
4,542 Views

As a workaround the Job-Status can be reset directly in the Database.

Doesn't fix the problem itself but helps with a cleaner status-overview.

Stop SnapCreator and execute the following (path may need to be adjusted to local environment):

export CLASSPATH=/usr/local/scServer4.0p1/engine/lib/derby.jar/usr/local/scServer4.0p1/engine/lib/derbytools.jar:.

cd /usr/local/scServer4.0p1/engine/

java org.apache.derby.tools.ij 

CONNECT 'jdbc:derby:snapcreator';

SET SCHEMA SNAPCREATOR;

SELECT JOBID, PROFILE, CONFIG, JOBSTATUS, STARTTIME, ENDTIME FROM JOB WHERE JOBSTATUS = 1;

UPDATE JOB SET JOBSTATUS = -1 WHERE JOBID = <jobid from previous command>;

EXIT;

Public