Workflow execution fails

[RoiBeci]

Hi,

IHAC whic is running his workflows from HP Operations Orchestrator (HPOO), and from time to time his workflows are failed, even before starting the workflow execution.

Here are some highlights from the wfa.log and server.log (it is a restricted site and I was needed to type this logs on my own, so excuse me for any typos).

wfa.log
~~~~~~~~

ERROR Job WorkflowJob {workflowId=76.....} failed: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
...

Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
...134 more
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Lock wait timeout exceeded; try restarting transaction
...137 more
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction

server.log
~~~~~~~~~~~
WARN [com.arjuna.ats.arjuna] (Transaction Reaper worker 5) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffffbe140661:75c4924f:5582b604:9aed0f in

state RUN

.. WARN: Abort of action id 0:ffffbe140661:75c4924f:5582b604:9aed0f invoked while multiple threads active within it.

... WARN [com.arjuna.ats.arjuna] (Transaction Reaper worker 5) ARJUNA012108: CheckedAction::check - atomic action 0:ffffbe140661:75c4924f:5582b604:9aed0f

aborting with 1 threads active!

... WARN [com.arjuna.ats.arjuna] (Transaction Reaper worker 5) ARJUNA012108: TransactionReaper::check timeout for TX 0:ffffbe140661:75c4924f:5582b604:9aed0f

in state CANCEL

... WARN [com.arjuna.ats.arjuna] (Transaction Reaper worker 5) ARJUNA012108: TransactionReaper::check timeout for TX 0:ffffbe140661:75c4924f:5582b604:9aed0f

in state CANCEL_INTERRUPTED

... WARN [com.arjuna.ats.arjuna] (Transaction Reaper worker 5) ARJUNA012120: TransactionReaper::check worker thread[TransactionReaper worker 5,5,main] not

responding to interrupt when cancelling TX 0:ffffbe140661:75c4924f:5582b604:9aed0f -- worker marked as zombie and TX scheduled for mark-as-rollback

WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012110: TransactionReaper::check successfuly marked TX 0:ffffbe140661:75c4924f:5582b604:9aed0f as

rollback only

...WARN [org.hibernate.engine.jdbc.spi.sqlExceptionHelper] (Thread-41061(HornetQ-client-global-threads-637449795)) SQL Error: 1025, SQLState: 41000

...ERROR [org.hibernate.engine.jdbc.spi.sqlExceptionHelper] (Thread-41061(HornetQ-client-global-threads-637449795)) Lock wait timeout exceeded; try

restarting transaction

..WARN [com.arjuna.ats.arjuna] (Thread-41061(HornetQ-client-global-threads-637449795)) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for

synchronizationImple<0:ffffbe140661:75c4924f:5582b604:9aed10, org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization@68a1753a

>:javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Lock wait timeout exceeded; try restarting transaction

.......

...WARN [com.arjuna.ats.arjuna] (Thread-41061(HornetQ-client-global-threads-637449795)) ARJUNA012077: Abort called on already aborted atomic action

0:ffffbe140661:75c4924f:5582b604:9aed0f

...WARN [org.jboss.jca.adapters.jdbc.WrappedConnection] (Transaction Reaper Worker 5) Cloosing a statement you left open, please do your own housekeeping:

java.lang.Throwable: STACKTRACE

seems like another transaction is running, preventing the workflow to start.

I appreciate your help!

Roi.

[sinhaa]

Roi,

    This error message : 

ARJUNA016053: Could not commit transaction

is a thrown by JBOSS. https://issues.jboss.org/browse/ENTMQ-509 . Apperently this is fixed in Jboss6.1 and above. WFA upgrded to JBOSS 7.1 from WFA2.1. So for any verison >=2.1 this shoudl not come. i.e. Ideally.

so the first question is: What is the WFA version the customer is using? 

There are are some tweaks which can be tried.

sinhaa

[RoiBeci]

Hi sinhaa,

WFA version is 3.0

[sinhaa]

Try to increase the JBoss transaction time-out. By default its set to 300 seconds, you can increase it to 500 or 800. But I don't know what is the impact of increasing it to extremely high numbers.

----

1. Open Windows Explorer and navigate to the following directory: WFA_install_location\WFA \bin

2. Double-click the ps.cmd file. A PowerShell command-line interface (CLI) prompt opens with Data ONTAP and WFA modules loaded in it.

3. At the prompt, enter the following Set-WfaConfig -Name TransactionTimeOut -Seconds NumericValue

Example:

Set-WfaConfig -Name TransactionTimeOut -Seconds 1000

4. When prompted, restart the WFA services.

sinhaa

[RoiBeci]

Thanks, will try and update.