Slow workflow execution

[francoisbnc]

I wonder why workflow execution is so slow at the first line of a command 2 minutes in this exemple with WFA 2.2 (same behavior with 2.1 version) without LDAP auth.

11:45:36.381 INFO  [Create volume] ### Command 'Create volume' ###

11:47:12.276 INFO  [Create volume] Executing command: ./Create_volume8173598297431862987.ps1 -AggregateName 'sas_t2_002' -Array '10.139.32.11' -Size '1000' -VolumeGuarantee 'none' -VolumeName 'test'

11:47:12.744 INFO  [Create volume] Get-WfaCredentials -Host 10.139.32.11

11:47:12.900 INFO  [Create volume] Credentials successfully provided for '10.139.32.11'

11:47:13.103 INFO  [Create volume] Connect-NaController (with credentials) -Name 10.139.32.11 -Timeout 60000 -ErrorAction Stop

11:47:24.662 INFO  [Create volume] Connected to controller

11:47:24.927 INFO  [Create volume] Creating a new volume - name 'test' , aggregate name 'sas_t2_002' , size '1000m' , volume guarantee 'none'

11:47:34.147 INFO  [Create volume] Command completed, took 117735 milliseconds

My productive wotkflows take more than 30 minutes just for cloning 5 volumes and mapping luns inside to igroup.

Any suggestions appreciate.

François

[adaikkap]

Hi Francois,

     Happy to see you back. Does preview or planning as well take the same time or is it just the execution alone ?

How may workflows were executed in parallel ?

Is it possible to share the workflow so that we can try it to see what is causing the problem ?

Regards

adai

[sinhaa]

I've seen it before but it was never consistently reproducible that I could draw some conclusion. After seeing your post, I just did 2 rounds of Test Create Volume on cm_storage command and I got it in 2nd time, first one was lightening quick in comparison. Is it consistent for you Francois? What is your browser? Are you running your server on non-default ports?

06:52:54.860 INFO  [Create volume] ### Command 'Create volume' ###

06:54:51.004 INFO  [Create volume] Executing command: ./Create_volume2491088665744029825.ps1 -AggregateName 'Test_aggregate' -SpaceGuarantee 'none' -Cluster '10.226.252.28' -Size '20' -VserverName 'vs1' -VolumeName 'myVol3003'

06:54:52.160 INFO  [Create volume] Get-WfaCredentials -Host 10.226.252.28

06:54:57.348 INFO  [Create volume] Credentials successfully provided for '10.226.252.28'

06:54:57.676 INFO  [Create volume] Connect-Controller -Type CLUSTER -Name 10.226.252.28 -Credential System.Management.Automation.PSCredential -Vserver  -Timeout 60000

06:54:58.567 INFO  [Create volume] Connect-NcController (with credentials) -Name 10.226.252.28 -Timeout 60000 -ErrorAction Stop

06:55:01.785 INFO  [Create volume] Connected to cluster node

06:55:02.145 INFO  [Create volume] Creating a new volume: New-NcVol -ErrorAction Stop -name myVol3003 -Aggregate Test_aggregate  -SpaceReserve none -State online -Size 20m -VserverContext vs1 -Type rw -JunctionPath /myVol3003

06:55:04.020 INFO  [Create volume] Configuring volume autosize attributes: Set-NcVolAutosize -ErrorAction Stop -name myVol3003 -VserverContext vs1 -Mode off

06:55:04.614 INFO  [Create volume] Command completed, took 18172 milliseconds

I've seen it with at times with commands doesn't even do anything real.

Working to dig the cause. Will update you.

sinhaa

[francoisbnc]

Hello Adai,

Happy as well you take care of WFA.

The preview goes fast and I just ran an workflow alone in a fresh installed WFA 2.2RC1.

Workflow is pretty simple, I just create a volume with certified "create volume" command . Every command takes 2 minutes at the first line.

Regards,

François

[francoisbnc]

Hi sinhaa,

I use chrome v.32 and server running on standard port. There are no difference using from ie.

Thanks

[adaikkap]

Thanks Francois,

     In the mean time pls open a case with NetApp Support for the same. Use a controller serial number along with the WFA System ID and ASUP.

Regards

adai

[abhit]

Hi Francois:

Couple of questions.

Where is time consumed more, is it that the command is taking more time to execute in the filer/cluster?

To verify you can run command line from the machine where WFA is running.

Try to connect to the cluster using powershell command or run a command from WFA and check how much time it is taking?

It may be an issue with the network or with the cluster also.

You have said that commands in the 1st row is taking more time?

In the 2nd row is it faster than 1st row?

Regards

Abhi

[francoisbnc]

Hello Abbit,

I don't think  that is relayed to PS or Network issue

PS commands from WFA itself run pretty fast.

In this exemple New-NaVol -Name test2 -Aggregate aggr0 -Size 1000m -SpaceReserve none that tooks some seconds.

Row is just the first line of in a execution , nothing to do with loops.

06:52:54.860 INFO  [Create volume] ### Command 'Create volume' ###

Regards,

François

[abhit]

Thanks for your prompt reply. It will be good to open a case with the logs.

We need to investigate why it is happening.

Regards

Abhi

[francoisbnc]

2004844286

Logs are uploaded to the case.

Thanks

[aparker]

Any update to this?   I'm seeing the same thing at a customer with 2.2.

[aparker]

The odd thing is that even the no-op (cluster mode) command takes almost 400 seconds in execute mode.  But in preview mode it runs as you would expect.

[mgoddard]

Have you tried running Powershell itself and time it, then load the "profile.ps1" in the <WFA>/PoSH folder and time that. (cd <wfa>\PoSH ;; .\profile.ps1)

I've found installing additional powershell extensions on the WFA machine, when there were configuration issues, or even the built-in ones, can cause long delays to launching and some logging commands.

Each time a command is run, even a No-Op, a new powershell.exe instance is launched and loads the profile that loads additional modules. If you run a lot of commands and there's a delay there, that time can add up.

Michael.

[sinhaa]

Adding to Michael's reply, can you run the following and send the output?

1. Open cmd prompt

2. Run command systeminfo and send me the output here or at sinhaa at netapp dot com

The thing is this issue is perhaps a JBOSS7 issue and JBOSS7 is what WFA2.1 onwards is using. We can give you a way to reduce the long delays.

warm regards,

Abhishek

[francoisbnc]

I opened a case, no answers for the moment.

Regards,

francois

[sinhaa]

Hi Francois and aparker,

DISCLAIMER: This solution is still under test and not guaranteed to work or not to cause any other issues. But its looks harmless to me.

      Please try the following JBoss config modification. Thanks to Sandeep for working on it.

1. Open file standalone-full.xml located at WFA\jboss\standalone\configuration for editing using some text editor like notepad++

2. Find the line : <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="http-executor"/>

and for HTTPS

     <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" executor="http-executor">

3. Remove the executor="http-executor" from them

4. Save the file.

5. Restart NetApp WFA Server service and wait for the service to be up.

6. Retest your command or workflow execution.

Let me know how this worked for you.

warm regards

sinhaa