Slow workflow execution

[francoisbnc]

I wonder why workflow execution is so slow at the first line of a command 2 minutes in this exemple with WFA 2.2 (same behavior with 2.1 version) without LDAP auth.

11:45:36.381 INFO  [Create volume] ### Command 'Create volume' ###

11:47:12.276 INFO  [Create volume] Executing command: ./Create_volume8173598297431862987.ps1 -AggregateName 'sas_t2_002' -Array '10.139.32.11' -Size '1000' -VolumeGuarantee 'none' -VolumeName 'test'

11:47:12.744 INFO  [Create volume] Get-WfaCredentials -Host 10.139.32.11

11:47:12.900 INFO  [Create volume] Credentials successfully provided for '10.139.32.11'

11:47:13.103 INFO  [Create volume] Connect-NaController (with credentials) -Name 10.139.32.11 -Timeout 60000 -ErrorAction Stop

11:47:24.662 INFO  [Create volume] Connected to controller

11:47:24.927 INFO  [Create volume] Creating a new volume - name 'test' , aggregate name 'sas_t2_002' , size '1000m' , volume guarantee 'none'

11:47:34.147 INFO  [Create volume] Command completed, took 117735 milliseconds

My productive wotkflows take more than 30 minutes just for cloning 5 volumes and mapping luns inside to igroup.

Any suggestions appreciate.

François

[jdowning]

Aparker is working with me onsite.  We tried the fix but that didn't help. 

The import of the modules is quick.

Here is the systeminfo from aparker's VM:

Host Name:                

OS Name:                   Microsoftr Windows Serverr 2008 Enterprise

OS Version:                6.0.6002 Service Pack 2 Build 6002

OS Manufacturer:           Microsoft Corporation

OS Configuration:          Standalone Server

OS Build Type:             Multiprocessor Free

Registered Owner:          Windows User

Registered Organization:

Product ID:                55041-381-2247303-76321

Original Install Date:     1/27/2014, 3:38:26 PM

System Boot Time:          2/19/2014, 8:14:06 AM

System Manufacturer:       VMware, Inc.

System Model:              VMware Virtual Platform

System Type:               x64-based PC

Processor(s):              2 Processor(s) Installed.

                           [01]: Intel64 Family 6 Model 29 Stepping 1 GenuineIntel ~2400 Mhz

                           [02]: Intel64 Family 6 Model 29 Stepping 1 GenuineIntel ~2400 Mhz

BIOS Version:              Phoenix Technologies LTD 6.00, 1/15/2013

Windows Directory:         C:\Windows

System Directory:          C:\Windows\system32

Boot Device:               \Device\HarddiskVolume1

System Locale:             en-us;English (United States)

Input Locale:              en-us;English (United States)

Time Zone:                 (GMT-08:00) Pacific Time (US & Canada)

Total Physical Memory:     8,191 MB

Available Physical Memory: 5,887 MB

Page File: Max Size:       16,429 MB

Page File: Available:      13,908 MB

Page File: In Use:         2,521 MB

Page File Location(s):     C:\pagefile.sys

Domain:                    WORKGROUP

Logon Server:             

Hotfix(s):                 2 Hotfix(s) Installed.

                           [01]: KB955430

                           [02]: KB968930

Network Card(s):           1 NIC(s) Installed.

                           [01]: Intel(R) PRO/1000 MT Network Connection

                                 Connection Name: Local Area Connection

                                 DHCP Enabled:    No

                                 IP address(es)

[aparker]

To follow-up on this, we may be seeing some different issues.    What we are seeing at our customer (a major US bank) is that it takes about 5 MINUTES for each WFA command block to start execution.      The server being used is pretty much cut off from the outside world, so I thought it may have had something to do with checking for certificate revocation.     But we tried disabling that by various methods, including as shown in this article:  http://velemental.com/2012/01/01/optimizing-the-execution-of-powershell-and-powercli/

Nothing made a difference.   If we open a powershell window on the server, it seems to open in a normal amount of time.    If we load the ONTAP module, it loads normally and quickly.   So we don't see the issue by using the powershell command window.

Our onsite guy at the customer is going to look into getting the server more outside access or adding it to the domain (which is currently isn't on).

-Alan Parker

[sinhaa]

Francois, Parker,

     I don't know what else can I suggest you to try. I just can't think of anything as of now.

Maybe you can try to upgrade the WFA server to .NET 4.5 version and if possible to Powershell 3.0 and see if it helps.

-sinhaa

[jdowning]

Upgraded to both...still doesn't help.

[jdowning]

Upgrade to 2008R2 earlier today and our 5 minute wait seemed to have  disappeared.  Not sure what fixed because the upgrade kept all the  settings and it wasn't a clean install.

[francoisbnc]

Hi sinhaa,

Do you know if some kind of certificates check is used for ONTAP module?

I traced communication during load of module, and process tried to access 50.63.243.228 port 80.

3573          184.174793000          10.140.16.45          50.63.243.228          TCP          66          52724 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

Following this fact,  I configured proxy in Server.

11:18:14.316 INFO  [Remove volume] ### Command 'Remove volume' ###

11:18:36.499 INFO  [Remove volume] Executing command: ./Remove_volume6766461420913838121.ps1 -Array '10.139.32.11' -VolumeName 'test'

11:18:36.655 INFO  [Remove volume] Get-WfaCredentials -Host 10.139.32.11

11:18:37.014 INFO  [Remove volume] Credentials successfully provided for '10.139.32.11'

11:18:37.139 INFO  [Remove volume] Connect-NaController (with credentials) -Name 10.139.32.11 -Timeout 60000 -ErrorAction Stop

11:18:42.536 INFO  [Remove volume] Connected to controller

11:18:44.861 INFO  [Remove volume] Removing all volume quotas, volume name: test

Workflow speed is miraculous good now

[aparker]

How much of a delay did you have before?

-Alan

[francoisbnc]

2 minutes

Francois

[sinhaa]

I'm glad it has  finally worked for both of you.

[francoisbnc]

sinhaa,

The idea was not to configure proxy settings in WFA server, can you tell me more about this unwanted traffic.

francois

[sinhaa]

Francois,

         Honestly, I don't know the answer right now. I'll post it once I find out.

[francoisbnc]

Regarding my case engineering opened a new bug for this issue.

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=802749

[francoisbnc]

Hi Sinaa,

Was there a change on Netappp side? I have to change to v3.0 because compatibility with ocum 6.2 and the slowdown problem reoccurs, event certificate unchecked.

I tried to play with bug solution 802749 and standalone-full.xml. When I removed lines, the WFA server service won't to start any more.

For HTTP: <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="http-executor"/>
For HTTPS: <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" executor="http-executor">

François

[abhit]

Francois:

Thanks for your reply.

Are you still facing the slow execution problem with WFA 3.0?

Regards

Abhi

[francoisbnc]

Hello abhit,

Yes the problem reappears since ver 3.0, the workaround worked for 2.x version but not works in the last version.

An example for a simple clone take 98 sec to complete., before in 10 sec.

09:36:01.626 INFO [ITS Clone volume on Last Snapshot] ### Command 'ITS Clone volume on Last Snapshot' ### 09:36:03.048 INFO [ITS Clone volume on Last Snapshot] Get-WfaCredentials -Host 10.139.32.75

09:36:03.126 INFO [ITS Clone volume on Last Snapshot] Credentials successfully provided for '10.139.32.75'

09:36:03.235 INFO [ITS Clone volume on Last Snapshot] Connect-NaController (with credentials) -Name 10.139.32.75 -Timeout 60000 -ErrorAction Stop -HTTP 09:36:04.188 INFO [ITS Clone volume on Last Snapshot] Connected to controller 09:36:54.345 INFO [ITS Clone volume on Last Snapshot] Create a clone for Volume sge_fcp_mssql_gdc01467_01 , named sge_fcp_mssql_gdc01840_01_clone

09:37:40.049 INFO [ITS Clone volume on Last Snapshot] Command completed, took 98423 milliseconds

François

[abhit]

Hi Francois:

Sorry to hear that the issue is reappearing.

We have opened a bug in our tracking system.

The number is 909331.

We have not observed this internally, so we may ask you to share

some more information if we require.

Regards

Abhi

[ag]

Hi Franc,

This is a custom command. Is it possible to provide the custom command? This will help with RCA.

[francoisbnc]

Hi,

I did some new tests again and it'seems the speed is back to normal now,  probably the FAS6240 has to much activities during the first of test.

WFA is not involved in the slowness.

14:38:48.051 INFO [Clone volume] ### Command 'Clone volume' ### 14:38:49.395 INFO [Clone volume] Using cached controller connection 14:38:51.098 INFO [Clone volume] Create a clone for Volume vol_source , named vol_source_clone 14:38:52.145 INFO [Clone volume] Command completed, took 4094 milliseconds

Apologize, my mistake.

François

[sinhaa]

Francois,

          I've not tried removing the http-executors in wfa3.0. If that is the case, I'm not sure.

 I can suggest another thing which has worked for some others.

Upgrade your .NET version to 4.5 or later. Then after run the .NET repair tool from here.

http://www.microsoft.com/en-us/download/details.aspx?id=30135

Do it even if the .NET upgrade went fine and didn't throw any errors. Just download the repair tool and run it

sinhaa

[francoisbnc]

Hello sinhaa,

Yes I did, but after removing this key, I cannot start WFA server service any more.

For .NET I run WFA on W2K12R2 so 4.5 is installed as feature already.

François