PowerShell toolkit 3.2 slow command execution

[AndrewR]

Hi all,

I'm having some issues with PowerShell toolkit v3.2 and it being very slow running a command for the first time.  System is 2008 R2, PowerShell v2.0 (same in v4.0), WFA 3.0 and PowerShell Toolkit v3.2, this is connecting to a cDOT environment running ONTAP 8.2.2P1.

I've already done the certification revokation modification and have checked all DNS settings are correct and resolution is working.

The behaviour we see is as follows:

PS> measure-command { connect-nccontroller $sourcecluster -credential $cred }
Days : 0
Hours : 0
Minutes : 0
Seconds : 18
Milliseconds : 641
Ticks : 186419303
TotalDays : 0.000215763082175926
TotalHours : 0.00517831397222222
TotalMinutes : 0.310698838333333
TotalSeconds : 18.6419303
TotalMilliseconds : 18641.9303

PS> measure-command { connect-nccontroller $sourcecluster -credential $cred }
Days : 0
Hours : 0
Minutes : 0
Seconds : 1
Milliseconds : 810
Ticks : 18107036
TotalDays : 2.09572175925926E-05
TotalHours : 0.000502973222222222
TotalMinutes : 0.0301783933333333
TotalSeconds : 1.8107036
TotalMilliseconds : 1810.7036

This is mirrored by every command we run, the first time it takes ~20 seconds, then every time after that 1-4 seconds regardless of how many times we run the command.  If we close a re-launch PowerShell we see the same behavior again.

[PADMANTECH]

Hi Andrew,

Interesting problem you have there. I am using Toolkit version 3.2.1 and have not seen this problem but thought i might ask a few questions to help naror down the problem:

1. Do you see the same problem when using another host apart from the WFA server?

2. What athentication are you using, Local or AD? If using AD, have you tired just local?

3. Are you connecting to the Cluster or Node SVM?

4. What interface does the LIF sit on that you are conecting too? MTU? Flow Control?

5. Tried Connecting over HTTP instead of HTTPS?

PS> measure-command { connect-nccontroller $sourcecluster -credential $cred -HTTP}

I know you have already siad that you have checked the certification revokation modification but really sounds like the same issue and would double check the setting has been changed to:

[HKEY_USERS\S-1-5-18\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing] "State"=dword:00023e00

Kind regards,

Jason

[AndrewR]

Hi Jason,

Thanks for the prompt response, in answer to your questions:

1. Yes, we have tried two hosts so far and am just building a standalone non AD joined VM to test on that

2. Local auth, have even tried with cluster "admin" account

3. Cluster SVM

4. LIF sits on a LACP and VLAN port, I have tried this on all four controllers in the cluster and get the same.  Port settings are matched to switch (MTU 9000 no flow control)

5. It does seem quicker over HTTP rather than HTTPS, will check out HTTPS configuration

Have checked this setting in registry and matches what I'd expect

Andrew

[PADMANTECH]

Hi Andrew,

The only thing that i would change is the MTU. I don't think it would be the smoking gun but for managment commands a MTU of 1500 (Default) would be better suited to the cluster managment LIF.

Out of interest:

1. When you connect over SSH do you see any delay in connecting or sending commands?

2. Any firewall's inbetween Host and Controller? Hops (Tracert)?

Hope this helps.

- Jason

[AndrewR]

No firewalls in place, and SSH connectivity seems similar in speed to any other system

[markweber]

since it is the first command, could the delay be caused by loading the dataontap module? or are you explicitly running import-module dataontap already?

[PADMANTECH]

Hi Andrew,

Just had a thought about the MTU 9000 as you mentioned that you also had this set as 9000 on the switches. Can I ask if you have the MTU set to exactly 9000 on the switches? The MTU should be higher like 9216 on the switch as the switch needs to add on header information to the packet and if the packet is sent as 9000 and the switch MTU is 9000 the the packet will get truncated.

- Jason

[AndrewR]

Hi all, thanks for all the replies!

So we load the modules separately (import-module dataontap) and the MTU sizing on the switches in 9126 (this is managed by the networks team)

I have managed to resolve this, or at least appear to resolve this, by importing the SSL certificates from the filers into Trusted Root on the server running PowerShell commands from.

[mark_schuren]

I had the same issues in some environments.

Found out that another windows "feature" was causing this in most: auto-update of SSL root authorities and auto-update of "bad certificates". It's not only cert revocation lists that are checked by the OS by default.

Just try the resolution of VMware KB 2084025, although not directly related, this can solve a lot of issues with any kind of "SSL-connect-delay" on a box that has no or partial internet connectivity, esp. when it cannot get to download.windowsupdate.com.

Found the original solution here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2084025

HTH