Powershell toolkit 3.0 7-mode commands are extremely slow

[aham_team]

Hello all,

I have a certain powershell script working very fast (takes about 2 sec) for a c-mode cluster that is making some sort of a calculation on the filers luns and tre quotas.

the script needs to load all volumes and luns from a filer/cluster.

when trying to convert the script from c-mode cmdlets to 7-mode i encounterd a problem where trying to use Get-NaVol and Get-NaLun is extremely slow in a way that makes the script unusable.

actually the get-NaVol usually timeouts after 60 sec which is the timeout (no point in raising it becuase calculation needs to take max 1 min.)

the get-NaLun takes a whole lot of time too.

is there any known issue with toolkit 3.0? should i consider Downgrading/Upgrading the toolkit for better performance with 7-mode filers?

is there anything else i can do?

please note that all operations like trying to lower the cpu usage on the filer and other similar operations are not relevant because it is production.

thank you

alon

[rdenyer001]

Hi All,

I have the same problem with a new FAS2552  running 8.2.2  7-Mode

Toolkit version 3.2  rev65

All Filers  connect to via https 

Get-NaVol   to   7-mode  7.3   49  Volumes      942  milli seconds

Get-NaVol   to   7-moe   8.1.1  52  Volumes     265  milli seconds

Get-NaVol   to  7-mode 8.2.2    21 volumes      fails with message  

get-navol : Object reference not set to an instance of an object.
At line:1 char:19
+ measure-command {(get-navol).count}
+ ~~~~~~~~~
+ CategoryInfo : InvalidOperation: (DC-FAS12:NaController) [Get-NaVol], NullReferenceException
+ FullyQualifiedErrorId : ApiException,DataONTAP.PowerShell.SDK.Cmdlets.Volume.GetNaVol

Regards,

Richard

[tdimarzio]

Same problem here, guys.

Get-NaAggr always works fine.

Get-NaVol is the problematic cmdlet.

Running Data ONTAP 8.2.2P1 7-Mode on FAS6220's.

With PowerShell ToolKit 3.1, the error was:

"Get-NaVol : Connection to xxxxxxxx on port 443 for protocol HTTPS timed out"

With PowerShell ToolKit 3.2, the error is:

"Get-NaVol : Object reference not set to an instance of an object."

Tried setting

$global:CurrentNaController.TimeoutMsec = 120000
$global:CurrentNaController.RetryCount = 5

... no help.

The failure occurs with both RPC and HTTPS protocols.

When the failure occurs, systat indicates filer CPU utilization around 30 to 50%.

In the end, I had no choice but to work around the problem by modifying the schedule of the task to run at the time of the day where Get-NaVol does not seem to fail. This is/was not ideal, but as NetApp has not been able to provide a root-cause or fix/resolution, I had no choice.

Hoping NetApp fixes this in the next release of the DataONTAP PowerShell ToolKit.

- Tony

[mbeattie]

Hi Alon,

Are you using the "-Terse" Parameter in the Get-NaVol" CmdLet? This avoids enumerating all volume properties which can be time consuming and utilizes more resources (without using the -terse paramater i've seen the CmdLet take a long time or result in a "out-of-memory" error on busy systems with a lot of volumes).

SYNTAX
    Get-NaVol [[-Name] <String[]>] [-Aggregate <String>] [-Terse] [-Controller <NaController>] [<CommonParameters>]

    -Terse
        If specified, omit retrieving some volume attributes, including "RaidGroups" in the "Plexes" array, "SnapAutoDelete", and "AutoSize".  Use wh
        en retrieving a large number of volumes and these attributes are not necessary.

        Required?                    false
        Position?                    named
        Default value
        Accept pipeline input?       false
        Accept wildcard characters?  false

Hope that helps

/matt

[sjallabais]

I have this same problem.

FlexPod with B200M3 blades and FAS8060 running 7-mode 8.2.1 with lots of CPU & bandwith, still Get-NaVol times-out.

At the moment I am not able to get get-navol to work at all even when triple the -Timeout value. The problem is maily with Get-NaVol, other applets seems to work.

PS> Get-NaToolkitVersion

Major Minor Build Revision
----- ----- ----- --------
3     1     1     181

PS>Connect-NaController nnnn -Credential (Get-Credential) -HTTPS

Name Address Ontapi Version
---- ------- ------ -------
nnnn n.n.n.n 1.21 NetApp Release 8.2.1 7-Mode: Fri Mar 21 14:48:58 PDT 2014

PS>Get-NaVol

Get-NaVol : Connection to nnnn on port 443 for protocol HTTPS timed out.
At line:1 char:10
+ get-navol <<<<
+ CategoryInfo : InvalidOperation: (psa1na01:NaController) [Get-NaVol], NaConnectionTimeoutException
+ FullyQualifiedErrorId : ApiException,DataONTAP.PowerShell.SDK.Cmdlets.Volume.GetNaVol

/Sjalla

[aham_team]

i should have mentioned that i am running the script on 2 ha-pairs that are almost identical considering cpu, snapmirror relations and capacity.

i am running ontap 8.1.2p4 and sysstat -m averages around the 30%-50% for all filers.

even so I am aware that the cpu is having a hard time on all of the filers because we have many snapmirror relationships that are updating every hour,so lots of kahuna operations are taking place.. so cpu is lower for normal operations obviously..

also the 7 mode filers have many more volumes than the c-mode (around 200 volumes per filer).

all of that said, still only trying to run get-NaVol should not take more than a minute..

anyone here has an environment of the same size/bigger and has normal times running commands?

thanks for the attention

[markweber]

get-navol with 129 volumes in under 2 seconds

PS E:\> (get-navol).count

129

PS E:\> measure-command {(get-navol).count}

Days              : 0

Hours             : 0

Minutes           : 0

Seconds           : 1

Milliseconds      : 844

Ticks             : 18449009

TotalDays         : 2.13530196759259E-05

TotalHours        : 0.000512472472222222

TotalMinutes      : 0.0307483483333333

TotalSeconds      : 1.8449009

TotalMilliseconds : 1844.9009

is it just get-navol that is taking 60+ seconds?

are you using the pipeline in your script?

[JBIDDLE2212]

What version of 7-mode are you running?

[JGPSHNTAP]

I'm running 3.0 without issues. What's the performance profile of the

filer look like?

What's sysstat -m 1 look like, is the filer busy?