WFA Quota Mgmt Pack for cDOT 8.3.x Issues

[RoiBeci]

Hi,

I have recently used the quota mgmt pack for clustered OnTap (8.3.2P1 system) and found the following issues on the Data Source Acquisition script:

1. Quotas with high disk limit (3TB and bigger) could not be set on the 'DiskLimit' variable, returning error regarding his type integer (number was too big)

2. The API command 'quota-list-entries-iter' did not returned quota rules from all vservers, but only one.

I've changed the data source acquisition script, and attached here so you can check it yourself.

<#
The script collects the following information from a Clustered ONTAP cluster:
1. Quota Policy
2. Quota Rule


It requires that credentials be set for the relevant cluster (it will ignore the credentials provided in the data source)

16.05.2016 - Roi Becidan
 - Changed quota variables from "int" to "double"
 - Changed API $quota_entry to powershell cmdlet "Get-NcQuota"
#>

$quotaPolicyFile = "./Quota_Policy.csv"
$quotaRuleFile = "./Quota_Rule.csv"


New-Item -Path $quotaPolicyFile -type file -force
New-Item -Path $quotaRuleFile -type file -force


# Ensure that dates are always returned in English
[System.Threading.Thread]::CurrentThread.CurrentCulture="en-US"

# Start stopwatch
$elapsedTime = [System.Diagnostics.Stopwatch]::StartNew()

# Get cluster information from data source
$connectionInfo = @{};
Try {
    $connectionInfo["host"] = Get-WfaRestParameter "host"
    $connectionInfo["port"] = Get-WfaRestParameter "port"
    $connectionInfo["credentials"] = Get-WfaCredentials
} catch [System.Exception] {
    $error = "Error getting data source credentials: $($_.Exception)"
    Get-WFALogger -message $error -Error
    Throw "Error getting data source credentials. Please see the log file for more details"
}

# Connect to NetApp cDOT cluster
try {
    Connect-WfaCluster $connectionInfo["host"]
} catch [Exception] {
    $error = "Error connecting to cluster: $($_.Exception)"
    Get-WFALogger -message $error -Error
    throw "Error connecting to cluster. Please see the log file for more details"
} 

# Get Cluster name
try {
    $cluster_name = (Get-NcCluster).ClusterName
} catch [Exception] {
    $error = "Error getting NetApp cDOT cluster name: $($_.Exception)"
    Get-WFALogger -message $error -Error
    throw "Error getting NetApp cDOT cluster name. Please see the log file for more details"
}

#Get Credential
 $credentials = Get-WfaCredentials -Host $connectionInfo["host"]

# Object to run Zapi
Try {
	$ZapiServer = New-WfaZapiServer -Host $connectionInfo["host"] -Type FILER -Credentials  $credentials 
	
}catch [Exception] {
    $error = "Error connect to zapi server : $($_.Exception)"
    Get-WFALogger -message $error -Error
    throw "Error connecting to ZAPI Server. Please see the log file for more details"
} 
$Host = $connectionInfo["host"]

# ============================================
# Get Quota Rules and write to flat file
# ============================================
try {
    
    $quota_entry =  Get-NcQuota
     	
    #  Get-WFALogger -Info -message $results.toString()

    
    if($quota_entry -ne $null)
    {
	
	
	foreach ($quota in $quota_entry){
		$UserName = ""
		$UserMapping = 'N'
		[double]$DiskLimit = 0
		[double]$FileLimit = 0
		[double]$SoftFileLimit = 0
		[double]$SoftDiskLimit = 0
		[double]$Threshold = 0
		$Volume = $quota.volume
		$Type = $quota.QuotaType
		$Qtree = $quota.qtree
		$Policy = $quota.policy
		$Vserver =$quota.vserver
		$Target = $quota.QuotaTarget

		if ( $quota.DiskLimit -ne "-" -And $quota.DiskLimit -ne "") {
			$DiskLimit = $quota.DiskLimit
		}
		if ( $quota.FileLimit -ne "-" -And  $quota.FileLimit -ne ""){
			$FileLimit = $quota.FileLimit
		}
		if ($quota.SoftFileLimit -ne "-"  -And $quota.SoftFileLimit -ne ""){
			$SoftFileLimit = $quota.SoftFileLimit
		}
		if ($quota.SoftDiskLimit -ne "-" -And $quota.SoftDiskLimit -ne ""){
			$SoftDiskLimit = $quota.SoftDiskLimit
		}
		if ($quota.threshold -ne "-" -And $quota.threshold -ne "") {
			$Threshold = $quota.threshold
		}
		
		if ($Type -eq 'user'){
			$Target = $Target.replace("\","\\")
			if ($quota.PerformUserMapping -eq 'false'){
				$UserMapping = 'N'
			}
			else{ $UserMapping = 'Y'}
			$UserName= $Target;		
		}	
		if ($Qtree -eq ''){	$Qtree = '""'}
		Add-Content $quotaRuleFile ([byte[]][char[]] "\N`t$Host`t$Vserver`t$Volume`t$Target`t$Qtree`t$Policy`t$Type`t$DiskLimit`t$FileLimit`t$SoftFileLimit`t$SoftDiskLimit`t$Threshold`t$UserName`t$UserMapping`n") -Encoding Byte
	}
    }    
} catch [Exception] {
    $error = "Error getting quota rules: $($_.Exception)"
    Get-WFALogger -message $error -Error
    throw "Error getting quota rules. Please see the log file for more details"
}

# ============================================
# Get Quota Policies and write to flat file
# ============================================
try {
       
    [NetApp.Manage.NaElement] $naElement = New-Object NetApp.Manage.naElement("quota-policy-get-iter")
        
    [NetApp.Manage.NaElement] $results = $ZapiServer.InvokeElem($naElement)

   $numRecords = $results.GetChildIntValue("num-records", 0)
    if($numRecords -gt 0)
    {
		[System.Collections.IList] $policy_info =  $results.GetChildByName("attributes-list").GetChildren()
		
		foreach ($policy in $policy_info){
			$PolicyName = $policy.GetChildContent("policy-name")
			$PolicyVserver = $policy.GetChildContent("vserver")	
			
		Add-Content $quotaPolicyFile ([byte[]][char[]] "\N`t$Host`t$PolicyVserver`t$PolicyName`n") -Encoding Byte
        # This is required to ensure that the output file is UNIX encoded, without which MySQL's LOAD DATA
        # command does not work
		}
    }
   else{
	Get-WFALogger -message "Non of the quota policy is assigned." -Info
   }
} catch [Exception] {
    $error = "Error getting quota policy information: $($_.Exception)"
    Get-WFALogger -message $error -Error
    throw "Error getting quota policy information. Please see the log file for more details"
}
Get-WFALogger -message "Cluster data acquisition completed in $($elapsedTime.Elapsed.ToString())" -Info

Roi Becidan.

[pavand]

Hi Roi,

The new version (1.2) of the quota mgmt pack has this fix for Higher disk limit, Please download the latest version.

We are working on the second issue, it will get fix in next version of quota pack.

Regards,

Pavan

[sinhaa · Accepted solution]

@RoiBeci

- Changed quota variables from "int" to "double"

 Changing Datatype maybe work in this case but I don't think this is right. I would suggest to use [int64] Data type.

 - Changed API $quota_entry to powershell cmdlet "Get-NcQuota"

API 'quota-list-entries-iter' does return all the quota rules. The cmdlet Get-NcQuota internally calls this API itself, so its not different. The problem is with how the API was called. API response has a next-tag if the returned list is more than default(20). So the code need to iterate using this next tag to get the next set of values. The code wan't doing it and hence all the results were not returned.

The new version shall have this fix.

sinhaa

[pavand]

Hi Roi,

New version of Quota management(1.3.0) pack is uploaded in store, It has fix for not listing all quota policies.

Regards,

Pavan

[dkorns]

LAB Setup:

 - Win2012R2

 - WFA 3.1P2

 - Quota Mgmt Pack 1.3 

Just testing Quota Pack v1.3 and getting this error from 'Set Quota' command step:

    Can't return outside a subroutine at ./Set_quota3818132062543445871.pl line 103, <STDIN> line 1.

Pulling the Perl code into an editor with line numbers shows this 'return' on line 103:

# Set Quota
#----------------------------------------------------------------------
$wfa_util->sendLog('INFO', 'Set Quota');
eval {

$server->quota_set_entry(%input_hash);

};
if($@)
{
$wfa_util->sendLog('ERROR', 'Unable to set quota for : ' .$QuotaTarget.'Error -'.$@);
return;  <<<=== line 103
}
else {
$wfa_util->sendLog('INFO', 'Quota set is successful');

}

#--------------------------------------------------------------------------

Any ideas? I'm executing the unmodified Workflow to set a quota on an existing qtree. Ive attached screenshot of my user inputs.

Even if I've made an improper run in some way, should I be getting this sort of error? Not a quick Perl reader or expert. 

[dkorns]

Attaching the Quota pack Workflow 'Execution Plan' screenshot for above error

[dkorns]

Figured it out, but there may still be a bug or two to report here.

The command 'Set Quota' was failing because it was not being passed a quota policy name. I did notice that the workflow UI didn't present any quota policies in the dropdown list which was odd. But THEN I noticed  that another cluster DID show values in the quota policy name dropdown list and Set Quota ran fne against that cluster. I confirmed both cluster and vservers did have the 'default' policies in place. 

So I looked at the cm_storage_quota.quota_policy table (via MySQL WB) and noticed the 'cluster' name field had the simple cluster name on the cluster that was working and the cluster that was not working had a FQDN, or hq-stor.acme.com.

That I was sure was making the UI SQL fail to find a match. Then I noticed that I had inadvertently used a FQDN in the Quota Data Source for one cluster (the one failing) and not for the other one (which was working).

 

Changed them both to simple names, Reset the Scheme, re-acquired and things are working. So maybe two issues or bugs identified here:

   1) The Quota 'Data Source' should probably not place fqdn's into the cluster name field .. because I believe they are always simple names in the cm_storage cluster table (though not sure)

   2) It looks like the 'Set Quota' has a 'return' statement that should only be used inside a perl subroutine which I don't think exists (though I'm not a perl expert) 

[sinhaa]

@dkorns

Thanks for pointing this out. WFA team will look into this.

sinhaa

[ArjonStellinga]

Hi,

Are you still looking into this, as we have updated to the last WFA version, and still have this issue?

Regards,

Arjon G. Stellinga

WFA version: 4.0.0.0.0

WFA Build    : 3858982

Quota Management package: 1.3.0 (released on 2016-07-11)