Subscribe

Compiling quota total commits to volume to then select Volume

We are working on workflows where the end state goal is to have WFA do the compiling of quotas on our utility volumes to check for oversubscription and then chose the proper volume.

 

Has anyone done something similiar to this?  If so, please share experiences.

 

The thought process is pull Volumes used size, total size and committed quotas and then go from there.

 

Any insight would be great.

 

Thanks

Re: Compiling quota total commits to volume to then select Volume

@JGPSHNTAP

 

You should have filters in place that can do most of this for you already. It's not clear what decision you are trying to get to though. Pick a volume for what? You mention quotas so are you talking for deploying new exports with user quotas? 

 

We are using filters to determine where to deploy new volumes based on aggregate utililzation and overcommitement %, happy to help if I can just need to clarify what you are after. 

 

-Nick

Re: Compiling quota total commits to volume to then select Volume

Nick,

 

I should have been more clear, i apologize if it was ambiguous.

 

So, we are doing deployment of what we are terming "general purpose" requests under 3TB to a dedicated utility volume.

 

So for example, the cluster will have 4 utility volumes.  As a request comes in, WFA calculates all the quotas committed, vol size used and then makes decisions on which volume to provision a  new qtree with new quota.  That is where we are heading.. Our DEV guys have made lots of progress on this in the past couple of days, by two methods, java or sql. 

 

The challenge we are running into now is with the reserveation database.  

 

Anything over 3TB, will get a dedicated volume.  The reason why we chose this design method is because of cluster vol limits and we realized that our provisioning was 80/20 rule.  Meaning 80% of our vols were <3TB.  Our Util vol starts at 10TB with autogrow max at 20TB.  

 

We also feel this will help out more with SVM-DR and mirror'd volume limits.

 

I will gladly take some feedback and any info on forcing our commands into the reservation database would be great.

 

Thanks

 

Re: Compiling quota total commits to volume to then select Volume

I've done similar queries in the past - I can send an example query over in the AM that might help.

Re: Compiling quota total commits to volume to then select Volume

^^

 

That would be great.  We have working queries, but we are not happy with the timesync of WFA-OCUM and reservation data that isn't updating, so we are hoping to have a query that updates the reservation cache.

Re: Compiling quota total commits to volume to then select Volume

JGPSHNTAP - 

 

Sure, there can always be a few minute delay between WFA and OCUM that can make things a little tricky if you're doing frequent updates.  

 

Is it just the quota create command that you need working reservations?  I can check if I have something written up for this already or not.

Re: Compiling quota total commits to volume to then select Volume

I found a custom command I wrote awhile ago to create new qtree disklimit quotas.  The reservation and verification code seems to work from some quick testing I did this morning, probably something you'll want to test further though.

 

Reservation Code

 

UPDATE
    cm_storage.qtree, cm_storage.volume, cm_storage.vserver, cm_storage.cluster
SET
    qtree.disk_limit_mb = ('${QuotaSizeGB}' * 1024)
WHERE
    qtree.volume_id = volume.id
    AND volume.vserver_id = vserver.id
    AND vserver.cluster_id = cluster.id
    AND qtree.name = '${Qtree}'
    AND volume.name = '${Volume}'
    AND vserver.name = '${Vserver}'
    AND (
        cluster.name='${Cluster}'
        OR cluster.primary_address='${Cluster}'
    )

 

Verification Code

 

SELECT
    qtree.id
FROM
    cm_storage.qtree,
    cm_storage.volume,
    cm_storage.vserver,
    cm_storage.cluster
WHERE
    qtree.disk_limit_mb = (
        '${QuotaSizeGB}' * 1024
    )
    AND qtree.volume_id = volume.id
    AND volume.vserver_id = vserver.id
    AND vserver.cluster_id = cluster.id
    AND qtree.name = '${Qtree}'
    AND volume.name = '${Volume}'
    AND vserver.name = '${Vserver}'
    AND (
        cluster.name='${Cluster}'
        OR cluster.primary_address='${Cluster}'
    )

Let me know if you need any help with the custom command or anything here.  Happy to help take a look.

Re: Compiling quota total commits to volume to then select Volume

[ Edited ]

Hi

 

i believe i have exactly what you need, but in PS not as a query, i gave up on WFA for my project..... you can maybe try and take the logic... . it will order the volumes for you for where you have the most available space based on Quota allocation. volume free space, aggregate free space and aggregate on the snapmirror destination (if exists) free space. you can later filter the volumes you don't want to show (for example all the root ones, volumes that have the quota full etc)

 

Good luck

Gidi

 

Exemple:

 

These are the volumes we found fit to your reqest:

ID Dedupe SpaceBottleneck SpaceBottleneckname
-- ------ --------------- -------------------
 0   True 9.5TB           VolumeFreeSpace    
 1   True 9.49TB          VolumeFreeSpace    
 2   True 9.47TB          VolumeFreeSpace    
 3   True 9.39TB          VolumeFreeSpace    
 4   True 9.25TB          AggrFreeSpace     
 5   True 8.93TB          TotalMinusQuota    
 6   True 8.46TB          TotalMinusQuota    
 7   True 8TB             TotalMinusQuota    
 8   True 7.79TB          DRAggrFreeSpace     
 9   True 7.72TB          TotalMinusQuota    
10   True 7TB             TotalMinusQuota    
11   True 6.22TB          TotalMinusQuota    
12   True 5.48TB          TotalMinusQuota    
13   True 5TB             TotalMinusQuota    
14   True 3.88TB          VolumeFreeSpace    
15   True 3.48TB          TotalMinusQuota    
16   True 2.8TB           TotalMinusQuota    
17   True 667.8GB         TotalMinusQuota    
22   True -735846.4MB     TotalMinusQuota    



please select a number from the list: 1
Selection made:

Name                      State       TotalSize  Used  Available Dedupe Aggregate                 Vserver                                                                                                    
----                      -----       ---------  ----  --------- ------ ---------                 -------                                                                                                    
Test_vol          online        10.0 TB    5%     9.5 TB  True  Test_Aggr         Test_SVM                                                                                                

 

 

 

Code:

 

 

 

<# Created by Gidi Marcus FEB 2018 for http://community.netapp.com/t5/OnCommand-Storage-Management-Software-Discussions/Compiling-quota-total-commits-to-volume-to-then-select-Volume/m-p/137922#M25017
 Tested on:
 $CdotClustersSessions | select -First 1 Version; Get-Module DataONTAP | select version;$PSVersionTable.PSVersion.ToString();(Get-WmiObject -class Win32_OperatingSystem).Caption
Version                                           
-------                                           
NetApp Release 9.1P6: Tue Jun 20 21:52:46 UTC 2017
4.1.0                                             
5.1.14393.1737
Microsoft Windows Server 2016 Standard
#>

Function Convert-BytesToSize
{[CmdletBinding()]
    Param([parameter(Mandatory=$False,Position=0)][int64]$Size)
    Switch ($Size){
        {$Size -gt 1TB}{
            $NewSize = “$([math]::Round(($Size / 1TB),2))TB”
            Break}
        {$Size -gt 1GB}{
            $NewSize = “$([math]::Round(($Size / 1GB),2))GB”
            Break}
        Default{
            $NewSize = “$([math]::Round(($Size / 1MB),2))MB”
            Break}
}Return $NewSize}

    
Function Select-FromOptions
{
    [CmdletBinding()]
    Param([parameter(Mandatory=$True,Position=0)]$OptionsList)
    While (-not $OptionSelected){
    $OptionSelected = $OptionsList[(Read-Host "please select a number from the list")]
    if ($OptionSelected)
        {
        Write-Host -ForegroundColor "Green" "Selection made:"
        Write-Host ($OptionSelected | Format-Table | Out-String)
        break;
        }
    else{Write-Warning "No valid selection made!, Please try again"}}
    Return $OptionSelected
}


function Get-CdotVars
{
    [CmdletBinding()]
    param($CdotClustersSessionss)
    $Results = @{}
    $Results.Aggregates = Get-NcAggr -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Volumes = Get-NcVol -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Vservers = Get-NcVserver -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Qtrees = Get-Ncqtree -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Quota = Get-NcQuota -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Quotapolicy = Get-NcQuotapolicy -Controller $CdotClustersSessions -ErrorVariable errors
    $Results.SnapMirror= Get-NcSnapmirror -Controller $CdotClustersSessions  -ErrorVariable errors
    if($errors){
        $Results.Error = $errors
        Write-Error -message $Results.Error -Category InvalidData
        return $Results
    }
    return $Results
}

function Get-CdotProdAplicableVolumes
{
    [CmdletBinding()]
    param($CdotVars,$CdotProdSVMs)
    $SVMAplicablePR = $CdotVars.Vservers | where Vserver -IN $CdotProdSVMs
    $Results = $CdotVars.Volumes  | where Vserver -in $SVMAplicablePR.vserver
    $Results | % {`
        $VolSelection_QuotaAllocated = [INT64](($CdotVars.Quota  | Where Vserver -eq $_.vserver |where Volume -eq $_.name | where DiskLimit -ne "-" | Measure-Object -property DiskLimit -sum).sum * 1024) #as NetApp not standarized it on our version
        $VolSelection_SanpMirror = ($CdotVars.SnapMirror | Where Policy -eq "DPDefault" | where SourceVserver -EQ $_.vserver | where SourceVolume -eq $_.name)
        $VolSelection_DRVolume = $CdotVars.Volumes | Where Name -eq $VolSelection_SanpMirror.DestinationVolume | where vserver -eq $VolSelection_SanpMirror.DestinationVserver
        $VolSelection_SpaceBottleneck = @{`
            "TotalMinusQuota" = ($_.TotalSize - $VolSelection_QuotaAllocated)
            "VolumeFreeSpace" = $_.Available
            "AggrFreeSpace" =  ($CdotVars.Aggregates | where name -eq $_.Aggregate).available
            "DRAggrFreeSpace" = ($CdotVars.Aggregates | where Name -eq $VolSelection_DRVolume.Aggregate).available
        }
        $_ | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name QuotaAllocated -value $VolSelection_QuotaAllocated | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name TotalMinusQuota -value $VolSelection_SpaceBottleneck.TotalMinusQuota | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name AggrFreeSpace -value  $VolSelection_SpaceBottleneck.AggrFreeSpace |`
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRSVM -value $VolSelection_SanpMirror.DestinationVserver |`
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRVolume -value $VolSelection_SanpMirror.DestinationVolume |`
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRController -value $VolSelection_DRVolume.NcController | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRAggr -value $VolSelection_DRVolume.Aggregate | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRAggrFreeSpace -value $VolSelection_SpaceBottleneck.DRAggrFreeSpace | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name SpaceBottlenecksize -Value (($VolSelection_SpaceBottleneck.GetEnumerator()  | Sort-Object -Property Value).Value | select -First 1) | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name SpaceBottleneckname -Value (($VolSelection_SpaceBottleneck.GetEnumerator()  | Sort-Object -Property Value).Name | select -First 1)
    }
}


$Cred = (Get-Credential -Message "login to clusters with Admin Accounts")
$CdotClustersSessions = Connect-NcController "Cluster1","Cluster2","Cluster3","Cluster4" -Credential $Cred -HTTPS -ErrorVariable errors # change clusters name name
$CdotVars = Get-CdotVars -CdotClustersSessions $CdotClustersSessions
$volumes = Get-CdotProdAplicableVolumes -CdotVars $CdotVars -CdotProdSVMs "ProdSVM1","ProdSVM2"   # Change SVM you conseider Prod
$Volumes = $volumes | ? name -notlike "*_root"  | Sort-Object SpaceBottlenecksize -Descending # exemple filter
Write-Host "These are the volumes we found fit to your reqest:"
$global:index=-1
Write-host ($volumes  | Format-Table -Property @{name="ID";expression={$global:index;$global:index+=1}},name,Dedupe,@{Expression={Convert-BytesToSize $_.SpaceBottlenecksize};Label=”SpaceBottleneck”},SpaceBottleneckname -AutoSize | Out-String)
$SelectedVolume = Select-FromOptions $volumes -ErrorAction Stop

 

Re: Compiling quota total commits to volume to then select Volume

^^

Nice powershell.. Yeah, powershell is always easier, and we have mocked up code in powershell pretty easy.  We need to use WFA as we plan on sending REST calls from ServiceNow front-end to our customers.

 

 

for your convert to size, you can use Netapp's builtin convertto-formattednumber cmdlet.  That might do the trick as well.

 

 

Re: Compiling quota total commits to volume to then select Volume

I also had this query - it's scoped at the SVM level but would be easy to expand.  Produces the following user input in a workflow:

 

wfa_quota.png

Would something like this be useful?