Active IQ Unified Manager Discussions

Compiling quota total commits to volume to then select Volume

JGPSHNTAP
7,864 Views

We are working on workflows where the end state goal is to have WFA do the compiling of quotas on our utility volumes to check for oversubscription and then chose the proper volume.

 

Has anyone done something similiar to this?  If so, please share experiences.

 

The thought process is pull Volumes used size, total size and committed quotas and then go from there.

 

Any insight would be great.

 

Thanks

14 REPLIES 14

NICKBARTON
7,770 Views

@JGPSHNTAP

 

You should have filters in place that can do most of this for you already. It's not clear what decision you are trying to get to though. Pick a volume for what? You mention quotas so are you talking for deploying new exports with user quotas? 

 

We are using filters to determine where to deploy new volumes based on aggregate utililzation and overcommitement %, happy to help if I can just need to clarify what you are after. 

 

-Nick

JGPSHNTAP
7,688 Views

Nick,

 

I should have been more clear, i apologize if it was ambiguous.

 

So, we are doing deployment of what we are terming "general purpose" requests under 3TB to a dedicated utility volume.

 

So for example, the cluster will have 4 utility volumes.  As a request comes in, WFA calculates all the quotas committed, vol size used and then makes decisions on which volume to provision a  new qtree with new quota.  That is where we are heading.. Our DEV guys have made lots of progress on this in the past couple of days, by two methods, java or sql. 

 

The challenge we are running into now is with the reserveation database.  

 

Anything over 3TB, will get a dedicated volume.  The reason why we chose this design method is because of cluster vol limits and we realized that our provisioning was 80/20 rule.  Meaning 80% of our vols were <3TB.  Our Util vol starts at 10TB with autogrow max at 20TB.  

 

We also feel this will help out more with SVM-DR and mirror'd volume limits.

 

I will gladly take some feedback and any info on forcing our commands into the reservation database would be great.

 

Thanks

 

joele
7,654 Views

I've done similar queries in the past - I can send an example query over in the AM that might help.

JGPSHNTAP
7,635 Views

^^

 

That would be great.  We have working queries, but we are not happy with the timesync of WFA-OCUM and reservation data that isn't updating, so we are hoping to have a query that updates the reservation cache.

joele
7,624 Views

JGPSHNTAP - 

 

Sure, there can always be a few minute delay between WFA and OCUM that can make things a little tricky if you're doing frequent updates.  

 

Is it just the quota create command that you need working reservations?  I can check if I have something written up for this already or not.

joele
7,615 Views

I found a custom command I wrote awhile ago to create new qtree disklimit quotas.  The reservation and verification code seems to work from some quick testing I did this morning, probably something you'll want to test further though.

 

Reservation Code

 

UPDATE
    cm_storage.qtree, cm_storage.volume, cm_storage.vserver, cm_storage.cluster
SET
    qtree.disk_limit_mb = ('${QuotaSizeGB}' * 1024)
WHERE
    qtree.volume_id = volume.id
    AND volume.vserver_id = vserver.id
    AND vserver.cluster_id = cluster.id
    AND qtree.name = '${Qtree}'
    AND volume.name = '${Volume}'
    AND vserver.name = '${Vserver}'
    AND (
        cluster.name='${Cluster}'
        OR cluster.primary_address='${Cluster}'
    )

 

Verification Code

 

SELECT
    qtree.id
FROM
    cm_storage.qtree,
    cm_storage.volume,
    cm_storage.vserver,
    cm_storage.cluster
WHERE
    qtree.disk_limit_mb = (
        '${QuotaSizeGB}' * 1024
    )
    AND qtree.volume_id = volume.id
    AND volume.vserver_id = vserver.id
    AND vserver.cluster_id = cluster.id
    AND qtree.name = '${Qtree}'
    AND volume.name = '${Volume}'
    AND vserver.name = '${Vserver}'
    AND (
        cluster.name='${Cluster}'
        OR cluster.primary_address='${Cluster}'
    )

Let me know if you need any help with the custom command or anything here.  Happy to help take a look.

GidonMarcus
7,605 Views

Hi

 

i believe i have exactly what you need, but in PS not as a query, i gave up on WFA for my project..... you can maybe try and take the logic... . it will order the volumes for you for where you have the most available space based on Quota allocation. volume free space, aggregate free space and aggregate on the snapmirror destination (if exists) free space. you can later filter the volumes you don't want to show (for example all the root ones, volumes that have the quota full etc)

 

Good luck

Gidi

 

Exemple:

 

These are the volumes we found fit to your reqest:

ID Dedupe SpaceBottleneck SpaceBottleneckname
-- ------ --------------- -------------------
 0   True 9.5TB           VolumeFreeSpace    
 1   True 9.49TB          VolumeFreeSpace    
 2   True 9.47TB          VolumeFreeSpace    
 3   True 9.39TB          VolumeFreeSpace    
 4   True 9.25TB          AggrFreeSpace     
 5   True 8.93TB          TotalMinusQuota    
 6   True 8.46TB          TotalMinusQuota    
 7   True 8TB             TotalMinusQuota    
 8   True 7.79TB          DRAggrFreeSpace     
 9   True 7.72TB          TotalMinusQuota    
10   True 7TB             TotalMinusQuota    
11   True 6.22TB          TotalMinusQuota    
12   True 5.48TB          TotalMinusQuota    
13   True 5TB             TotalMinusQuota    
14   True 3.88TB          VolumeFreeSpace    
15   True 3.48TB          TotalMinusQuota    
16   True 2.8TB           TotalMinusQuota    
17   True 667.8GB         TotalMinusQuota    
22   True -735846.4MB     TotalMinusQuota    



please select a number from the list: 1
Selection made:

Name                      State       TotalSize  Used  Available Dedupe Aggregate                 Vserver                                                                                                    
----                      -----       ---------  ----  --------- ------ ---------                 -------                                                                                                    
Test_vol          online        10.0 TB    5%     9.5 TB  True  Test_Aggr         Test_SVM                                                                                                

 

 

 

Code:

 

 

 

<# Created by Gidi Marcus FEB 2018 for http://community.netapp.com/t5/OnCommand-Storage-Management-Software-Discussions/Compiling-quota-total-commits-to-volume-to-then-select-Volume/m-p/137922#M25017
 Tested on:
 $CdotClustersSessions | select -First 1 Version; Get-Module DataONTAP | select version;$PSVersionTable.PSVersion.ToString();(Get-WmiObject -class Win32_OperatingSystem).Caption
Version                                           
-------                                           
NetApp Release 9.1P6: Tue Jun 20 21:52:46 UTC 2017
4.1.0                                             
5.1.14393.1737
Microsoft Windows Server 2016 Standard
#>

Function Convert-BytesToSize
{[CmdletBinding()]
    Param([parameter(Mandatory=$False,Position=0)][int64]$Size)
    Switch ($Size){
        {$Size -gt 1TB}{
            $NewSize = “$([math]::Round(($Size / 1TB),2))TB”
            Break}
        {$Size -gt 1GB}{
            $NewSize = “$([math]::Round(($Size / 1GB),2))GB”
            Break}
        Default{
            $NewSize = “$([math]::Round(($Size / 1MB),2))MB”
            Break}
}Return $NewSize}

    
Function Select-FromOptions
{
    [CmdletBinding()]
    Param([parameter(Mandatory=$True,Position=0)]$OptionsList)
    While (-not $OptionSelected){
    $OptionSelected = $OptionsList[(Read-Host "please select a number from the list")]
    if ($OptionSelected)
        {
        Write-Host -ForegroundColor "Green" "Selection made:"
        Write-Host ($OptionSelected | Format-Table | Out-String)
        break;
        }
    else{Write-Warning "No valid selection made!, Please try again"}}
    Return $OptionSelected
}


function Get-CdotVars
{
    [CmdletBinding()]
    param($CdotClustersSessionss)
    $Results = @{}
    $Results.Aggregates = Get-NcAggr -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Volumes = Get-NcVol -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Vservers = Get-NcVserver -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Qtrees = Get-Ncqtree -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Quota = Get-NcQuota -Controller $CdotClustersSessions  -ErrorVariable errors
    $Results.Quotapolicy = Get-NcQuotapolicy -Controller $CdotClustersSessions -ErrorVariable errors
    $Results.SnapMirror= Get-NcSnapmirror -Controller $CdotClustersSessions  -ErrorVariable errors
    if($errors){
        $Results.Error = $errors
        Write-Error -message $Results.Error -Category InvalidData
        return $Results
    }
    return $Results
}

function Get-CdotProdAplicableVolumes
{
    [CmdletBinding()]
    param($CdotVars,$CdotProdSVMs)
    $SVMAplicablePR = $CdotVars.Vservers | where Vserver -IN $CdotProdSVMs
    $Results = $CdotVars.Volumes  | where Vserver -in $SVMAplicablePR.vserver
    $Results | % {`
        $VolSelection_QuotaAllocated = [INT64](($CdotVars.Quota  | Where Vserver -eq $_.vserver |where Volume -eq $_.name | where DiskLimit -ne "-" | Measure-Object -property DiskLimit -sum).sum * 1024) #as NetApp not standarized it on our version
        $VolSelection_SanpMirror = ($CdotVars.SnapMirror | Where Policy -eq "DPDefault" | where SourceVserver -EQ $_.vserver | where SourceVolume -eq $_.name)
        $VolSelection_DRVolume = $CdotVars.Volumes | Where Name -eq $VolSelection_SanpMirror.DestinationVolume | where vserver -eq $VolSelection_SanpMirror.DestinationVserver
        $VolSelection_SpaceBottleneck = @{`
            "TotalMinusQuota" = ($_.TotalSize - $VolSelection_QuotaAllocated)
            "VolumeFreeSpace" = $_.Available
            "AggrFreeSpace" =  ($CdotVars.Aggregates | where name -eq $_.Aggregate).available
            "DRAggrFreeSpace" = ($CdotVars.Aggregates | where Name -eq $VolSelection_DRVolume.Aggregate).available
        }
        $_ | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name QuotaAllocated -value $VolSelection_QuotaAllocated | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name TotalMinusQuota -value $VolSelection_SpaceBottleneck.TotalMinusQuota | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name AggrFreeSpace -value  $VolSelection_SpaceBottleneck.AggrFreeSpace |`
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRSVM -value $VolSelection_SanpMirror.DestinationVserver |`
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRVolume -value $VolSelection_SanpMirror.DestinationVolume |`
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRController -value $VolSelection_DRVolume.NcController | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRAggr -value $VolSelection_DRVolume.Aggregate | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name DRAggrFreeSpace -value $VolSelection_SpaceBottleneck.DRAggrFreeSpace | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name SpaceBottlenecksize -Value (($VolSelection_SpaceBottleneck.GetEnumerator()  | Sort-Object -Property Value).Value | select -First 1) | `
        Add-Member -MemberType NoteProperty -PassThru -force -Name SpaceBottleneckname -Value (($VolSelection_SpaceBottleneck.GetEnumerator()  | Sort-Object -Property Value).Name | select -First 1)
    }
}


$Cred = (Get-Credential -Message "login to clusters with Admin Accounts")
$CdotClustersSessions = Connect-NcController "Cluster1","Cluster2","Cluster3","Cluster4" -Credential $Cred -HTTPS -ErrorVariable errors # change clusters name name
$CdotVars = Get-CdotVars -CdotClustersSessions $CdotClustersSessions
$volumes = Get-CdotProdAplicableVolumes -CdotVars $CdotVars -CdotProdSVMs "ProdSVM1","ProdSVM2"   # Change SVM you conseider Prod
$Volumes = $volumes | ? name -notlike "*_root"  | Sort-Object SpaceBottlenecksize -Descending # exemple filter
Write-Host "These are the volumes we found fit to your reqest:"
$global:index=-1
Write-host ($volumes  | Format-Table -Property @{name="ID";expression={$global:index;$global:index+=1}},name,Dedupe,@{Expression={Convert-BytesToSize $_.SpaceBottlenecksize};Label=”SpaceBottleneck”},SpaceBottleneckname -AutoSize | Out-String)
$SelectedVolume = Select-FromOptions $volumes -ErrorAction Stop

 

Gidi Marcus (Linkedin) - Storage and Microsoft technologies consultant - Hydro IT LTD - UK

JGPSHNTAP
7,588 Views

^^

Nice powershell.. Yeah, powershell is always easier, and we have mocked up code in powershell pretty easy.  We need to use WFA as we plan on sending REST calls from ServiceNow front-end to our customers.

 

 

for your convert to size, you can use Netapp's builtin convertto-formattednumber cmdlet.  That might do the trick as well.

 

 

joele
7,575 Views

I also had this query - it's scoped at the SVM level but would be easy to expand.  Produces the following user input in a workflow:

 

wfa_quota.png

Would something like this be useful?

 

 

JGPSHNTAP
6,796 Views

Joele,

 

Yes, that would also be helpful.  We will share ours shortly and see what optimizations can be done.

 

We have figured out the reservation cache as well, just need to adjust our query

joele
6,786 Views

Here's the query for the user input variable I've been using:

 

SELECT
    volume.name AS 'volume',
    ROUND (( (volume.size_mb * ((100 - volume.snapshot_reserved_percent) / 100)) / 1024 ),0 ) AS 'Total Volume Size (GB)',
    ROUND ((volume.used_size_mb / 1024),0) AS 'Used Volume Size (GB)',
    ROUND ((volume.available_size_mb / 1024),0) AS 'Available Volume Size (GB)',
    COUNT(qtree.id) AS 'Qtree Count',
    ROUND ((SUM(qtree.disk_limit_mb) / 1024),0) AS 'Quota Allocated (GB)'
FROM
    cm_storage.volume as volume,
    cm_storage.vserver as vserver,
    cm_storage.cluster as cluster,
    cm_storage.qtree as qtree
WHERE
    qtree.volume_id = volume.id
    AND volume.vserver_id = vserver.id
    AND vserver.cluster_id = cluster.id
    AND qtree.name != ''
    AND volume.name NOT LIKE '%_root%'
    AND vserver.name = '${VserverName}'
    AND (
        cluster.name = '${ClusterName}'
        OR cluster.primary_address = '${ClusterName}'
    )
GROUP BY
    volume.id
ORDER BY
    SUM(qtree.disk_limit_mb)

 

 

Curious to see what you've been working on around the cache query.  Have a good one!

JGPSHNTAP
6,774 Views

Ok, here is our query the team has wrote, and could use some advice on this.

 

SELECT
    cv.id,
    cv.name AS 'name',
    cv.used_size_mb AS 'used',
    cv.size_mb AS 'total_size',
    vserver.name AS 'vserver.name',
    cluster.primary_address AS 'vserver.cluster.primary_address',
    (((SELECT
        sum(qtree.disk_limit_mb)                                                                                
    FROM
        cm_storage.qtree                                                                                          
    WHERE
        qtree.volume_id = cv.id )/cv.size_mb)*.5 + (cv.used_size_mb/cv.size_mb)) AS 'Averaged'        
FROM
    cm_storage.volume cv,
    cm_storage.vserver,
    cm_storage.cluster                               
WHERE
    cv.security_style = '${security_style}'                                                   
    AND cv.used_size_mb / cv.size_mb < .90                                                      
    AND cv.name LIKE '${VolPrefix}%'                                                   
    AND vserver.id = cv.vserver_id                                                                                     
    AND cluster.id = vserver.cluster_id                                                                                     
    AND vserver.name NOT LIKE '%_dr'          
    AND          (
        cv.size_mb * 2 > (
            SELECT
                sum(qtree.disk_limit_mb)                                                                                                                            
            FROM
                cm_storage.qtree                                                                                                                                      
            WHERE
                qtree.volume_id = cv.id 
        )     
        OR cv.used_size_mb = 0
    ) 
ORDER BY
    Averaged ASC        

We have to do same subquery 2x to reference.  

 

 

For reservations we had to go to commands we defined mandatory variables 

 

  [parameter(Mandatory=$false, HelpMessage="Disk Limit")]
  [long]$disk_limit_mb,

  [parameter(Mandatory=$false, HelpMessage="Disk Soft Limit")]
  [long]$disk_soft_limit_mb

Added parameter defintions, add parameter mappings, and then in reservation tab

added this

'${disk_limit_mb}' AS disk_limit_mb, -- Default value for disk limit, soft disk limit and filelimit is 0
'${disk_soft_limit_mb}' AS disk_soft_limit_mb,

 

Now that updates the reservation table.   

 

Thanks for checking our query

GidonMarcus
6,458 Views

Hi @joele

 

can you share ther query you used for this table ?

 

G

Gidi Marcus (Linkedin) - Storage and Microsoft technologies consultant - Hydro IT LTD - UK

joele
6,446 Views

Hi Gidon,

 

 

The SQL query syntax is a few posts up for that table, here's a direct link:

 

https://community.netapp.com/t5/OnCommand-Storage-Management-Software-Discussions/Compiling-quota-total-commits-to-volume-to-then-select-Volume/m-p/13...

 

Hope this helps and let me know if you have questions.

 

Joel

Public