Aggregate over commit percentage

JON_MELIA · ‎2014-06-30

Hi

I've been trying to find a powershell command that will return the above. The dfm is able to email this info out as a warning, but thus far I've not been able to track down a way of doing this.. Get-NaAggr and Get-NAAggrSpace don't return this info..

HONIG2012 · ‎2014-06-30

well, i think DFM calculates it - sum all volume sizes - compare with aggr total space and you should be able to calc the percentage value.

best andreas

markweber · ‎2014-07-01

this doesn't take into account the snap reserve, but you can do something like this:

PS E:\> get-navol | Group-Object -Property ContainingAggregate | %{$_.name; ($_.group | measure-object -property totalsize -sum).sum/(get-naaggr $_.name).totalsize*100}

aggr0

126.98877560655991701782664328

aggr5

95.82054648597344067824994915

aggr9

43.057449986213111943013075340

JGPSHNTAP · ‎2014-07-02

This is interesting.. I never thought to do it that way...

I'm a script junkie, but I used to use the excel api to dump to a dashboard and then use the power of excel to do the calcs. I'm going to play around with this today if I have time to see if I can also come up with a solution...

JON_MELIA · ‎2014-07-02

That's the kind of thing I'm after, thanks.... I'm normally pretty good at borrowing ps from the web and tweaking it for my needs, but this is beyond my skill level

I'm trying to figure a way of incorporating that into my report.... I use the below to pull back Aggr info and write it into a html report

Is it possible to parse your suggestion into my code somehow below so that I could get a Committed Use %age for each aggr or is it liekely I'll have to create a new fuinction?

ForEach ($netapp in $netapplist) {

write-host "Connecting to ... "$netapp

WriteAggrTableHeader-v2 $netapp

Connect-nacontroller $netapp -Credential $cred

Get-NaAggrspace | Sort-Object AggregateName | %{

$Agg = {} | select AggregateName,SizeFree,SizeMetadata,SizeNominal,SizeSnapUsed,SizeUsed,Used

$Agg.AggregateName = $_.AggregateName

$Agg.SizeFree = $_.SizeFree

$Agg.SizeMetadata = $_.SizeMetadata

$Agg.SizeNominal = $_.SizeNominal

$Agg.SizeSnapUsed = $_.SizeSnapUsed

$Agg.SizeUsed = $_.SizeUsed

$Agg.Used = $_.Used

writeAgg-v2Info $netapp $Agg.AggregateName $Agg.SizeFree $Agg.SizeMetadata $Agg.SizeNominal $Agg.SizeSnapUsed $Agg.SizeUsed $Agg.Used

}

Add-Content $fileName "</table>"

Function writeAgg-v2Info
{
param($Netapp,$AggName,$AggFree,$AggMeta,$AggNom,$AggSnap,$AggSUsed,$AggPFree,$CommittedUse)

$AggFree = "{0:N2}" -f (([Math]::Round(($AggFree/1099511627776 ),2)))
$AggMeta = "{0:N2}" -f (([Math]::Round(($AggMeta/1099511627776 ),2)))
$AggNom = "{0:N2}" -f (([Math]::Round(($AggNom/1099511627776 ),2)))
$AggSnap = "{0:N2}" -f (([Math]::Round(($AggSnap/1099511627776 ),2)))
$AggSUsed = "{0:N2}" -f (([Math]::Round(($AggSUsed/1099511627776 ),2)))
$AggPFree = "{0:N2}" -f (([Math]::Round(($AggFree/$AggNom*100 ),2)))
$CommittedUse = "{0:N2}" -f (([Math]::Round(($CommittedUse),2)))

billyd · ‎2014-07-02

See if this is what you are after:

foreach ($aggr in Get-NaAggr){

$voltotal = foreach ($volume in get-navol -Aggregate $aggr) {Get-NaVolSize $volume.name | select @{name="Total";e={[math]::truncate($_.VolumeSize / 1GB)}}}

$aggrused = Get-NaAggr -name $aggr |Select @{name="Used";e={[math]::truncate($_.SizeUsed / 1GB)}}

$committed = ($voltotal | Measure-Object 'Total' -Sum).Sum

Get-NaAggr -name $aggr | select @{n="Controller";e={$controller}},Name,@{name="Total";e={[math]::truncate($_.TotalSize / 1GB)}},@{name="Used";e={$aggrused.used}},@{name="Used%";e={[math]::round(([decimal] $_.sizeused / $_.totalsize)*100)}},@{name="Committed";e={$committed}},@{name="Committed%";e={[math]::round(([decimal] $committed / ($_.TotalSize / 1GB))*100)}}

}

This will loop through all of the aggregates on the controller

$voltotal = Pulls all of the volume sizes from the aggregate

$aggrused = Gets to used space from the aggregate
$committed = SUM of all of the volume sizes

Then outputs the information using expressions.

JON_MELIA · ‎2014-07-02

Thats a great bit of code which I think almost has it.....

The figures however do not match what dfm tells me... The figure produced from your code is actually 15% lower than the dfm figure... Could that be related to the Snap reserve as an earlier poster mentioned

billyd · ‎2014-07-02

Good catch - I didn't account for SnapReserve. This will. I changed from using get-navolsize to get-navol and used TotalSize and SnapshotBlocksReserved fields to load $voltotal.

foreach ($aggr in Get-NaAggr){

$voltotal = foreach ($volume in get-navol -Aggregate $aggr) {Get-NaVol $volume.name | select @{name="Total";e={[math]::truncate($_.TotalSize / 1GB)}},@{name="Snap";e={[math]::truncate($_.SnapshotBlocksReserved / 1GB)}}}

$aggrused = Get-NaAggr -name $aggr |Select @{name="Used";e={[math]::truncate($_.SizeUsed / 1GB)}}

$committed = ($voltotal | Measure-Object 'Total' -Sum).Sum

Get-NaAggr -name $aggr | select @{n="Controller";e={$controller}},Name,@{name="Total";e={[math]::truncate($_.TotalSize / 1GB)}},@{name="Used";e={$aggrused.used}},@{name="Used%";e={[math]::round(([decimal] $_.sizeused / $_.totalsize)*100)}},@{name="Committed";e={$committed}},@{name="Committed%";e={[math]::round(([decimal] $committed / ($_.TotalSize / 1GB))*100)}}

}

JON_MELIA · ‎2014-07-02

Hi billyd

I really appreciate the effort on this, but for some reason the committed %age has gone lower than previously dfm says 157%. Your original code 142% and now its down to 130%...

billyd · ‎2014-07-02

Could you send the output of the command along with what DFM is showing?

Also - if you have a report in DFM that you want to see in excel, you could do this. This script could be run as a scheduled task on the DFM server to automatically report and save the committed space from week to week.

Powershell script to run report and send output to CSV:

Import-Module DataONTAP -ErrorAction SilentlyContinue

$aggregatefile = "C:\scripts\data\aggregates\"+(Get-Date -format MM_dd_yyyy)+".csv"

dfm report view -F csv Aggr_Report | Convertfrom-csv | Export-Csv -LiteralPath $aggregatefile -NoTypeInformation

If you don't have a custom DFM report, here's what a basic one for Committed space would look like. Then you can use the same script above to run it.

DFM CLI to create report:

dfm report create -R Aggregate -f Aggregate.StorageSystem,Aggregate.Name,Aggregate.TotalSpace,Aggregate.BytesCommittedPct Aggr_Report

JON_MELIA · ‎2014-07-02

Hi billyd

Command output shows this

Controller :
Name       : aggr0_SATA
Total      : 31235
Used       : 14869
Used%      : 48
Committed : 26643
Committed% : 85

Controller :
Name       : aggr1_SAS
Total      : 9351
Used       : 4888
Used%      : 52
Committed : 12189
Committed% : 130

DFM Email shows this for the aggr1 SAS

Aggregate Overcommitted.

Committed 14.4 TB (157.51%) out of 9.13 TB available;
Using 4.77 TB (52.27%) out of 9.13 TB available

I'll start looking directly at the DFM Reports

billyd · ‎2014-07-02

Hey Jon,

I didn't add the Snap Reserve to the committed amount. I pasted the entire script and all changes are in bold. From what you sent the Total,Used,Used% are all accurate. Hopefully getting the SnapReserve portion into the calculation will get us closer.

foreach ($aggr in Get-NaAggr){

$voltotal = foreach ($volume in get-navol -Aggregate $aggr) {Get-NaVol $volume.name | select @{name="Total";e={[math]::truncate($_.TotalSize / 1GB)}},@{name="Snap";e={[math]::truncate($_.SnapshotBlocksReserved / 1024 / 1024)}}}

$aggrused = Get-NaAggr -name $aggr |Select @{name="Used";e={[math]::truncate($_.SizeUsed / 1GB)}}

$committed = ($voltotal | Measure-Object 'Total' -Sum).Sum

$committed += ($voltotal | Measure-Object 'Snap' -sum).Sum

Get-NaAggr -name $aggr | select @{n="Controller";e={$controller}},Name,@{name="Total";e={[math]::truncate($_.TotalSize / 1GB)}},@{name="Used";e={$aggrused.used}},@{name="Used%";e={[math]::round(([decimal] $_.sizeused / $_.totalsize)*100)}},@{name="Committed";e={$committed}},@{name="Committed%";e={[math]::round(([decimal] $committed / ($_.TotalSize / 1GB))*100)}}

}

JON_MELIA · ‎2014-07-02

Hi Billy

Nearly there

Controller :
Name       : aggr0_SATA
Total      : 31235
Used       : 14859
Used%      : 48
Committed : 28237
Committed% : 90

Controller :
Name       : aggr1_SAS
Total      : 9351
Used       : 4890
Used%      : 52
Committed : 13255
Committed% : 142

I'm just trying out some DFM reports to see if I can get what I need that way, This is great stuff by the way...

tenali · ‎2015-06-10

Hi,

I tried to execute the script mentioned, but some how i couldn't get the complete output and also the script hung after generating values for couple of aggregates.

Controller : clip-fascls1

Name : n01_aggr01

Total : 111735

Used : 0

Used% : 0

Committed : 294949

Committed% : 264

Name : n01_aggr02

Total : 744

Used : 0

Used% : 0

Committed : 2824

Committed% : 379

Name : n02_aggr01

Total : 111735

Used : 0

Used% : 0

Committed : 260017

Committed% : 233

Name : n02_aggr02

Total : 744

Used : 0

Used% : 0

Committed : 2824

Committed% : 379

I'm not a scripting guy, can u assist?

JON_MELIA · ‎2014-07-07

Here is my full powershell (with identifying text redacted)...

What I would ideally like is to create another row under the WriteAggrTableHeader-v2 section to reflect the Committed size.... I'm really struggling to get this to work whichever way I try it... I had thought I could return a dfm report as a csv file and then import-csv to get this info into my report, but I can't get that to work either...

There are some additional bits in there that pull in stats from our UCS that can safely be ignored..

I'm now back thinking that the best way would be just to work out the aggregate committed size as Billy has done above, but have been unable to get these figures to match the dfm report output...

# Import the NetApp Data ONTAP module
Import-Module DataONTAP

# Create a credential object to parse into the connection strings
$password = ConvertTo-SecureString "" -AsPlainText -Force
$cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList "",$password

# Email recipients
$emailNotifications=("")
# SMTP server
$emailServer=""
# Get todays Date
$Now=Get-Date
# File name
$fileName = "WeeklyStats.htm"
# File path
$Path = "C:\Scripts\WeeklyStats-DCA"
# File path and name
$FilePath = $Path + "\" + $filename

# List the filers you want to scan
$NetappList=("")

##############################

Remove-Item $FilePath
New-Item -ItemType file -Path $Path -Name $filename

# Function to write HTML Header
Function writeHtmlHeader
{
param($fileName)
$date = Get-Date -format D
Add-Content $fileName "<html>"
Add-Content $fileName "<head>"
Add-Content $fileName "<meta http-equiv='Content-Type' content='text/html; charset=iso-8859-1'>"
Add-Content $fileName '<title>DCA Capacity Planning Report</title>'
Add-Content $fileName '<STYLE TYPE="text/css">'
Add-Content $fileName ""
Add-Content $fileName "</style>"
Add-Content $fileName "</head>"
Add-Content $fileName "<body>"
Add-Content $fileName "<table width='100%'>"
Add-Content $fileName "<tr bgcolor='#CCCCCC'>"
Add-Content $fileName "<td colspan='7' height='25' align='center'>"
Add-Content $fileName "<font face='tahoma' color='#003399' size='4'><strong>DCA Capacity Planning Report</strong></font>"
Add-Content $fileName "</td>"
Add-Content $fileName "</tr>"
Add-Content $fileName "</table>"
add-content $fileName "<table width='100%'>"
add-content $fileName "<tr bgcolor='#CCCCCC'>"
add-content $fileName "<td width='50%' height='25' align='center'>"
add-content $fileName "<font face='tahoma' color='#003399' size='2'><strong>Report generated on - $($ENV:Computername)</strong></font>"
add-content $fileName "</td>"
add-content $fileName "<td width='50%' height='25' align='center'>"
add-content $fileName "<font face='tahoma' color='#003399' size='2'><strong>$date</strong></font>"
add-content $fileName "</td>"
add-content $fileName "</tr>"
add-content $fileName "</table>"
}

# Function to write the Table Header to the file
Function writeTableHeader
{
param($fileName,$title)
add-content $fileName "<table width='100%'>"
add-content $fileName "<tr bgcolor='#CCCCCC'>"
add-content $fileName "<td height='25' align='left'>"
add-content $fileName "<font face='tahoma' color='#003399' size='2'><strong>$title</strong></font>"
add-content $fileName "</td>"
add-content $fileName "</tr>"
add-content $fileName "</table>"
}

# Function to write the Aggregate Info Table Header to the file
Function WriteAggrTableHeader-v2
{
param($title)
Add-Content $fileName "<table width='100%'><tbody>"
Add-Content $fileName "<tr bgcolor=#CCCCCC><td colspan='7'><strong>$title</strong></td></tr>"
Add-Content $fileName "<tr bgcolor=#CCCCCC>"
Add-Content $fileName "<td width='16%' align='center'><strong>Name</strong></td>"
Add-Content $fileName "<td width='12%' align='center'><strong>Free</strong></td>"
Add-Content $fileName "<td width='12%' align='center'><strong>Metadata</strong></td>"
Add-Content $fileName "<td width='12%' align='center'><strong>Nominal</strong></td>"
Add-Content $fileName "<td width='14%' align='center'><strong>Snap Used</strong></td>"
Add-Content $fileName "<td width='12%' align='center'><strong>Used</strong></td>"
Add-Content $fileName "<td width='12%' align='center'><strong>% Free</strong></td>"
Add-Content $fileName "</tr>"
}

# Function to write the CPU Table Header to the file
Function writeCPUTableheader2
{
Add-Content $fileName "<table width='100%'><tbody>"
Add-Content $fileName "<tr bgcolor=#CCCCCC>"
Add-Content $fileName "<td width='15%' align='center'><strong>Total CPU Cores</strong></td>"
Add-Content $fileName "<td width='14%' align='center'><strong>Assigned CPU Cores</strong></td>"
Add-Content $fileName "<td width='14%' align='center'><strong>CPU Cores % Used</strong></td>"
Add-Content $fileName "<td width='14%' align='center'><strong>CPU Ratio</strong></td>"
Add-Content $fileName "<td width='14%' align='center'><strong>Total RAM</strong></td>"
Add-Content $fileName "<td width='14%' align='center'><strong>Assigned RAM</strong></td>"
Add-Content $fileName "<td width='15%' align='center'><strong>RAM % Used</strong></td>"
Add-Content $fileName "</tr>"
}

#Function to get Aggregate Info
Function get-AggInfo-v2
{
   ForEach ($netapp in $netapplist) {
      write-host "Connecting to ... "$netapp

   WriteAggrTableHeader-v2 $netapp
   Connect-nacontroller $netapp -Credential $cred
   Get-NaAggrspace | Sort-Object AggregateName | %{
      $Agg = {} | select AggregateName,SizeFree,SizeMetadata,SizeNominal,SizeSnapUsed,SizeUsed,Used

   $Agg.AggregateName = $_.AggregateName
   $Agg.SizeFree = $_.SizeFree
   $Agg.SizeMetadata = $_.SizeMetadata
   $Agg.SizeNominal = $_.SizeNominal
   $Agg.SizeSnapUsed = $_.SizeSnapUsed
   $Agg.SizeUsed = $_.SizeUsed
   $Agg.Used = $_.Used


   writeAgg-v2Info $netapp $Agg.AggregateName $Agg.SizeFree $Agg.SizeMetadata $Agg.SizeNominal $Agg.SizeSnapUsed $Agg.SizeUsed $Agg.Used


    }
   Add-Content $fileName "</table>"

   }

}

#Function to get CPU Info
Function get-CPUInfo
{
   $csvinfo1 = Import-CSV vieUCS_vRAM_vCPU_TotalUsed.txt -Header @("AssCPU","AssRAM")
   $csvinfo2 = Import-CSV vieUCS_pRAM_pCPU_TotalInstalled.txt -Header @("TotalCPU","TotalRAM")

   WriteCPUTableHeader2

   $TotalCPU = "{0:N0}" -f [Decimal]$csvinfo2.TotalCPU
   $TotalRAM = "{0:N0}" -f [Decimal]$csvinfo2.TotalRAM
   $AssCPU = "{0:N0}" -f [Decimal]$csvinfo1.AssCPU
   $AssRAM = "{0:N0}" -f [Decimal]$csvinfo1.AssRAM
   $PercCPU = "{0:N1}" -f [Decimal]($csvinfo1.AssCPU / $csvinfo2.TotalCPU * 100)
   $PercRAM = "{0:N1}" -f [Decimal]($csvinfo1.AssRAM / $csvinfo2.TotalRAM * 100)
   $CPURatio = "{0:N1}" -f [Decimal]($AssCPU / ($TotalCPU / 2))
   $csvdate = Get-Date -format D

   $toCSV = $csvdate + "," + $csvinfo2.TotalCPU + "," + $csvinfo2.TotalRAM + "," + $csvinfo1.AssCPU + "," + $csvinfo1.AssRAM + "," + $PercCPU + "," + $PercRAM
   Add-Content -Path "C:\Scripts\WeeklyStats-DCA\CPU.csv" -Value $toCSV

   $tableEntry = "<tr><td align='center'>"+ $TotalCPU +"</td><td align='center'>"+ $AssCPU +"</td><td align='center'>"+ $PercCPU +" %</td><td align='center'>"+ $CPURatio +" : 1</td><td align='center'>"+ $TotalRAM +" Gb</td><td align='center'>"+ $AssRAM +" Gb</td><td align='center'>"+ $PercRAM +" %</td></tr>"
   Add-Content $fileName $tableEntry
}

# Function to Write Aggregate Info
Function writeAgg-v2Info
{
param($Netapp,$AggName,$AggFree,$AggMeta,$AggNom,$AggSnap,$AggSUsed,$AggPFree)

$AggFree = "{0:N2}" -f (([Math]::Round(($AggFree/1099511627776 ),2)))
$AggMeta = "{0:N2}" -f (([Math]::Round(($AggMeta/1099511627776 ),2)))
$AggNom = "{0:N2}" -f (([Math]::Round(($AggNom/1099511627776 ),2)))
$AggSnap = "{0:N2}" -f (([Math]::Round(($AggSnap/1099511627776 ),2)))
$AggSUsed = "{0:N2}" -f (([Math]::Round(($AggSUsed/1099511627776 ),2)))
$AggPFree = "{0:N2}" -f (([Math]::Round(($AggFree/$AggNom*100 ),2)))

If (($NetApp -eq "") -or ($NetApp -eq "")) {
     If ($AggName -eq "aggr0_SATA") {
        $tableEntry = "<tr bgcolor=#F5A9A9><td align='center'>$AggName</td>"
     }
     If ($AggName -eq "aggr1_SAS") {
        $tableEntry = "<tr bgcolor=#F4FA58><td align='center'>$AggName</td>"
     }
     If ($AggName -eq "aggr2_SATA") {
        $tableEntry = "<tr bgcolor=#58ACFA><td align='center'>$AggName</td>"
     }
}
   ElseIf (($NetApp -eq "") -or ($NetApp -eq "")) {
     If ($AggName -eq "SAS_aggr0") {
        $tableEntry = "<tr bgcolor=#F4FA58><td align='center'>$AggName</td>"
     }
     If ($AggName -eq "SAS_aggr1") {
        $tableEntry = "<tr bgcolor=#F4FA58><td align='center'>$AggName</td>"
     }

   }

$tableEntry += "<td align='center'>$AggFree Tb</td><td align='center'>$AggMeta Tb</td><td align='center'>$AggNom Tb</td><td align='center'>$AggSnap Tb</td><td align='center'>$AggSUsed Tb</td><td align='center'>$AggPFree %</td></tr>"
Add-Content $fileName $tableEntry

}

# Function to write Table Footer
Function writeHtmlFooter
{
param($fileName)
Add-Content $fileName "</table>"
Add-Content $fileName "</body>"
Add-Content $fileName "</html>"
}

# Function to Send Email
Function sendEmail
{ param($from,$to,$subject,$smtphost,$htmlFileName)
$body = Get-Content $htmlFileName
$smtp= New-Object System.Net.Mail.SmtpClient $smtphost
Foreach ($name in $to) {
    write-host "Sending Email to... "$name
    $msg = New-Object System.Net.Mail.MailMessage $from, $name, $subject, $body
$msg.isBodyhtml = $true
    $smtp.send($msg)
}

}

# Main Script

# Write HTML Header
writehtmlheader $fileName

# Compile CPU Info
writeTableheader $fileName "CPU & Memory Info"
get-CPUInfo
writehtmlfooter $fileName

# Compile Aggregate Info
writeTableheader $fileName "Aggregate Info"
get-AggInfo-v2
writehtmlfooter $fileName

# Send Email
sendEmail $emailNotifications "DCA Capacity Planning Report" $emailServer $fileName

JGPSHNTAP · ‎2014-07-07

John,

Looks good

A couple of suggestions.. Don't know how indepth you want to get, but wouldn't it be easier to create custom objects and pipe it to convertto-html cmdlet with a custom header.

Now, if you really want to it it out of the park, what i've done is i've used javascript jquery on the backend with datatables plug and it really blows away any tables you can create standard with html.

I can post some of what i've done if your interested...

JON_MELIA · ‎2014-07-07

As I say, I'm no coding whizz. I've managed to beg, borrow and steal most of the above from other scripts and it nearly does everything I need...

I am however certainly interested in seeing other ways of doing things, especially if it makes things simpler... I'll have a look at the convertto-html and see if I can utilise that...

Post up some of your jquery stuff as I interested to see how that works..

JGPSHNTAP · ‎2014-07-07

Jon -

Google jquery and datatables you will see what we can produce. Also, you need to have the js put either on an apache server or windows host and dump your scripts to there.

If I get time today or sometime this week i can hopefully dig through your script and see what i can do to help... Also i can tell you right off the back, there are easier ways to do things for sure..

Here', i'll give you one quick tip.. Windows PS 2.0, to send email it makes it a lot easier.

You want to use the send-mailmessage cmdlet

Send-MailMessage -To $recipients -From $sendMailAs -Subject $subjectLine -Body "put body here" -Priority $priority -SmtpServer $smtpServer