NDMP Thread Count Monitoring

[RP_MCDOUGALL]

Is there by chance an API call I can use to track NDMP Thread Counts via Powershell Toolkit?

Thanks!

Ryan

[vinith]

Hi Ryan,

The PS ToolKit has several cmdlets to work with NDMP copy.

Just type Get-NaHelp -Category ndmp to get all cmdlets in the category "NDMP"

I guess you can use Get-NaNdmpCopy to track the status of a running NDMP copy operation.

[RP_MCDOUGALL]

Thanks for the reply Vinith! I did come across this particular cmdlet, but it didn't appear to have the information I was looking for as it doesn't list how many NDMP threads are open for an NDMP operation on a given filer form what I can see. We have a very crude way of doing this currently, but I was just looking to see if there was a more efficient way to go about it. The purpose of all this is to monitor the count of these threads, as there is a cap on how many are allowed, which we've found useful in troubleshooting certain situations. We gather information via ndmpd status and ndmp probe. Does this maybe give some insight on what I am looking for more?

[vinith]

Hi Ryan,

Can you try Invoke-NaSsh cmdlet, it sends a Data ONTAP CLI command via SSH.

Invoke-NaSsh -Name <ControllerName> -Credential root "ndmpd status"

Invoke-NaSsh -Name <ControllerName> -Credential root "ndmp probe"

Here are some more details regarding this cmdlet.

Invoke-NaSsh -Command <String[]> [-Credential <PSCredential>] [-Port <Int32>] [-Timeout <Int32>] [-WhatIf]
[-Confirm] [<CommonParameters>]
PARAMETERS
    -Command <String[]>
        The command string to send to Data ONTAP.  If the command string contains hyphens, enclose the command in
        quotes lest PowerShell attempt to interpret those as cmdlet arguments.
    -Credential <PSCredential>
        Use this argument to explicitly provide SSH credentials for authenticating with Data ONTAP.  If credentials
        are provided, they take precedence over any other credentials that may be available.  If not provided, the
        cmdlet will look for valid credentials from either the specified NaController object or
        $global:CurrentNaController as appropriate.  Failing that, credentials will be sought in the Toolkit's
        credential cache.
    -Port <Int32>
        The port on which to connect to the storage controller.
    -Timeout <Int32>
        Connection timeout in milliseconds.
    -WhatIf
    -Confirm
    -Name <String>
        The name or address of the controller to connect to.  If a hostname is specified, it must be resolvable to an
        IP address.  Specify either this argument or the -Controller argument, but not both.
    -Controller <NaController>
        The controller to connect to, embodied in an NaController object as returned by Connect-NaController.  Specify
        either this argument or the -Name argument, but not both.

[RP_MCDOUGALL]

I totally overlooked this one! I think this might do the trick, I will keep you posted.

[vinith]

Hi Ryan, do let us know if the solution worked for you

[RP_MCDOUGALL · Accepted solution]

It did work! I totally forgot to reply with the solution I came up with, it was actually rather simple.

#ndmp_threadmon.ps1

#NDMP Session Tracking

#Ryan McDougall

#2/6/2013

Import-Module DataONTAP

#Filer hostnames as strings

<filer hostname strings>

#Mail Stuff

$sendMailAs    = ""

$recipients = ""

$subjectLine = "Backup Filer NDMP Threshold Warning!"

$smtpServer = "10.11.23.15"

$priority = "high"

#Script Params

$thread_thresh = "194"

$sw = new-object system.diagnostics.stopwatch

#Filers object

$backup_filers = <filer name variables>

foreach($filer in $backup_filers)

                                        {

                                                  #Determine number of NDMP sessions

                                                  $lcount = (Invoke-NaSsh -Name $filer "ndmpd status" | Select-String -AllMatches "Session: " | foreach {$_.matches}|select value | Measure-Object).Count

                                                  #Determine if session count is greater than determined threshold (194 min for 3210's)

                                                  if($lcount -ige $thread_thresh)

                                                            {

                                                                      #Start timer

                                                                      $sw.start();

                                                                      #Send initial message and track current filer

                                                                      Send-MailMessage -To $recipients -From $sendMailAs -Subject $subjectLine -Body "The NDMP Thread count on $current_filer reached $lcount, this may require investigation - Time: $dateCurrent" -Priority $priority -SmtpServer $smtpServer

                                                                      $current_filer = $filer

                                                                      #Recheck loop

                                                                      while($lcount -ige $thread_thresh)

                                                                      {

                                                                                #Kill script after 30 minutes

                                                                                if($sw.elapsed.minutes -gt 30)

                                                                                {

                                                                                          Send-MailMessage -To $recipients -From $sendMailAs -Subject $subjectLine -Body "The script has been running for 30 minutes or more, killing script. Please investigate NDMP issues." -Priority $priority -SmtpServer $smtpServer

                                                                                          exit

                                                                                }

                                                                                #Initiate recheck

                                                                                $recheck = (Invoke-NaSsh -Name $filer "ndmpd status" | Select-String -AllMatches "Session: " | foreach {$_.matches}|select value | Measure-Object).Count

                                                                                $dateCurrent = Get-Date

                                                                                if($recheck -ige $lcount)

                                                                                {

                                                                                  #Retry after 5 minutes

                                                                                  sleep -Seconds 300

                                                                                  Send-MailMessage -To $recipients -From $sendMailAs -Subject $subjectLine -Body "The NDMP Thread count on $current_filer is currently $lcount, please investigate. - Time: $dateCurrent" -Priority $priority -SmtpServer $smtpServer

                                                                                }

                                                                                else

                                                                                {

                                                                                  #Send message if below threshold

                                                                                  Send-MailMessage -To $recipients -From $sendMailAs -Subject $subjectLine -Body "The NDMP Thread count on $current_filer has fallen to $recheck, which is below the threshold of $thread_thresh. Time: $dateCurrent" -Priority $priority -SmtpServer $smtpServer

                                                                                }

                                                                      }

                                                            }

                                                            elseif($lcount -lt $thread_thresh)

                                                            {

                                                                      Send-MailMessage -To $recipients -From $sendMailAs -Subject $subjectLine -Body "$filer is currently below the NDMP session threshold." -Priority $priority -SmtpServer $smtpServer

                                                            }

                                        }

[vinith]

Superb!, Nice to know that it worked out fine.