Using the nextTag field for more than 500 results

[slcfreddy]

Created a PowerShell script that will pull SnapMirror information as part of an drive to change our DP SnapMirrors to XDP SnapMirrors. The query returns more than 500 results and so I'm attempting to use the nextTag field to get all the rest. The first query works fine, however, I am getting the following message when I substitute in the nextTag parameter in subsequent queries: 

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At line:1 char:11
+ $sminfo = Invoke-RestMethod `
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

I assume that I am formatting this incorrectly but can't find documentation on the correct method. Here is my Powershell code: 

Do {

   if ($null -eq $next) {
      $sminfo = Invoke-RestMethod `
      -Method Get `
      -Credential $global:netappapicred `
      -Uri "https://$($APIServer)/api/6.0/ontap/snap-mirrors?relationship_type=data_protection" `
      -ContentType "application/json"

      $next=$sminfo.result.nextTag
   }
   else {
      $sminfo = Invoke-RestMethod `
      -Method Get `
      -Credential $global:netappapicred `
      -Uri "https://$($APIServer)/api/6.0/ontap/snap-mirrors?nextTag=$($next))" `
      -ContentType "application/json"
   
      $next=$sminfo.result.nextTag
   }

   foreach ($record in $sminfo.result.records) {
      $sm = New-Object PSCustomObject
      $source = $record.source_location
      $destination = $record.destination_location
      $type = $record.relationship_type

      $sm | Add-Member -Name source -Value $source -MemberType NoteProperty
      $sm | Add-Member -Name destination -Value $destination -MemberType NoteProperty
      $sm | Add-Member -Name type -Value $type -MemberType NoteProperty
      $data_dp.add($sm)
  }
} While ($null -NE $next)

 Any suggestion how I fix this? Using PowerShell 5.1 and the ONTAP API v6.

[slcfreddy]

FYI, I used the solution from this link as a basis for my solution:

https://community.netapp.com/t5/Software-Development-Kit-SDK-and-API-Discussions/maxRecords-query-parameter-over-500/m-p/126802#M2216

[RobertBlackhart]

Can you point to what the "ONTAP API v6" is? That's not something that I'm familiar with.

Since you're not sending the request directly to the cluster and you don't have the error message attached, I'm not sure what the problem might be. It would depend on what that API proxy is doing with the nextTag value. It's also unclear if it's the API proxy returning a 400 or ONTAP.

I would suggest as a next step in your debugging you should print out the response to the request which might help inform what's going wrong.

[slcfreddy]

API version 6 is an older version of the ONTAP REST API. I believe it shipped with AIQ 9.6.

We have multiple clusters so I query the AIQ server instead of going to each one individually which saves time and cycles on the clusters. Unfortunately, this is all the information being returned:

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At line:1 char:11
+ $sminfo = Invoke-RestMethod `
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

I can only assume that I have an error somewhere in the Invoke-RestMethod and the AIQ server is expecting the command to be formatted differently when using the nextTag than what I am doing. There just isn't any documentation that tells me what format the server is expecting.

[RobertBlackhart]

I'm afraid I'm not familiar with AIQ, only ONTAP so I'm not sure where to point you for documentation or examples.

But from the thread you linked, the suggested fix used "NextTag" (with the N capitalized) instead of "nextTag" like you have in your code below. Could that be the difference?

[slcfreddy]

The API is the same whether querying the cluster directly or using AIQ as a clearinghouse. AIQ has the advantage of all the data in one place so I can get the data I need without sending queries to 10 different clusters. 

I tried NextTag and nextTag, both fail. 

[slcfreddy]

Found out what is happening, but I don't know how to fix it. When I assign the variable $sminfo to the output of the Invoke-RestMethod PowerShell is changing some of the characters in the nextTag field. Here is a partial copy of the original nextTag value compared to the value after assigning it to a variable:

(original) nextTag=PiM%2BMjlmNj
(changed) nextTag=PiM+MjlmNj

In PowerShell % is an arithmetic operator and the only thing I can think of is that when the original value is assigned PowerShell is performing the operation instead of treating it as a string character. Any thoughts on this?

[RobertBlackhart · Accepted solution]

"%2B" is the url-encoded form of "+". I'm assuming that AIQ is expecting the nextTag that is sent to it to still be in a url-encoded form.

You might try re-encoding the tag before use to preserve it.

[slcfreddy]

That makes more sense than an arithmetic operation and gave me the information needed to solve the puzzle. Here is the final code I used: 

      $sminfo = Invoke-RestMethod `
         -Method Get `
         -Credential $cred `
         -Uri "https://$($apiserver)/api/6.0/ontap/snap-mirrors?relationship_type=data_protection" `
         -ContentType "application/json"

      $tempvalue=$sminfo.result.nextTag
      $next = [uri]::EscapeDataString($tempvalue)

      $sminfo = Invoke-RestMethod `
         -Method Get `
         -Credential $cred `
         -Uri "https://$($apiserver)/api/6.0/ontap/snap-mirrors?nextTag=$next" `
         -ContentType "application/json"

 At this point $next is formatted properly and works to grab the next page of results. Still some things to iron out to make the whole thing work but thanks for getting me past this problem.