Software Development Kit (SDK) and API Discussions
Software Development Kit (SDK) and API Discussions
Hello, I am a TSE with this Active case open - https://smartsolve.netapp.com/case?case_id=2006299816
I am creating this post for the customer as he believes that there is a bug related to the netgorups-file-get-iter call. I will maintain the case Active for monitoring and to provide assistance where needed.
Problem Summary:
The tag mechanism of the "netgroups-file-get-iter" zapi call doesn't appear to work. The first call returns 20 records, but calling it again with a tag always returns zero records, even when the filer definitely has more than 20 netgroup entries.
Increasing the "max-records" parameter results in more records being returned on the first call, but the second call still returns zero records.
Test performed:
I attached a perl script text below that can reproduce the bug, as follows:
1. Use "netgroup load" on the cluster to populate the local netgroup database. The loaded netgroup file should have 40 or more entries.
2. Edit the attached perl script to supply a username and password for a user with the capability to run netgroups-file-get-iter. A user with the "readonly" role should work.
3. Run the script, supplying the cluster management address as the first parameter.
Although the cluster has more than 20 netgroup entries, you'll get
Received 20 entries
Received no entries
What you should get is
Received 20 entries
Received 20 entries
You can compare this behavior to that of "volume-get-iter" or "qtree-list-iter" by changing the $API variable. Those calls will give you the correct output (if run against a cluster with at least 40 volumes or qtrees, respectively).
Perl script text:
#!/usr/bin/perl
use strict;
use warnings;
use NaServer;
my $API = 'netgroups-file-get-iter';
my $USER = '<insert username here>';
my $PASS = '<insert password here>';
{
my $s = NaServer->new($ARGV[0], 1, 31);
$s->set_admin_user($USER, $PASS);
# first query
my $res = $s->invoke($API);
if (!$res || $res->results_status() eq 'failed') {
die "$API failed";
}
my $itertag = $res->child_get_string('next-tag')
or die "Invalid test: data all fits in one call";
my $attrs = $res->child_get('attributes-list');
if ($attrs) {
print "Received ",scalar($attrs->children_get())," entries\n";
} else {
print "Received no entries\n";
}
# second query
my $query = NaElement->new($API);
if (defined($itertag)) {
$query->child_add_string('tag', $itertag);
}
$res = $s->invoke_elem($query);
if (!$res || $res->results_status() eq 'failed') {
die "$API failed";
}
$attrs = $res->child_get('attributes-list');
if ($attrs) {
print "Received ",scalar($attrs->children_get())," entries\n";
} else {
print "Received no entries\n";
}
}
any updates?
, the “netgroups-file-get-iter” is a new API introduced with SDK version 5.5 (API version 1.3.1)
I downloaded the latest SDK and had a look at the structure compared to “qtree-list-iter” (which has the same parameters) EG:
Here is the example code in Perl to invoke the ZAPI from zexplore (set the path as required):
require 5.6.1;
use lib '<path_to_nmsdk_root>/lib/perl/NetApp';
use strict;
use warnings;
use NaServer;
use NaElement;
my $s = new NaServer('<server name or IP address>', 1 , 31);
$s->set_server_type('FILER');
$s->set_transport_type('HTTPS');
$s->set_port(443);
$s->set_style('LOGIN');
$s->set_admin_user('<user name>', '<password>');
my $api = new NaElement('netgroups-file-get-iter');
my $xi = new NaElement('desired-attributes');
$api->child_add($xi);
my $xi1 = new NaElement('netgroups-file-config-info');
$xi->child_add($xi1);
$xi1->child_add_string('domain','<domain>');
$xi1->child_add_string('host','<host>');
$xi1->child_add_string('member-netgroup','<member-netgroup>');
$xi1->child_add_string('netgroup-name','<netgroup-name>');
$xi1->child_add_string('user','<user>');
$xi1->child_add_string('vserver','<vserver>');
$api->child_add_string('max-records','<max-records>');
my $xi2 = new NaElement('query');
$api->child_add($xi2);
my $xi3 = new NaElement('netgroups-file-config-info');
$xi2->child_add($xi3);
$xi3->child_add_string('domain','<domain>');
$xi3->child_add_string('host','<host>');
$xi3->child_add_string('member-netgroup','<member-netgroup>');
$xi3->child_add_string('netgroup-name','<netgroup-name>');
$xi3->child_add_string('user','<user>');
$xi3->child_add_string('vserver','<vserver>');
$api->child_add_string('tag','<tag>');
my $xo = $s->invoke_elem($api);
if ($xo->results_status() eq 'failed') {
print 'Error:\n';
print $xo->sprintf();
exit 1;
}
print 'Received:\n';
print $xo->sprintf();
Another method to determine if the issue is caused by the ZAPI or the PERL code would be to test the ZAPI using the “apitest” utility within the SDK
There is a PERL example in the SDK which contains the required syntax. See (“<path_to_nmsdk_root>/src/sample/Data_ONTAP/Perl/apitest.pl”)
next step:
waiting for customer to test the ZAPI
Hi Logan,
I believe the issue relates the "max-records" property default value being set to 20. I do not think there is a bug with the API as a low default value would have probably been defined to ensure the ZAPI does not return all records. You need to explicitly set the 'max-records' property of the NaElement object BEFORE you invoke the 'netgroups-file-get-iter' ZAPI to ensure that you can iterate through each netgroup rather than just the default of 20.
Here is an example in PowerShell which uses the "manageontap.dll" from the NMSDK 5.5 (ZAPI version 1.31 introduces the 'netgroups' API)
<#'----------------------------------------------------------------------------- 'Script Name : NetGroups.ps1 'Author : Matthew Beattie 'Email : mbeattie@netapp.com 'Created : 2016-07-07 'Description : This script invokes the "netgroups-get-iter" ZAPI to enumerate ' : and iterate through a list of netgroups on a vserver '-----------------------------------------------------------------------------#> param( [Parameter(Mandatory = $True, HelpMessage = "The name of the storage controller to connect to")] [String]$ClusterName, [Parameter(Mandatory = $False, HelpMessage = "The name of the vFiler to connect to")] [String]$VserverName ) #'------------------------------------------------------------------------------ #'Initialization Section. Define Global Variables. #'------------------------------------------------------------------------------ [String]$scriptPath = Split-Path($MyInvocation.MyCommand.Path) [String]$scriptSpec = $MyInvocation.MyCommand.Definition [String]$scriptBaseName = (Get-Item $scriptSpec).BaseName [String]$scriptName = (Get-Item $scriptSpec).Name [String]$fileSpec = "$scriptPath\ManageOntap.dll" #'------------------------------------------------------------------------------ #'Ensure the ManageONTAP.dll file exists in the scripts working directory. #'------------------------------------------------------------------------------ If(-Not(Test-Path -Path $fileSpec)){ Write-Warning "The file ""$fileSpec"" does not exist. Exiting" Exit } #'------------------------------------------------------------------------------ #'Load the ManageONTAP.dll file #'------------------------------------------------------------------------------ Try{ [Reflection.Assembly]::LoadFile($fileSpec) | Out-Null Write-Host "Loaded file ""$fileSpec""" }Catch{ Write-Warning -Message $("Failed loading file ""$fileSpec"". Error " + $_.Exception.Message) Exit -1 } #'------------------------------------------------------------------------------ #'Create a ZAPI connection to the cluster using ZAPI version 1.31 (required for netgroups). #'------------------------------------------------------------------------------ [NetApp.Manage.NaServer]$naServer = New-Object NetApp.Manage.NaServer($ClusterName, "1", "31") #'------------------------------------------------------------------------------ #'Enuemate credentials to connect ot the cluster. #'------------------------------------------------------------------------------ $credentials = $host.ui.PromptForCredential("Connect to ""$ClusterName""", "Please enter the user name and password", "", "") #'------------------------------------------------------------------------------ #'Enumerate the username and password from the PowerShell credential object. #'------------------------------------------------------------------------------ [String]$domain = $credentials.GetNetworkCredential().domain [String]$user = $credentials.GetNetworkCredential().username [String]$password = $credentials.GetNetworkCredential().password If($domain -ne "" -Or $domain -ne $Null){ If($domain.Contains(".")){ [String]$userName = "$user`@$domain" }Else{ If($credentials.UserName.Contains("\") -And $credentials.UserName.SubString(0, 1) -ne "\"){ [String]$userName = $credentials.UserName }Else{ [String]$userName = $user } } } #'------------------------------------------------------------------------------ #'Set the username and password for the NaServer object. #'------------------------------------------------------------------------------ $naServer.SetAdminUser($userName, $password) $naServer.ServerType = 'FILER' $naServer.TransportType = 'HTTPS' #'------------------------------------------------------------------------------ #'Invoke the ZAPI to list the netgroups #'------------------------------------------------------------------------------ [String]$zapiName = "netgroups-file-get-iter" [Int]$maxRecords = 100 Write-Host "Invoking ZAPI ""$zapiName""" Try{ $naElement = New-Object NetApp.Manage.NaElement("$zapiName") $naElement.AddNewChild("max-records", $maxRecords) $results = $NaServer.InvokeElem($naElement) $records = $results.GetChildContent("num-records") }Catch{ Write-Warning -Message $("Failed invoking ""$zapiName"". Error " + $_.Exception.Message) Exit -1 } Write-Host "There are $records netgroups on vserver $VserverName" #'------------------------------------------------------------------------------ #'Iterate through each netgroup. #'------------------------------------------------------------------------------ $netGroups = $results.GetChildByName("attributes-list").GetChildren(); Write-Host "NetGroupName,Users" ForEach($netGroup In $netGroups){ Write-Host $($netGroup.GetChildContent("netgroup-name") + "," + $netGroup.GetChildContent("user")); } #'------------------------------------------------------------------------------
Note: The above script assumes you have downloaded the NMSDK 5.5 and copied the 'manageontap.dll' to the scripts working directory.
Example output:
PS C:\Scripts\PowerShell\Projects\NetGroups> .\NetGroups1.ps1 -ClusterName cluster3 -VserverName testnfs1 Loaded file "C:\Scripts\PowerShell\Projects\NetGroups\ManageOntap.dll" Invoking ZAPI "netgroups-file-get-iter" There are 25 netgroups on vserver testnfs1 NetGroupName,Users group1,user1 group2,user2 group3,user3 group4,user4 group5,user5 group6,user6 group7,user7 group8,user8 group9,user9 group10,user10 group11,user11 group12,user12 group13,user13 group14,user14 group15,user15 group16,user16 group17,user17 group18,user18 group19,user19 group20,user20 group21,user21 group22,user22 group23,user23 group24,user24 group25,user25
Hope that helps.
/Matt
The bug is with the tag parameter, not with the max-records parameter. The first call (with no tag parameter) always works correctly; it just returns the first max-records records and a next-tag field. The problem happens when you try to use that next-tag field in a second invocation of the API call.
I'm not a powershell programmer, but you need to do something like this: leave the max-records alone, but get the "next-tag" field from the first call:
$naElement = New-Object NetApp.Manage.NaElement($zapiName) $results = $NaServer.InvokeElem($naElement) $tag = $results.GetChildContent("next-tag")
...then use that in a second invocation:
$naElement = New-Object NetApp.Manage.NaElement($zapiName) $naElement.AddNewChild("tag", $tag) $results = $NaServer.InvokeElem($naElement)
This time $results will contain no entries, while it should contain records 21-40. And indeed, if you replace the zapi call with another -iter, say, qtree-list-iter or volume-list-iter, the tag mechanism works fine, and each call gets you the next 20 records.
Hi,
You are correct and appologies for the delayed response, I think there is a BUG with the "netgroups-file-get-iter" ZAPI when used with the tag mechanism as it does not return more than 20 returns. I confirmed this by testing another ZAPI "qtree-list-iter" using the tag mechanism to iterate through the results and it works successfully. EG:
<#'----------------------------------------------------------------------------- 'Script Name : GetQtrees.ps1 'Author : Matthew Beattie 'Email : mbeattie@netapp.com 'Created : 2016-08-02 'Description : This script invokes the "qtree-list-iter" ZAPI to enumerate ' : and iterate through a list of qtrees in a volume on a vserver. '-----------------------------------------------------------------------------#> param( [Parameter(Mandatory = $True, HelpMessage = "The name of the cluster")] [String]$ClusterName, [Parameter(Mandatory = $False, HelpMessage = "The name of the vserver")] [String]$VserverName, [Parameter(Mandatory = $False, HelpMessage = "The name of the volume")] [String]$VolumeName ) #'------------------------------------------------------------------------------ #'Initialization Section. Define Global Variables. #'------------------------------------------------------------------------------ [String]$scriptPath = Split-Path($MyInvocation.MyCommand.Path) [String]$scriptSpec = $MyInvocation.MyCommand.Definition [String]$scriptBaseName = (Get-Item $scriptSpec).BaseName [String]$scriptName = (Get-Item $scriptSpec).Name [String]$fileSpec = "$scriptPath\ManageOntap.dll" #'------------------------------------------------------------------------------ #'Ensure the ManageONTAP.dll file exists in the scripts working directory. #'------------------------------------------------------------------------------ If(-Not(Test-Path -Path $fileSpec)){ Write-Warning "The file ""$fileSpec"" does not exist. Exiting" Exit } #'------------------------------------------------------------------------------ #'Load the ManageONTAP.dll file #'------------------------------------------------------------------------------ Try{ [Reflection.Assembly]::LoadFile($fileSpec) | Out-Null Write-Host "Loaded file ""$fileSpec""" }Catch{ Write-Warning -Message $("Failed loading file ""$fileSpec"". Error " + $_.Exception.Message) Exit -1 } #'------------------------------------------------------------------------------ #'Create a ZAPI connection to the cluster using ZAPI version 1.31 (required for netgroups). #'------------------------------------------------------------------------------ [NetApp.Manage.NaServer]$naServer = New-Object NetApp.Manage.NaServer($ClusterName, "1", "31") #'------------------------------------------------------------------------------ #'Enuemate credentials to connect ot the cluster. #'------------------------------------------------------------------------------ $credentials = $host.ui.PromptForCredential("Connect to ""$ClusterName""", "Please enter the user name and password", "", "") #'------------------------------------------------------------------------------ #'Enumerate the username and password from the PowerShell credential object. #'------------------------------------------------------------------------------ [String]$domain = $credentials.GetNetworkCredential().domain [String]$user = $credentials.GetNetworkCredential().username [String]$password = $credentials.GetNetworkCredential().password If($domain -ne "" -Or $domain -ne $Null){ If($domain.Contains(".")){ [String]$userName = "$user`@$domain" }Else{ If($credentials.UserName.Contains("\") -And $credentials.UserName.SubString(0, 1) -ne "\"){ [String]$userName = $credentials.UserName }Else{ [String]$userName = $user } } } #'------------------------------------------------------------------------------ #'Set the username and password for the NaServer object. #'------------------------------------------------------------------------------ $naServer.SetAdminUser($userName, $password) $naServer.ServerType = 'FILER' $naServer.TransportType = 'HTTPS' #'------------------------------------------------------------------------------ #'Invoke the ZAPI to list the netgroups #'------------------------------------------------------------------------------ [String]$zapiName = "qtree-list-iter" [String]$tag = "start" Write-Host "Invoking ZAPI ""$zapiName""" Write-Host "QtreeName" While($tag){ $api = New-Object NetApp.Manage.NaElement("$zapiName") $xi1 = New-Object NetApp.Manage.NaElement("query"); $api.AddChildElement($xi1); $xi2 = New-Object NetApp.Manage.NaElement("qtree-info"); $xi1.AddChildElement($xi2); $xi2.AddNewChild("vserver", $VserverName); $xi2.AddNewChild("volume", $volumeName); If($tag -ne "start"){ $api.AddNewChild("tag", $tag) } Try{ $results = $NaServer.InvokeElem($api) }Catch{ Write-Warning -Message $("Failed invoking ""$zapiName"". Error " + $_.Exception.Message) Exit -1 } $tag = $results.GetChildContent("next-tag") $records = $results.GetChildContent("num-records") Write-Host "There are $records records in the current iteration" If($results -ne $Null -And $records -ne 0){ ForEach($result in ($results.GetChildByName("attributes-list").GetChildren())){ [String]$qtreeName = $result.GetChildContent("qtree") Write-Host $qtreeName } }Else{ Write-Host "There were $records records in the iteration" } } Write-Host "Done" #'------------------------------------------------------------------------------
Example output:
PS C:\Scripts\PowerShell\Projects> cd .\Qtrees PS C:\Scripts\PowerShell\Projects\Qtrees> .\Qtrees.ps1 -ClusterName cluster3 -VserverName testnfs1 -VolumeName testvol1 Loaded file "C:\Scripts\PowerShell\Projects\Qtrees\ManageOntap.dll" Invoking ZAPI "qtree-list-iter" QtreeName There are 20 records in the current iteration qtree_01 qtree_02 qtree_03 qtree_04 qtree_05 qtree_06 qtree_07 qtree_08 qtree_09 qtree_10 qtree_11 qtree_12 qtree_13 qtree_14 qtree_15 qtree_16 qtree_17 qtree_18 qtree_19 There are 20 records in the current iteration qtree_20 qtree_21 qtree_22 qtree_23 qtree_24 qtree_25 qtree_26 qtree_27 qtree_28 qtree_29 qtree_30 qtree_31 qtree_32 qtree_33 qtree_34 qtree_35 qtree_36 qtree_37 qtree_38 qtree_39 There are 11 records in the current iteration qtree_40 qtree_41 qtree_42 qtree_43 qtree_44 qtree_45 qtree_46 qtree_47 qtree_48 qtree_49 qtree_50 Done
Now if i apply exactly the same principle using the tag mechanism but using the "netgroups-file-get-iter" it will only show the first 20 results (even though there were 25 netgroups on my vserver). EG
<#'----------------------------------------------------------------------------- 'Script Name : NetGroups.ps1 'Author : Matthew Beattie 'Email : mbeattie@netapp.com 'Created : 2016-08-02 'Description : This script invokes the "netgroups-get-iter" ZAPI to enumerate ' : and iterate through a list of netgroups on a vserver '-----------------------------------------------------------------------------#> param( [Parameter(Mandatory = $True, HelpMessage = "The name of the cluster")] [String]$ClusterName, [Parameter(Mandatory = $False, HelpMessage = "The name of the vserver")] [String]$VserverName ) #'------------------------------------------------------------------------------ #'Initialization Section. Define Global Variables. #'------------------------------------------------------------------------------ [String]$scriptPath = Split-Path($MyInvocation.MyCommand.Path) [String]$scriptSpec = $MyInvocation.MyCommand.Definition [String]$scriptBaseName = (Get-Item $scriptSpec).BaseName [String]$scriptName = (Get-Item $scriptSpec).Name [String]$fileSpec = "$scriptPath\ManageOntap.dll" #'------------------------------------------------------------------------------ #'Ensure the ManageONTAP.dll file exists in the scripts working directory. #'------------------------------------------------------------------------------ If(-Not(Test-Path -Path $fileSpec)){ Write-Warning "The file ""$fileSpec"" does not exist. Exiting" Exit } #'------------------------------------------------------------------------------ #'Load the ManageONTAP.dll file #'------------------------------------------------------------------------------ Try{ [Reflection.Assembly]::LoadFile($fileSpec) | Out-Null Write-Host "Loaded file ""$fileSpec""" }Catch{ Write-Warning -Message $("Failed loading file ""$fileSpec"". Error " + $_.Exception.Message) Exit -1 } #'------------------------------------------------------------------------------ #'Create a ZAPI connection to the cluster using ZAPI version 1.31 (required for netgroups). #'------------------------------------------------------------------------------ [NetApp.Manage.NaServer]$naServer = New-Object NetApp.Manage.NaServer($ClusterName, "1", "31") #'------------------------------------------------------------------------------ #'Enuemate credentials to connect ot the cluster. #'------------------------------------------------------------------------------ $credentials = $host.ui.PromptForCredential("Connect to ""$ClusterName""", "Please enter the user name and password", "", "") #'------------------------------------------------------------------------------ #'Enumerate the username and password from the PowerShell credential object. #'------------------------------------------------------------------------------ [String]$domain = $credentials.GetNetworkCredential().domain [String]$user = $credentials.GetNetworkCredential().username [String]$password = $credentials.GetNetworkCredential().password If($domain -ne "" -Or $domain -ne $Null){ If($domain.Contains(".")){ [String]$userName = "$user`@$domain" }Else{ If($credentials.UserName.Contains("\") -And $credentials.UserName.SubString(0, 1) -ne "\"){ [String]$userName = $credentials.UserName }Else{ [String]$userName = $user } } } #'------------------------------------------------------------------------------ #'Set the username and password for the NaServer object. #'------------------------------------------------------------------------------ $naServer.SetAdminUser($userName, $password) $naServer.ServerType = 'FILER' $naServer.TransportType = 'HTTPS' #'------------------------------------------------------------------------------ #'Invoke the ZAPI to list the netgroups #'------------------------------------------------------------------------------ [String]$zapiName = "netgroups-file-get-iter" [String]$tag = "start" Write-Host "Invoking ZAPI ""$zapiName""" Write-Host "NetGroupName,Users" While($tag){ $api = New-Object NetApp.Manage.NaElement("$zapiName") $xi1 = New-Object NetApp.Manage.NaElement("query"); $api.AddChildElement($xi1); $xi2 = New-Object NetApp.Manage.NaElement("netgroups-file-config-info"); $xi1.AddChildElement($xi2); $xi2.AddNewChild("vserver", $VserverName); If($tag -ne "start"){ $api.AddNewChild("tag", $tag) } Try{ $results = $NaServer.InvokeElem($api) }Catch{ Write-Warning -Message $("Failed invoking ""$zapiName"". Error " + $_.Exception.Message) Exit -1 } $tag = $results.GetChildContent("next-tag") $records = $results.GetChildContent("num-records") Write-Host "There are $records records in the current iteration" If($results -ne $Null -And $records -ne 0){ ForEach($result in ($results.GetChildByName("attributes-list").GetChildren())){ [String]$netGroupName = $result.GetChildContent("netgroup-name") [String]$netGroupUser = $result.GetChildContent("user") Write-Host "$netGroupName,$netGroupUser" } }Else{ Write-Host "There were $records records in the iteration" } } Write-Host "Done" #'------------------------------------------------------------------------------
Output:
PS C:\Scripts\PowerShell\Projects\NetGroups> .\NetGroups.ps1 -ClusterName cluster3 -VserverName testnfs1 Loaded file "C:\Scripts\PowerShell\Projects\NetGroups\ManageOntap.dll" Invoking ZAPI "netgroups-file-get-iter" NetGroupName,Users There are 20 records in the current iteration group1,user1 group2,user2 group3,user3 group4,user4 group5,user5 group6,user6 group7,user7 group8,user8 group9,user9 group10,user10 group11,user11 group12,user12 group13,user13 group14,user14 group15,user15 group16,user16 group17,user17 group18,user18 group19,user19 group20,user20 There are 0 records in the current iteration There were 0 records in the iteration Done
CLI output
cluster3::> netgroup file show -vserver testnfs1 -fields netgroup,user vserver netgroup user -------- -------- -------- testnfs1 group1 user1 testnfs1 group2 user2 testnfs1 group3 user3 testnfs1 group4 user4 testnfs1 group5 user5 testnfs1 group6 user6 testnfs1 group7 user7 testnfs1 group8 user8 testnfs1 group9 user9 testnfs1 group10 user10 testnfs1 group11 user11 testnfs1 group12 user12 testnfs1 group13 user13 testnfs1 group14 user14 testnfs1 group15 user15 testnfs1 group16 user16 testnfs1 group17 user17 testnfs1 group18 user18 testnfs1 group19 user19 testnfs1 group20 user20 testnfs1 group21 user21 testnfs1 group22 user22 testnfs1 group23 user23 testnfs1 group24 user24 testnfs1 group25 user25 25 entries were displayed.
I think this is a BUG with the "netgroups-file-get-iter" ZAPI.
/matt