Connecting with PS Toolkit

[srijan]

Category1: File System
Category2: Data ONTAP/WAFL
Detailed Symptom: [CORE]Issues connecting to filer noted
Priority: 3
OS SW Version: 8.1.4P2 7-MODE
Model: FAS6280-R5
What have you done so far?: 
Netapp powershell Toolkit 4
- Adrian has Netapp power toolkit installed on the host 
- Andrian has shown me over the web-ex with an example of 2 nodes.
- 1 node for 7 mode the customer is able to login successfully and for 1 node the login process takes up to much of time and does not proceed anywhere.
- checked the user mapping that was correct and identical accept for the node which does not connect has membership to "builtin\user"
- checked the httpd options and could see that it is identical , except the user has provided httpd.rootdir as "/vol/vol0/home/http"
- checked the configuration of useradmin group list ,which shows "sdlg_group" as extra field on the node which user is able to connect using Netapp power toolkit.
- I have collected the screen shot of the issues and I am attaching the same to the case.
- when we tried to collect logs from the powershell it gave an error stating "Access denied"

Documents refereed : 

https://www.netapp.com/us/media/tr-4475.pdf

https://kb.netapp.com/support/s/article/ka31A0000008hTaQAI/how-to-collect-debug-logs- for-the-data-ontap-powershell-toolkit?language=en_US

https://kb.netapp.com/support/s/article/ka11A0000001RGPQA2/how-to-set-minimum- rights-for-the-powershell-toolkit-login-with-clustered-data-ontap
( no document I could find for 7 mode)

I have also collected the ouput for  " get-nahelp "
What do you need help with?: 
we need to know why the filer is not connecting on Netapp Powershell toolkit

- the credentials appear to be mapped to PC user 
- the host is windows server 2008 
- the Netapp powershell toolkit version is 4
Detailed Error message: 
- the images are updated on the case as "case attachments" as the stingray and light  is not working.

Outputs for the follows are collected:
- wcc -s 
- options httpd
- Powershell screen shots for the node which is able to connect and for he node where is is not able to connect to.
- Powershell Log.txt  we couldnt collect it.

[mbeattie]

Hi,

What’s the output of the following commands from a controller in the environment that does work compared to the one that fails to connect:

•    options httpd
•    options ssl
•    options tls
•    secureadmin status

I'd assume based on the case notes that customer has setup the authorized_keys in sshd for passwordless authentication on the controller and are attempting to connect to via RPC

https://kb.netapp.com/support/s/article/ka31A0000000oqlQAA/how-to-setup-ssh-public-key-authentication-on-windows-using-putty?language=en_US

Have you attempted to connect to the controller using Zexplore (from the NMSDK)? If you can't connect using Zexplore then it's likely to be a configuration issue (not a "PowerShell" problem). Here are some instuctions to connect to the conntroller using ZExplore, it requires JRE to be installed as it's a a Java application.

• Download the NMSDK here: https://mysupport.netapp.com/NOW/download/software/nmsdk/5.7/
• Extract Zexplore.exe and ZExplore.jar from the "zedi" in the NMSDK files
• From a command prompt run "java -jar zexplore.jar" or open the zexplore.exe (and wait for the application to load)
• Select the ZAPI version

• Select "Preferences\Connect" and enter the hostname and credentials to connect to the controller

• Select the vFiler (if required) and click connect

If your ZExplore connection fails then it's a configuration issue on the controller and is not related to the PowerShell toolkit (which is just a C# wrapper around the NMSDK

Hope that helps with your troubleshooting

/Matt

[srijan]

Hello,

Below are the output from a failing controller

emomni0117c01> options httpd

httpd.access                 legacy

httpd.admin.access           legacy

httpd.admin.enable           on

httpd.admin.hostsequiv.enable off

httpd.admin.max_connections  512

httpd.admin.ssl.enable       on

httpd.admin.top-page.authentication off

httpd.autoindex.enable       off

httpd.bypass_traverse_checking off

httpd.enable                 off

httpd.ipv6.enable            off

httpd.log.format             common     (value might be overwritten in takeover)

httpd.method.trace.enable    off

httpd.rootdir                /vol/vol0/home/http

httpd.timeout                300        (value might be overwritten in takeover)

httpd.timewait.enable        off        (value might be overwritten in takeover)

emomni0117c01>

emomni0117c01> options ssl

ssl.enable                   on

ssl.v2.enable                off        (same value required in local+partner)

ssl.v3.enable                off        (same value required in local+partner)

emomni0117c01>

emomni0117c01>

emomni0117c01> options tls

tls.enable                   on         (same value required in local+partner)

emomni0117c01>

emomni0117c01> secureadmin show

Usage:

        secureadmin setup [-f] [-q] ssh

        secureadmin setup [-f] [-q] ssl

        secureadmin addcert ssl [<path to CA signed cert>]

        secureadmin enable  all|ssh|ssh1|ssh2|ssl

        secureadmin disable all|ssh|ssh1|ssh2|ssl

        secureadmin status

emomni0117c01> secureadmin status

ssh2    - active

ssh1    - inactive

ssl     - active

emomni0117c01>

here are the same options in a working filer….

emomni0120c01> options httpd

httpd.access                 legacy

httpd.admin.access           legacy

httpd.admin.enable           on

httpd.admin.hostsequiv.enable off

httpd.admin.max_connections  512

httpd.admin.ssl.enable       on

httpd.admin.top-page.authentication off

httpd.autoindex.enable       on

httpd.bypass_traverse_checking off

httpd.enable                 off

httpd.ipv6.enable            off

httpd.log.format             common     (value might be overwritten in takeover)

httpd.method.trace.enable    off

httpd.rootdir                /vol/vol0/home/http

httpd.timeout                300        (value might be overwritten in takeover)

httpd.timewait.enable        off        (value might be overwritten in takeover)

emomni0120c01>

emomni0120c01> options ssl

ssl.enable                   on

ssl.v2.enable                off        (same value required in local+partner)

ssl.v3.enable                off        (same value required in local+partner)

emomni0120c01>

emomni0120c01> options tls

tls.enable                   on         (same value required in local+partner)

emomni0120c01>

emomni0120c01> secureadmin status

ssh2    - active

ssh1    - inactive

ssl     - active

emomni0120c01>

the only difference, between the 2 , that the customer can see is

httpd.autoindex.enable      

is set to off on 117c01 ( failing) , and its set on on on 0120c01 ( working)

Also, customer is able to  connect to the controller using Zexplore.

I have attached more findings on the case from the email with the customer.

Thank You.

[mbeattie]

Hi,

Thanks for the update, when comparing the configuration to the 7-Mode simulator im using for testing i can't see any issue.

What security context did the customer connect to the controller as when using ZExplore (as root or using domain credentials?)

Is the customer able to connect to the controller using PowerShell as the root user either via HTTPS or RPC

Import-Module DataONTAP
$controllerName = "x.x.x.x" #Update to set this variable to the controller hostname or IP Address.
$credentials = Get-Credential -Credential root

# test connection as root via HTTPS 
Connect-NaController -Name $controllerName -HTTPS -Credential $credentials

# test connection as root via RPC
Connect-NaController -Name $controllerName -RPC -Credential $credentials

If the answer is yes to both then i'd suspect that the issue is related to the users authorized_keys in sshd.

Have you tried copying the authorized_keys from the working controller to it's partner?

From: \\emomni0120c01\etc$\sshd\<%USERNAME%>\.ssh\authorized_keys
To: \\emomni0117c01\etc$\sshd\<%USERNAME%>\.ssh\authorized_keys

/Matt

[srijan]

Hello

Pease see the below ouput of the commands on the failing and the working controller

Here are some of the options in a failing filer….

emomni0117c01> options httpd

httpd.access                 legacy

httpd.admin.access           legacy

httpd.admin.enable           on

httpd.admin.hostsequiv.enable off

httpd.admin.max_connections  512

httpd.admin.ssl.enable       on

httpd.admin.top-page.authentication off

httpd.autoindex.enable       off

httpd.bypass_traverse_checking off

httpd.enable                 off

httpd.ipv6.enable            off

httpd.log.format             common     (value might be overwritten in takeover)

httpd.method.trace.enable    off

httpd.rootdir                /vol/vol0/home/http

httpd.timeout                300        (value might be overwritten in takeover)

httpd.timewait.enable        off        (value might be overwritten in takeover)

emomni0117c01>

emomni0117c01> options ssl

ssl.enable                   on

ssl.v2.enable                off        (same value required in local+partner)

ssl.v3.enable                off        (same value required in local+partner)

emomni0117c01>

emomni0117c01>

emomni0117c01> options tls

tls.enable                   on         (same value required in local+partner)

emomni0117c01>

emomni0117c01> secureadmin show

Usage:

        secureadmin setup [-f] [-q] ssh

        secureadmin setup [-f] [-q] ssl

        secureadmin addcert ssl [<path to CA signed cert>]

        secureadmin enable  all|ssh|ssh1|ssh2|ssl

        secureadmin disable all|ssh|ssh1|ssh2|ssl

        secureadmin status

emomni0117c01> secureadmin status

ssh2    - active

ssh1    - inactive

ssl     - active

emomni0117c01>

here are the same options ina working filer….

emomni0120c01> options httpd

httpd.access                 legacy

httpd.admin.access           legacy

httpd.admin.enable           on

httpd.admin.hostsequiv.enable off

httpd.admin.max_connections  512

httpd.admin.ssl.enable       on

httpd.admin.top-page.authentication off

httpd.autoindex.enable       on

httpd.bypass_traverse_checking off

httpd.enable                 off

httpd.ipv6.enable            off

httpd.log.format             common     (value might be overwritten in takeover)

httpd.method.trace.enable    off

httpd.rootdir                /vol/vol0/home/http

httpd.timeout                300        (value might be overwritten in takeover)

httpd.timewait.enable        off        (value might be overwritten in takeover)

emomni0120c01>

emomni0120c01> options ssl

ssl.enable                   on

ssl.v2.enable                off        (same value required in local+partner)

ssl.v3.enable                off        (same value required in local+partner)

emomni0120c01>

emomni0120c01> options tls

tls.enable                   on         (same value required in local+partner)

emomni0120c01>

emomni0120c01> secureadmin status

ssh2    - active

ssh1    - inactive

ssl     - active

emomni0120c01>

the only difference, between the 2 , that the customer can see is

httpd.autoindex.enable      

is set to off on 117c01 ( failing) , and its set on on on 0120c01 ( working)

Also customer is able to connect to the controller using Zexplore.

I have attached more findings on the case which the customer has shared over email. 

[ekashpureff]

srijan -

Per your post -

"

- the credentials appear to be mapped to PC user 

I don't think the default 'pcuser' account has any admin or api priviledges.

You probably want to connect as a dedicated script user or as 'admin'.

See post from asuliva above.

I hope this response has been helpful to you.

At your service,

Eugene E. Kashpureff, Sr.
Independent NetApp Consultant http://www.linkedin.com/in/eugenekashpureff
Senior NetApp Instructor, Fast Lane US http://www.fastlaneus.com/
(P.S. I appreciate 'kudos' on any helpful posts.)

[asulliva]

Hello @srijan,

In order to adjust the log level for the PSTK you'll need to run the PowerShell prompt as an administrator.  

Is authentication being attempted using RPC?  Does it succeed when credentials are explicitly provided, e.g.:

# use root or "domain\username" + password
$credential = Get-Credential

# optionally supply the connection protocol using -HTTP or -HTTPS
Connect-NaController -Name x.x.x.x -Credential $credential

Andrew