Network Storage Protocols Discussions

Access Denied setting up NFSv3 on OnTap 9.2

I’m trying to test an existing NFS configuration before moving from Ontap 8.3 to 9.2. We’ve setup a 9.2 simulator (details at the end) with cifs and nfs shared and ntfs permissions. This works fine with Windows 7 & 10 clients, but getting “access denied” errors when mounting to nfs on CentOS 6.9. Do you have any ideas why?

 

Here’s the Error:

 

[root@centos6 ~]# echo 32767 > /proc/sys/sunrpc/nfs_debug

[root@centos6 ~]# mount -a
mount.nfs: Connection timed out
mount.nfs: access denied by server while mounting simshare.bu.edu:/cifs_test

 

The CentOs target can see the share, as can Windows clients:

 

[root@centos6 ~]# showmount -e simshare
Export list for simshare:
/cifs_test (everyone)
/          (everyone)

 

Here’s the corresponding “error -13” from /var/log/messages with NFS debugging on

 

Jun 14 09:03:11 centos6 kernel: NFS: nfs mount opts='soft,sec=krb5,nolock,noacl,rsize=8192,wsize=8192,addr=10.241.33.108,vers
=3,proto=tcp,mountvers=3,mountproto=tcp,mountport=635'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'soft'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'sec=krb5'
Jun 14 09:03:11 centos6 kernel: NFS: parsing sec=krb5 option
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'nolock'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'noacl'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'rsize=8192'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'wsize=8192'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'addr=10.241.33.108'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'vers=3'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'proto=tcp'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'mountvers=3'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'mountproto=tcp'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'mountport=635'
Jun 14 09:03:11 centos6 kernel: NFS: MNTPATH: '/cifs_test'
Jun 14 09:03:11 centos6 kernel: NFS: sending MNT request for simshare.bu.edu:/cifs_test
Jun 14 09:03:11 centos6 kernel: NFS: MNT server returned result -13
Jun 14 09:03:11 centos6 kernel: NFS: unable to mount server simshare.bu.edu, error -13



 

Here’s the configuration on my target CentOS 6 system

 

/etc/fstab (The relevant line)

 

simshare.bu.edu:/sim_test /sim/sim_test nfs \

vers=3,rw,tcp,soft,sec=krb5,nolock,noacl,rsize=8192,wsize=8192,noatime 0 0

/etc/krb5.conf [with edits]

 

[libdefaults]
default_realm = AD.BU.EDU
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
allow_weak_crypto = true          # will remove this once things work

[realms]
AD.BU.EDU = {
 kdc = ad.bu.edu.
}

# Mapping of domains to kerberos realms.
#
# These entries will at least map any reference to an active directory hostname
# to the realm, and if we wanted we could also point bu.edu to that as well.
# As per the docs on krb5.conf, an entry starting with a period is for a whole
# domain, while one without specifies an actual host.
[domain_realm]
.ad.bu.edu = AD.BU.EDU
ad.bu.edu = AD.BU.EDU

[appdefaults]
pam = {
 minimum_uid = 3000
}

/etc/sssd/sssd.conf

 

# This configures SSSD to use generic LDAP and krb5 interfaces to connect to
# AD.  There is also an AD provider, but it is fairly recent and not as well
# documented.

[sssd]
config_file_version = 2
domains = AD
services = nss, pam

[nss]

### Options for the sss entries in /etc/nsswitch.conf.

override_homedir = /home/%u

# Maybe there's a shell entry in LDAP somewhere, but by default it can't find
# one.  This works fine, though.
default_shell = /bin/bash

[pam]

### Options for the sss entries in /etc/pam.d/.
# We don't need to configure anything here.

[domain/AD]

### Select providers for each service
id_provider     = ldap
auth_provider   = krb5
chpass_provider = krb5
access_provider = ldap

### General settings

# If the system's hostname isn't under ad.bu.edu, this will make SSSD still use
# ad.bu.edu for discovering LDAP/kerberos/etc. servers.
dns_discovery_domain = ad.bu.edu



For reference here’s the OnTap 9.2 Simulator setup

 

simshare-clu::*> vserver nfs show

Virtual      General

Server       Access v3    v4.0 v4.1 UDP      TCP

------------ ------- -------- -------- -------- -------- --------

svm1         true enabled  enabled disabled enabled  enabled

 

simshare-clu::*> set -privilege advanced

simshare-clu::*> nfs show -vserver svm1

 

                                          Vserver: svm1

                               General NFS Access: true

            RPC GSS Context Cache High Water Mark: 0

                             RPC GSS Context Idle: 0

                                           NFS v3: enabled

                                         NFS v4.0: enabled

                                     UDP Protocol: enabled

                                     TCP Protocol: enabled

                             Default Windows User: guest

                      Enable NFSv3 EJUKEBOX error: true

Require All NFSv3 Reads to Return Read Attributes: false

Show Change in FSID as NFSv3 Clients Traverse Filesystems: enabled

Enable the Dropping of a Connection When an NFSv3 Request is Dropped: enabled

               Vserver NTFS Unix Security Options: use_export_policy

                    Vserver Change Ownership Mode: use_export_policy

                       NFS Response Trace Enabled: false

                   NFS Response Trigger (in secs): 60

                UDP Maximum Transfer Size (bytes): 32768

                TCP Maximum Transfer Size (bytes): 65536

              NFSv3 TCP Maximum Read Size (bytes): 65536

             NFSv3 TCP Maximum Write Size (bytes): 65536

                              NFSv4.0 ACL Support: disabled

                  NFSv4.0 Read Delegation Support: disabled

                 NFSv4.0 Write Delegation Support: disabled

Show Change in FSID as NFSv4 Clients Traverse Filesystems: enabled

                         NFSv4.0 Referral Support: disabled

                          NFSv4 ID Mapping Domain: bu.edu

NFSv4 Validate UTF-8 Encoding of Symbolic Link Data: disabled

              NFSv4 Lease Timeout Value (in secs): 30

              NFSv4 Grace Timeout Value (in secs): 45

Preserves and Modifies NFSv4 ACL (and NTFS File Permissions in Unified Security Style): enabled

                    NFSv4.1 Minor Version Support: disabled

                                    Rquota Enable: enabled

                 NFSv4.1 Implementation ID Domain: netapp.com

                   NFSv4.1 Implementation ID Name: NetApp Release 9.2

                   NFSv4.1 Implementation ID Date: Mon Jun 19 18:20:04 2017

                     NFSv4.1 Parallel NFS Support: enabled

                         NFSv4.1 Referral Support: disabled

                              NFSv4.1 ACL Support: disabled

                             NFS vStorage Support: disabled

              NFSv4 Support for Numeric Owner IDs: enabled

                            Default Windows Group: Everyone

                  NFSv4.1 Read Delegation Support: disabled

                 NFSv4.1 Write Delegation Support: disabled

Number of Slots in the NFSv4.x Session slot tables: 180

Size of the Reply that will be Cached in Each NFSv4.x Session Slot (in bytes): 640

                   Maximum Number of ACEs per ACL: 400
                              NFS Mount Root Only: disabled
                                    NFS Root Only: disabled
                 AUTH_SYS Extended Groups Enabled: disabled                   
   AUTH_SYS and RPCSEC_GSS Auxillary Groups Limit: 32
Validation of Qtree IDs for Qtree File Operations: enabled
                            NFS Mount Daemon Port: 635
                        Network Lock Manager Port: 4045
                      Network Status Monitor Port: 4046
                            NFS Quota Daemon Port: 4049
              Permitted Kerberos Encryption Types: des, des3, aes-128, aes-256
                                Showmount Enabled: enabled
Set the Protocol Used for Name Services Lookups for Exports: udp
          Map Unknown UID to Default Windows User: enable
 DNS Domain Search Enabled During Netgroup Lookup: enabled
Trust No-Match Result from Any Name Service Switch Source During Netgroup Lookup: disabled
 Display maximum NT ACL Permissions to NFS Client: disabled
                      NFSv3 MS-DOS Client Support: disabled
      Ignore the NT ACL Check for NFS User 'root': disabled
Time To Live Value (in msecs) of a Positive Cached Credential: 86400000
Time To Live Value (in msecs) of a Negative Cached Credential: 7200000
Skip Permission Check for NFS Write Calls from Root/Owner: disabled
         Use 64 Bits for NFSv3 FSIDs and File IDs: disabled
Ignore Client Specified Mode Bits and Preserve Inherited NFSv4 ACL When Creating New Files or Directories: disabled
          Fallback to Unconverted Filename Search: disabled
             I/O Count to Be Grouped as a Session: 5000
Duration for I/O to Be Grouped as a Session (Secs): 120
      Enable or disable Checksum for Replay-Cache: enabled



 

simshare-clu::*> vserver cifs options show -vserver svm1

Vserver: svm1

                           Client Session Timeout: 900
                             Copy Offload Enabled: true
                               Default Unix Group: -
                                Default Unix User: guest
                                  Guest Unix User: -
              Are Administrators mapped to 'root': true
          Is Advanced Sparse File Support Enabled: true
                 Direct-Copy Copy Offload Enabled: true
                          Export Policies Enabled: false
           Grant Unix Group Permissions to Others: false
                         Is Advertise DFS Enabled: false
    Is Client Duplicate Session Detection Enabled: true
              Is Client Version Reporting Enabled: true
                                   Is DAC Enabled: false
                     Is Fake Open Support Enabled: true
                        Is Hide Dot Files Enabled: false
                             Is Large MTU Enabled: false
                            Is Local Auth Enabled: true
                Is Local Users and Groups Enabled: true
           Is NetBIOS over TCP (port 139) Enabled: true
              Is NBNS over UDP (port 137) Enabled: false
                              Is Referral Enabled: false
            Is Search Short Names Support Enabled: false
 Is Trusted Domain Enumeration And Search Enabled: true
                       Is UNIX Extensions Enabled: false
         Is Use Junction as Reparse Point Enabled: true
                              Max Multiplex Count: 255
             Max Same User Session Per Connection: 2050
                Max Same Tree Connect Per Session: 4096
                     Max Opens Same File Per Tree: 800
                         Max Watches Set Per Tree: 100
                  Is Path Component Cache Enabled: true
   NT ACLs on UNIX Security Style Volumes Enabled: true
                                 Read Grants Exec: disabled
                                 Read Only Delete: disabled
                 Reported File System Sector Size: 4096
                               Restrict Anonymous: no-restriction
                             Shadowcopy Dir Depth: 5
                               Shadowcopy Enabled: true
                                     SMB1 Enabled: true
                 Max Buffer Size for SMB1 Message: 65535
                                     SMB2 Enabled: true
                                     SMB3 Enabled: true
                                   SMB3.1 Enabled: true
           Map Null User to Windows User or Group: -
                                     WINS Servers: -
        Report Widelink as Reparse Point Versions: SMB1



simshare-clu::*> vserver nfs kerberos interface show

              Logical

Vserver        Interface Address         Kerberos SPN

-------------- ------------- --------------- -------- -----------------------

svm1           lif_1 10.241.###.###  enabled nfs/simshare.bu.edu@AD.BU.EDU



simshare-clu::*> vserver services unix-user show
              User User   Group Full
Vserver        Name ID     ID Name
-------------- --------------- ------ ------ --------------------------------
svm1           guest 65533  65534
svm1           nfs 500    0
svm1           nobody 65535  65535
svm1           pcuser 65534  65534
svm1           root 0      1
svm1           test 65532  65532




Here’s possibly some important differences between the simulator and actual NetApp running 8.2. The root, pcuser, and guest accounts differ:

 

nas-clu::> vserver services unix-user show
              User User   Group Full
Vserver        Name ID     ID Name
-------------- --------------- ------ ------ --------------------------------
engnas         guest 65534  65534
engnas         nfs 500    0
engnas         nobody 65535  65535 -
engnas         root 0      0
engnas         test 65532  65532 -

7 REPLIES 7

Re: Access Denied setting up NFSv3 on OnTap 9.2

Hi

 

i think you covered most of the configuration basics. but i can't see a fault in it.

You can use "vserver security trace" command to see why the filer deny the request (if it does).if it's not - a tcpdump from the client. or pktt from the filer will show a bit more.

 

other than that we missing a few more outputs in regard to permissions, a few output that i had in mind while looking on it:

 vserver nfs show -fields access,name-service-lookup-protocol,mount-rootonly,default-win-user,v4.0-acl,nfs-rootonly

vserver security file-directory show -path /sim_test -vserver sim1

vserver export-policy rule show

qtree show -fields security-style,qtree-path,export-policy

 

 

 

Gidi

Gidi Marcus (Linkedin) - Storage and Microsoft technologies consultant - Hydro IT LTD - UK

Re: Access Denied setting up NFSv3 on OnTap 9.2

Hi Gidi,

 

Thanks for your response. Sorry mine was delayed, but I was on vacation. Here is the output of the queries you recommended. You'll see I have 2 other shares (i.e. app_test and cifs_test) that I didn't mention early and don't think we need to consider here. 

 

 

simshare-clu::*> vserver nfs show -fields access,name-service-lookup-protocol,mount-rootonly,default-win-user,v4.0-acl,nfs-rootonly
vserver access default-win-user v4.0-acl mount-rootonly nfs-rootonly name-service-lookup-protocol
------- ------ ---------------- -------- -------------- ------------ ----------------------------
svm1    true   guest            disabled disabled       disabled     udp


simshare-clu::*> vserver security file-directory show -path /sim_test -vserver svm1

               Vserver: svm1
             File Path: /sim_test
     File Inode Number: 64
        Security Style: ntfs
       Effective Style: ntfs
        DOS Attributes: 10
DOS Attributes in Text: ----D---
Expanded Dos Attributes: -
          UNIX User Id: 0
         UNIX Group Id: 0
        UNIX Mode Bits: 777
UNIX Mode Bits in Text: rwxrwxrwx
                  ACLs: NTFS Security Descriptor
                        Control:0x8004
                        Owner:BUILTIN\Administrators
                        Group:BUILTIN\Administrators
                        DACL - ACEs
                          ALLOW-Everyone-0x1f01ff
                          ALLOW-Everyone-0x10000000-OI|CI|IO

 

simshare-clu::*> vserver export-policy rule show
             Policy          Rule   Access   Client                RO
Vserver      Name            Index  Protocol Match                 Rule
------------ --------------- ------ -------- --------------------- ---------
svm1         app_test        1      any      0.0.0.0/0             any
svm1         cifs_first      1      cifs,    0.0.0.0/0             any
                                    nfs
svm1         nfs_first       1      nfs      0.0.0.0/0             krb5
svm1         nfs_first       2      cifs     0.0.0.0/0             any




simshare-clu::*> qtree show -fields security-style,qtree-path,export-policy
vserver volume   qtree qtree-path    security-style export-policy
------- -------- ----- ------------- -------------- -------------
svm1    app_test ""    /vol/app_test mixed          app_test
svm1    cifs_test
                 ""    /vol/cifs_test
                                     ntfs           nfs_first
svm1    sim_test ""    /vol/sim_test ntfs           cifs_first
svm1    svm_root ""    /vol/svm_root ntfs           default


 

Could the issue be the export policy on svm_root? I'll take a look.

Re: Access Denied setting up NFSv3 on OnTap 9.2

Thanks D_BEREZENKO.

 

I'll start on the troublshooting guide.

Re: Access Denied setting up NFSv3 on OnTap 9.2

Looks like root volume export policy was denying access.

 

simshare-clu::*> check-access -vserver svm1 -volume sim_test -client-ip 10.241.185.35 \

  -authentication-method krb5 -protocol nfs3 -access-type read
(vserver export-policy check-access)
                                         Policy    Policy     Rule
Path                          Policy     Owner     Owner Type Index  Access
----------------------------- ---------- --------- ---------- ------ ----------
/                             default    svm_root  volume     0      denied

 

 

I changed the export policy on svm_root from default, which had no rule, to base_vol, which allows any cifs,nfs,flexcache connection. I’m not sure this is the proper configuration, but now access checks out and I can mount the shares (albeit with permission errors).

 

simshare-clu::*> check-access -vserver svm1 -volume sim_test -client-ip 10.241.185.35 \

  -authentication-method krb5 -protocol nfs3 -access-type read
(vserver export-policy check-access)
                                         Policy    Policy     Rule
Path                          Policy     Owner     Owner Type Index  Access
----------------------------- ---------- --------- ---------- ------ ----------

/                             root_vol   svm_root  volume     1      read

/sim_test                     cifs_first sim_test  volume     1      read

 

 

It seems wrong to have / wide open to Everyone. Any thoughts?

Re: Access Denied setting up NFSv3 on OnTap 9.2

hi.

 

You should allow only read on the root and for subnets you trust (like 10.0.0.0)

 

G

Gidi Marcus (Linkedin) - Storage and Microsoft technologies consultant - Hydro IT LTD - UK

Re: Access Denied setting up NFSv3 on OnTap 9.2

Thanks, G.

 

engnasim-clu::> vserver export-policy rule show
             Policy          Rule   Access   Client                RO
Vserver      Name            Index  Protocol Match                 Rule
------------ --------------- ------ -------- --------------------- ---------
svm1         cifs_first      1      cifs,nfs 0.0.0.0/0             any
svm1         root_vol        1      cifs,nfs 128.197.0.0/16        any
                                    flexcache
svm1         root_vol        2      cifs,nfs 10.0.0.0/8            any
                                    flexcache

Forums