Network and Storage Protocols

Access Denied setting up NFSv3 on OnTap 9.2

james48
31,159 Views

I’m trying to test an existing NFS configuration before moving from Ontap 8.3 to 9.2. We’ve setup a 9.2 simulator (details at the end) with cifs and nfs shared and ntfs permissions. This works fine with Windows 7 & 10 clients, but getting “access denied” errors when mounting to nfs on CentOS 6.9. Do you have any ideas why?

 

Here’s the Error:

 

[root@centos6 ~]# echo 32767 > /proc/sys/sunrpc/nfs_debug

[root@centos6 ~]# mount -a
mount.nfs: Connection timed out
mount.nfs: access denied by server while mounting simshare.bu.edu:/cifs_test

 

The CentOs target can see the share, as can Windows clients:

 

[root@centos6 ~]# showmount -e simshare
Export list for simshare:
/cifs_test (everyone)
/          (everyone)

 

Here’s the corresponding “error -13” from /var/log/messages with NFS debugging on

 

Jun 14 09:03:11 centos6 kernel: NFS: nfs mount opts='soft,sec=krb5,nolock,noacl,rsize=8192,wsize=8192,addr=10.241.33.108,vers
=3,proto=tcp,mountvers=3,mountproto=tcp,mountport=635'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'soft'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'sec=krb5'
Jun 14 09:03:11 centos6 kernel: NFS: parsing sec=krb5 option
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'nolock'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'noacl'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'rsize=8192'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'wsize=8192'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'addr=10.241.33.108'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'vers=3'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'proto=tcp'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'mountvers=3'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'mountproto=tcp'
Jun 14 09:03:11 centos6 kernel: NFS:   parsing nfs mount option 'mountport=635'
Jun 14 09:03:11 centos6 kernel: NFS: MNTPATH: '/cifs_test'
Jun 14 09:03:11 centos6 kernel: NFS: sending MNT request for simshare.bu.edu:/cifs_test
Jun 14 09:03:11 centos6 kernel: NFS: MNT server returned result -13
Jun 14 09:03:11 centos6 kernel: NFS: unable to mount server simshare.bu.edu, error -13



 

Here’s the configuration on my target CentOS 6 system

 

/etc/fstab (The relevant line)

 

simshare.bu.edu:/sim_test /sim/sim_test nfs \

vers=3,rw,tcp,soft,sec=krb5,nolock,noacl,rsize=8192,wsize=8192,noatime 0 0

/etc/krb5.conf [with edits]

 

[libdefaults]
default_realm = AD.BU.EDU
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
allow_weak_crypto = true          # will remove this once things work

[realms]
AD.BU.EDU = {
 kdc = ad.bu.edu.
}

# Mapping of domains to kerberos realms.
#
# These entries will at least map any reference to an active directory hostname
# to the realm, and if we wanted we could also point bu.edu to that as well.
# As per the docs on krb5.conf, an entry starting with a period is for a whole
# domain, while one without specifies an actual host.
[domain_realm]
.ad.bu.edu = AD.BU.EDU
ad.bu.edu = AD.BU.EDU

[appdefaults]
pam = {
 minimum_uid = 3000
}

/etc/sssd/sssd.conf

 

# This configures SSSD to use generic LDAP and krb5 interfaces to connect to
# AD.  There is also an AD provider, but it is fairly recent and not as well
# documented.

[sssd]
config_file_version = 2
domains = AD
services = nss, pam

[nss]

### Options for the sss entries in /etc/nsswitch.conf.

override_homedir = /home/%u

# Maybe there's a shell entry in LDAP somewhere, but by default it can't find
# one.  This works fine, though.
default_shell = /bin/bash

[pam]

### Options for the sss entries in /etc/pam.d/.
# We don't need to configure anything here.

[domain/AD]

### Select providers for each service
id_provider     = ldap
auth_provider   = krb5
chpass_provider = krb5
access_provider = ldap

### General settings

# If the system's hostname isn't under ad.bu.edu, this will make SSSD still use
# ad.bu.edu for discovering LDAP/kerberos/etc. servers.
dns_discovery_domain = ad.bu.edu



For reference here’s the OnTap 9.2 Simulator setup

 

simshare-clu::*> vserver nfs show

Virtual      General

Server       Access v3    v4.0 v4.1 UDP      TCP

------------ ------- -------- -------- -------- -------- --------

svm1         true enabled  enabled disabled enabled  enabled

 

simshare-clu::*> set -privilege advanced

simshare-clu::*> nfs show -vserver svm1

 

                                          Vserver: svm1

                               General NFS Access: true

            RPC GSS Context Cache High Water Mark: 0

                             RPC GSS Context Idle: 0

                                           NFS v3: enabled

                                         NFS v4.0: enabled

                                     UDP Protocol: enabled

                                     TCP Protocol: enabled

                             Default Windows User: guest

                      Enable NFSv3 EJUKEBOX error: true

Require All NFSv3 Reads to Return Read Attributes: false

Show Change in FSID as NFSv3 Clients Traverse Filesystems: enabled

Enable the Dropping of a Connection When an NFSv3 Request is Dropped: enabled

               Vserver NTFS Unix Security Options: use_export_policy

                    Vserver Change Ownership Mode: use_export_policy

                       NFS Response Trace Enabled: false

                   NFS Response Trigger (in secs): 60

                UDP Maximum Transfer Size (bytes): 32768

                TCP Maximum Transfer Size (bytes): 65536

              NFSv3 TCP Maximum Read Size (bytes): 65536

             NFSv3 TCP Maximum Write Size (bytes): 65536

                              NFSv4.0 ACL Support: disabled

                  NFSv4.0 Read Delegation Support: disabled

                 NFSv4.0 Write Delegation Support: disabled

Show Change in FSID as NFSv4 Clients Traverse Filesystems: enabled

                         NFSv4.0 Referral Support: disabled

                          NFSv4 ID Mapping Domain: bu.edu

NFSv4 Validate UTF-8 Encoding of Symbolic Link Data: disabled

              NFSv4 Lease Timeout Value (in secs): 30

              NFSv4 Grace Timeout Value (in secs): 45

Preserves and Modifies NFSv4 ACL (and NTFS File Permissions in Unified Security Style): enabled

                    NFSv4.1 Minor Version Support: disabled

                                    Rquota Enable: enabled

                 NFSv4.1 Implementation ID Domain: netapp.com

                   NFSv4.1 Implementation ID Name: NetApp Release 9.2

                   NFSv4.1 Implementation ID Date: Mon Jun 19 18:20:04 2017

                     NFSv4.1 Parallel NFS Support: enabled

                         NFSv4.1 Referral Support: disabled

                              NFSv4.1 ACL Support: disabled

                             NFS vStorage Support: disabled

              NFSv4 Support for Numeric Owner IDs: enabled

                            Default Windows Group: Everyone

                  NFSv4.1 Read Delegation Support: disabled

                 NFSv4.1 Write Delegation Support: disabled

Number of Slots in the NFSv4.x Session slot tables: 180

Size of the Reply that will be Cached in Each NFSv4.x Session Slot (in bytes): 640

                   Maximum Number of ACEs per ACL: 400
                              NFS Mount Root Only: disabled
                                    NFS Root Only: disabled
                 AUTH_SYS Extended Groups Enabled: disabled                   
   AUTH_SYS and RPCSEC_GSS Auxillary Groups Limit: 32
Validation of Qtree IDs for Qtree File Operations: enabled
                            NFS Mount Daemon Port: 635
                        Network Lock Manager Port: 4045
                      Network Status Monitor Port: 4046
                            NFS Quota Daemon Port: 4049
              Permitted Kerberos Encryption Types: des, des3, aes-128, aes-256
                                Showmount Enabled: enabled
Set the Protocol Used for Name Services Lookups for Exports: udp
          Map Unknown UID to Default Windows User: enable
 DNS Domain Search Enabled During Netgroup Lookup: enabled
Trust No-Match Result from Any Name Service Switch Source During Netgroup Lookup: disabled
 Display maximum NT ACL Permissions to NFS Client: disabled
                      NFSv3 MS-DOS Client Support: disabled
      Ignore the NT ACL Check for NFS User 'root': disabled
Time To Live Value (in msecs) of a Positive Cached Credential: 86400000
Time To Live Value (in msecs) of a Negative Cached Credential: 7200000
Skip Permission Check for NFS Write Calls from Root/Owner: disabled
         Use 64 Bits for NFSv3 FSIDs and File IDs: disabled
Ignore Client Specified Mode Bits and Preserve Inherited NFSv4 ACL When Creating New Files or Directories: disabled
          Fallback to Unconverted Filename Search: disabled
             I/O Count to Be Grouped as a Session: 5000
Duration for I/O to Be Grouped as a Session (Secs): 120
      Enable or disable Checksum for Replay-Cache: enabled



 

simshare-clu::*> vserver cifs options show -vserver svm1

Vserver: svm1

                           Client Session Timeout: 900
                             Copy Offload Enabled: true
                               Default Unix Group: -
                                Default Unix User: guest
                                  Guest Unix User: -
              Are Administrators mapped to 'root': true
          Is Advanced Sparse File Support Enabled: true
                 Direct-Copy Copy Offload Enabled: true
                          Export Policies Enabled: false
           Grant Unix Group Permissions to Others: false
                         Is Advertise DFS Enabled: false
    Is Client Duplicate Session Detection Enabled: true
              Is Client Version Reporting Enabled: true
                                   Is DAC Enabled: false
                     Is Fake Open Support Enabled: true
                        Is Hide Dot Files Enabled: false
                             Is Large MTU Enabled: false
                            Is Local Auth Enabled: true
                Is Local Users and Groups Enabled: true
           Is NetBIOS over TCP (port 139) Enabled: true
              Is NBNS over UDP (port 137) Enabled: false
                              Is Referral Enabled: false
            Is Search Short Names Support Enabled: false
 Is Trusted Domain Enumeration And Search Enabled: true
                       Is UNIX Extensions Enabled: false
         Is Use Junction as Reparse Point Enabled: true
                              Max Multiplex Count: 255
             Max Same User Session Per Connection: 2050
                Max Same Tree Connect Per Session: 4096
                     Max Opens Same File Per Tree: 800
                         Max Watches Set Per Tree: 100
                  Is Path Component Cache Enabled: true
   NT ACLs on UNIX Security Style Volumes Enabled: true
                                 Read Grants Exec: disabled
                                 Read Only Delete: disabled
                 Reported File System Sector Size: 4096
                               Restrict Anonymous: no-restriction
                             Shadowcopy Dir Depth: 5
                               Shadowcopy Enabled: true
                                     SMB1 Enabled: true
                 Max Buffer Size for SMB1 Message: 65535
                                     SMB2 Enabled: true
                                     SMB3 Enabled: true
                                   SMB3.1 Enabled: true
           Map Null User to Windows User or Group: -
                                     WINS Servers: -
        Report Widelink as Reparse Point Versions: SMB1



simshare-clu::*> vserver nfs kerberos interface show

              Logical

Vserver        Interface Address         Kerberos SPN

-------------- ------------- --------------- -------- -----------------------

svm1           lif_1 10.241.###.###  enabled nfs/simshare.bu.edu@AD.BU.EDU



simshare-clu::*> vserver services unix-user show
              User User   Group Full
Vserver        Name ID     ID Name
-------------- --------------- ------ ------ --------------------------------
svm1           guest 65533  65534
svm1           nfs 500    0
svm1           nobody 65535  65535
svm1           pcuser 65534  65534
svm1           root 0      1
svm1           test 65532  65532




Here’s possibly some important differences between the simulator and actual NetApp running 8.2. The root, pcuser, and guest accounts differ:

 

nas-clu::> vserver services unix-user show
              User User   Group Full
Vserver        Name ID     ID Name
-------------- --------------- ------ ------ --------------------------------
engnas         guest 65534  65534
engnas         nfs 500    0
engnas         nobody 65535  65535 -
engnas         root 0      0
engnas         test 65532  65532 -

1 ACCEPTED SOLUTION
7 REPLIES 7

GidonMarcus
31,051 Views

Hi

 

i think you covered most of the configuration basics. but i can't see a fault in it.

You can use "vserver security trace" command to see why the filer deny the request (if it does).if it's not - a tcpdump from the client. or pktt from the filer will show a bit more.

 

other than that we missing a few more outputs in regard to permissions, a few output that i had in mind while looking on it:

 vserver nfs show -fields access,name-service-lookup-protocol,mount-rootonly,default-win-user,v4.0-acl,nfs-rootonly

vserver security file-directory show -path /sim_test -vserver sim1

vserver export-policy rule show

qtree show -fields security-style,qtree-path,export-policy

 

 

 

Gidi

Gidi Marcus (Linkedin) - Storage and Microsoft technologies consultant - Hydro IT LTD - UK

james48
30,922 Views

Hi Gidi,

 

Thanks for your response. Sorry mine was delayed, but I was on vacation. Here is the output of the queries you recommended. You'll see I have 2 other shares (i.e. app_test and cifs_test) that I didn't mention early and don't think we need to consider here. 

 

 

simshare-clu::*> vserver nfs show -fields access,name-service-lookup-protocol,mount-rootonly,default-win-user,v4.0-acl,nfs-rootonly
vserver access default-win-user v4.0-acl mount-rootonly nfs-rootonly name-service-lookup-protocol
------- ------ ---------------- -------- -------------- ------------ ----------------------------
svm1    true   guest            disabled disabled       disabled     udp


simshare-clu::*> vserver security file-directory show -path /sim_test -vserver svm1

               Vserver: svm1
             File Path: /sim_test
     File Inode Number: 64
        Security Style: ntfs
       Effective Style: ntfs
        DOS Attributes: 10
DOS Attributes in Text: ----D---
Expanded Dos Attributes: -
          UNIX User Id: 0
         UNIX Group Id: 0
        UNIX Mode Bits: 777
UNIX Mode Bits in Text: rwxrwxrwx
                  ACLs: NTFS Security Descriptor
                        Control:0x8004
                        Owner:BUILTIN\Administrators
                        Group:BUILTIN\Administrators
                        DACL - ACEs
                          ALLOW-Everyone-0x1f01ff
                          ALLOW-Everyone-0x10000000-OI|CI|IO

 

simshare-clu::*> vserver export-policy rule show
             Policy          Rule   Access   Client                RO
Vserver      Name            Index  Protocol Match                 Rule
------------ --------------- ------ -------- --------------------- ---------
svm1         app_test        1      any      0.0.0.0/0             any
svm1         cifs_first      1      cifs,    0.0.0.0/0             any
                                    nfs
svm1         nfs_first       1      nfs      0.0.0.0/0             krb5
svm1         nfs_first       2      cifs     0.0.0.0/0             any




simshare-clu::*> qtree show -fields security-style,qtree-path,export-policy
vserver volume   qtree qtree-path    security-style export-policy
------- -------- ----- ------------- -------------- -------------
svm1    app_test ""    /vol/app_test mixed          app_test
svm1    cifs_test
                 ""    /vol/cifs_test
                                     ntfs           nfs_first
svm1    sim_test ""    /vol/sim_test ntfs           cifs_first
svm1    svm_root ""    /vol/svm_root ntfs           default


 

Could the issue be the export policy on svm_root? I'll take a look.

james48
30,919 Views

Thanks D_BEREZENKO.

 

I'll start on the troublshooting guide.

james48
30,915 Views

Looks like root volume export policy was denying access.

 

simshare-clu::*> check-access -vserver svm1 -volume sim_test -client-ip 10.241.185.35 \

  -authentication-method krb5 -protocol nfs3 -access-type read
(vserver export-policy check-access)
                                         Policy    Policy     Rule
Path                          Policy     Owner     Owner Type Index  Access
----------------------------- ---------- --------- ---------- ------ ----------
/                             default    svm_root  volume     0      denied

 

 

I changed the export policy on svm_root from default, which had no rule, to base_vol, which allows any cifs,nfs,flexcache connection. I’m not sure this is the proper configuration, but now access checks out and I can mount the shares (albeit with permission errors).

 

simshare-clu::*> check-access -vserver svm1 -volume sim_test -client-ip 10.241.185.35 \

  -authentication-method krb5 -protocol nfs3 -access-type read
(vserver export-policy check-access)
                                         Policy    Policy     Rule
Path                          Policy     Owner     Owner Type Index  Access
----------------------------- ---------- --------- ---------- ------ ----------

/                             root_vol   svm_root  volume     1      read

/sim_test                     cifs_first sim_test  volume     1      read

 

 

It seems wrong to have / wide open to Everyone. Any thoughts?

GidonMarcus
30,859 Views

hi.

 

You should allow only read on the root and for subnets you trust (like 10.0.0.0)

 

G

Gidi Marcus (Linkedin) - Storage and Microsoft technologies consultant - Hydro IT LTD - UK

james48
30,844 Views

Thanks, G.

 

engnasim-clu::> vserver export-policy rule show
             Policy          Rule   Access   Client                RO
Vserver      Name            Index  Protocol Match                 Rule
------------ --------------- ------ -------- --------------------- ---------
svm1         cifs_first      1      cifs,nfs 0.0.0.0/0             any
svm1         root_vol        1      cifs,nfs 128.197.0.0/16        any
                                    flexcache
svm1         root_vol        2      cifs,nfs 10.0.0.0/8            any
                                    flexcache

Public