Network and Storage Protocols
Network and Storage Protocols
I’m trying to test an existing NFS configuration before moving from Ontap 8.3 to 9.2. We’ve setup a 9.2 simulator (details at the end) with cifs and nfs shared and ntfs permissions. This works fine with Windows 7 & 10 clients, but getting “access denied” errors when mounting to nfs on CentOS 6.9. Do you have any ideas why?
Here’s the Error:
[root@centos6 ~]# echo 32767 > /proc/sys/sunrpc/nfs_debug
[root@centos6 ~]# mount -a
mount.nfs: Connection timed out
mount.nfs: access denied by server while mounting simshare.bu.edu:/cifs_test
The CentOs target can see the share, as can Windows clients:
[root@centos6 ~]# showmount -e simshare
Export list for simshare:
/cifs_test (everyone)
/ (everyone)
Here’s the corresponding “error -13” from /var/log/messages with NFS debugging on
Jun 14 09:03:11 centos6 kernel: NFS: nfs mount opts='soft,sec=krb5,nolock,noacl,rsize=8192,wsize=8192,addr=10.241.33.108,vers
=3,proto=tcp,mountvers=3,mountproto=tcp,mountport=635'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'soft'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'sec=krb5'
Jun 14 09:03:11 centos6 kernel: NFS: parsing sec=krb5 option
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'nolock'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'noacl'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'rsize=8192'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'wsize=8192'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'addr=10.241.33.108'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'vers=3'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'proto=tcp'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'mountvers=3'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'mountproto=tcp'
Jun 14 09:03:11 centos6 kernel: NFS: parsing nfs mount option 'mountport=635'
Jun 14 09:03:11 centos6 kernel: NFS: MNTPATH: '/cifs_test'
Jun 14 09:03:11 centos6 kernel: NFS: sending MNT request for simshare.bu.edu:/cifs_test
Jun 14 09:03:11 centos6 kernel: NFS: MNT server returned result -13
Jun 14 09:03:11 centos6 kernel: NFS: unable to mount server simshare.bu.edu, error -13
Here’s the configuration on my target CentOS 6 system
/etc/fstab (The relevant line)
simshare.bu.edu:/sim_test /sim/sim_test nfs \
vers=3,rw,tcp,soft,sec=krb5,nolock,noacl,rsize=8192,wsize=8192,noatime 0 0
/etc/krb5.conf [with edits]
[libdefaults]
default_realm = AD.BU.EDU
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
allow_weak_crypto = true # will remove this once things work
[realms]
AD.BU.EDU = {
kdc = ad.bu.edu.
}
# Mapping of domains to kerberos realms.
#
# These entries will at least map any reference to an active directory hostname
# to the realm, and if we wanted we could also point bu.edu to that as well.
# As per the docs on krb5.conf, an entry starting with a period is for a whole
# domain, while one without specifies an actual host.
[domain_realm]
.ad.bu.edu = AD.BU.EDU
ad.bu.edu = AD.BU.EDU
[appdefaults]
pam = {
minimum_uid = 3000
}
/etc/sssd/sssd.conf
# This configures SSSD to use generic LDAP and krb5 interfaces to connect to
# AD. There is also an AD provider, but it is fairly recent and not as well
# documented.
[sssd]
config_file_version = 2
domains = AD
services = nss, pam
[nss]
### Options for the sss entries in /etc/nsswitch.conf.
override_homedir = /home/%u
# Maybe there's a shell entry in LDAP somewhere, but by default it can't find
# one. This works fine, though.
default_shell = /bin/bash
[pam]
### Options for the sss entries in /etc/pam.d/.
# We don't need to configure anything here.
[domain/AD]
### Select providers for each service
id_provider = ldap
auth_provider = krb5
chpass_provider = krb5
access_provider = ldap
### General settings
# If the system's hostname isn't under ad.bu.edu, this will make SSSD still use
# ad.bu.edu for discovering LDAP/kerberos/etc. servers.
dns_discovery_domain = ad.bu.edu
For reference here’s the OnTap 9.2 Simulator setup
simshare-clu::*> vserver nfs show
Virtual General
Server Access v3 v4.0 v4.1 UDP TCP
------------ ------- -------- -------- -------- -------- --------
svm1 true enabled enabled disabled enabled enabled
simshare-clu::*> set -privilege advanced
simshare-clu::*> nfs show -vserver svm1
Vserver: svm1
General NFS Access: true
RPC GSS Context Cache High Water Mark: 0
RPC GSS Context Idle: 0
NFS v3: enabled
NFS v4.0: enabled
UDP Protocol: enabled
TCP Protocol: enabled
Default Windows User: guest
Enable NFSv3 EJUKEBOX error: true
Require All NFSv3 Reads to Return Read Attributes: false
Show Change in FSID as NFSv3 Clients Traverse Filesystems: enabled
Enable the Dropping of a Connection When an NFSv3 Request is Dropped: enabled
Vserver NTFS Unix Security Options: use_export_policy
Vserver Change Ownership Mode: use_export_policy
NFS Response Trace Enabled: false
NFS Response Trigger (in secs): 60
UDP Maximum Transfer Size (bytes): 32768
TCP Maximum Transfer Size (bytes): 65536
NFSv3 TCP Maximum Read Size (bytes): 65536
NFSv3 TCP Maximum Write Size (bytes): 65536
NFSv4.0 ACL Support: disabled
NFSv4.0 Read Delegation Support: disabled
NFSv4.0 Write Delegation Support: disabled
Show Change in FSID as NFSv4 Clients Traverse Filesystems: enabled
NFSv4.0 Referral Support: disabled
NFSv4 ID Mapping Domain: bu.edu
NFSv4 Validate UTF-8 Encoding of Symbolic Link Data: disabled
NFSv4 Lease Timeout Value (in secs): 30
NFSv4 Grace Timeout Value (in secs): 45
Preserves and Modifies NFSv4 ACL (and NTFS File Permissions in Unified Security Style): enabled
NFSv4.1 Minor Version Support: disabled
Rquota Enable: enabled
NFSv4.1 Implementation ID Domain: netapp.com
NFSv4.1 Implementation ID Name: NetApp Release 9.2
NFSv4.1 Implementation ID Date: Mon Jun 19 18:20:04 2017
NFSv4.1 Parallel NFS Support: enabled
NFSv4.1 Referral Support: disabled
NFSv4.1 ACL Support: disabled
NFS vStorage Support: disabled
NFSv4 Support for Numeric Owner IDs: enabled
Default Windows Group: Everyone
NFSv4.1 Read Delegation Support: disabled
NFSv4.1 Write Delegation Support: disabled
Number of Slots in the NFSv4.x Session slot tables: 180
Size of the Reply that will be Cached in Each NFSv4.x Session Slot (in bytes): 640
Maximum Number of ACEs per ACL: 400
NFS Mount Root Only: disabled
NFS Root Only: disabled
AUTH_SYS Extended Groups Enabled: disabled
AUTH_SYS and RPCSEC_GSS Auxillary Groups Limit: 32
Validation of Qtree IDs for Qtree File Operations: enabled
NFS Mount Daemon Port: 635
Network Lock Manager Port: 4045
Network Status Monitor Port: 4046
NFS Quota Daemon Port: 4049
Permitted Kerberos Encryption Types: des, des3, aes-128, aes-256
Showmount Enabled: enabled
Set the Protocol Used for Name Services Lookups for Exports: udp
Map Unknown UID to Default Windows User: enable
DNS Domain Search Enabled During Netgroup Lookup: enabled
Trust No-Match Result from Any Name Service Switch Source During Netgroup Lookup: disabled
Display maximum NT ACL Permissions to NFS Client: disabled
NFSv3 MS-DOS Client Support: disabled
Ignore the NT ACL Check for NFS User 'root': disabled
Time To Live Value (in msecs) of a Positive Cached Credential: 86400000
Time To Live Value (in msecs) of a Negative Cached Credential: 7200000
Skip Permission Check for NFS Write Calls from Root/Owner: disabled
Use 64 Bits for NFSv3 FSIDs and File IDs: disabled
Ignore Client Specified Mode Bits and Preserve Inherited NFSv4 ACL When Creating New Files or Directories: disabled
Fallback to Unconverted Filename Search: disabled
I/O Count to Be Grouped as a Session: 5000
Duration for I/O to Be Grouped as a Session (Secs): 120
Enable or disable Checksum for Replay-Cache: enabled
simshare-clu::*> vserver cifs options show -vserver svm1
Vserver: svm1
Client Session Timeout: 900
Copy Offload Enabled: true
Default Unix Group: -
Default Unix User: guest
Guest Unix User: -
Are Administrators mapped to 'root': true
Is Advanced Sparse File Support Enabled: true
Direct-Copy Copy Offload Enabled: true
Export Policies Enabled: false
Grant Unix Group Permissions to Others: false
Is Advertise DFS Enabled: false
Is Client Duplicate Session Detection Enabled: true
Is Client Version Reporting Enabled: true
Is DAC Enabled: false
Is Fake Open Support Enabled: true
Is Hide Dot Files Enabled: false
Is Large MTU Enabled: false
Is Local Auth Enabled: true
Is Local Users and Groups Enabled: true
Is NetBIOS over TCP (port 139) Enabled: true
Is NBNS over UDP (port 137) Enabled: false
Is Referral Enabled: false
Is Search Short Names Support Enabled: false
Is Trusted Domain Enumeration And Search Enabled: true
Is UNIX Extensions Enabled: false
Is Use Junction as Reparse Point Enabled: true
Max Multiplex Count: 255
Max Same User Session Per Connection: 2050
Max Same Tree Connect Per Session: 4096
Max Opens Same File Per Tree: 800
Max Watches Set Per Tree: 100
Is Path Component Cache Enabled: true
NT ACLs on UNIX Security Style Volumes Enabled: true
Read Grants Exec: disabled
Read Only Delete: disabled
Reported File System Sector Size: 4096
Restrict Anonymous: no-restriction
Shadowcopy Dir Depth: 5
Shadowcopy Enabled: true
SMB1 Enabled: true
Max Buffer Size for SMB1 Message: 65535
SMB2 Enabled: true
SMB3 Enabled: true
SMB3.1 Enabled: true
Map Null User to Windows User or Group: -
WINS Servers: -
Report Widelink as Reparse Point Versions: SMB1
simshare-clu::*> vserver nfs kerberos interface show
Logical
Vserver Interface Address Kerberos SPN
-------------- ------------- --------------- -------- -----------------------
svm1 lif_1 10.241.###.### enabled nfs/simshare.bu.edu@AD.BU.EDU
simshare-clu::*> vserver services unix-user show
User User Group Full
Vserver Name ID ID Name
-------------- --------------- ------ ------ --------------------------------
svm1 guest 65533 65534
svm1 nfs 500 0
svm1 nobody 65535 65535
svm1 pcuser 65534 65534
svm1 root 0 1
svm1 test 65532 65532
Here’s possibly some important differences between the simulator and actual NetApp running 8.2. The root, pcuser, and guest accounts differ:
nas-clu::> vserver services unix-user show
User User Group Full
Vserver Name ID ID Name
-------------- --------------- ------ ------ --------------------------------
engnas guest 65534 65534
engnas nfs 500 0
engnas nobody 65535 65535 -
engnas root 0 0
engnas test 65532 65532 -
Solved! See The Solution
Hi
i think you covered most of the configuration basics. but i can't see a fault in it.
You can use "vserver security trace" command to see why the filer deny the request (if it does).if it's not - a tcpdump from the client. or pktt from the filer will show a bit more.
other than that we missing a few more outputs in regard to permissions, a few output that i had in mind while looking on it:
vserver nfs show -fields access,name-service-lookup-protocol,mount-rootonly,default-win-user,v4.0-acl,nfs-rootonly vserver security file-directory show -path /sim_test -vserver sim1 vserver export-policy rule show qtree show -fields security-style,qtree-path,export-policy
Gidi
Hi Gidi,
Thanks for your response. Sorry mine was delayed, but I was on vacation. Here is the output of the queries you recommended. You'll see I have 2 other shares (i.e. app_test and cifs_test) that I didn't mention early and don't think we need to consider here.
simshare-clu::*> vserver nfs show -fields access,name-service-lookup-protocol,mount-rootonly,default-win-user,v4.0-acl,nfs-rootonly
vserver access default-win-user v4.0-acl mount-rootonly nfs-rootonly name-service-lookup-protocol
------- ------ ---------------- -------- -------------- ------------ ----------------------------
svm1 true guest disabled disabled disabled udp
simshare-clu::*> vserver security file-directory show -path /sim_test -vserver svm1
Vserver: svm1
File Path: /sim_test
File Inode Number: 64
Security Style: ntfs
Effective Style: ntfs
DOS Attributes: 10
DOS Attributes in Text: ----D---
Expanded Dos Attributes: -
UNIX User Id: 0
UNIX Group Id: 0
UNIX Mode Bits: 777
UNIX Mode Bits in Text: rwxrwxrwx
ACLs: NTFS Security Descriptor
Control:0x8004
Owner:BUILTIN\Administrators
Group:BUILTIN\Administrators
DACL - ACEs
ALLOW-Everyone-0x1f01ff
ALLOW-Everyone-0x10000000-OI|CI|IO
simshare-clu::*> vserver export-policy rule show
Policy Rule Access Client RO
Vserver Name Index Protocol Match Rule
------------ --------------- ------ -------- --------------------- ---------
svm1 app_test 1 any 0.0.0.0/0 any
svm1 cifs_first 1 cifs, 0.0.0.0/0 any
nfs
svm1 nfs_first 1 nfs 0.0.0.0/0 krb5
svm1 nfs_first 2 cifs 0.0.0.0/0 any
simshare-clu::*> qtree show -fields security-style,qtree-path,export-policy
vserver volume qtree qtree-path security-style export-policy
------- -------- ----- ------------- -------------- -------------
svm1 app_test "" /vol/app_test mixed app_test
svm1 cifs_test
"" /vol/cifs_test
ntfs nfs_first
svm1 sim_test "" /vol/sim_test ntfs cifs_first
svm1 svm_root "" /vol/svm_root ntfs default
Could the issue be the export policy on svm_root? I'll take a look.
Thanks D_BEREZENKO.
I'll start on the troublshooting guide.
Looks like root volume export policy was denying access.
simshare-clu::*> check-access -vserver svm1 -volume sim_test -client-ip 10.241.185.35 \
-authentication-method krb5 -protocol nfs3 -access-type read
(vserver export-policy check-access)
Policy Policy Rule
Path Policy Owner Owner Type Index Access
----------------------------- ---------- --------- ---------- ------ ----------
/ default svm_root volume 0 denied
I changed the export policy on svm_root from default, which had no rule, to base_vol, which allows any cifs,nfs,flexcache connection. I’m not sure this is the proper configuration, but now access checks out and I can mount the shares (albeit with permission errors).
simshare-clu::*> check-access -vserver svm1 -volume sim_test -client-ip 10.241.185.35 \
-authentication-method krb5 -protocol nfs3 -access-type read
(vserver export-policy check-access)
Policy Policy Rule
Path Policy Owner Owner Type Index Access
----------------------------- ---------- --------- ---------- ------ ----------
/ root_vol svm_root volume 1 read
/sim_test cifs_first sim_test volume 1 read
It seems wrong to have / wide open to Everyone. Any thoughts?
hi.
You should allow only read on the root and for subnets you trust (like 10.0.0.0)
G
Thanks, G.
engnasim-clu::> vserver export-policy rule show
Policy Rule Access Client RO
Vserver Name Index Protocol Match Rule
------------ --------------- ------ -------- --------------------- ---------
svm1 cifs_first 1 cifs,nfs 0.0.0.0/0 any
svm1 root_vol 1 cifs,nfs 128.197.0.0/16 any
flexcache
svm1 root_vol 2 cifs,nfs 10.0.0.0/8 any
flexcache