normal user access vs root user access

esxi · ‎2021-02-26

Hi

I have centos7 box

NFS Qtree used for home dir from Netapp is able to mount just fine

Exoport policy shows :

any any any ...ie super user access as well is any

When i am root user on client box , can cd to user home dir of any user

We have SSSD setup & use can login with AD id

when the normal user login The home dir of the user is not able to mount & error is

##
su - userxxxx
Last login: Fri Feb 26 19:17:03 EST 2021 from s...
su: warning: cannot change directory to .../..: Permission denied
-bash: .../.bash_profile: Permission denied
-bash-4.2$

Here is tcpdump shows

###

tcpdump -s 192 port nfs -i ens192
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 192 bytes

19:13:34.696861 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [S], seq 4251450372, win 29200, options [mss 1460,sackOK,TS val 1258809 ecr 0,nop,wscale 7], length 0
19:13:34.697147 IP NetappXXXX.com.nfs > CLIENTXXX.busboy: Flags [S.], seq 3336189244, ack 4251450373, win 65535, options [mss 8960,nop,wscale 8,sackOK,TS val 1699851922 ecr 1258809], length 0
19:13:34.697169 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [.], ack 1, win 229, options [nop,nop,TS val 1258809 ecr 1699851922], length 0
19:13:34.697184 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [P.], seq 1:137, ack 1, win 229, options [nop,nop,TS val 1258809 ecr 1699851922], length 136: NFS request xid 2803191295 132 access [|nfs]
19:13:34.707092 IP NetappXXXX.com.nfs > CLIENTXXX.busboy: Flags [P.], seq 1:125, ack 137, win 257, options [nop,nop,TS val 1699851932 ecr 1258809], length 124: NFS reply xid 2803191295 reply ok 120 access c 0003
19:13:34.707101 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [.], ack 125, win 229, options [nop,nop,TS val 1258819 ecr 1699851932], length 0
19:13:34.707147 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [P.], seq 137:281, ack 125, win 229, options [nop,nop,TS val 1258819 ecr 1699851932], length 144: NFS request xid 2819968511 140 lookup [|nfs]
19:13:34.707621 IP NetappXXXX.com.nfs > CLIENTXXX.busboy: Flags [P.], seq 125:389, ack 281, win 257, options [nop,nop,TS val 1699851932 ecr 1258819], length 264: NFS reply xid 2819968511 reply ok 260 lookup fh Unknown/01000000A916668000000000F0F93B00CEC24854A91666800000000061000000
19:13:34.747554 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [.], ack 389, win 237, options [nop,nop,TS val 1258860 ecr 1699851932], length 0
19:14:34.799582 IP CLIENTXXX.busboy > NetappXXXX.com.nfs: Flags [.], ack 389, win 237, options [nop,nop,TS val 1318912 ecr 1699851932], length 0
19:14:34.799848 IP NetappXXXX.com.nfs > CLIENTXXX.busboy: Flags [.], ack 281, win 257, options [nop,nop,TS val 1699912024 ecr 1258860], length 0

#######################################

fstab entry :-
xxx:/vol_home/home xxx nfs vers=3,bg,soft,retrans=4 0 0
#######################################

mount options from client end
mount -v |grep -i nfs

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
xx:/vol_home/home on xx type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,soft,proto=tcp,timeo=600,retrans=4,sec=sys,mountaddr=xxx,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=xxx)
#######################################

Any suggestions Please

esxi · ‎2021-02-26

I just noticed even as root user i am not able to do below

ls -l /xx/home/userxxx

but i can see below works

ls -ld /xx/home/userxxx

Here are permission which looks ok

drwx------ 16 xxx xxx 4096 Feb 26 16:43 /xxx/home/userxxc

parisi · ‎2021-03-01

Have a look at TR-4067.

https://www.netapp.com/us/media/tr-4067.pdf

That packet trace is kind of useless here, as all we see are the headers, which tell us "access denied" but have no corresponding info from packet details.

But generally speaking, if your permissions are 700 and the user is the owner of the volume and cannot access it, that means ONTAP doesn't know who the user attempting access is. I'd suggest having a look at the TR and opening a support case.

Mjizzini · ‎2021-03-02

Do you have any other working NFS mounts?

"userxxx" has to be a local user to the cluster or a valid LDAP user if LDAP is configured.

You can run the below command to validate "userxxx".

nas-cm96::*> getxxbyyy getpwbyname -node nodex -vserver vserverx -username userxxx

parisi · ‎2021-03-03

You don't need local users for NFSv3. Those come in as numeric IDs and will be able to access based on permissions.

esxi · ‎2021-03-03

Sorry for late reply

Here is investigation so far , Summary is its seems client side issue ... at the same time at times both client & server at times can do config changes to accomodate issues .. so will still keep open

I did rise a netapp case BTW

1 did 2 identaical client .. centos 7 + SSSD & other one Centos + SSSD + CIS hardening

CIS node it fails to mount

CIS does over 200 changes .. i tried to revert back few no luck...

working node home dir permission

-sh-4.2$ pwd
/mnt/xxx/home/xxx
-sh-4.2$ ls -ld .
drwx------ 17 xxx xxxx 4096 Mar 2 15:21 .
-sh-4.2$

Extensive client side log does not give any info ...

netapp too share a good KB to check that too does not give much to help identify the root coz ..

https://mysupport.netapp.com/site/article?lang=en&type=howto&page=%2FAdvice_and_Troubleshooting%2FData_Storage_Software%2FONTAP_OS%2FHow_to_Troublesho...

if i am able to get to bottom of this i will post .. at this time as i had work-around i am not pushing

BTW the subject of the post i realized is wrong now ... it has nothing to do with Root or other user ... in fact when i tried to cd to user home dir it said permission denied

on the same working box the user was able to work ok as per the above output

esxi · ‎2021-03-03

Netapp gave good KB but no luck

https://mysupport.netapp.com/site/article?lang=en&type=howto&page=%2FAdvice_and_Troubleshooting%2FData_Storage_Software%2FONTAP_OS%2FHow_to_Troublesho...

this is client side issue is my finding so far .. i did exact same steps on a box with no CIS hardening works fine