Re: How to enforce all_squash for NFSv3 on NetApp?

VincentValentine · ‎2016-07-15

Hi there,

I'm having some hard time setting up NFSv3 with anonymous access on NetApp. I'm unable to squash anonymous users.

I’ve read the NetApp’s man page for “exports”. The below is the list of options that I get:

path=
anon=
nosuid
ro=
rw=
root=
sec=

I have no clue how one is supposed to set “all_squash” in this case.

The man exports says that by default, the effective user ID of all anonymous and root NFS client users is 65534. This is my exprots file:

FAS> rdfile /etc/exports
/vol/NFS_WEB      -sec=sys,rw=10.10.0.149:10.10.0.147,root=10.10.0.147:10.10.0.149,anon=65534,nosuid

Ran the following after setting the exports file:

FAS> exportfs -r

This is the current security context:

FAS>  fsecurity show /vol/NFS_WEB

[/vol/NFS_WEB - Directory (inum 64)]

  Security style: Unix
  Effective style: Unix

   DOS attributes: 0x0010 (----D---)

   Unix security:

    uid: 65534 (pcuser)
    gid: 65534 (pcuser)
    mode: 0755 (rwxr-xr-x)

   No security descriptor available.

These may help as well:

FAS> rdfile /etc/passwd
root:_<removed>:0:1::/:
pcuser::65534:65534::/:
nobody::65535:65535::/:
ftp::65533:65533:FTP Anonymous:/home/ftp:

FAS> rdfile /etc/group
daemon:*:1:
pcuser:*:65534:
nobody:*:65535:

So it’s got uid/gid of 65534 on the NetApp.

Now on the NFS client (CentOS 6), I mount the share as NFSv3:

# mount|grep netapp
10.10.0.51:/vol/NFS_WEB/ on /mnt/netapp_nfs type nfs (rw,vers=3,sec=sys,clientaddr=10.10.0.147,addr=10.10.0.51)

These are the uid/gid:

# ls -ld /mnt/netapp_nfs/
drwxr-xr-x. 3 nfsnobody nfsnobody 4096 Jul 15 11:11 /mnt/netapp_nfs/

# ls -lnd /mnt/netapp_nfs/
drwxr-xr-x. 3 65534 65534 4096 Jul 15 11:11 /mnt/netapp_nfs/

We see that uid/gid math with the ones we have on the NetApp.

With NFSv3, I should be able to write to the share as a regular user:

$ id
uid=3001(test) gid=504(test) groups=504(test),10(wheel)
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

$ touch /mnt/netapp_nfs/test
touch: cannot touch `/mnt/netapp_nfs/test': Permission denied

$ ls -ld /mnt/netapp_nfs/
drwxr-xr-x. 3 nfsnobody nfsnobody 4096 Jul 15 11:29 /mnt/netapp_nfs/

$ ls -lnd /mnt/netapp_nfs/
drwxr-xr-x. 3 65534 65534 4096 Jul 15 11:29 /mnt/netapp_nfs/

SELinux is set to permissive:

$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   permissive
Mode from config file:          permissive
Policy version:                 24
Policy from config file:        targeted

I’m trying to understand why NFSv3 doesn’t work.

And one more thing to show that it’s not squashing on NFSv3, take a look at the below:

# touch /mnt/netapp_nfs/root
# ls -l /mnt/netapp_nfs/root
-rw-r--r--. 1 root root 0 Jul 15 11:38 /mnt/netapp_nfs/root

The owner of the file is root and not nfsnobody.

Now if I do the following:

# usermod -aG nfsnobody test

# su - test

$ id
uid=3001(test) gid=504(test) groups=504(test),10(wheel),65534(nfsnobody) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

$ touch /mnt/netapp_nfs/test

$ ls -l /mnt/netapp_nfs/test
-rw-rw-r--. 1 test test0 Jul 15 11:39 /mnt/netapp_nfs/test

It’s owned by my user test and not nfsnobody (and yes, I’m able to write to the share if my user is a member of the nfsnobody group and when share permissions are 775 – no idea why that’s required).

aborzenkov · ‎2016-07-15

All_squash is default if you do not explicitly give root= option.

VincentValentine · ‎2016-07-15

I wasn't aware of that, thanks. However, that still doesn't work for me.

This is what I have on NetApp now:

FAS> exportfs -v
/vol/NFS_WEB      -sec=sys,rw=10.10.0.149:10.10.0.147,nosuid

On the client, if I try to write as root, it's squashed (root_squash is invoked I suspect):

# touch /mnt/netapp_nfs/root

# ls -l /mnt/netapp_nfs/root
-rw-r--r--. 1 nfsnobody nfsnobody 0 Jul 15 13:08 /mnt/netapp_nfs/root

However, my regular user still has no access:

# su - test

$ touch /mnt/netapp_nfs/newtest
touch: cannot touch `/mnt/netapp_nfs/newtest': Permission denied

This implies that all_squash doesn't work (but root_squash does).

aborzenkov · ‎2016-07-15

I apologize - I misunderstood your question. Try sec=none option; then all connections are treated as anonymous.

VincentValentine · ‎2016-07-15

I just tried with sec=none, still get permission denied when trying to write with my regular user.

FAS> exportfs -r

FAS> exportfs -v
/vol/NFS_WEB      -sec=none,rw=10.10.0.149:10.10.0.147,nosuid

I re-mounted the share on the client, it's now owned by the root (was nfsnobody previously):

# ls -ld /mnt/netapp_nfs/
drwxr-xr-x. 2 root root 4096 Jul 15 16:03 /mnt/netapp_nfs/

# touch /mnt/netapp_nfs/rootnew

# ls -l /mnt/netapp_nfs/rootnew
-rw-r--r--. 1 root root 0 Jul 15 16:03 /mnt/netapp_nfs/rootnew

Regular user denied:

# su - test

$ touch /mnt/netapp_nfs/testnew
touch: cannot touch `/mnt/netapp_nfs/testnew': Permission denied

VincentValentine · ‎2016-07-15

Apologies, I just realised that the share hasn't even been mounted - I tried to write to the folder that resides on a local system, hence root permissions.

I run the following on the client:

# mount -v -t nfs -o vers=3,sec=none 10.10.0.51:/vol/NFS_WEB /mnt/netapp_nfs/
mount.nfs: timeout set for Fri Jul 15 16:18:45 2016
mount.nfs: trying text-based options 'vers=3,sec=none,addr=10.10.0.51'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.10.0.51 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.10.0.51 prog 100005 vers 3 prot UDP port 4046
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 10.10.0.51:/vol/NFS_WEB

The file /var/log/messages on the client says:

 localhost kernel: RPC: server 10.10.0.51 requires stronger authentication.

The NetApp tells me that:

[FAS:nfsd.auth.status.bad:warning]: Client 10.10.0.147 has an authentication error 5

So I cannot even mount it now.

aborzenkov · ‎2016-07-15

You need to export with sec=none on NetApp side.

VincentValentine · ‎2016-07-15

It's been done, see my previous reply.

FAS> exportfs -v
/vol/NFS_WEB      -sec=none,rw=10.10.0.149:10.10.0.147,nosuid

aborzenkov · ‎2016-07-15

Looks like feature is broken in 8.x 7-Mode. I got the same result on 8.2. Testing with "exportfs -c" succeeds, as well as mount request from client, but then client got error response when actually accessing volume. I also verified that client does send AUTH_NONE as expected. I then tested on 9.0RC1 (C-mode obviously) and it works as expected (setting sec=none as the only accepted authentication will effectively result in all_squash behavior). I do not have Data ONTAP 7.x to test right now, but I am fairly confident it was working in the past.

I suggest you open case with NetApp support. May be there is some hidden option to do it, but I could not find one. If you get it working I appreciate if you share information (I found old post two years ago with the same question and the same problem with sec=none not working). Thank you!

VincentValentine · ‎2016-07-16

Our ONTAP version is 8.2.2P2.

I do have a case open with NetApp support for over a week now, and so far they had zero luck fixing the issue (that's basically the point where I turned for help to community).

If I manage to get it working, I'll post the solution. Thanks for your help.

aborzenkov · ‎2016-07-19

I tested with 7.3.7 and now I can confirm that this is broken in 8.x:

filer1-1-co> exportfs -io sec=none,rw /vol/t

cn1:~ # mount filer1-1-co:/vol/t /mnt
cn1:~ # touch /mnt/foo

cn1:~ # su - tele
tele@cn1:~> touch /mnt/bar
tele@cn1:~> ll /mnt
total 0
-rw-r--r-- 1 nobody nogroup 0 Jul 19 12:22 bar
-rw-r--r-- 1 nobody nogroup 0 Jul 19 11:52 foo

While on 8.2 I am not even able to mount it. Of course, I also have different host OS here (RHEL vs. SLES) but as I get error reply from Data ONTAP I do not think it depends on client.

I do have a case open with NetApp support for over a week now, and so far they had zero luck fixing the issue (that's basically the point where I turned for help to community).

I do not mean opening case and asking "how to implement all_squash". I mean opening case about sec=none being completely broken in 8.x. May be there is already bug about it in this case the more people complain the more chances it gets fixed.

VincentValentine · ‎2016-07-28

Finally got all squashing working with NFSv4 (still no luck with NFSv3 though).

On the filer, security context needs to be sec=none, but the client must mount the share with sec=sys.

aborzenkov · ‎2016-07-28

On the filer, security context needs to be sec=none, but the client must mount the share with sec=sys.

Yes, that's how it works on 7G as well, except it also works using NFS v3.

VincentValentine · ‎2016-07-28

I was told that it (NFSv3) does work on ONTAP 8 if the client that's in use is RHEL 5.

I'm on RHEL 6 and therefore unable to test myself.

VincentValentine · ‎2016-07-28

Iv'e got the following explanation from NetApp support. Hope this helps people who've got the same issue.

ONTAP 8.x with RHEL 6 client.

Configuring sec=sys on the client and sec=none on the filer, with NFSv3, the client makes a "AUTH_UNIX" call and the filer responds with "AUTH_NULL".

However, this is not the response that the client would expect. Therefore we reply incorrectly with only 1 flavor (we should be replying with 2), because our code does not allow for "AUTH_NULL".

There are a number of internal cases open within NetApp, but there is no expected fix as NFSv4 will take care of the issues encountered with NFSv3.

So the advice from NetApp would be to use, sec=sys on the client, sec=none on the filer, using NFSv4 from RHEL6 and upwards.