Permission denied on NTFS volume shared by NFS

sassynatan · ‎2011-08-15

Hi All,

Well the question is easy, but the answer is hard 🙂

I have in my infrastcture 10 Linux machine running RHEL 5.0.

The 10 Linux machines are powers boxes (each with 24 cores and 96GB of RAM)

All my users are loging to these machine using NX client.

The Linux machine are configure to work with the Microsoft Active Driecroty servivces.

So all users infromation coming from the LDAP (UID, GID, gecons etc....)

Authentication is done by kerberos.

I also configure the LDAP options on the NetApp and it seems to work fine (getXXbyYY works!)

Anyway, this setup was working great for the last 3 years. No issues at all!

The problem is with RHEL 6.0!

This version was installed, and used the same setup as in version 5.0.

Authentication is done by kerberos, and I used the nslcd option to get info from the ldap server.

So far everything is good.

I also manage to mount the NFS shares on the machine.

But the problem I now facing is that on NFS share, which is a NTFS qtree based I get permssion denied when trying to copy a file.

Here is a copy paste:

RHEL6.0:

[sassy.natan@cpu06 /]# mount

/dev/sda1 on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

/tmp on /tmp type none (rw,bind)

/var/tmp on /var/tmp type none (rw,bind)

/home on /home type none (rw,bind)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

filer:/vol/Users on /home type nfs (rw,addr=172.16.0.6) This is a UNIX QTREE!

filer:/vol/Softwares on /mnt/Softwares type nfs (rw,addr=172.16.0.6)

filer:/vol/Common on /mnt/Common type nfs (rw,addr=172.16.0.6)

filer:/vol/Groups on /mnt/Groups type nfs (rw,addr=172.16.0.6) - This is the NTFS QTREE!

AppSrv:/var/www/soconline/soconline on /mnt/Soconline type nfs (rw,addr=172.16.0.5)

[sassy.natan@cpu06]/home/sassy.natan(85): cd /mnt/Groups/IT NTFS Directory where user sassy.natan as a full permssion!

[sassy.natan@cpu06]/mnt/Groups/IT(86): ll

total 9108

drwx------ 9 root root 4096 Aug 15 16:12 .

dr-xr-xr-x 21 root root 4096 Jun 14 12:33 ..

drwx------ 2 azhar.sharoof Domain Users 4096 Jan 1 2009 2008

drwx------ 3 azhar.sharoof Domain Users 4096 Aug 1 09:46 2009

drwx------ 2 azhar.sharoof Domain Users 4096 Jun 21 19:33 2010

drwx------ 2 10041 Domain Users 4096 Mar 14 15:44 :DA96000

drwx------ 12 root bin 4096 Mar 2 14:32 How To

drwx------ 8 10041 Domain Users 4096 Nov 3 2008 Licence

drwx------ 2 root bin 4096 Nov 10 2009 Operations

-rwx------ 1 sassy.natan Domain Users 9240576 Aug 15 16:11 output.file

-rwx------ 1 azhar.sharoof Domain Users 11431 Feb 9 2011 Printer Mapping.xlsx

-rwx------ 1 root bin 16013 Sep 28 2010 Telephony.xlsx

[sassy.natan@cpu06]/mnt/Groups/IT(87): touch file Here I touch a simple file

[sassy.natan@cpu06]/mnt/Groups/IT(88): ls -la

total 9108

drwx------ 9 root root 4096 Aug 15 16:58 .

dr-xr-xr-x 21 root root 4096 Jun 14 12:33 ..

drwx------ 2 azhar.sharoof Domain Users 4096 Jan 1 2009 2008

drwx------ 3 azhar.sharoof Domain Users 4096 Aug 1 09:46 2009

drwx------ 2 azhar.sharoof Domain Users 4096 Jun 21 19:33 2010

drwx------ 2 10041 Domain Users 4096 Mar 14 15:44 :DA96000

-rwx------ 1 sassy.natan Domain Users 0 Aug 15 16:58 file and here it is 🙂

drwx------ 12 root bin 4096 Mar 2 14:32 How To

drwx------ 8 10041 Domain Users 4096 Nov 3 2008 Licence

drwx------ 2 root bin 4096 Nov 10 2009 Operations

-rwx------ 1 sassy.natan Domain Users 9240576 Aug 15 16:11 output.file

-rwx------ 1 azhar.sharoof Domain Users 11431 Feb 9 2011 Printer Mapping.xlsx

-rwx------ 1 root bin 16013 Sep 28 2010 Telephony.xlsx

[sassy.natan@cpu06]/mnt/Groups/IT(89): mkdir test here I create a dir

[sassy.natan@cpu06]/mnt/Groups/IT(91): ls -la

total 9112

drwx------ 10 root root 4096 Aug 15 16:59 .

dr-xr-xr-x 21 root root 4096 Jun 14 12:33 ..

drwx------ 2 azhar.sharoof Domain Users 4096 Jan 1 2009 2008

drwx------ 3 azhar.sharoof Domain Users 4096 Aug 1 09:46 2009

drwx------ 2 azhar.sharoof Domain Users 4096 Jun 21 19:33 2010

drwx------ 2 10041 Domain Users 4096 Mar 14 15:44 :DA96000

-rwx------ 1 sassy.natan Domain Users 0 Aug 15 16:58 file

drwx------ 12 root bin 4096 Mar 2 14:32 How To

drwx------ 8 10041 Domain Users 4096 Nov 3 2008 Licence

drwx------ 2 root bin 4096 Nov 10 2009 Operations

-rwx------ 1 sassy.natan Domain Users 9240576 Aug 15 16:11 output.file

-rwx------ 1 azhar.sharoof Domain Users 11431 Feb 9 2011 Printer Mapping.xlsx

-rwx------ 1 root bin 16013 Sep 28 2010 Telephony.xlsx

drwx------ 2 sassy.natan Domain Users 4096 Aug 15 16:59 test and here it is 🙂

And now the problem:

Here I going to copy file output.file to the same folder

[sassy.natan@cpu06]/mnt/Groups/IT(92): cp -rf output.file output.file.test

cp: cannot create regular file `output.file.test': Permission denied

Exit 1

Doing ls - seems to create the file in size 0:

[sassy.natan@cpu06]/mnt/Groups/IT(93): ls -la

total 9112

drwx------ 10 root root 4096 Aug 15 17:02 .

dr-xr-xr-x 21 root root 4096 Jun 14 12:33 ..

drwx------ 2 azhar.sharoof Domain Users 4096 Jan 1 2009 2008

drwx------ 3 azhar.sharoof Domain Users 4096 Aug 1 09:46 2009

drwx------ 2 azhar.sharoof Domain Users 4096 Jun 21 19:33 2010

drwx------ 2 10041 Domain Users 4096 Mar 14 15:44 :DA96000

-rwx------ 1 sassy.natan Domain Users 0 Aug 15 16:58 file

drwx------ 12 root bin 4096 Mar 2 14:32 How To

drwx------ 8 10041 Domain Users 4096 Nov 3 2008 Licence

drwx------ 2 root bin 4096 Nov 10 2009 Operations

-rwx------ 1 sassy.natan Domain Users 9240576 Aug 15 16:11 output.file

-rwx------ 1 sassy.natan Domain Users 0 Aug 15 17:02 output.file.test file size is 0

-rwx------ 1 azhar.sharoof Domain Users 11431 Feb 9 2011 Printer Mapping.xlsx

-rwx------ 1 root bin 16013 Sep 28 2010 Telephony.xlsx

drwx------ 2 sassy.natan Domain Users 4096 Aug 15 16:59 test

Doing again copy now seems to work:

[sassy.natan@cpu06]/mnt/Groups/IT(94): cp -rf output.file output.file.test

cp: overwrite `output.file.test'? y

[sassy.natan@cpu06]/mnt/Groups/IT(95): ll

total 18156

drwx------ 10 root root 4096 Aug 15 17:02 .

dr-xr-xr-x 21 root root 4096 Jun 14 12:33 ..

drwx------ 2 azhar.sharoof Domain Users 4096 Jan 1 2009 2008

drwx------ 3 azhar.sharoof Domain Users 4096 Aug 1 09:46 2009

drwx------ 2 azhar.sharoof Domain Users 4096 Jun 21 19:33 2010

drwx------ 2 10041 Domain Users 4096 Mar 14 15:44 :DA96000

-rwx------ 1 sassy.natan Domain Users 0 Aug 15 16:58 file

drwx------ 12 root bin 4096 Mar 2 14:32 How To

drwx------ 8 10041 Domain Users 4096 Nov 3 2008 Licence

drwx------ 2 root bin 4096 Nov 10 2009 Operations

-rwx------ 1 sassy.natan Domain Users 9240576 Aug 15 16:11 output.file

-rwx------ 1 sassy.natan Domain Users 9240576 Aug 15 17:03 output.file.test FILE COPIED!

-rwx------ 1 azhar.sharoof Domain Users 11431 Feb 9 2011 Printer Mapping.xlsx

-rwx------ 1 root bin 16013 Sep 28 2010 Telephony.xlsx

drwx------ 2 sassy.natan Domain Users 4096 Aug 15 16:59 test

[sassy.natan@cpu06]/mnt/Groups/IT(96):

In the RHEL 5.0 this works without a problem 🙂

Here is a copy of the /proc/mounts on RHEL 6.0 vs RHEL 5.0

RHEL 6.0:

filer:/vol/Users /home nfs rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.6,mountvers=3,mountport=4046,mountproto=udp,addr=172.16.0.6 0 0

filer:/vol/Softwares /mnt/Softwares nfs rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.6,mountvers=3,mountport=4046,mountproto=udp,addr=172.16.0.6 0 0

filer:/vol/Common /mnt/Common nfs rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.6,mountvers=3,mountport=4046,mountproto=udp,addr=172.16.0.6 0 0

filer:/vol/Groups /mnt/Groups nfs rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.6,mountvers=3,mountport=4046,mountproto=udp,addr=172.16.0.6 0 0

RHEL 5.0:

filer:/vol/Users /home nfs rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=filer 0 0

filer:/vol/Softwares /mnt/Softwares nfs rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=filer 0 0

filer:/vol/Common /mnt/Common nfs rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=filer 0 0

filer:/vol/Groups /mnt/Groups nfs rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=filer 0 0

/etc/fstab on both server looks like this:

filer:/vol/Users /home nfs defaults 0 0

filer:/vol/Softwares /mnt/Softwares nfs defaults 0 0

filer:/vol/Common /mnt/Common nfs defaults 0 0

filer:/vol/Groups /mnt/Groups nfs defaults 0 0

Any ideas?

Thanks

Sassy

sassynatan · ‎2011-08-15

FYI, I still don't know what is the different, but moving to NFSv4 on RHEL 6.0 solved the problem.

I do however think this is a bug or some configuration error, and will be happy if someone can tell what is the cause of it.

As I see it there no reason why this shouldn't working the same as in version 5.0 of RedHat.

Thanks

Sassy

aborzenkov · ‎2011-08-15

Start with stracing cp command to find out what system call exactly returns EPERM. I suspect it is attempt to (explicitly) change file mask/permissions, because file is obviously created.

thomasnetzker · ‎2012-02-21

I've experienced exactly the same problem: its happen only with RHEL6, NFS v3 and qtree sec-style NTFS. It first creates an empty file while saying permission denied and fills the content of the file on the second copy. Strace showed the following while trying to copy ADR.flt for the first time:

..

f_files=2485504, f_ffree=2388770, f_fsid={2063430184, 312728360}, f_namelen=255, f_frsize=4096}) = 0

brk(0) = 0x1da3000

brk(0x1dc4000) = 0x1dc4000

open("/proc/filesystems", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff61c995000

read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 1024) = 342

read(3, "", 1024) = 0

close(3) = 0

munmap(0x7ff61c995000, 4096) = 0

open("/usr/lib/locale/locale-archive", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=99158752, ...}) = 0

mmap(NULL, 99158752, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff616af2000

close(3) = 0

open("/proc/filesystems", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff61c995000

read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 1024) = 342

read(3, "", 1024) = 0

close(3) = 0

munmap(0x7ff61c995000, 4096) = 0

geteuid() = 2693

stat("./", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0

stat("/dxm/dxmakt/dbsave/110718/ADR.flt", {st_mode=S_IFREG|0600, st_size=558830, ...}) = 0

stat("./ADR.flt", 0x7fff961c8990) = -1 ENOENT (No such file or directory)

open("/dxm/dxmakt/dbsave/110718/ADR.flt", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG|0600, st_size=558830, ...}) = 0

open("./ADR.flt", O_WRONLY|O_CREAT|O_EXCL, 0600) = -1 EACCES (Permission denied)

open("/usr/share/locale/locale.alias", O_RDONLY) = 4

fstat(4, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff61c995000

read(4, "# Locale name alias data base.\n#"..., 4096) = 2512

read(4, "", 4096) = 0

close(4) = 0

munmap(0x7ff61c995000, 4096) = 0

open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = 4

fstat(4, {st_mode=S_IFREG|0644, st_size=435, ...}) = 0

mmap(NULL, 435, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7ff61c995000

close(4) = 0

write(2, "cp: ", 4) = 4

write(2, "cannot create regular file `./AD"..., 38) = 38

open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

write(2, ": Permission denied", 19) = 19

write(2, "\n", 1) = 1

close(3) = 0

close(0) = 0

close(1) = 0

close(2) = 0

exit_group(1) = ?

gopinathp · ‎2012-05-18

We had the similar issue with some of our Linux clients. One solution was to set this hidden option on the affected filers:

filer>options cifs.ntfs_ignore_unix_security_ops on

For more info

https://kb.netapp.com/support/index?page=content&id=3011859