NFS Volumes became inaccessible.

[seftekhari]

  Hi,

I really hope someone has came across this in the past.

We have  ESX server located in Boston. This Tuesday all three NFS Volumes became inaccessible to the ESX host. To my knowledge there has been no changes to ESX host server and NetApp. Nothing was changed.

I have checked the export list on the Filer and it checks out fine. I am not quite sure what could be cause of this. I checked the Qtree and it is set to UNIX.

From the ESX host i can see all security permission and they look okay:

[root@bosvi4blsrvxx BOSVMLUN2]$ less -SRi /var/log/vmkernel

/var/log/vmkernel: Permission denied

[root@bosvi4blsrvxx BOSVMW7CN001_1]$ file BOSVMW7CN001.vmdk

BOSVMW7CN001.vmdk: writable, regular file, no read permission

[root@bosvi4blsrvxx BOSVMW7CN001_1]$ cat BOSVMW7CN001.vmdk

cat: BOSVMW7CN001.vmdk: Permission denied

[root@bosvi4blsrvxx BOSVMW7CN001_1]$ ls -laht

[root@bosvi4blsrv02 BOSVMW7CN001_1]# file BOSVMW7CN001.vmdk

BOSVMW7CN001.vmdk: ERROR: cannot read `BOSVMW7CN001.vmdk' (Permission denied)

[admin@bosvi4blsrv02 volumes]$ ls -ltra BOSVMLUN1

lrwxr-xr-x 1 root root 17 Jul 13 10:26 BOSVMLUN1 -> 732a15a0-0df07409

[admin@bosvi4blsrv02 volumes]$ ls -ltra BOSVMLUN2

lrwxr-xr-x 1 root root 17 Jul 13 10:27 BOSVMLUN2 -> ca6fb646-a19a7a0c

[admin@bosvi4blsrv02 volumes]$ ls -ltra BOSVMLUN3

lrwxr-xr-x 1 root root 17 Jul 13 10:27 BOSVMLUN3 -> 1e04a735-999ca990

[admin@bosvi4blsrv02 volumes]$ ls -laht BOSVMLUN1

lrwxr-xr-x 1 root root 17 Jul 13 10:29 BOSVMLUN1 -> 732a15a0-0df07409

[admin@bosvi4blsrv02 volumes]$ ls -laht BOSVMLUN2

lrwxr-xr-x 1 root root 17 Jul 13 10:28 BOSVMLUN2 -> ca6fb646-a19a7a0c

[admin@bosvi4blsrv02 volumes]$ ls -laht BOSVMLUN3

lrwxr-xr-x 1 root root 17 Jul 13 10:28 BOSVMLUN3 -> 1e04a735-999ca990

NetApp Exports:

#Auto-generated by setup Tue Feb 20 16:07:27 GMT 2007

/vol/BOSVMLUN1  -sec=sys,rw,root=10.40.2.100

/vol/BOSVMLUN2  -sec=sys,rw,root=10.40.2.100

/vol/BOSVMLUN3  -sec=sys,rw,root=10.40.2.100

/vol/BOSVMNFS1  -sec=sys,rw,root=10.40.2.100

Does anyone have any ideas what could be wrong?

Any help is greatly appreciate it.

[forgette]

Double check the output of the 'exportfs' and compare it to the output you listed.

Also, check the vmKernel interfaces in ESX.  Are there any other vmKernel interfaces on the 10.40.2.0 network?  Its possible that ESX is trying to mount the export using an interface with an ip address other than 10.40.2.100.

Lastly, in my experience, all "strange" NFS behaviors (ie: nothing has changed, but its now broken) seem to be name service related.  Are your datastores mounted using ip addresses or host names?  Is it possible that DNS records have changed for the host or the filer?

Hope this helps,
-Eric