I recently upgrade our FAS 2240-4 system ( EOSL) from ONTAP 8.3.2P12 version to 9.1P20. All went well, until we started seeing our RHEL 7.7 machines encountering NFS4 mount issues after reboot. The NFS4 exports works well until the server actually rebooted. Suddenly the mount points doesn't show up with "df -h" command and this only happens after reboot of the Linux server. NFS3 works well with UDP protocol but not with NFS4.
Please note, the export policy and export-policy check does show that the client has RW access. I do not see any errors on the NetApp logs. However, with pktt trace shows that NFS server throwing "NFS4ERR_DENIED" error. Please find below NetApp packet trace details excerpts from wireshark:
I'm unable to understand why NetApp NFS server is showing up this error. To give you some perspective from the redhat client server is recently (gap of 1 week) we have installed McAfee AV as well. And, the server running IBM MQ application services in HA mode (meaning primary and secondary servers as active-standby).
Please find below details of NFS4 mount error -
[root@server101 user_name]# mount -vvv SVM:/vol/vol10/UAT_MQHA_MQ /UAT_MQHA_MQ mount.nfs: timeout set for Tue Nov 17 11:59:15 2020 mount.nfs: trying text-based options 'vers=4.1,addr=10.XXX.XXX.156,clientaddr=10.XXX.XXX.36' mount.nfs: mount(2): Protocol not supported mount.nfs: trying text-based options 'vers=4.0,addr=10.XXX.XXX.156,clientaddr=10.XXX.XXX.36'
It stuck at this step. Redhat says that it is NFS server which is causing this error. Unfortunately we do not have any support on NetApp. Can anyone please help me understand where could be the issue?
Please note that redhat servers do not have any AD or LDAP authentication. Users are locally maintained.
I would appreciate if the community helps me with the issue. Thanks much!
Not sure why running the command with the -path did not work. looking at the packet trace the LOCK Status: NFS4ERR_DENIED means that an attempt to lock a file is denied. Are you able to mount vol10 only with nfsv4 without the whole path:
Per example: mount -vvv SVM:/vol/vol10 /UAT_MQHA_MQ
The mounted path is same i.e. vol/vol10 /UAT_MQHA_MQ and it is still mounted with NFS4. The problem starts only if we reboot the server then the mount point will fail to mount. Luckily we have VM snapshot maintained to revert in working condition. NFS3 mounts perfectly - but we need NFS4 mount point as it need to support to support multi-mode for IBM MQ application. NFS3 doesn't support multimode.
What could be the possible reason for this issue? The NFS mount works perfectly as long as server UP and running. Is it related to Kerberos?
If you see my first post, I have already given the excerpt from the packet trace output. The problem starts only when the client server restarted else already mount NFS FS does not have any issue. I tried to migrate the LIF as well last time but it did not helped. Luckily we had VM snapshots available prior to reboot, we reverted the Snapshot and it started working.