NFS daemon is not coming up

[Melsa]

I have a query regarding the compatibility between nfsd and glibc. In our system, we've upgraded glibc to version 2.40 and are using NFS version 2.1.1 with NFSv3. Previously, with glibc 2.23, everything was working fine, and we weren’t using libtirpc. However, after the glibc upgrade, libtirpc was included and enabled in NFS as well. Now, none of the NFS-related services (nfsd, rpc.statd, rpc.mountd, portmap) are running.

When attempting to start nfsd, the following errors occur:

"unable to set any sockets for nfsd"
"writing fd to kernel failed: errno 89 (Destination address required)" or "errno 111 (Connection refused)"
Console logs show:
"svc: failed to register nfsdv3 RPC service (errno 111)."
After upgrading glibc, the --enable-obsolete-rpc option has been removed from glibc. Can anyone provide guidance on how to debug or resolve this issue?

[AmitKerkar]

I am hoping your network does not have any issue . Error 89 possibly mean network binding. check your /etc/exports and test any single export in your system.

also, can you update NFS utils and restart services to verify ? (yum update nfs-utils)

this should solve the issue.

Else,  is your rpcbind running ?

can you get me output of

1- systemctl status rpcbind

2- netstat -tuln | grep 111

3- lsmod | grep nfsd

4- cat /proc/filesystems | grep nfsd

5- cat /etc/netconfig 

[Melsa]

Thanks, Amit, for responding to this query.

There haven't been any changes in the network; the only difference is that we've upgraded glibc.
With the previous version of glibc, NFS was working fine. We were using portmap instead of rpcbind, so rpcbind isn't running now.

In our current system, here are the outputs:

ps aux | grep nfsd
root 9064 0.0 0.0 4052 340 pts/2 S+ 23:48 0:00 grep nfsd
# ps aux | grep rpcbind
root 9231 0.0 0.0 4052 356 pts/2 S+ 23:48 0:00 grep rpcbind
# netstat -tuln | grep 111
udp 0 0 0.0.0.0:30111 0.0.0.0:*
# lsmod | grep nfsd
nfsd 372736 1 - Live 0xffffffffc05f1000
# cat /proc/filesystems | grep nfsd
nodev nfsd
# cat /etc/netconfig
cat: can't open '/etc/netconfig': No such file or directory
# cat /etc/exports   --> no data, we are using this command in the startup process of nfsd.
exportfs -i -orw,sync,insecure,no_subtree_check,no_root_squash 172.16.0.0/18:/mnt/raid/data

Just tried to copy netconfig file from libtirpc/doc/netconfig and could see change in these outputs.
# netstat -tuln | grep 111
udp 0 0 0.0.0.0:111 0.0.0.0:*  --> portmap is up with udp protocol in 111 port
# cat /etc/netconfig
#
# The network configuration file. This file is currently only used in
# conjunction with the TI-RPC code in the libtirpc library.
#
# Entries consist of:
#
# <network_id> <semantics> <flags> <protofamily> <protoname> \
# <device> <nametoaddr_libs>
#
# The <device> and <nametoaddr_libs> fields are always empty in this
# implementation.
#
udp tpi_clts v inet udp - -
tcp tpi_cots_ord v inet tcp - -
rawip tpi_raw - inet - - -
local tpi_cots_ord - loopback - - -
unix tpi_cots_ord - loopback - - -
But still nfsd is not up. 

[AmitKerkar]

error 89 means networking binding issue. Did you check your interfaces.

other way you could check is use "/etc/exports" for valid up range.

can you pl test with simple export ?

this way we can isolate the issue is not related to binding or invalid addr.

after doing that pl restart nfsd with

exportfs -ra

sysctl restart nfs-server

[Melsa]

Interfaces are up.
There is no error while trying with this command and it is executed successfully
exportfs -i -orw,sync,insecure,no_subtree_check,no_root_squash 172.16.0.0/18:/mnt/raid/data
But I tried a simple export using this command and gave some messages.
exportfs -i -orw,sync 192.168.1.0/18:/tmp/test
exportfs: /tmp/test requires fsid= for NFS export
Or
exportfs -i -orw,sync *:/tmp/test
exportfs: /tmp/test requires fsid= for NFS export


Additionally, some other errors or messages bringing to notice

/usr/sbin/rpc.statd -F -d -name=master
rpc.statd: Version 2.1.1 starting
rpc.statd: Flags: No-Daemon Log-STDERR TI-RPC
sm-notify: Version 2.1.1 starting
sm-notify: Already notifying clients; Exiting!
rpc.statd: Local NSM state number: 3
rpc.statd: Failed to unregister program 100024, version 1
rpc.statd: Running as root. chown /var/lib/nfs to choose different user
rpc.statd: Failed to access local netconfig database: Netconfig database not found
rpc.statd: failed to create RPC listeners, exiting

[Melsa]

The main issue we've observed is that before the glibc upgrade, we were using code where libtirpc was not enabled. However, after the upgrade, nfsd is using the code if libtirpc is enabled. Is there a way to bypass the changes related to libtirpc and make nfs to work?

[AmitKerkar]

1- your /tmp/test does not have FSID. Pl either add fsid or use proper mount point.

2- please resole rpc issue.

systemctl enable rpcbind

systemctl start rpcbind

prcinfo -p

pl send me output 

3- can you pl have following in netconfig ?

echo "udp tpi_clts v inet udp - -" > /etc/netconfig

echo "tcp tpi_cots v inet tcp - -" >> /etc/netconfig

then again run following commands

pkill -9 rpc.statd

rm -f /var/lib/nfs/sm/*

systemctl start rcp.statd

systemctl status rpc.statd

rcpinfo -p  | grep 100024

systemctl restart rpcbind

systemctl restart rpc.statd

systemctl restart rpc.mountd

systemctl restart nfs-kernel-server

exportfs -v

send me output of

netstat -tuln | grep 2049

and then finally try to mount

mount -t nfsd nfsd /proc/fs/nfsd

3- libtirpc is standard external lib for RPC and I do not recommend bypassing it.

you could use glibc 2.39 (downgrade glibc , glibc-common and glibc-devel

[Melsa]

1)Tried using a proper mount point and performed a simple export, which worked without errors. Here's the command I ran:
exportfs -i -orw,sync 192.168.1.0/18:/mnt/raid/data
However, when I attempted to start rpc.nfsd, I received the following error:
/usr/sbin/rpc.nfsd -N2 -N4 -Hmaster
rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
rpc.nfsd: unable to set any sockets for nfsd

I checked the running processes, and no nfsd, rpc.statd appears:

ps aux | grep nfsd
root 12977 0.0 0.0 4052 348 pts/2 S+ 06:33 0:00 grep nfsd
ps aux | grep rpc
root 86 0.0 0.0 0 0 ? I< 06:10 0:00 [rpciod]
root 13003 0.0 0.0 4052 352 pts/2 S+ 06:33 0:00 grep rpc

netstat -tuln | grep 2049
netstat -tuln | grep 111
udp 0 0 0.0.0.0:111 0.0.0.0:*
udp 0 0 0.0.0.0:30111 0.0.0.0:*

One more thing to update here is if we are trying to bring up NFS using V4 version, NFS will come up but then all functionalities are not working.

# netstat -tuln | grep 2049
tcp 0 0 172.16.63.1:2049 0.0.0.0:* LISTEN
udp 0 0 172.16.63.1:2049 0.0.0.0:*

2)We don't have rpcbind installed, but instead, we're using portmap and sunrpc functions. Is it mandatory to use rpcbind if NFS is linked with libtirpc? If so, what is the compatible version of NFS, rpcbind, and libtirpc? We tried installing rpcbind, but it’s not starting up and we're having trouble debugging it. Are there any additional dependencies required to bring up rpcbind?

3)netconfig contains the data given above. Is this netconfig file fine?
laas-setup:card1-cpu0# cat /etc/netconfig
#
# The network configuration file. This file is currently only used in
# conjunction with the TI-RPC code in the libtirpc library.
#
# Entries consist of:
#
# <network_id> <semantics> <flags> <protofamily> <protoname> \
# <device> <nametoaddr_libs>
#
# The <device> and <nametoaddr_libs> fields are always empty in this
# implementation.
#
udp tpi_clts v inet udp - -
tcp tpi_cots v inet tcp - -
udp6 tpi_clts v inet6 udp - -
tcp6 tpi_cots_ord v inet6 tcp - -
rawip tpi_raw - inet - - -
local tpi_cots_ord - loopback - - -
unix tpi_cots_ord - loopback - - -

We had issues with portmap not coming up initially, but after adding the netconfig file, portmap is now up. However, when we check the ports for 111, it is only listening on UDP and not TCP:
netstat -tuln | grep 111
udp 0 0 0.0.0.0:111 0.0.0.0:*

Are the ports being added based on the data in the netconfig file, or do we need to make other adjustments to listen on TCP?

4)

Since we cannot downgrade glibc at this time, we need to proceed with using the libtirpc library. Is this the correct course of action?

?

[Melsa]

Any suggestion on this issue?