[ B2B ] - What Happens When a NFS Share is Mounted

RodrigoNascimento · ‎2025-07-21

I'm excited to kick off a new blog series called Back to Basics (B2B). The goal is to revisit fundamental concepts that often slip through the cracks because the world is going too fast, life happens, and you have to keep all the plates spinning without breaking any of them.

Mounting an NFS share might seem like a straightforward task - but have you ever wondered what really happens behind the scenes when you do it on a Linux client? To answer this question, we'll take a closer look at the NFS protocol and use some powerful tools to peek under the hood from the client's perspective. Let's dive in!

Network File System (NFS)

Let's take a quick look at the history of NFS. If you'd rather skip the details, here is the key takeaway:

NFS is a mature, battle-tested protocol with decades of engineering behind it. It's an industry-standard solution for file sharing, well-suited for modern workloads like AI/ML pipelines, and needless to say it's fully embraced by NetApp. That means you get top-tier performance from your NetApp storage without needing to install any proprietary agents, modules, or "magic" components on your systems.

In 1984, Sun Microsystems developed a solution to enable file access over a network as if the files were local. This solution, known as the Network File System (NFS), was initially used internally at Sun.

In 1989, NFS was publicly available for the first time with the release of NFSv2 under RFC 1094. This version introduced a stateless protocol using UDP, it supported 32-bit file sizes and offsets (limiting files to 4GB) [1], and it gained widespread adoption due its simplicity and portability.

By 1995, NFSv3 was introduced via RFC 1813, bringing significant enhancements such as the support for 64-bit file sizes and offsets, TCP as an optional transport protocol, and asynchronous writes for improved performance [2].

Then, a major redesign followed with NFSv4.0, published in 2003 under RFC 3530. This version transitioned NFS to a stateful protocol, incorporating built-in support for security mechanisms like Kerberos and Access Control Lists (ACLs), the integration of file locking and mounting into the protocol, and compound operations to reduce network round-trips [3].

In 2010, NFSv4.1 was released under RFC 5661, introducing parallel NFS (pNFS) for scalable and parallel access to storage, a session model for enhanced recovery and performance (client ID trunking and session trunking), and it improved delegation and layout handling [4].

Then, in 2016, NFSv4.2 was published under RFC 7862, adding several modern features such as server-side copy to offload file copy operations to the server, the support for sparse file for efficient storage of files with holes, the application I/O hints to optimize server-side caching, among other features to improve security, and space management [5].

Now that we've covered the history, let's dive into what happens when you mount an NFS share on a Linux client. For this walkthrough, I'll be using NFSv4.1, as features like pNFS and session trunking are essential for delivering the performance and scalability required by modern AI/ML workloads.

Looking Under the Hood

To understand the interaction between the NFS client and NFS server during a mount operation, the first step is to enable kernel tracing for the sunrpc, nfs, and nfsv4 modules. Before enabling tracing, ensure that these modules are loaded into the kernel. You can do this using the modprobe command:

sudo modprobe sunrpc
sudo modprobe nfs
sudo modprobe nfsv4
sudo modprobe nfs_layout_nfsv41_files

Once the modules are loaded, enable tracing by writing 1 to the enabled file for each event in the kernel tracing subsystem:

echo 1 | sudo tee /sys/kernel/tracing/events/sunrpc/enabled
echo 1 | sudo tee /sys/kernel/tracing/events/nfs/enabled
echo 1 | sudo tee /sys/kernel/tracing/events/nfs4/enabled

Next, I opened 2 (two) terminal windows. In the first window, I ran a command to continuously read tracing information from the kernel's tracing subsystem:

cat /sys/kernel/tracing/trace_pipe

This streams live trace events as they occur, allowing us to observe the interaction between the NFS client and server in real-time.

In the second window, I ran the following mount command:

sudo mount -t nfs4 -o vers=4.1,rsize=65536,wsize=65536,proto=tcp 10.26.128.221:/vol_cp_5node /cp

The original tracing output was 899 lines long. To keep things consice and focused, I filtered the data to include only the relevant events needed to understand the communication flow between the NFS client and server, from the client's perspective.

The trace begins with the mount command parsing the mount options:

           <...>-3331    [012] .....   949.293271: nfs_mount_option: option source
           <...>-3331    [012] .....   949.293276: nfs_mount_option: option vers
           <...>-3331    [012] .....   949.293277: nfs_mount_assign: option vers=4.1
           <...>-3331    [012] .....   949.293278: nfs_mount_option: option rsize
           <...>-3331    [012] .....   949.293279: nfs_mount_option: option wsize
           <...>-3331    [012] .....   949.293280: nfs_mount_option: option proto
           <...>-3331    [012] .....   949.293280: nfs_mount_assign: option proto=tcp
           <...>-3331    [012] .....   949.293282: nfs_mount_option: option addr
           <...>-3331    [012] .....   949.293282: nfs_mount_assign: option addr=10.26.128.221
       mount.nfs-3331    [012] .....   949.293283: nfs_mount_option: option clientaddr
       mount.nfs-3331    [012] .....   949.293283: nfs_mount_assign: option clientaddr=10.26.56.192
       mount.nfs-3331    [012] .....   949.293286: nfs_mount_path: path='/vol_cp_5node'

Next, it prepares the RPC transport where the xprt_create and rpc_clnt_new initialize the RPC client to communicate with the NFS server at port 2049:

       mount.nfs-3331    [012] .....   949.293390: xprt_create: peer=[10.26.128.221]:2049 state=BOUND
       mount.nfs-3331    [012] .....   949.293405: rpc_clnt_new: client=00000000 peer=[10.26.128.221]:2049 program=nfs server=10.26.128.221 xprtsec=none flags=DISCRTRY|INFINITE_SLOTS|NO_RETRANS_TIMEOUT
       mount.nfs-3331    [012] .....   949.293407: rpc_task_begin: task:00000001@00000000 flags=NULLCREDS|DYNAMIC|SOFT|SOFTCONN|CRED_NOREF runstate=ACTIVE status=0 action=0x0
       mount.nfs-3331    [012] .....   949.294496: rpc_connect_status: task:00000001@00000000 status=-11

With the communication established between client and server, now the system needs to establish a NFS session. It starts by giving a unique identity for the NFS client on the server. The client sends an EXCHANGE_ID RPC to the server:

       mount.nfs-3331    [012] .....   949.295698: rpc_request: task:00000002@00000000 nfsv4 EXCHANGE_ID (sync)

... and the server responds with a client ID (xid=0x734ec2e0):

       mount.nfs-3331    [012] .....   949.296538: rpc_stats_latency: task:00000002@00000000 xid=0x734ec2e0 nfsv4 EXCHANGE_ID backlog=11 rtt=719 execute=841
       mount.nfs-3331    [012] .....   949.296541: nfs4_exchange_id: error=0 (OK) dstaddr=10.26.128.221

Next, it's time to create the session. The client uses the client ID from EXCHANGE_ID to request a session:

   10.26.128.221-m-3333    [001] .....   949.297657: rpc_request: task:00000004@00000000 nfsv4 CREATE_SESSION (sync)
   10.26.128.221-m-3333    [001] .....   949.298389: rpc_stats_latency: task:00000004@00000000 xid=0x754ec2e0 nfsv4 CREATE_SESSION backlog=10 rtt=583 execute=731

Then, the server responds with session parameters like session id (0x87e0d01f), number of slots, sequence ID and callback information. This session now will be used for all subsequent NFS operations:

   10.26.128.221-m-3333    [001] .....   949.298393: nfs4_create_session: error=0 (OK) dstaddr=10.26.128.221
   10.26.128.221-m-3333    [001] .....   949.299115: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=1 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()

Then, the client sends a LOOKUP_ROOT, and SECINFO_NO_NAME request to the server. The LOOKUP_ROOT operation retrieves the filehandle of the root directory of the exported NFS file system, and the SECINFO_NO_NAME determines the security mechanisms (authentication methods) supported by the server for accessing a specific path component. These operations are important to establish the starting point of the NFS namespace on the client side. Once the client gathered and validated the root filehandle, it can begin traversing the directory tree:

       mount.nfs-3331    [012] .....   949.299259: rpc_request: task:00000001@00000001 nfsv4 SECINFO_NO_NAME (sync)
       mount.nfs-3331    [012] .....   949.300992: rpc_stats_latency: task:00000001@00000001 xid=0x774ec2e0 nfsv4 SECINFO_NO_NAME backlog=718 rtt=1612 execute=2444
       mount.nfs-3331    [012] .....   949.305147: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=3 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.305148: rpc_request: task:00000002@00000001 nfsv4 LOOKUP_ROOT (sync)
       mount.nfs-3331    [012] .....   949.306163: rpc_stats_latency: task:00000002@00000001 xid=0x784ec2e0 nfsv4 LOOKUP_ROOT backlog=10 rtt=782 execute=1020
       mount.nfs-3331    [012] .....   949.306164: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=3 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.306168: nfs4_lookup_root: error=0 (OK) fileid=00:00:64 fhandle=0x507c44a0 valid=TYPE|MODE|NLINK|OWNER|GROUP|SIZE|FSID|FILEID|ATIME|MTIME|CTIME|CHANGE|0x400200

Next, the client probes the server about the file system capabilities with a FSINFO and SERVER_CAPS request, and gets its attributes issuing a GETATTR operation retrieving file type, size, permissions, timestamps, etc:

       mount.nfs-3331    [012] .....   949.306786: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=5 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.306786: rpc_request: task:00000004@00000001 nfsv4 FSINFO (sync)
       mount.nfs-3331    [012] .....   949.307693: rpc_stats_latency: task:00000004@00000001 xid=0x7a4ec2e0 nfsv4 FSINFO backlog=3 rtt=897 execute=907
       mount.nfs-3331    [012] .....   949.307693: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=5 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.307694: nfs4_fsinfo: error=0 (OK) fileid=00:00:64 fhandle=0x507c44a0 valid=TYPE|MODE|NLINK|OWNER|GROUP|SIZE|FSID|FILEID|ATIME|MTIME|CTIME|CHANGE|0x400200
       mount.nfs-3331    [012] .....   949.307696: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=6 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.307697: rpc_request: task:00000005@00000001 nfsv4 SERVER_CAPS (sync)
       mount.nfs-3331    [012] .....   949.308249: rpc_stats_latency: task:00000005@00000001 xid=0x7b4ec2e0 nfsv4 SERVER_CAPS backlog=3 rtt=452 execute=552
       mount.nfs-3331    [012] .....   949.308249: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=6 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.309855: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=10 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.309855: rpc_request: task:00000009@00000001 nfsv4 GETATTR (sync)
       mount.nfs-3331    [012] .....   949.310555: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=10 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 () 
       mount.nfs-3331    [012] .....   949.310557: nfs4_getattr: error=0 (OK) fileid=00:2f:64 fhandle=0x507c44a0 valid=TYPE|MODE|NLINK|OWNER|GROUP|SIZE|FSID|FILEID|ATIME|MTIME|CTIME|CHANGE|0x400200

Then, it verifies for the client access permissions, and resolves the full path /vol_cp_5node:

       mount.nfs-3331    [012] .....   949.310575: nfs_access_enter: fileid=00:2f:64 fhandle=0x507c44a0 version=1881467492116116952
       mount.nfs-3331    [012] .....   949.310577: nfs_access_exit: error=-10 (CHILD) fileid=00:2f:64 fhandle=0x507c44a0 type=4 (DIR) version=1881467492116116952 size=4096 cache_validity=0x0 () nfs_flags=0x0 () mask=0x81 permitted=0xffffffff
       mount.nfs-3331    [012] .....   949.310577: nfs_access_enter: fileid=00:2f:64 fhandle=0x507c44a0 version=1881467492116116952
       mount.nfs-3331    [012] .....   949.310580: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=11 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.310580: rpc_request: task:0000000a@00000001 nfsv4 ACCESS (sync)
       mount.nfs-3331    [012] .....   949.311414: rpc_stats_latency: task:0000000a@00000001 xid=0x804ec2e0 nfsv4 ACCESS backlog=3 rtt=688 execute=834
       mount.nfs-3331    [012] .....   949.311415: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=11 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.311421: nfs4_access: error=0 (OK) fileid=00:2f:64 fhandle=0x507c44a0
       mount.nfs-3331    [012] .....   949.311422: nfs_access_exit: error=0 (OK) fileid=00:2f:64 fhandle=0x507c44a0 type=4 (DIR) version=1881467492116116952 size=4096 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET) mask=0x1 permitted=0x7
       mount.nfs-3331    [012] .....   949.311428: nfs_lookup_enter: flags=0x5 (FOLLOW|AUTOMOUNT) name=00:2f:64/vol_cp_5node fileid=0
       mount.nfs-3331    [012] .....   949.311429: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=12 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.312312: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=12 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.312317: nfs4_lookup: error=0 (OK) name=00:2f:64/vol_cp_5node
       mount.nfs-3331    [012] .....   949.312320: nfs_lookup_exit: error=0 (OK) flags=0x5 (FOLLOW|AUTOMOUNT) name=00:2f:64/vol_cp_5node fileid=105

Finally, the mount completes successfully and the RPC resources are released:

       mount.nfs-3331    [012] .....   949.315199: nfs_mount_option: option source
       mount.nfs-3331    [012] .....   949.315201: nfs_mount_path: path='/vol_cp_5node'
       mount.nfs-3331    [012] .....   949.316737: rpc_clnt_shutdown: client=00000002
       mount.nfs-3331    [012] .....   949.316738: rpc_clnt_release: client=00000002
    kworker/12:1-450     [012] .....   949.316760: rpc_clnt_free: client=00000002
       mount.nfs-3331    [012] .....   949.328981: rpc_clnt_shutdown: client=00000001
       mount.nfs-3331    [012] .....   949.328990: rpc_clnt_release: client=00000001
    kworker/12:1-450     [012] .....   949.329007: rpc_clnt_free: client=00000001

A Bunch of Tracing Messages, Now What?

Understanding the communication flow by analyzing the sequence of tracing messages is incredibly valuable to me. It significantly enhances my ability to troubleshoot deployment and performance issues. Let me give you a couple practical examples.

Suppose you issue an NFS mount command and it hangs without any obvious reason. In such cases, you can terminate the command. Next, enable tracing for the sunrpc and nfs subsystems, then re-run the mount operation. With tracing enabled, you can observe the communication flow and identify exactly where the process is getting stuck.

Now, supose your system begins performing more I/O than usual due to a subtle increase in user activity. You can enable tracing for the NFS subsystem and monitor the highest_slotid, target_highest_slotid, and highest_used_slotid in the nfs4_setup_sequence and nfs4_sequence_done trace events.

If you observe that highest_used_slotid from nfs4_setup_sequence frequently matches the highest_slotid reported in nfs4_sequence_done, this indicates that your system is hitting a session slot bottleneck. In such cases, you should consider increasing the number of available slots to improve concurrency and performance. Not sure what the NFSv4 slot table is or how it works? Don't worry, we'll break it down in the next post of this series.

Tracing isn't just about collecting logs. It's about gaining visibility into the inner workings of your system. Whether you're troubleshooting a stalled mount or diagnosing a session slot bottleneck, tracing empowers you to move from guesswork to informed action. The more fluent you become in reading these messages, the more effective you'll be at tuning and scaling your systems with confidence.

Wrapping Up

In the next post, we'll explore what really happens under the hood when you read from or write to a file on an NFS server with pNFS enabled. We'll trace the data paths, examine how layout operations work, and see how the client coordinates with multiple data servers to boost performance. If you've ever wondered how NFS scales I/O across nodes, stay tuned for the next post in the Back to Basics series.

References

[1] Nowicki, B. (1989). RFC 1094: NFS Network File System Protocol Specification. Available from: <https://www.rfc-editor.org/rfc/pdfrfc/rfc1094.txt.pdf>

[2] Callaghan, B., Pawlowaki, B., Staubach, P. (1995). RFC 1813: NFS Version 3 Protocol Specification. Available from: <https://www.rfc-editor.org/rfc/pdfrfc/rfc1813.txt.pdf>

[3] Shepler, S., et al. (2003). RFC 3530: Network File System (NFS) version 4 Protocol. Available from: <https://www.rfc-editor.org/rfc/pdfrfc/rfc3530.txt.pdf>

[4] Shepler, S., Eisler, M., Noveck, D. (2010). RFC 5661: Network File System (NFS) Version 4 Minor Version 1 Protocol. Available from: <https://www.rfc-editor.org/rfc/pdfrfc/rfc5661.txt.pdf>

[5] Haynes, T. (2016). RFC 7862: Network File System (NFS) Version 4 Minor Version 2 Protocol. Available from: <https://www.rfc-editor.org/rfc/pdfrfc/rfc7862.txt.pdf>