If you have been following this blog for a while, you have probably noticed that the majority of my recent posts have dealt with KV cache offloading. This is not by accident. As AI initiatives reach operational maturity, ROI is coming under increasing scrutiny while compute infrastructure remains structurally scarce and expensive. To maximize hardware efficiency, KV cache offloading has become a critical strategy for generating more tokens using the exact same compute footprint. The KV cache offloading software ecosystem continues to expand accordingly. My previous posts on offloading your KV cache to ONTAP pNFS and StorageGRID S3 have focused on LMCache. LMCache is a popular open-source KV cache offloading framework for vLLM, which is itself one of the leading open-source LLM serving engines. In this post, I will explore two new options for offloading your KV cache to shared storage with vLLM.
KV cache offloading refresher
If you read my previous posts and are already familiar with KV cache offloading, you can skip this section. If it's a new concept for you, read on.
First off, what is a KV cache? Simply put, KV (key-value) caching is a technique that is used to optimize large language model (LLM) inference by storing previously calculated values in a KV cache so that these values don't need to be calculated again for every new token that is generated, which would otherwise be necessary. This article from HuggingFace provides a good overview.
KV cache offloading is simply the offloading of a previously-processed prompt's KV cache to an external target outside of GPU/accelerator memory, such as system RAM, local disk, or shared storage. You can think of these KV cache offload "targets" as functioning as tiered "swap spaces" for GPU memory.
New vLLM offloading options
vLLM's native offloading connector
Earlier this year, the vLLM project introduced a new native offloading connector that implements similar functionality to the LMCache connector that I wrote about previously. A key difference is that this native connector invokes an offloading framework built directly into vLLM. It doesn't require the installation of a separate framework (e.g. LMCache). At the time of this writing, the native connector supports offloading to CPU memory (system RAM) and any locally-mounted filesystem.
With the native connector, we can implement the same three-tier KV cache setup that I previously demonstrated using LMCache:
- Hot tier - GPU memory
- Tier 2 - CPU memory (100 GB capacity)
- Tier 3 - ONTAP pNFS over RDMA (capped by capacity of ONTAP volume)
I have included an example vLLM server startup command for this three-tier setup below, for reference. Note that you may want to tune the block size if you serve very long prompts.
#Start vLLM server with native offloading connector
PYTHONHASHSEED=0 \
vllm serve \
Qwen/Qwen3-8B \
--tensor-parallel-size 1 \
--kv-transfer-config '{
"kv_connector": "OffloadingConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"spec_name": "TieringOffloadingSpec",
"cpu_bytes_to_use": 100000000000,
"block_size": 256,
"eviction_policy": "lru",
"offload_prompt_only": false,
"secondary_tiers": [
{
"type": "fs",
"root_dir": "/mnt/kvcache",
"n_read_threads": 32,
"n_write_threads": 16
}
]
}
}' \
--port 8000
LMCache MP mode
This past spring, the LMCache project released a new architecture called multiprocess (MP) mode. When using MP mode, LMCache runs as a standalone service outside of the vLLM process, and vLLM attaches to LMCache using a new connector (LMCacheMPConnector). The original architecture, in which LMCache is invoked by a different connector (LMCacheConnectorV1) and runs inside the vLLM process, was dubbed in-process mode. LMCache MP mode supports offloading to CPU memory and many different storage backends. We can use the NIXL dynamic or FS native backend with ONTAP pNFS over RDMA and implement the same three-tier setup I previously described.
I have included example LMCache server and vLLM server startup commands for this three-tier setup below, for reference. Note that you may want to tune the chunk size if you serve very long prompts.
#Option 1 - Start LMCache server (NIXL dynamic backend with GDS)
PYTHONHASHSEED=0 \
lmcache server \
--host localhost \
--port 5555 \
--http-port 8080 \
--max-workers 32 \
--l1-size-gb 100 \
--eviction-policy LRU \
--l2-adapter '{
"type": "nixl_store_dynamic",
"backend": "GDS",
"backend_params": {
"file_path": "/mnt/kvcache",
"use_direct_io": "true",
"max_capacity_gb": "10000"
}
}' \
--chunk-size 256
#Option 2 - Start LMCache server (FS native backend)
PYTHONHASHSEED=0 \
lmcache server \
--host localhost \
--port 5555 \
--http-port 8080 \
--max-workers 32 \
--l1-size-gb 100 \
--eviction-policy LRU \
--l2-adapter '{
"type": "fs_native",
"base_path": "/mnt/kvcache",
"num_workers": 32
}' \
--chunk-size 256
#Start vLLM server with LMCache MP connector
PYTHONHASHSEED=0 \
vllm serve \
Qwen/Qwen3-8B \
--tensor-parallel-size 1 \
--kv-transfer-config '{
"kv_connector": "LMCacheMPConnector",
"kv_connector_module_path": "lmcache.integration.vllm.lmcache_mp_connector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"lmcache.mp.host": "localhost",
"lmcache.mp.port": 5555
}
}' \
--port 8000
Performance comparison
We ran a benchmark to compare the performance of LMCache's in-process mode, LMCache's MP mode, and vLLM's native offloading connector when offloading to ONTAP pNFS over RDMA.
Benchmark
For this testing, we used the same benchmark that we used in our previous testing, the LMCache project's multi-round QA benchmark. Again, we tested against two separate vLLM instances behind a load balancer. The five scenarios that we tested this time are outlined in the following table.
Scenario | Description |
|---|
Baseline for comparison purposes | KV cache tiers: - GPU memory
- CPU memory (100 GB capacity)
|
vLLM with LMCache in-process mode (filesystem backend for pNFS tier)* *same setup we tested previously | KV cache tiers: - GPU memory
- CPU memory (100 GB capacity)
- ONTAP pNFS over RDMA (10 TB capacity)
|
vLLM with LMCache MP mode (NIXL dynamic backend with GDS for pNFS tier) | KV cache tiers: - GPU memory
- CPU memory (100 GB capacity)
- ONTAP pNFS over RDMA (10 TB capacity)
|
vLLM with LMCache MP mode (FS native backend for pNFS tier) | KV cache tiers: - GPU memory
- CPU memory (100 GB capacity)
- ONTAP pNFS over RDMA (10 TB capacity)
|
vLLM with native offloading connector | KV cache tiers: - GPU memory
- CPU memory (100 GB capacity)
- ONTAP pNFS over RDMA (10 TB capacity)
|
Consistent with our last round of testing, we held the shared system prompt constant at 1,000 tokens, the maximum number of concurrent users constant at 15, the number of rounds per user constant at 20, the benchmark runtime constant at 20 minutes, and the queries per second (QPS) constant at 2. We once again re-ran the benchmark with different values for the length of the user-specific context. This value represents the length of the unique context included in each user's first message. For each vLLM instance, we served the Qwen3-8B model on a single GPU. I've included the exact benchmark command that we used below. You can use this to run the same tests against your own environment.
python3 $LMCACHE_REPO_PATH/benchmarks/multi_round_qa/multi-round-qa.py \
--num-users 15 \
--num-rounds 20 \
--qps 2 \
--shared-system-prompt 1000 \
--user-history-prompt $USER_CONTEXT_LENGTH \
--answer-len 100 \
--time 1200 \
--model Qwen/Qwen3-8B \
--output $WORKING_DIR/${SCENARIO_NAME}_${USER_CONTEXT_LENGTH}.csv \
--base-url http://localhost:8000/v1
The benchmark begins by immediately creating 15 user sessions, staggered in simulated time so they appear to have started at evenly spaced offsets in the past (roughly 9.5 seconds apart, given QPS=2). Each user's first request includes the 1,000-token shared system prompt concatenated with their user-specific context, followed by a question. In subsequent rounds, the full accumulated chat history is sent with each request, so prompt length grows with every round. Each user sends one request every 7.5 seconds (15 users / 2 QPS) for up to 20 rounds. New user sessions are added by the manager on a fixed time interval (roughly every 9.5 seconds). When the 20-minute time limit is reached, the simulation loop exits and the benchmark waits for all in-flight requests to complete before writing results and exiting.
Results
The results of our testing are captured in the following chart. This chart shows the overall system throughput across multiple initial prompt lengths for each of the five deployment setups.
As you can see from the results, vLLM's native offloading connector delivered the best performance overall, and LMCache's in-process mode outperformed MP mode for longer prompts. At the time of this writing, the LMCache project has not yet implemented support for preempted prompts in the MP mode connector. We believe this is the reason for MP mode's underperformance. This system was clearly under a heavy load, and vLLM needed to preempt prompts to maintain maximum efficiency. Whenever a previously-preempted prompt was reloaded, MP mode forced the recalculation of the entire KV cache.
Recommended tooling
If you are using vLLM to serve your model(s) and wish to implement a multi-tier KV cache setup using ONTAP pNFS over RDMA, we recommend using vLLM's native offloading connector or LMCache's in-process mode (with the filesystem backend). At this time, we do not recommend LMCache's MP mode. When support for preempted prompts is added in MP mode, we will revisit this testing and post an updated recommendation.
Conclusion
KV cache offloading is an increasingly important technique, and the inclusion of a shared storage tier can significantly enhance your LLM inference performance. When exploring your options, remember that NetApp supports KV cache offloading using industry-standard protocols and tooling. No custom proprietary clients are required. You can continue to use the same NFS and S3 protocols that you have used for years. To learn more about NetApp's platforms and services, visit netapp.com.
