In my last post, I showed how to use vLLM's native offloading connector and LMCache's MP mode, two new frameworks that have emerged this year, to offload your KV cache to NetApp pNFS. I also compared the performance of these two frameworks to LMCache's legacy in-process mode, which I had written about previously. In this post, I'll introduce an additional framework into the comparison.
LLM serving engines
All of my previous posts on KV cache offloading have focused on the vLLM ecosystem. However, there are two leading open-source LLM serving engines - vLLM and SGLang. While vLLM has historically seen wider adoption, SGLang has emerged as a top-tier engine that is catching up rapidly. In this post, I will show you how to offload your KV cache to NetApp pNFS (Novus or ONTAP) using SGLang's HiCache framework, and I will compare HiCache's performance to the other frameworks that we've tested.
SGLang HiCache
HiCache is a flexible KV cache offloading framework that is built into SGLang. You can find more details on HiCache, including an architecture deep-dive, here. We can use HiCache's NIXL backend to implement the same three-tier KV cache setup that I previously demonstrated with vLLM's native offloading connector and LMCache:
- Hot tier - GPU memory
- Tier 2 - CPU memory (100 GB capacity)
- Tier 3 - NetApp pNFS over RDMA (Novus or ONTAP)
I have included an example NIXL backend config file and SGLang server startup command for this three-tier setup below, for reference. Note that you may want to tune the page size if you serve very long prompts.
NIXL backend config file
[plugin.gds_mt]
thread_count = 32
active = true
SGLang server startup command
PYTHONHASHSEED=0 \
SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR=/mnt/kvcache \
python3 -m sglang.launch_server \
--model-path "Qwen/Qwen3-8B" \
--host 0.0.0.0 \
--port 8000 \
--page-size 256 \
--enable-hierarchical-cache \
--hicache-size 100 \
--hicache-write-policy write_through \
--hicache-storage-prefetch-policy wait_complete \
--hicache-storage-backend nixl \
--hicache-storage-backend-extra-config "@nixl_backend_config.toml"
Performance comparison
We tested this three-tier SGLang HiCache setup using the same benchmark that we previously ran against the vLLM ecosystem frameworks. The results of our testing are captured in the following chart. This chart shows the overall system throughput across multiple initial prompt lengths for all of the different setups that we've tested.
As you can see from the results, vLLM's native offloading connector delivered the best performance overall.
Conclusion
KV cache offloading is an increasingly important technique, and the inclusion of a shared storage tier can significantly enhance your LLM inference performance. When exploring your options, remember that NetApp supports KV cache offloading using industry-standard protocols and tooling. No custom proprietary clients are required. You can continue to use the same NFS and S3 protocols that you have used for years. To learn more about NetApp's platforms and services, visit netapp.com.
