Since we announced NetApp Novus, I've been getting a lot of questions about my past posts on KV cache offloading. The most common question has been, "do those posts apply to Novus?" The good news is that all of my posts covering ONTAP pNFS also apply to Novus since it also uses pNFS. So that you don't have to go digging through those old posts, I'm posting this primer on implementing KV cache offloading with Novus.
KV cache offloading refresher
If you read my previous posts and are already familiar with KV cache offloading, you can skip this section. If it's a new concept for you, read on.
First off, what is a KV cache? Simply put, KV (key-value) caching is a technique that is used to optimize large language model (LLM) inference by storing previously calculated values in a KV cache so that these values don't need to be calculated again for every new token that is generated, which would otherwise be necessary. This article from HuggingFace provides a good overview.
KV cache offloading is simply the offloading of a previously-processed prompt's KV cache to an external target outside of GPU/accelerator memory, such as system RAM, local disk, or shared storage. You can think of these KV cache offload "targets" as functioning as tiered "swap spaces" for GPU memory. It has helpful to have a KV cache tier that is shared across your entire fleet so that individual nodes can retrieve KV cache blocks that were generated by other nodes. With a shared tier, you can route an incoming request to any node, and no matter where it goes, a pre-existing KV cache block will never need to be recalculated.
LLM inference ecosystem
Before digging into KV cache offloading frameworks, it is helpful to have a basic understanding of the LLM inference ecosystem. There are two leading open-source engines for serving LLMs - vLLM and SGLang. While vLLM has historically seen wider adoption, SGLang has emerged as a top-tier engine that is catching up rapidly.
In this post, I will show you how to offload your KV cache to NetApp Novus using vLLM's native offloading connector, SGLang's HiCache framework, and LMCache, a popular KV cache offloading framework that is primarily used alongside vLLM. You can pick the framework that is best suited for your deployment.
Note that all of the examples included in this post implement the same three-tier KV cache setup:
- Hot tier - GPU memory
- Tier 2 - CPU memory (100 GB capacity)
- Tier 3 (shared across GPU servers) - NetApp Novus pNFS
vLLM
vLLM's native offloading connector
There is a KV cache offloading connector that is built directly into vLLM. We can use this native offloading connector to implement our three-tier setup. I have included an example vLLM server startup command below, for reference. Note that you may want to tune the block size if you serve very long prompts.
#Start vLLM server with native offloading connector
PYTHONHASHSEED=0 \
vllm serve \
Qwen/Qwen3-8B \
--tensor-parallel-size 1 \
--kv-transfer-config '{
"kv_connector": "OffloadingConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"spec_name": "TieringOffloadingSpec",
"cpu_bytes_to_use": 100000000000,
"block_size": 256,
"eviction_policy": "lru",
"offload_prompt_only": false,
"secondary_tiers": [
{
"type": "fs",
"root_dir": "/mnt/novus",
"n_read_threads": 32,
"n_write_threads": 16
}
]
}
}' \
--port 8000
LMCache (MP mode)
LMCache is a popular open-source KV cache offloading framework that is commonly used alongside vLLM. LMCache currently supports two deployment modes, in-process mode and multiprocess (MP) mode. In-process mode existed first, but MP mode is the recommended mode going forward. When using MP mode, LMCache runs as a standalone service outside of the vLLM process, and vLLM attaches to LMCache using a connector (LMCacheMPConnector). LMCache features many different L2 storage backends, several of which will work with a mounted filesystem like Novus pNFS. We have validated the "nixl_store_dynamic" and "fs_native" backends.
I have included example LMCache server and vLLM server startup commands for our three-tier setup below, for reference. Note that you may want to tune the chunk size if you serve very long prompts.
#Option 1 - Start LMCache server (nixl_store_dynamic backend with GDS)
PYTHONHASHSEED=0 \
lmcache server \
--host localhost \
--port 5555 \
--http-port 8080 \
--max-workers 8 \
--l1-size-gb 100 \
--eviction-policy LRU \
--l2-prefetch-max-in-flight 128 \
--l2-adapter '{
"type": "nixl_store_dynamic",
"backend": "GDS",
"backend_params": {
"file_path": "/mnt/novus",
"use_direct_io": "true",
"max_capacity_gb": "10000"
}
}' \
--chunk-size 256
#Option 2 - Start LMCache server (fs_native backend)
PYTHONHASHSEED=0 \
lmcache server \
--host localhost \
--port 5555 \
--http-port 8080 \
--max-workers 8 \
--l1-size-gb 100 \
--eviction-policy LRU \
--l2-prefetch-max-in-flight 128 \
--l2-adapter '{
"type": "fs_native",
"base_path": "/mnt/novus",
"num_workers": 128
}' \
--chunk-size 256
#Start vLLM server with LMCache MP connector
PYTHONHASHSEED=0 \
vllm serve \
Qwen/Qwen3-8B \
--tensor-parallel-size 1 \
--kv-transfer-config '{
"kv_connector": "LMCacheMPConnector",
"kv_connector_module_path": "lmcache.integration.vllm.lmcache_mp_connector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"lmcache.mp.host": "localhost",
"lmcache.mp.port": 5555
}
}' \
--port 8000
SGLang
HiCache
HiCache is a flexible KV cache offloading framework that is built into SGLang. You can find more details on HiCache, including an architecture deep-dive, here. HiCache's NIXL backend can be used to implement our three-tier setup. I have included an example NIXL backend config file and SGLang server startup command below, for reference. Note that you may want to tune the page size if you serve very long prompts.
NIXL backend config file
[plugin.gds_mt]
thread_count = 32
active = true
SGLang server startup command
PYTHONHASHSEED=0 \
SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR=/mnt/novus \
python3 -m sglang.launch_server \
--model-path "Qwen/Qwen3-8B" \
--host 0.0.0.0 \
--port 8000 \
--page-size 256 \
--enable-hierarchical-cache \
--hicache-size 100 \
--hicache-write-policy write_through \
--hicache-storage-prefetch-policy wait_complete \
--hicache-storage-backend nixl \
--hicache-storage-backend-extra-config "@nixl_backend_config.toml"
Performance
During preliminary testing in our lab, we observed a 20x reduction in average TTFT (time to first token) per request and an almost 2x increase in prefill throughput when compared to a baseline two-tier setup with GPU and CPU memory only. Our testing efforts are ongoing, so stay tuned.
With NetApp Novus, you can generate more tokens with the same GPU infrastructure. To learn more about Novus, click here.
