Several blogs (From hours to minutes: Trident 26.06 controller concurrency at 1000 volume scale and Trident Controller Parallelism) have described the improvements made, starting in Trident 25.06, to add concurrency to the Trident controller to increase throughput and scalability. Here we will do a deep dive into how this is achieved.
The Trident controller primarily orchestrates the relationship between storage backends (such as ONTAP) and Kubernetes resources (such as PersistentVolumeClaims). This can involve many API requests back and forth with the backend, that must be ordered. For example, with Trident-managed export policies and the ONTAP-NAS backend, the FlexVol must be created in ONTAP before the export policy is attached. This means for most Trident operations, only one thing can happen to a volume at a time, so we need to coordinate access to resources.
Concurrent Cache
Trident has always kept an internal cache of backends, volumes, etc. There were no limits placed on accessing these resources, because the global lock serialized all access. With concurrency, we introduce the concurrent cache. The primary principle under which the concurrent cache operates is lock ordering: if locks are always taken and released in the same order, there will never be a deadlock.
An early insight during this effort was 'hierarchical locks:' locking a snapshot's volume dependency before the snapshot prevents the volume from being deleted while the snapshot is being modified. This hierarchy is defined by the schema below (from Trident 26.06.0):
// schema is the authoritative source for relationships between resources. For example,
// a snapshot depends on a volume, and a volume depends on a backend.
var schema = map[resource][]resource{
root: nil, // root is not a schema dependency; injected synthetically during consistent locks
node: nil,
storageClass: nil,
groupSnapshot: nil, // no schema dependencies; constituents are locked as separate snapshot trees
backend: nil,
autogrowPolicy: nil,
volume: {backend},
subordinateVolume: {volume},
volumePublication: {volume, node},
snapshot: {volume},
}
The schema can be extended with new types, and will handle sorting automatically.
Nested locks
In Trident 26.06.0, we introduced nested locks. Previously it was illegal to call Lock() when a previous set of locks were held (List and InconsistentRead queries were excepted). Lock() now returns a context with information attached to allow NestedLock to be called. The most important information attached to the context is the 'lastLockHeld'. Because the lock order is stable, by recording the last lock held by the previous Lock() call we can compare all the new Queries, and if any Subquery would sort before this last lock NestedLock() returns an error.
Query structure
The cache interface was inspired by SQL rows: all rows in a sql.Result will contain the same columns. Lock() (and NestedLock()) returns a slice of Results. Each Result can point to one of every possible resource, and functions to create, update, or delete each resource. Lock() is called with an arbitrary set of Queries, that contain Subqueries. Each Query may only contain one Subquery for each resource type (plus Lists), and Lock() may be called with any number of Queries. Results will be returned in the same order as a Queries were added to Lock.
Example
This example walks through each step of locking, using, and unlocking multiple queries to publish a volume. volume1, backend1, and node1 already exist in the cache.
_, results, unlocker, err := db.Lock(ctx,
db.Query(
db.UpsertVolumePublication("volume1", "node1"),
db.ListVolumePublications(),
db.ListVolumes(),
db.ReadBackend(""),
db.UpsertVolume("volume1", ""),
),
db.Query(
db.UpsertBackend("backend1", "", ""),
db.UpsertSnapshot("volume1", "snap1"),
),
)
defer unlocker()
if err != nil {
return err
}
if results[0].VolumePublication.Read != nil {
return fmt.Errorf("volume publication already exists")
}
results[0].VolumePublication.Upsert(&models.VolumePublication{
NodeName: "node1",
VolumeName: "volume1",
})
if len(results[0].VolumePublications) != 0 {
return fmt.Errorf("unexpected existing publications")
}
if len(results[0].Volumes) <= 0 {
return fmt.Errorf("no volumes listed")
}
if results[0].Backend.Read == nil {
return fmt.Errorf("no backend found")
}
if results[1].Backend.Read == nil {
return fmt.Errorf("no backend found in results[1]")
}
// Mutate backend state and persist the copy held in the result.
results[1].Backend.Read.SetState(storage.Unknown)
results[1].Backend.Upsert(results[1].Backend.Read)
results[1].Snapshot.Upsert(&storage.Snapshot{
Config: &storage.SnapshotConfig{
Name: "snap1",
VolumeName: "volume1",
},
})
UpsertVolumePublication does not need to be in a separate query from UpsertVolume as written in production code. In the actual concurrent core it is isolated into its own query to allow subordinate volumes to be set in another Result; here the second query instead carries unrelated backend and snapshot work to show how multiple queries merge into a single lock order.
The dependency trees built from each query look like this:
Step 1: Dedupe and build trees
Lock() begins by calling assembleQueries() on each Query slice. dedupe() sorts subqueries and returns an error if more than one non-list operation targets the same resource type. Because this query has no duplicates, the first query is sorted to:
[ListVolumePublications(), ListVolumes(), ReadBackend(""), UpsertVolume("volume1", ""), UpsertVolumePublication("volume1", "node1")]
Next, buildTrees() walks each subquery, appends synthetic read subqueries for any missing schema dependencies, and records tree roots.
Query 0 — fill in missing dependency nodes:
ListVolumes() → list op, complete tree
ListVolumePublications() → list op, complete tree
ReadBackend("") → no schema dependencies, complete tree
UpsertVolume("volume1", "") → append Subquery{read, backend}; dependencies = [backend index]
UpsertVolumePublication("volume1", "node1")
→ append Subquery{read, volume}, Subquery{read, node}; dependencies = [volume, node]
Query 0 — find roots:
ListVolumes() → list op → root
ListVolumePublications() → list op → root
ReadBackend("") → volume is in query (inverse schema) → not a root
UpsertVolume("volume1", "") → no dependents in query → root
UpsertVolumePublication(...) → no dependents in query → root
Subquery{read, volume} → UpsertVolumePublication is in query → not a root
Subquery{read, node} → not a root
Subquery{read, backend} → volume is in query → not a root
roots = [ListVolumes, ListVolumePublications, UpsertVolume, UpsertVolumePublication]
The second query builds similarly:
UpsertBackend("backend1") → complete tree → root
UpsertSnapshot("volume1", "snap1") → append Subquery{read, volume}; dependencies = [volume index] → root
Subquery{read, volume} → UpsertSnapshot is in query → not a root
Step 2: Fill in IDs
Under read locks on every cache touched by either query (lockCachesAndFillInIDs), each tree is walked from its roots to resolve missing IDs before any per-resource locks are taken.
Query 0:
root: ListVolumes() → list, skip
root: ListVolumePublications() → list, skip
root: UpsertVolume("volume1", "")
id already set to "volume1"
dependency: read backend
cast volume to BackendDependent → set backend.id = "backend1"
root: UpsertVolumePublication("volume1", "node1")
setDependencyIDs:
Subquery{read, volume}.id = "volume1"
Subquery{read, node}.id = "node1"
recursive fillInIDs on volume:
dependency: read backend → set backend.id = "backend1" (from cached volume)
After ID resolution, query 0 looks like:
[Subquery{list, volumePublication},
Subquery{list, volume},
Subquery{read, backend, "backend1"},
Subquery{upsert, volume, "volume1"},
Subquery{upsert, volumePublication, "volume1.node1"},
Subquery{read, volume, "volume1"},
Subquery{read, node, "node1"}]
Query 1:
root: UpsertBackend("backend1") → id = "backend1"
root: UpsertSnapshot("volume1", "snap1")
setDependencyIDs → Subquery{read, volume}.id = "volume1"
Step 3: Merge and sort
Both queries are merged. Because at least one subquery requires a consistent lock, a synthetic read on root is prepended to block shutdown during the operation. The combined list is sorted by list operations first, then resource rank (dependencies before dependents), resource type, ID, and operation:
# Subquery{<operation>, <resource>, <id>, <setResults>, <result index>}
[
Subquery{read, root, ".", nil, -},
Subquery{list, volume, "", setResults(), 0},
Subquery{list, volumePublication, "", setResults(), 0},
Subquery{upsert, backend, "backend1", setResults(), 1},
Subquery{read, backend, "backend1", setResults(), 0},
Subquery{read, node, "node1", nil, 0},
Subquery{upsert, volume, "volume1", setResults(), 0},
Subquery{read, volume, "volume1", nil, 0},
Subquery{read, volume, "volume1", nil, 1},
Subquery{upsert, volumePublication, "volume1.node1", setResults(), 0},
Subquery{upsert, snapshot, "snap1", setResults(), 1},
]
Note that ReadBackend from query 0 and UpsertBackend from query 1 target the same backend, and both queries append synthetic ReadVolume dependencies. Sorting places the upsert (write) before the read on the same (resource, id) pair, so only the most restrictive lock is taken. Synthetic read subqueries without a setResults callback still participate in lock ordering but are skipped at lock time when a write on the same ID is already held.
Lock acquisition order:
1. root (synthetic read)
2. backend (upsert, query 1) ← write lock; satisfies read on same ID
3. node (read, query 0)
4. volume (upsert, query 0) ← write lock; satisfies reads on same ID
5. volumePublication (upsert, query 0)
6. snapshot (upsert, query 1)
Step 4: Acquire locks and populate results
lockNewKeys runs first (no new keys in this example). Then lockQuery walks the merged list:
Subquery{list, volume, ...}:
list does not take per-resource locks; call setResults()
take read lock on volume cache
deep-copy each volume → results[0].Volumes
release volume cache read lock
Subquery{list, volumePublication, ...}:
call setResults()
take read lock on volumePublication cache
deep-copy each publication → results[0].VolumePublications
release volumePublication cache read lock
Subquery{upsert, backend, "backend1", ...}:
take write lock on backend1; append unlock; call setResults()
backend1 exists → shallow copy → results[1].Backend.Read
set results[1].Backend.Upsert
Subquery{read, backend, "backend1", ...}:
write lock already held for backend1; call setResults()
shallow copy → results[0].Backend.Read
Subquery{read, node, "node1", ...}:
take read lock on node1; setResults is nil (synthetic dependency)
Subquery{read, volume, "volume1", ...} (both instances):
skipped — upsert on volume1 will take the write lock
Subquery{upsert, volume, "volume1", ...}:
take write lock on volume1; call setResults()
volume1 exists → deep copy → results[0].Volume.Read
set results[0].Volume.Upsert
Subquery{upsert, volumePublication, "volume1.node1", ...}:
take write lock on volume1.node1; call setResults()
publication does not exist → results[0].VolumePublication.Read stays nil
set results[0].VolumePublication.Upsert
Subquery{upsert, snapshot, "snap1", ...}:
take write lock on snap1; call setResults()
snapshot does not exist → results[1].Snapshot.Read stays nil
set results[1].Snapshot.Upsert
Step 5: Use results in the workflow
After Lock() returns, the caller performs backend I/O and then commits cache changes through the result callbacks:
results[0].VolumePublication.Upsert(vp)
take write lock on volumePublication cache
set volumePublication at "volume1.node1" to vp
release write lock
results[1].Backend.Upsert(b)
take read lock on backend cache
set backend at "backend1" to b
release read lock
results[1].Snapshot.Upsert(s)
take write lock on snapshot cache
set snapshot at "snap1" to s
release write lock
Each Upsert callback acquires its own cache lock when invoked. The per-resource locks held by Lock() remain active until unlocker() runs; the callbacks take separate cache-level locks to commit changes safely. The copies in results[n].*.Read are the caller's working copies—mutations do not affect the cache until the corresponding Upsert is called.
Step 6: Unlock
defer unlocker() releases all per-resource locks in reverse acquisition order:
snap1 write lock released
volume1.node1 write lock released
volume1 write lock released
node1 read lock released
backend1 write lock released
root read lock released
The returned context carries a lockContext recording every held lock and the last lock acquired (snapshot upsert), enabling a subsequent NestedLock call if the operation needs to touch additional resources later in the workflow. Any new locks from NestedLock must sort after the snapshot upsert; attempting to lock the volume or backend for write would return an error because those locks are already held as writes.
Nested lock example
The first Lock() call cannot know every resource a long workflow will touch up front. NestedLock() extends the locks already held in ctx as long as new locks sort after lastLockHeld, do not upgrade an existing read lock to a write lock, and do not include newKey updates. This pattern appears when updating a backend and then locking each of its volumes individually—after listing them under the backend write lock.
backend1 already exists with three volumes (volume1, volume2, volume3). The workflow holds the backend for write, lists its volumes, then acquires a write lock on each volume in a second locking round.
ctx, results, unlocker, err := db.Lock(ctx,
db.Query(
db.ListVolumesForBackend("backend1"),
db.UpsertBackend("backend1", "", ""),
),
)
if err != nil {
return err
}
defer unlocker()
backend := results[0].Backend.Read
volumes := results[0].Volumes
// One UpsertVolume subquery per listed volume. Each must be wrapped in its own db.Query
// because dedupe rejects multiple non-list operations on the same resource type in one Query.
upsertVolumes := make([]db.Subquery, 0, len(volumes))
for _, vol := range volumes {
upsertVolumes = append(upsertVolumes, db.UpsertVolume(vol.Config.Name, vol.BackendUUID))
}
lockQuery := make([][]db.Subquery, len(upsertVolumes))
for i, sq := range upsertVolumes {
lockQuery[i] = db.Query(sq)
}
nestedCtx, nestedResults, nestedUnlocker, err := db.NestedLock(ctx, lockQuery...)
if err != nil {
return err
}
defer nestedUnlocker()
for _, volResult := range nestedResults {
vol := volResult.Volume.Read
if vol == nil {
continue
}
vol.BackendUUID = backend.BackendUUID()
// ... check whether the volume still exists on the backend ...
volResult.Volume.Upsert(vol)
}
The two locking phases and the lockContext carried between them:
Dependency trees for each nested query (identical shape, different volume ID):
Phase A: Initial Lock
Step A1: Dedupe and build trees
The single query contains a list and an upsert on different resource types, so dedupe() succeeds and sorts to:
[ListVolumesForBackend("backend1"), UpsertBackend("backend1", "", "")]
buildTrees() finds two roots—both subqueries are complete trees with no schema dependencies to append:
ListVolumesForBackend("backend1") → list op → root
UpsertBackend("backend1", "", "") → root
roots = [ListVolumesForBackend, UpsertBackend]
Step A2: Fill in IDs
root: ListVolumesForBackend → list, skip
root: UpsertBackend → id = "backend1"
Step A3: Merge and sort
A synthetic read on root is prepended. The merged, sorted list is:
[
Subquery{read, root, ".", nil, -},
Subquery{list, volume, "", setResults(), 0},
Subquery{upsert, backend, "backend1", setResults(), 0},
]
Lock acquisition order:
1. root (synthetic read)
2. backend (upsert) ← write lock on backend1
The list subquery runs without per-resource locks; it takes a brief read lock on the volume cache inside setResults(), deep-copies volumes whose BackendUUID matches "backend1", and appends them to results[0].Volumes.
Step A4: Context returned
After Lock() returns, the context carries:
lockContext{
qs: { (backend, "backend1") → write },
lastLock: Subquery{upsert, backend, "backend1"},
}
The outer unlocker still holds the root read lock and the backend1 write lock until defer unlocker() runs.
Phase B: NestedLock
Step B1: Assemble nested queries
Each UpsertVolume("volumeN", "backend1") subquery gets its own Query slice. buildTrees() appends a synthetic Subquery{read, backend} for each:
Query 0: [UpsertVolume("volume1", "backend1"), Subquery{read, backend, dependencies=[backend]}]
Query 1: [UpsertVolume("volume2", "backend1"), Subquery{read, backend, ...}]
Query 2: [UpsertVolume("volume3", "backend1"), Subquery{read, backend, ...}]
setDependencyIDs on each upsert sets the synthetic backend's id to "backend1" directly from the argument—no cache walk is needed because the caller supplied the owner ID.
Step B2: Validate against held locks
lock() receives the existing querySet and lastLock from ctx. Three checks run before any new locks are taken:
No lock upgrades — every synthetic ReadBackend matches (backend, "backend1"), which is already in qs as a write. No read-to-write upgrade is attempted.
Ordering — the first new consistent lock is UpsertVolume("volume1"). Comparing against lastLockHeld (backend upsert):
compareSubqueries(backend upsert, volume1 upsert):
rank backend (0) < rank volume (1) → volume sorts after backend ✓
If the caller instead tried NestedLock(ctx, db.Query(db.UpsertBackend("backend1", "", ""))) on a backend that was read-locked, or attempted to lock backend2 (which sorts before a held backend1 write in a different goroutine's ordering), NestedLock would return an error rather than deadlock.
No key updates — nested locks reject any subquery with newKey set.
Step B3: Merge and sort nested queries
Because the outer lock already injected a synthetic root read, mergeQueries does not add a second one. The merged list is:
[
Subquery{upsert, volume, "volume1", setResults(), 0},
Subquery{read, backend, "backend1", nil, 0},
Subquery{upsert, volume, "volume2", setResults(), 1},
Subquery{read, backend, "backend1", nil, 1},
Subquery{upsert, volume, "volume3", setResults(), 2},
Subquery{read, backend, "backend1", nil, 2},
]
Step B4: Acquire new locks and populate results
lockQuery skips any subquery whose (resource, id) is already in the held set. The synthetic backend reads are skipped; only the volume write locks are taken:
Subquery{upsert, volume, "volume1"}:
take write lock on volume1; call setResults()
volume1 exists → deep copy → nestedResults[0].Volume.Read
set nestedResults[0].Volume.Upsert
Subquery{read, backend, "backend1"}:
skipped — backend1 write lock already held
Subquery{upsert, volume, "volume2"}:
take write lock on volume2; populate nestedResults[1]
Subquery{upsert, volume, "volume3"}:
take write lock on volume3; populate nestedResults[2]
Lock acquisition order for new locks only:
1. volume1 (upsert, write)
2. volume2 (upsert, write)
3. volume3 (upsert, write)
The updated context records the expanded held set and a new lastLockHeld:
lockContext{
qs: {
(backend, "backend1") → write,
(volume, "volume1") → write,
(volume, "volume2") → write,
(volume, "volume3") → write,
},
lastLock: Subquery{upsert, volume, "volume3"},
}
Step B5: Use nested results
Each nestedResults[i] corresponds to one UpsertVolume query. The caller mutates the deep copy in Volume.Read and commits through Volume.Upsert. Because the backend write lock is still held from phase A, no other goroutine can delete backend1 while volumes are being updated.
Step B6: Unlock in two stages
defer nestedUnlocker() releases only the locks taken by NestedLock, in reverse order:
volume3 write lock released
volume2 write lock released
volume1 write lock released
defer unlocker() from the outer Lock() then releases:
backend1 write lock released
root read lock released
A third NestedLock(nestedCtx, ...) could still be called before the outer unlocker runs, but any new consistent lock must sort after volume3 upsert—for example an UpsertSnapshot on volume1. Attempting to NestedLock with an UpsertBackend would fail because backend sorts before volume3.
Interior mutability
Restricting access using locks would not improve performance if there was only one backend, which would effectively serialize all volume operations. We borrow the concept of "interior mutability" from Rust. In Rust, only one mutable reference to a variable may be held at a time, but types may relax these rules by coordinating access internally. Thus immutable references can make mutations. We use this concept to allow mutations while only holding read locks: backends are the biggest example. This allows multiple volumes to be created or modified on the same backend concurrently, with the backend holding locks for the brief times it needs to.
