We first introduced the Trident concurrent core as a tech preview feature in Trident release 25.06. With traditional Trident, all operations in the k8s cluster were effectively serialized with a global lock. With concurrent core enabled, Trident can safely perform many provisioning operations concurrently, drastically increasing the performance at scale.
With trident 26.06, we are proud to offer General Availability (GA) support for even more drivers, including ONTAP-NAS-Economy, ONTAP-SAN-Economy and Azure NetApp Files (ANF).
Please view our controller scalability documentation to learn how to enable the controller scalability feature and see the full list of supported drivers.
The Big Picture
Average operational rate increase, aggregated across all benchmarks, all operations
Trident Backend | Average speed increase with concurrency |
|---|
ONTAP NAS | 707% (~8.1× throughput) |
ONTAP NAS Economy | 472% (~5.7× throughput) |
ONTAP SAN (iSCSI) | 449% (~5.5× throughput) |
ONTAP SAN Economy | 285% (~3.9× throughput) |
How we measured
Measuring the timing of each operation uses the same pattern. For each object, we submit a large batch and then wait for all of them to reach their terminal state.
- Start the clock
- Submit a large batch of new objects. Example: Run
kubectl apply -f on a YAML file that defines 1000 individual PVCs. - Poll until all are done. Example: Poll until all PVCs are in the Bound state.
- Stop the clock
Test Environment
Kubernetes cluster
Server version | v1.34.9 |
|---|
Total K8s Nodes | 100 |
|---|
Per Node resources: | 56 CPU, 250GB RAM |
|---|
Trident CSI
ONTAP
Property | Value |
|---|
ONTAP Version | 9.18.1P2 |
Hardware | AFF A70 |
Note on snapshot performance:
https://github.com/kubernetes-csi/external-snapshotter/issues/1344
The k8s snapshot controller deployment default settings throttle Trident’s concurrent core snapshot performance. To achieve maximum snapshot performance with concurrent core enabled in trident, modify the QPS/Burst values on the snapshot controller deployment in the k8s cluster. Without this setting, the concurrent core snapshot performance will not show an improvement.
k edit deployment -n kube-system snapshot-controller
app.kubernetes.io/name: snapshot-controller
spec:
containers:
- args:
- --v=5
- --leader-election=true
- --worker-threads=25
- --kube-api-qps=75 # Modified
- --kube-api-burst=150 # Modified
100 concurrent volume operations: concurrency on vs off
Trident CSI benchmark at 100 volumes (100 vols, 25 pods, 4 PVCs/pod, 1 Gi, Trident v26.06.0).
ONTAP NAS
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Create Volume | 6m 57s · 0.24 vol/s | 41s · 2.44 vol/s | 10.2× |
Publish (Attach) | 2m 20s · 0.71 vol/s | 21s · 4.76 vol/s | 6.7× |
Unpublish (Detach) | 2m 20s · 0.71 vol/s | 26s · 3.85 vol/s | 5.4× |
Create Snapshot | 3m 03s · 0.55 vol/s | 16s · 6.25 vol/s | 11.4× |
Clone from Snapshot | 7m 40s · 0.22 vol/s | 42s · 2.38 vol/s | 11.0× |
Delete Clone | 2m 07s · 0.79 vol/s | 32s · 3.12 vol/s | 4.0× |
Delete Snapshot | 2m 04s · 0.81 vol/s | 16s · 6.25 vol/s | 7.7× |
Delete Volume | 2m 04s · 0.81 vol/s | 26s · 3.85 vol/s | 4.8× |
Sum of step times | 28m 35s | 3m 40s | ~7.8× overall |
ONTAP NAS economy
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Create Volume | 11m 33s · 0.14 vol/s | 1m 29s · 1.12 vol/s | 7.8× |
Publish (Attach) | 4m 10s · 0.40 vol/s | 2m 05s · 0.80 vol/s | 2.0× |
Unpublish (Detach) | 3m 13s · 0.52 vol/s | 32s · 3.12 vol/s | 6.0× |
Delete Volume | 1m 56s · 0.86 vol/s | 21s · 4.76 vol/s | 5.5× |
Sum of step times | 20m 52s | 4m 27s | ~4.7× overall |
ONTAP SAN
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Create Volume | 2m 36s · 0.64 vol/s | 58s · 1.72 vol/s | 2.7× |
Publish (Attach) | 1m 23s · 1.20 vol/s | 31s · 3.23 vol/s | 2.7× |
Unpublish (Detach) | 22s · 4.55 vol/s | 22s · 4.55 vol/s | 1.0× |
Create Snapshot | 3m 02s · 0.55 vol/s | 16s · 6.25 vol/s | 11.4× |
Clone from Snapshot | 3m 58s · 0.42 vol/s | 32s · 3.12 vol/s | 7.4× |
Delete Clone | 2m 01s · 0.83 vol/s | 43s · 2.33 vol/s | 2.8× |
Delete Snapshot | 2m 04s · 0.81 vol/s | 15s · 6.67 vol/s | 8.3× |
Delete Volume | 2m 05s · 0.80 vol/s | 21s · 4.76 vol/s | 6.0× |
Sum of step times | 17m 31s | 3m 58s | ~4.4× overall |
ONTAP SAN economy
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Create Volume | 3m 18s · 0.51 vol/s | 58s · 1.72 vol/s | 3.4× |
Publish (Attach) | 1m 23s · 1.20 vol/s | 31s · 3.23 vol/s | 2.7× |
Unpublish (Detach) | 27s · 3.70 vol/s | 22s · 4.55 vol/s | 1.2× |
Delete Volume | 4m 33s · 0.37 vol/s | 1m 13s · 1.37 vol/s | 3.7× |
Sum of step times | 9m 41s | 3m 04s | ~3.2× overall |
1000 concurrent volume operations: concurrency on vs off
Trident CSI benchmark at 1000 volumes (1000 vols, 250 pods, 4 PVCs/pod, 1 Gi, Trident v26.06.0).
Increasing the batch size by a factor of ten, Trident concurrent core maintains and even exceeds the operational rate seen with batch size of 100.
ONTAP NAS
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Create Volume | 57m 31s · 0.29 vol/s | 10m 30s · 1.59 vol/s | 5.5× |
Publish (Attach) | 23m 04s · 0.72 vol/s | 2m 47s · 5.99 vol/s | 8.3× |
Unpublish (Detach) | 23m 40s · 0.70 vol/s | 2m 36s · 6.41 vol/s | 9.1× |
Create Snapshot | 30m 05s · 0.55 vol/s | 2m 39s · 6.29 vol/s | 11.4× |
Clone from Snapshot | 1h 41m · 0.17 vol/s | 6m 41s · 2.49 vol/s | 15.1× |
Delete Clone | 24m 12s · 0.69 vol/s | 5m 02s · 3.31 vol/s | 4.8× |
Delete Snapshot | 20m 44s · 0.80 vol/s | 2m 04s · 8.06 vol/s | 10.0× |
Delete Volume | 20m 36s · 0.81 vol/s | 5m 18s · 3.14 vol/s | 3.9× |
Sum of step times | 5h 00m 52s | 37m 37s | ~8.0× overall |
ONTAP NAS-economy
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Create Volume | 1h 11m 27s · 0.23 vol/s | 11m 50s · 1.41 vol/s | 6.0× |
Publish (Attach) | 43m 15s · 0.39 vol/s | 19m 01s · 0.88 vol/s | 2.3× |
Unpublish (Detach) | 23m 06s · 0.72 vol/s | 2m 35s · 6.45 vol/s | 8.9× |
Delete Volume | 19m 25s · 0.86 vol/s | 2m 43s · 6.13 vol/s | 7.1× |
Sum of step times | 2h 37m 13s | 36m 09s | ~4.3× overall |
ONTAP SAN
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Create Volume | 27m 28s · 0.61 vol/s | 7m 57s · 2.10 vol/s | 3.5× |
Publish (Attach) | 12m 57s · 1.29 vol/s | 2m 35s · 6.45 vol/s | 5.0× |
Unpublish (Detach) | 3m 53s · 4.29 vol/s | 2m 37s · 6.37 vol/s | 1.5× |
Create Snapshot | 30m 03s · 0.55 vol/s | 2m 44s · 6.10 vol/s | 11.0× |
Clone from Snapshot | 56m 24s · 0.30 vol/s | 13m 25s · 1.24 vol/s | 4.2× |
Delete Clone | 24m 14s · 0.69 vol/s | 4m 17s · 3.89 vol/s | 5.7× |
Delete Snapshot | 20m 38s · 0.81 vol/s | 2m 03s · 8.13 vol/s | 10.1× |
Delete Volume | 20m 38s · 0.81 vol/s | 4m 23s · 3.80 vol/s | 4.7× |
Sum of step times | 3h 16m 15s | 40m 01s | ~4.9× overall |
ONTAP SAN-economy
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Operation | Concurrency OFF | Concurrency ON | ON vs OFF (throughput) |
|---|
Create Volume | 44m 49s · 0.37 vol/s | 9m 41s · 1.72 vol/s | 4.6× |
Publish (Attach) | 13m 47s · 1.21 vol/s | 2m 53s · 5.78 vol/s | 4.8× |
Unpublish (Detach) | 4m 55s · 3.39 vol/s | 2m 37s · 6.37 vol/s | 1.9× |
Delete Volume | 46m 55s · 0.36 vol/s | 5m 33s · 3.00 vol/s | 8.5× |
Sum of step times | 1h 50m 26s | 20m 44s | ~5.3× overall |
Try it yourself!
We used this automated BASH script to capture these numbers. The script is supplied here so that you can see exactly how the measurements were performed.
If you want to try it yourself, simply run the script as shown below. A new directory will be created containing a JSON report of all step timings and the manifests of the resources created in Kubernetes. Note that the script automatically cleans up the resources created for the benchmark.
Example command line:
./Trident_csi_benchmark.sh -n 100 -s my-storage-class -N my-benchmark-namespace
