Best Of

From hours to minutes: Trident 26.06 controller concurrency at 1000 volume scale

We first introduced the Trident concurrent core as a tech preview feature in Trident release 25.06. With traditional Trident, all operations in the k8s cluster were effectively serialized with a global lock. With concurrent core enabled, Trident can safely perform many provisioning operations concurrently, drastically increasing the performance at scale.

With trident 26.06, we are proud to offer General Availability (GA) support for even more drivers, including ONTAP-NAS-Economy, ONTAP-SAN-Economy and Azure NetApp Files (ANF).

Please view our controller scalability documentation to learn how to enable the controller scalability feature and see the full list of supported drivers.

The Big Picture

Average operational rate increase, aggregated across all benchmarks, all operations

Trident Backend	Average speed increase with concurrency
ONTAP NAS	707% (~8.1× throughput)
ONTAP NAS Economy	472% (~5.7× throughput)
ONTAP SAN (iSCSI)	449% (~5.5× throughput)
ONTAP SAN Economy	285% (~3.9× throughput)

How we measured

Measuring the timing of each operation uses the same pattern. For each object, we submit a large batch and then wait for all of them to reach their terminal state.

Start the clock
Submit a large batch of new objects. Example: Run kubectl apply -f on a YAML file that defines 1000 individual PVCs.
Poll until all are done. Example: Poll until all PVCs are in the Bound state.
Stop the clock

Test Environment

Kubernetes cluster

Server version	v1.34.9
Total K8s Nodes	100
Per Node resources:	56 CPU, 250GB RAM

Trident CSI

Trident version	v26.06.0

ONTAP

Property	Value
ONTAP Version	9.18.1P2
Hardware	AFF A70

Note on snapshot performance:

https://github.com/kubernetes-csi/external-snapshotter/issues/1344

The k8s snapshot controller deployment default settings throttle Trident’s concurrent core snapshot performance. To achieve maximum snapshot performance with concurrent core enabled in trident, modify the QPS/Burst values on the snapshot controller deployment in the k8s cluster. Without this setting, the concurrent core snapshot performance will not show an improvement.

k edit deployment -n kube-system snapshot-controller
        app.kubernetes.io/name: snapshot-controller
    spec:
      containers:
      - args:
        - --v=5
        - --leader-election=true
        - --worker-threads=25
        - --kube-api-qps=75           # Modified
        - --kube-api-burst=150        # Modified

100 concurrent volume operations: concurrency on vs off

Trident CSI benchmark at 100 volumes (100 vols, 25 pods, 4 PVCs/pod, 1 Gi, Trident v26.06.0).

ONTAP NAS

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)
Create Volume	6m 57s · 0.24 vol/s	41s · 2.44 vol/s	10.2×
Publish (Attach)	2m 20s · 0.71 vol/s	21s · 4.76 vol/s	6.7×
Unpublish (Detach)	2m 20s · 0.71 vol/s	26s · 3.85 vol/s	5.4×
Create Snapshot	3m 03s · 0.55 vol/s	16s · 6.25 vol/s	11.4×
Clone from Snapshot	7m 40s · 0.22 vol/s	42s · 2.38 vol/s	11.0×
Delete Clone	2m 07s · 0.79 vol/s	32s · 3.12 vol/s	4.0×
Delete Snapshot	2m 04s · 0.81 vol/s	16s · 6.25 vol/s	7.7×
Delete Volume	2m 04s · 0.81 vol/s	26s · 3.85 vol/s	4.8×
Sum of step times	28m 35s	3m 40s	~7.8× overall

ONTAP NAS economy

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)
Create Volume	11m 33s · 0.14 vol/s	1m 29s · 1.12 vol/s	7.8×
Publish (Attach)	4m 10s · 0.40 vol/s	2m 05s · 0.80 vol/s	2.0×
Unpublish (Detach)	3m 13s · 0.52 vol/s	32s · 3.12 vol/s	6.0×
Delete Volume	1m 56s · 0.86 vol/s	21s · 4.76 vol/s	5.5×
Sum of step times	20m 52s	4m 27s	~4.7× overall

ONTAP SAN

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)
Create Volume	2m 36s · 0.64 vol/s	58s · 1.72 vol/s	2.7×
Publish (Attach)	1m 23s · 1.20 vol/s	31s · 3.23 vol/s	2.7×
Unpublish (Detach)	22s · 4.55 vol/s	22s · 4.55 vol/s	1.0×
Create Snapshot	3m 02s · 0.55 vol/s	16s · 6.25 vol/s	11.4×
Clone from Snapshot	3m 58s · 0.42 vol/s	32s · 3.12 vol/s	7.4×
Delete Clone	2m 01s · 0.83 vol/s	43s · 2.33 vol/s	2.8×
Delete Snapshot	2m 04s · 0.81 vol/s	15s · 6.67 vol/s	8.3×
Delete Volume	2m 05s · 0.80 vol/s	21s · 4.76 vol/s	6.0×
Sum of step times	17m 31s	3m 58s	~4.4× overall

ONTAP SAN economy

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)
Create Volume	3m 18s · 0.51 vol/s	58s · 1.72 vol/s	3.4×
Publish (Attach)	1m 23s · 1.20 vol/s	31s · 3.23 vol/s	2.7×
Unpublish (Detach)	27s · 3.70 vol/s	22s · 4.55 vol/s	1.2×
Delete Volume	4m 33s · 0.37 vol/s	1m 13s · 1.37 vol/s	3.7×
Sum of step times	9m 41s	3m 04s	~3.2× overall

1000 concurrent volume operations: concurrency on vs off

Trident CSI benchmark at 1000 volumes (1000 vols, 250 pods, 4 PVCs/pod, 1 Gi, Trident v26.06.0).

Increasing the batch size by a factor of ten, Trident concurrent core maintains and even exceeds the operational rate seen with batch size of 100.

ONTAP NAS

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)
Create Volume	57m 31s · 0.29 vol/s	10m 30s · 1.59 vol/s	5.5×
Publish (Attach)	23m 04s · 0.72 vol/s	2m 47s · 5.99 vol/s	8.3×
Unpublish (Detach)	23m 40s · 0.70 vol/s	2m 36s · 6.41 vol/s	9.1×
Create Snapshot	30m 05s · 0.55 vol/s	2m 39s · 6.29 vol/s	11.4×
Clone from Snapshot	1h 41m · 0.17 vol/s	6m 41s · 2.49 vol/s	15.1×
Delete Clone	24m 12s · 0.69 vol/s	5m 02s · 3.31 vol/s	4.8×
Delete Snapshot	20m 44s · 0.80 vol/s	2m 04s · 8.06 vol/s	10.0×
Delete Volume	20m 36s · 0.81 vol/s	5m 18s · 3.14 vol/s	3.9×
Sum of step times	5h 00m 52s	37m 37s	~8.0× overall

ONTAP NAS-economy

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)
Create Volume	1h 11m 27s · 0.23 vol/s	11m 50s · 1.41 vol/s	6.0×
Publish (Attach)	43m 15s · 0.39 vol/s	19m 01s · 0.88 vol/s	2.3×
Unpublish (Detach)	23m 06s · 0.72 vol/s	2m 35s · 6.45 vol/s	8.9×
Delete Volume	19m 25s · 0.86 vol/s	2m 43s · 6.13 vol/s	7.1×
Sum of step times	2h 37m 13s	36m 09s	~4.3× overall

ONTAP SAN

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)
Create Volume	27m 28s · 0.61 vol/s	7m 57s · 2.10 vol/s	3.5×
Publish (Attach)	12m 57s · 1.29 vol/s	2m 35s · 6.45 vol/s	5.0×
Unpublish (Detach)	3m 53s · 4.29 vol/s	2m 37s · 6.37 vol/s	1.5×
Create Snapshot	30m 03s · 0.55 vol/s	2m 44s · 6.10 vol/s	11.0×
Clone from Snapshot	56m 24s · 0.30 vol/s	13m 25s · 1.24 vol/s	4.2×
Delete Clone	24m 14s · 0.69 vol/s	4m 17s · 3.89 vol/s	5.7×
Delete Snapshot	20m 38s · 0.81 vol/s	2m 03s · 8.13 vol/s	10.1×
Delete Volume	20m 38s · 0.81 vol/s	4m 23s · 3.80 vol/s	4.7×
Sum of step times	3h 16m 15s	40m 01s	~4.9× overall

ONTAP SAN-economy

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)

Operation	Concurrency OFF	Concurrency ON	ON vs OFF (throughput)
Create Volume	44m 49s · 0.37 vol/s	9m 41s · 1.72 vol/s	4.6×
Publish (Attach)	13m 47s · 1.21 vol/s	2m 53s · 5.78 vol/s	4.8×
Unpublish (Detach)	4m 55s · 3.39 vol/s	2m 37s · 6.37 vol/s	1.9×
Delete Volume	46m 55s · 0.36 vol/s	5m 33s · 3.00 vol/s	8.5×
Sum of step times	1h 50m 26s	20m 44s	~5.3× overall

Try it yourself!

Trident_csi_benchmark.zip

We used this automated BASH script to capture these numbers. The script is supplied here so that you can see exactly how the measurements were performed.

If you want to try it yourself, simply run the script as shown below. A new directory will be created containing a JSON report of all step timings and the manifests of the resources created in Kubernetes. Note that the script automatically cleans up the resources created for the benchmark.

Example command line:

./Trident_csi_benchmark.sh -n 100 -s my-storage-class -N my-benchmark-namespace

kjackson

Community Migration: Complete

The new NetApp Community is LIVE

I'm happy to welcome you to the new NetApp Community! Our migration seems to have gone smoothly and the site is open again for business.

We're still tidying up a few things, but if something seems out of place let me know. Account problems or missing content are key.

Report issues here: https://community.netapp.com/community/categories/new-to-netapp
Or on our Discord: https://discord.gg/NetApp (@ me in the Digital-Svcs-Help channel)
You can also email: community@netapp.com

Thanks for your patience during our read-only period.

Cheers,
-Drew

Previous message:

I'm excited to share the news that we'll be migrating to a new community platform!
To prepare for the migration, the current site will enter Read Only mode at approximately 12:00 PM Eastern, Thursday July 23, 2026. During this time, all existing content will remain available for viewing, but posting, replying, and other content updates will be temporarily disabled.
Our new Community is planned to launch on July 30, bringing a modernized experience while preserving the valuable knowledge and discussions that make the Community a trusted resource.
We intend to migrate nearly everything. Your account, posts, comments, and badges will transfer to the new platform. No action is required on your part. This change comes with new features and improvements to the category structure to make the site easier to navigate.
We’re also taking the opportunity to do some housekeeping. To help keep the new platform organized and concise:
Content created before January 1, 2015 will not be migrated.
Content created between January 1, 2015 and December 31, 2018 will be archived internally.
Inactive user accounts may not be migrated if they have no associated content.
When the new Community launches, you'll still sign in using your NetApp Single Sign-On. Your content will be here, your subscriptions will be here. We expect existing links to continue working whenever possible.
If you need community assistance during the migration period, you can find us on Discord. As always, the Knowledge Base, the NetApp Support Site, and our engineers remain available 24/7.
As with any major platform migration, you may notice small issues during the first few days after launch. We appreciate your patience and welcome your feedback as we continue to improve the experience.
If something doesn't seem right after the migration, please let us know at: community@netapp.com
Thank you for your patience and support as we work to deliver the next generation of the NetApp Community.

Cheers,

-Drew

Drew_C