Tech ONTAP Blogs
Tech ONTAP Blogs
Trident 25.06 adds support for Kubernetes Volume Group Snapshots (tested with v1beta1) with ONTAP-SAN volumes. This feature enables point-in-time, crash-consistent snapshots of multiple Persistent Volumes (PVs) by leveraging NetApp ONTAP’s Consistency Group (CG) capabilities.
Trident has long supported Kubernetes Volume Snapshots, allowing admins to snapshot individual Kubernetes PVs with ONTAP storage. However, this doesn't work for scenarios where point-in-time consistency is required between multiple snapshots. With standard Kubernetes Volume Snapshots, neither Kubernetes nor Trident can guarantee point-in-time consistency.
That’s where Kubernetes Volume Group Snapshots come in. This feature allows you to take point-in-time consistent snapshots of a group of PVs.
To support this, Kubernetes introduces three new Custom Resources (CRs):
VolumeGroupSnapshot
VolumeGroupSnapshotContent
VolumeGroupSnapshotClass
At the time of writing, Kubernetes 1.33 will support v1beta1 volume group snapshot CRDs. Trident will automatically detect if these CRDs are present to enable the relevant feature gates in our CSI sidecars. You will also need to enable the group snapshot feature gate in the "external-snapshotter" in Kubernetes.
Trident leverages ONTAP's Consistency Group (CG) feature to achieve point-in-time consistency for multiple PVs. Trident creates an ONTAP CG to take a group snapshot of a set of volumes. Once the group snapshot is complete, Trident removes the ONTAP CG. This innovation treats CGs in an ephemeral manner.
First, administrators will need to group volumes. This can be accomplished by applying a Kubernetes label consisting of their own custom key-value pair. A single Persistent Volume (PV) may be part of multiple groups, and those groups may be entirely disjoint or partially overlapping. For this example, we'll only map the PVC to a single group through the label: "group: myGroup". Labels may also be applied to existing PVCs that are already bound to a PV.
Example
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvc1 labels: group: myGroup spec: resources: requests: storage: 500Mi storageClassName: sc1
Then, define a VolumeGroupSnapshotClass
CR that specifies all fields necessary. Similar to the Kubernetes VolumeSnapshotClass
, a VolumeGroupSnapshotClass
CR may be reused for multiple volume group snapshot instances. This connects the group snapshot to the right CSI driver (like Trident). The "driver" field must reference Trident, or the group snapshot will not be created. Depending on your use cases, it may be wise to specify the proper deletion policy.
You do not have to create a unique VolumeGroupSnapshotClass
CR for every group snapshot.
Example
apiVersion: groupsnapshot.storage.k8s.io/v1beta1 kind: VolumeGroupSnapshotClass metadata: name: vgsclass1 driver: csi.trident.netapp.io deletionPolicy: Delete
Next, create a VolumeGroupSnapshot
CR.
Be sure to specify the aforementioned VolumeGroupSnapshotClass
CR and appropriate label.
Example
apiVersion: groupsnapshot.storage.k8s.io/v1beta1 kind: VolumeGroupSnapshot metadata: name: vgs1 namespace: trident spec: volumeGroupSnapshotClassName: vgsclass1 source: selector: matchLabels: group: myGroup
At this point, Trident receives a CSI request to create a group snapshot of the selected volumes.
This process creates several resources:
VolumeGroupSnapshot
and VolumeGroupSnapshotContent
custom resource (CR)VolumeSnapshot
and VolumeSnapshotContent
CR for each volume in the groupTridentGroupSnapshot
CRTridentSnapshot
CR for each individual snapshot in the group
In this concrete example, I've created a group snapshot for a group of two volumes.
Let's walk through each resource, its purpose, and provide an example.
Let's take a closer look at the following resources:
VolumeGroupSnapshot
CRVolumeGroupSnapshotContent
CRVolumeSnapshot
CRsVolumeSnapshotContent
CRs
This resource represents the request to create a group snapshot.
Example
$ kubectl get volumegroupsnapshot -n trident -o yaml apiVersion: v1 items: - apiVersion: groupsnapshot.storage.k8s.io/v1beta1 kind: VolumeGroupSnapshot metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"groupsnapshot.storage.k8s.io/v1beta1","kind":"VolumeGroupSnapshot","metadata":{"annotations":{},"name":"vgs1","namespace":"trident"},"spec":{"source":{"selector":{"matchLabels":{"group":"myGroup"}}},"volumeGroupSnapshotClassName":"csi-group-snap-class"}} creationTimestamp: "2025-06-26T18:32:42Z" finalizers: - groupsnapshot.storage.kubernetes.io/volumegroupsnapshot-bound-protection generation: 1 name: vgs1 namespace: trident resourceVersion: "200373" uid: 411d2111-89ea-4129-a454-015d1fd8ae85 spec: source: selector: matchLabels: group: myGroup volumeGroupSnapshotClassName: csi-group-snap-class status: boundVolumeGroupSnapshotContentName: groupsnapcontent-411d2111-89ea-4129-a454-015d1fd8ae85 creationTime: "2025-06-26T18:01:21Z" readyToUse: true
This resource represents the fulfillment of the VolumeGroupSnapshot
request.
Example
$ kubectl get volumegroupsnapshotcontent -n trident -o yaml apiVersion: v1 items: - apiVersion: groupsnapshot.storage.k8s.io/v1beta1 kind: VolumeGroupSnapshotContent metadata: creationTimestamp: "2025-06-26T18:32:42Z" finalizers: - groupsnapshot.storage.kubernetes.io/volumegroupsnapshotcontent-bound-protection generation: 1 name: groupsnapcontent-411d2111-89ea-4129-a454-015d1fd8ae85 resourceVersion: "200347" uid: df01c3d7-8e91-4b0f-b992-eddc0a99e6ab spec: deletionPolicy: Delete driver: csi.trident.netapp.io source: volumeHandles: - pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500 - pvc-cf598c63-5fc6-42af-83b0-f490180d23aa volumeGroupSnapshotClassName: csi-group-snap-class volumeGroupSnapshotRef: apiVersion: groupsnapshot.storage.k8s.io/v1beta1 kind: VolumeGroupSnapshot name: vgs1 namespace: trident resourceVersion: "200331" uid: 411d2111-89ea-4129-a454-015d1fd8ae85 status: creationTime: "2025-06-26T18:01:21Z" readyToUse: true volumeGroupSnapshotHandle: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 volumeSnapshotHandlePairList: - snapshotHandle: pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 volumeHandle: pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500 - snapshotHandle: pvc-cf598c63-5fc6-42af-83b0-f490180d23aa/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 volumeHandle: pvc-cf598c63-5fc6-42af-83b0-f490180d23aa
These resources represent the request for a volume snapshot. For group snapshots, these will have a new "ownerReferences" entry that points to their parent VolumeGroupSnapshot
as the owner.
Read more about owner references.
Example
$ kubectl get volumesnapshot -n trident -o yaml apiVersion: v1 items: - apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: creationTimestamp: "2025-06-26T18:32:47Z" finalizers: - snapshot.storage.kubernetes.io/volumesnapshot-in-group-protection - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection generation: 1 name: snapshot-08f47b907f2f006f5d890a80afe11fbb977f63554498833f217435d95e435045 namespace: trident ownerReferences: - apiVersion: groupsnapshot.storage.k8s.io/v1beta1 kind: VolumeGroupSnapshot name: vgs1 uid: 411d2111-89ea-4129-a454-015d1fd8ae85 resourceVersion: "200378" uid: 2345f17a-5e8a-4ff6-a77d-47c2cc7b34d9 spec: source: persistentVolumeClaimName: pvc2-2 status: boundVolumeSnapshotContentName: snapcontent-08f47b907f2f006f5d890a80afe11fbb977f63554498833f217435d95e435045 creationTime: "2025-06-26T18:01:21Z" readyToUse: true restoreSize: 100Mi volumeGroupSnapshotName: vgs1 - apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: creationTimestamp: "2025-06-26T18:32:47Z" finalizers: - snapshot.storage.kubernetes.io/volumesnapshot-in-group-protection - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection generation: 1 name: snapshot-258720deb7c66ef2f4198286be15515d5908186da7d1888c08af3800e3bfed1e namespace: trident ownerReferences: - apiVersion: groupsnapshot.storage.k8s.io/v1beta1 kind: VolumeGroupSnapshot name: vgs1 uid: 411d2111-89ea-4129-a454-015d1fd8ae85 resourceVersion: "200370" uid: 0637880a-6134-428d-a8fc-67214c726e99 spec: source: persistentVolumeClaimName: pvc2 status: boundVolumeSnapshotContentName: snapcontent-258720deb7c66ef2f4198286be15515d5908186da7d1888c08af3800e3bfed1e creationTime: "2025-06-26T18:01:21Z" readyToUse: true restoreSize: 100Mi volumeGroupSnapshotName: vgs1 kind: List metadata: resourceVersion: ""
These resources represent the result of creating the requested VolumeSnapshot
.
Example
$ kubectl get volumesnapshotcontent -n trident -o yaml apiVersion: v1 items: - apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: annotations: groupsnapshot.storage.k8s.io/volumeGroupSnapshotHandle: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 creationTimestamp: "2025-06-26T18:32:47Z" finalizers: - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection generation: 2 name: snapcontent-08f47b907f2f006f5d890a80afe11fbb977f63554498833f217435d95e435045 resourceVersion: "200376" uid: 8e161f10-40f9-478a-a2f1-04a2d70fa8a4 spec: deletionPolicy: Delete driver: csi.trident.netapp.io source: volumeHandle: pvc-cf598c63-5fc6-42af-83b0-f490180d23aa sourceVolumeMode: Filesystem volumeSnapshotRef: kind: VolumeSnapshot name: snapshot-08f47b907f2f006f5d890a80afe11fbb977f63554498833f217435d95e435045 namespace: trident uid: 2345f17a-5e8a-4ff6-a77d-47c2cc7b34d9 status: creationTime: 1750960881000000000 readyToUse: true restoreSize: 104857600 snapshotHandle: pvc-cf598c63-5fc6-42af-83b0-f490180d23aa/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 volumeGroupSnapshotHandle: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 - apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: annotations: groupsnapshot.storage.k8s.io/volumeGroupSnapshotHandle: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 creationTimestamp: "2025-06-26T18:32:47Z" finalizers: - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection generation: 2 name: snapcontent-258720deb7c66ef2f4198286be15515d5908186da7d1888c08af3800e3bfed1e resourceVersion: "200362" uid: 5631b489-1bdf-4d47-a38d-153a08b944cf spec: deletionPolicy: Delete driver: csi.trident.netapp.io source: volumeHandle: pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500 sourceVolumeMode: Filesystem volumeSnapshotRef: kind: VolumeSnapshot name: snapshot-258720deb7c66ef2f4198286be15515d5908186da7d1888c08af3800e3bfed1e namespace: trident uid: 0637880a-6134-428d-a8fc-67214c726e99 status: creationTime: 1750960881000000000 readyToUse: true restoreSize: 104857600 snapshotHandle: pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 volumeGroupSnapshotHandle: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 kind: List metadata: resourceVersion: ""
Trident adds its own custom resources when creating a volume group snapshot. These Trident CRs are not meant to be manipulated by users directly but may be viewed via kubectl or tridentctl.
As part of a group snapshot, Trident creates the following CRs:
TridentGroupSnapshot
CRTridentSnapshot
CR
This stores useful metadata about the group snapshot for Trident. Creating a group snapshot in Kubernetes should result in a 1-1 relationship between the TridentGroupSnapshot
and Kubernetes VolumeGroupSnapshotContent
.
Example
$ kubectl get tridentgroupsnapshot -n trident -o yaml apiVersion: v1 items: - apiVersion: trident.netapp.io/v1 dateCreated: "2025-06-26T18:01:21.000Z" kind: TridentGroupSnapshot metadata: creationTimestamp: "2025-06-26T18:32:47Z" finalizers: - trident.netapp.io generation: 1 name: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 namespace: trident resourceVersion: "200340" uid: e905c9b1-73c4-48b2-92e6-ecedaaadbd39 snapshotIDs: - pvc-cf598c63-5fc6-42af-83b0-f490180d23aa/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 - pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 spec: internalName: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 name: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 version: "1" volumeNames: - pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500 - pvc-cf598c63-5fc6-42af-83b0-f490180d23aa kind: List metadata: resourceVersion: ""
This stores useful metadata about the individual snapshot for Trident. Creating a group snapshot in Kubernetes should result in a 1-1 relationship between Trident Snapshots and Kubernetes Volume Snapshot Contents.
Example
$ kubectl get tridentsnapshot -n trident -o yaml apiVersion: v1 items: - apiVersion: trident.netapp.io/v1 dateCreated: "2025-06-26T18:01:21.000Z" kind: TridentSnapshot metadata: creationTimestamp: "2025-06-26T18:32:47Z" finalizers: - trident.netapp.io generation: 1 name: pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500-snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 namespace: trident resourceVersion: "200342" uid: 6e4e197e-2f13-4705-98f8-64df0a0fc475 size: 104857600 spec: groupSnapshotName: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 importNotManaged: false internalName: snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 name: snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 volumeInternalName: websterj_pvc_373e0043_ca12_42ab_bcd9_4360f0ddc500 volumeName: pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500 state: online - apiVersion: trident.netapp.io/v1 dateCreated: "2025-06-26T18:01:21.000Z" kind: TridentSnapshot metadata: creationTimestamp: "2025-06-26T18:32:47Z" finalizers: - trident.netapp.io generation: 1 name: pvc-cf598c63-5fc6-42af-83b0-f490180d23aa-snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 namespace: trident resourceVersion: "200341" uid: 881bbe33-1553-485a-a0c7-f79142918bdd size: 104857600 spec: groupSnapshotName: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 importNotManaged: false internalName: snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 name: snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 volumeInternalName: websterj_pvc_cf598c63_5fc6_42af_83b0_f490180d23aa volumeName: pvc-cf598c63-5fc6-42af-83b0-f490180d23aa state: online kind: List metadata: resourceVersion: ""
Trident's CLI, tridentctl, may be used to view group snapshots via the new "groupsnapshot" subcommand.
Example
$ tridentctl get groupsnapshot -h -n trident Get one or more group snapshots from Trident Usage: tridentctl get groupsnapshot [<group snapshot name>...] [flags] Aliases: groupsnapshot, gs, gsnap, groupsnapshots Flags: -h, --help help for groupsnapshot --max-snapshots int Set the maximum number of snapshots (inclusive) to display when displaying grouped snapshots. Setting this to 0 will show all snapshots in the group. (default 3) Global Flags: -d, --debug Set the log level to debug -k, --kubeconfig string Kubernetes config path --log-level string Log level (trace, debug, warn, info, error, fatal (default "info") -n, --namespace string Namespace of Trident deployment -o, --output string Output format. One of json|yaml|name|wide|ps (default) -s, --server string Address/port of Trident REST interface (127.0.0.1 or [::1] only)
You may also specify an output format: "-o json|yaml|wide
":
$ tridentctl get groupsnapshot -n trident +----------------------------------------------------+ | NAME | +----------------------------------------------------+ | groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | +----------------------------------------------------+ $ tridentctl get groupsnapshot -n trident -o wide +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+ | NAME | CREATED | SNAPSHOTS | SNAPSHOT COUNT | +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+ | groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | 2025-06-26T18:01:21.000Z | pvc-cf598c63-5fc6-42af-83b0-f490180d23aa/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | 2 | | | | pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | | +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+ $ tridentctl get groupsnapshot -n trident -o json { "items": [ { "version": "1", "name": "groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85", "internalName": "groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85", "volumeNames": [ "pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500", "pvc-cf598c63-5fc6-42af-83b0-f490180d23aa" ], "snapshotIDs": [ "pvc-cf598c63-5fc6-42af-83b0-f490180d23aa/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85", "pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85" ], "dateCreated": "2025-06-26T18:01:21.000Z" } ] } $ tridentctl get groupsnapshot -n trident -o yaml items: - dateCreated: "2025-06-26T18:01:21.000Z" internalName: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 name: groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 snapshotIDs: - pvc-cf598c63-5fc6-42af-83b0-f490180d23aa/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 - pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 version: "1" volumeNames: - pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500 - pvc-cf598c63-5fc6-42af-83b0-f490180d23aa
The wide
format option will display 3 additional columns:
Because there may be a large number of snapshots in a single group snapshot, the number of snapshot handles is truncated to 3 but will display more should you supply the --max-snapshots
flag. Specifying "0" will display all snapshots in the group.
$ tridentctl get groupsnapshot -o wide -n trident --max-snapshots=0 +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+ | NAME | CREATED | SNAPSHOTS | SNAPSHOT COUNT | +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+ | groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | 2025-06-26T18:01:21.000Z | pvc-cf598c63-5fc6-42af-83b0-f490180d23aa/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | 2 | | | | pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | | +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+ # If you specify a number less than the total count, but greater than 0, the list of constituent snapshot handles is truncated. $ tridentctl get groupsnapshot -o wide -n trident --max-snapshots=1 +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+ | NAME | CREATED | SNAPSHOTS | SNAPSHOT COUNT | +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+ | groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | 2025-06-26T18:01:21.000Z | pvc-cf598c63-5fc6-42af-83b0-f490180d23aa/snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | 2 | | | | ... (+1 more) | | +----------------------------------------------------+--------------------------+----------------------------------------------------------------------------------------+----------------+
Additionally, Trident Snapshots taken as a group will have an extra field ("GroupSnapshot") identifying its Trident Group Snapshot parent:
$ tridentctl get snapshot -o wide -n trident +-----------------------------------------------+------------------------------------------+--------------------------+---------+-------+---------+----------------------------------------------------+ | NAME | VOLUME | CREATED | SIZE | STATE | MANAGED | GROUPSNAPSHOT | +-----------------------------------------------+------------------------------------------+--------------------------+---------+-------+---------+----------------------------------------------------+ | snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | pvc-373e0043-ca12-42ab-bcd9-4360f0ddc500 | 2025-06-26T18:01:21.000Z | 100 MiB | | true | groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | | snapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | pvc-cf598c63-5fc6-42af-83b0-f490180d23aa | 2025-06-26T18:01:21.000Z | 100 MiB | | true | groupsnapshot-411d2111-89ea-4129-a454-015d1fd8ae85 | +-----------------------------------------------+------------------------------------------+--------------------------+---------+-------+---------+----------------------------------------------------+
tridentctl
while their parent TridentGroupSnapshot
CR exists.TridentGroupSnapshot
with tridentctl
is not recommended, as it may cause inconsistencies between Kubernetes and Trident.VolumeGroupSnapshot
resource.VolumeGroupSnapshot
also deletes all constituent snapshots, the TridentGroupSnapshot
, and related TridentSnapshot
resources.
In future releases, the Trident team plans to broaden support for volume group snapshots to the ONTAP-SAN-Economy, ONTAP-NAS, and ONTAP-SAN (NVMe) drivers. Try out our new volume group snapshot feature with ONTAP-SAN volumes and let us know what you think.
Thanks for reading!