Welcome to my blog! Today, we're diving into the essential best practices for optimizing your OpenShift Virtualization setup with Trident and NetApp storage. Let's start with three key guidelines you should follow:
1. Install Trident and set up Trident Storage Class as default before installing OpenShift Virtualization
Begin by installing Trident and setting up the Trident storage class as the default storage class for your virtualization workloads. This foundational step ensures seamless integration and optimal performance.
2. Set Up Trident VolumeSnapshotClass as Default
Create a Trident VolumeSnapshotClass and configure it as the default. This setup allows the CloneStrategy parameter in StorageProfile to be set to snapshot, leveraging NetApp FlexClone Technology for fast and efficient operations with OpenShift VMs.
Note: You can set up the default storage class and default VolumeSnapshotClass by editing the annotation as follows:
kubectl patch storageclass <storage-class-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl patch volumesnapshotclass <volumesnapshotclass-name> --type=merge -p '{"metadata":{"annotations":{"snapshot.storage.kubernetes.io/is-default-class":"true"}}}'
kubectl patch storageclass <storage-class-name> -p '{"metadata": {"annotations":{"storageclass.kubevirt.io/is-default-virt-class":"true"}}}'
kubectl patch volumesnapshotclass <volumesnapshotclass-name> --type=merge -p '{"metadata":{"annotations":{"snapshot.kubevirt.io/is-default-virt-class":"true"}}}'When creating VMs, ensure the root disk uses the same storage class as the one used for the golden image. Cloning a PVC is only possible if the target PVC matches the source's storage class. Starting with Trident 25.06.1, you'll encounter errors if this consistency isn't maintained.
You might be wondering, why start with best practices? Because in this blog, I’ll show you what happens when these practices are ignored. By the end, you'll see why adhering to these guidelines is crucial for harnessing the full potential of Trident and NetApp storage.
So, let's get started!
Key Concepts of StorageProfile
In the OpenShift Container Platform, a StorageProfile is a Kubernetes custom resource that provides recommended storage settings and behaviors for a specific StorageClass, especially in the context of OpenShift Virtualization and the Containerized Data Importer (CDI). These profiles are automatically created for each storage class when using OpenShift Virtualization, offering tailored storage settings that can be customized for optimal performance.
CDI, responsible for importing, uploading, and cloning VM disks, uses StorageProfiles to interface with various storage providers. If CDI doesn't natively recognize a storage provider, administrators can customize the StorageProfile to ensure successful volume allocation and operations.
In OpenShift Virtualization, StorageProfiles play a crucial role in optimizing and standardizing storage provisioning for virtual machines. They influence aspects such as access modes (e.g., ReadWriteMany), volume modes (e.g., Block), and boot source handling.
A StorageProfile allows administrators to define parameters and behaviors for a StorageClass, particularly when provisioning PersistentVolumeClaims (PVCs) for VMs or other CDI-managed workloads. These settings are applied unless explicitly configured in the DataVolume object requesting the storage.
One of the critical functions of StorageProfile is defining cloning strategies for a StorageClass. This determines how volumes are cloned (e.g., using snapshots, CSI clones, or host-assisted copies), optimizing performance and resource usage based on the capabilities of the underlying storage vendor.
Now, let’s dive in!
In the screenshot below, you can see that there is no StorageProfile object created for the StorageClass defined in the cluster. This is because I have not yet installed Openshift Virtualization in the cluster.
After installing OpenShift Virtualization, a storageprofile class is created for each storage class in the cluster. (In this example, I have thin-csi available as a storage class since this cluster is on VMware. But you could be having other storage classes on your cluster set as default. I am using thin-csi only to illustrate what happens if you have any other storage class other than a Trident storage class set as default before you install OpenShift Virtualization).
Let us look at the StorageProfile parameters of interest.
[root@localhost ~]# oc describe storageprofile/thin-csi
Name: thin-csi
Namespace:
Labels: app=containerized-data-importer
app.kubernetes.io/component=storage
app.kubernetes.io/managed-by=cdi-controller
app.kubernetes.io/part-of=hyperconverged-cluster
app.kubernetes.io/version=4.19.1
cdi.kubevirt.io=
Annotations: <none>
API Version: cdi.kubevirt.io/v1beta1
Kind: StorageProfile
Metadata:
Creation Timestamp: 2025-07-31T15:25:29Z
Generation: 1
Owner References:
API Version: cdi.kubevirt.io/v1beta1
Block Owner Deletion: true
Controller: true
Kind: CDI
Name: cdi-kubevirt-hyperconverged
UID: d0ee7e11-e408-4f92-9a74-93a1edee154c
Resource Version: 1083543
UID: d595abc0-7670-4d15-b674-a6e2b67852b4
Spec:
Status:
Claim Property Sets:
Access Modes:
ReadWriteOnce
Volume Mode: Block
Access Modes:
ReadWriteOnce
Volume Mode: Filesystem
Clone Strategy: snapshot
Data Import Cron Source Format: pvc
Provisioner: csi.vsphere.vmware.com
Snapshot Class: csi-vsphere-vsc
Storage Class: thin-csi
Events: <none>
You can see that the storage Profile parameters indicate that the Source for the Data Import is in a PVC and the Clone Strategy is set to snapshot. These parameters are automatically set to the values shown for the provisioner csi.vsphere.vmware.com.
They may not be the same for other provisioners or even for Trident csi provisioner, which we will see later in this blog.
With the thin-csi as the default storage class, the images for the different OSes are downloaded and available in PVCs/PVs.
There are no volume snapshots for the os images. The images are just available as PVCs/PVs.
Next, I have installed Trident, created a trident backend and storage class for nas in the OpenShift cluster.
I also created Trident VolumeSnapshotClass.
And as before, a storage profile for sc-nas has been created automatically
Let us look at the parameters of interest in the StorageProfile for the Trident Storage Class.
[root@localhost storageProfile]# oc describe storageprofile sc-nas
Name: sc-nas
Namespace:
Labels: app=containerized-data-importer
app.kubernetes.io/component=storage
app.kubernetes.io/managed-by=cdi-controller
app.kubernetes.io/part-of=hyperconverged-cluster
app.kubernetes.io/version=4.19.1
cdi.kubevirt.io=
Annotations: <none>
API Version: cdi.kubevirt.io/v1beta1
Kind: StorageProfile
Metadata:
Creation Timestamp: 2025-07-31T16:19:18Z
Generation: 1
Owner References:
API Version: cdi.kubevirt.io/v1beta1
Block Owner Deletion: true
Controller: true
Kind: CDI
Name: cdi-kubevirt-hyperconverged
UID: d0ee7e11-e408-4f92-9a74-93a1edee154c
Resource Version: 1115443
UID: f906e71d-b77c-40f5-be41-555dbee4cecb
Spec:
Status:
Claim Property Sets:
Access Modes:
ReadWriteMany
Volume Mode: Filesystem
Access Modes:
ReadWriteOnce
Volume Mode: Filesystem
Clone Strategy: snapshot
Data Import Cron Source Format: snapshot
Provisioner: csi.trident.netapp.io
Snapshot Class: trident-snapshotclass
Storage Class: sc-nas
Events: <none>
Note that the clone strategy is set to snapshot. And the source Format is snapshot. These are the defaults for the Trident CSI (when you have a Trident VolumeSnapshot class available in the cluster). Remember, that the source PVC for the golden image was created using thin-csi provisioner.
Now, if I create a VM using sc-nas as the Storage Class for the root disk of the VM, a new PVC should be created using storage class sc-nas as a clone of the original PVC having the OS image according to the parameters in the Storage Profile. Does this happen? Let us see:
Issue: You can see that the cloning cannot be completed as Trident cannot take a snapshot of the PVC created using a different storage class.
So let us now reconfigure to follow best practice #1:
I deleted all the data volumes created using the thin-csi storage class, deleted OpenShift Virtualization, created additional storage classes in trident and setup iscsi as the default storage class. I also deleted the trident VolumeSnapshot Class.
After this, I installed OpenShift Virtualization, in effect following the best practice #1 where Trident storage class is installed and configured as default storage class before OpenShift Virtualization is installed.
[root@localhost storageProfile]# oc describe storageprofile/sc-iscsi
Name: sc-iscsi
Namespace:
Labels: app=containerized-data-importer
app.kubernetes.io/component=storage
app.kubernetes.io/managed-by=cdi-controller
app.kubernetes.io/part-of=hyperconverged-cluster
app.kubernetes.io/version=4.18.11
cdi.kubevirt.io=
Annotations: <none>
API Version: cdi.kubevirt.io/v1beta1
Kind: StorageProfile
Metadata:
Creation Timestamp: 2025-08-01T01:54:43Z
Generation: 2
Owner References:
API Version: cdi.kubevirt.io/v1beta1
Block Owner Deletion: true
Controller: true
Kind: CDI
Name: cdi-kubevirt-hyperconverged
UID: cb2047e0-8f00-44da-bdc4-8d4a9782c194
Resource Version: 18330078
UID: 4c8443fd-747d-4f19-8298-1da9a41c7353
Spec:
Status:
Claim Property Sets:
Access Modes:
ReadWriteMany
Volume Mode: Block
Clone Strategy: copy
Data Import Cron Source Format: pvc
Provisioner: csi.trident.netapp.io
Snapshot Class: trident-snapshotclass
Storage Class: sc-iscsi
Events: <none>
This time, the StorageProfile has the Clone Strategy set to copy and the source format set to PVC. (Clone Strategy is copy because we don’t have the Trident VolumeSnapshotclass in the cluster).
Impact of not following best practice #2:
When best practice #2 is not followed, i.e Trident snapshot class is not installed, the storage profile has the sourceFormat set to pvc and the clone strategy set to copy. You can see that the golden images are in the PVCs/PVs and there are no volume snapshots.
I created a new VM using the template source available for centos vm using the default storage class. The golden image from the pvc/pv gets copied (not cloned) onto the root disk pvc.
The result: From the screenshot below, you can see it takes about 4 minutes for the image to get copied. Slow and inefficient.
Let us check the volume allocated for the root disk in ONTAP. We can use the internalName available from the oc describe command for the PV.
[root@localhost storageProfile]# oc describe pv/pvc-b803f61d-3080-43d2-8012-4e2c40ff810a
Name: pvc-b803f61d-3080-43d2-8012-4e2c40ff810a
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.trident.netapp.io
volume.kubernetes.io/provisioner-deletion-secret-name:
volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers: [external-provisioner.volume.kubernetes.io/finalizer kubernetes.io/pv-protection external-attacher/csi-trident-netapp-io]
StorageClass: sc-iscsi
Status: Bound
Claim: default/centos-using-copy-strategy
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Block
Capacity: 30Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.trident.netapp.io
FSType:
VolumeHandle: pvc-b803f61d-3080-43d2-8012-4e2c40ff810a
ReadOnly: false
VolumeAttributes: backendUUID=ff20099f-a1a9-4b5a-a18a-0e60f75004c5
internalName=openshift_iscsi_openshift_virtualization_os_images_tmp_pvc_c9bdea1e_48cb_494d_9b72_626f49c42010_b803f
name=pvc-b803f61d-3080-43d2-8012-4e2c40ff810a
protocol=block
storage.kubernetes.io/csiProvisionerIdentity=1753299154552-7067-csi.trident.netapp.io
Events: <none>
From the System Manager, we can see that the backing volume in ONTAP is a FlexVol Volume and does not have a clone hierarchy.
Follow Best Practices #1 and #2 to get the benefit:
Now, let us change the configuration again. And this time, let us follow best practices #1 and #2, i.e install Trident storage class AND Trident Volume Snapshot class BEFORE installing OpenShift Virtualization.
[root@localhost storageProfile]# oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
sc-fc csi.trident.netapp.io Delete Immediate true 14d
sc-iscsi (default) csi.trident.netapp.io Delete Immediate true 30d
sc-nas csi.trident.netapp.io Delete Immediate true 30d
sc-nvme csi.trident.netapp.io Delete Immediate true 30d
sc-san-eco csi.trident.netapp.io Delete Immediate true 8d
[root@localhost storageProfile]# oc get volumesnapshotclass
NAME DRIVER DELETIONPOLICY AGE
trident-snapshotclass csi.trident.netapp.io Delete 10m
[root@localhost storageProfile]# oc get storageprofile
NAME AGE
sc-fc 7m40s
sc-iscsi 7m40s
sc-nas 7m40s
sc-nvme 7m40s
sc-san-eco 7m40s
[root@localhost storageProfile]# oc describe storageprofile/sc-iscsi
Name: sc-iscsi
Namespace:
Labels: app=containerized-data-importer
app.kubernetes.io/component=storage
app.kubernetes.io/managed-by=cdi-controller
app.kubernetes.io/part-of=hyperconverged-cluster
app.kubernetes.io/version=4.18.11
cdi.kubevirt.io=
Annotations: <none>
API Version: cdi.kubevirt.io/v1beta1
Kind: StorageProfile
Metadata:
Creation Timestamp: 2025-08-01T11:43:53Z
Generation: 1
Owner References:
API Version: cdi.kubevirt.io/v1beta1
Block Owner Deletion: true
Controller: true
Kind: CDI
Name: cdi-kubevirt-hyperconverged
UID: 33833bb4-a627-40cd-abb5-a6a79de947db
Resource Version: 18615470
UID: d22cf18c-7816-4373-b7be-2986c066a3b3
Spec:
Status:
Claim Property Sets:
Access Modes:
ReadWriteMany
Volume Mode: Block
Clone Strategy: snapshot
Data Import Cron Source Format: snapshot
Provisioner: csi.trident.netapp.io
Snapshot Class: trident-snapshotclass
Storage Class: sc-iscsi
Events: <none>
As you can see above, the SourceFormat and Clone strategy is set to snapshot. What this means is that the golden image will be available as volume snapshots, and when a new VM is created a clone of this volume snapshot will be created for the root disk of the new VM.
Here is the list of os images available as volume snapshots after you install OpenShift Virtualization. For each image, the temporary PVC is deleted after creating the snapshot.
Now let us create a VM (using the default storage class for the root disk).
The root disk of the VM is created by creating a clone from the snapshot and this is super-fast compared to the copy strategy used previously, as you can see in the screenshot below.(It only takes a few seconds as opposed to a few minutes when the Clone Strategy was copy)
[root@localhost storageProfile]# oc get vm
NAME AGE STATUS READY
fedora-using-clone-strategy 58s Running True
[root@localhost storageProfile]# oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
fedora-using-clone-strategy Bound pvc-59ebe2ba-8cf4-40f0-80e7-7de2d97f7da5 30Gi RWX sc-iscsi <unset> 64s
[root@localhost storageProfile]# oc describe pv/pvc-59ebe2ba-8cf4-40f0-80e7-7de2d97f7da5
Name: pvc-59ebe2ba-8cf4-40f0-80e7-7de2d97f7da5
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.trident.netapp.io
volume.kubernetes.io/provisioner-deletion-secret-name:
volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers: [external-provisioner.volume.kubernetes.io/finalizer kubernetes.io/pv-protection external-attacher/csi-trident-netapp-io]
StorageClass: sc-iscsi
Status: Bound
Claim: default/fedora-using-clone-strategy
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Block
Capacity: 30Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.trident.netapp.io
FSType:
VolumeHandle: pvc-59ebe2ba-8cf4-40f0-80e7-7de2d97f7da5
ReadOnly: false
VolumeAttributes: backendUUID=ff20099f-a1a9-4b5a-a18a-0e60f75004c5
internalName=openshift_iscsi_openshift_virtualization_os_images_tmp_pvc_39fc52f8_231b_4d49_930f_c137774e1ad2_59ebe
name=pvc-59ebe2ba-8cf4-40f0-80e7-7de2d97f7da5
protocol=block
storage.kubernetes.io/csiProvisionerIdentity=1753299154552-7067-csi.trident.netapp.io
Events: <none>
Let us look at the volume for the root disk in ONTAP backend using the internalName seen above.
From ONTAP CLI:
HCG-NetApp-C400-E9U9::> volume show -volume openshift_iscsi_openshift_virtualization_os_images_tmp_pvc_39fc52f8_231b_4d49_930f_c137774e1ad2_59ebe -instance
Vserver Name: openshift
Volume Name: openshift_iscsi_openshift_virtualization_os_images_tmp_pvc_39fc52f8_231b_4d49_930f_c137774e1ad2_59ebe
Aggregate Name: HCG_NetApp_C400_E9U9a_SSD_CAP_1
List of Aggregates for FlexGroup Constituents: HCG_NetApp_C400_E9U9a_SSD_
CAP_1
Encryption Type: none
List of Nodes Hosting the Volume: HCG-NetApp-C400-E9U9a
Volume Size: 33GB
Volume Data Set ID: 37477
Volume Master Data Set ID: 2162962060
Volume State: online
Volume Style: flex
Extended Volume Style: flexvol
FlexCache Endpoint Type: none
Is Cluster-Mode Volume: true
Is Constituent Volume: false
Number of Constituent Volumes: -
Export Policy: default
User ID: 0
Group ID: 0
Security Style: unix
UNIX Permissions: ---rwxrwxrwx
Junction Path: -
Junction Path Source: -
Junction Active: -
Junction Parent Volume: -
Comment:
Available Size: 32.40GB
Filesystem Size: 33GB
Total User-Visible Size: 33GB
Used Size: 618.8MB
Used Percentage: 1%
-----
Language: C.UTF-8
Clone Volume: true
Node name: HCG-NetApp-C400-E9U9a
Clone Parent Vserver Name: openshift
FlexClone Parent Volume: openshift_iscsi_openshift_virtualization_os_images_prime_b39d0f76_ee5d_4b13_860d_6a063da71133_070ca
NVFAIL Option: on
Volume's NVFAIL State: false
Force NVFAIL on MetroCluster Switchover: off
Is File System Size Fixed: false
Press <space> to page down, <return> for next line, or 'q' to quit...
You can see that the root disk is a FlexClone of the snapshot of the volume with the golden image in ONTAP backend.
And as a final note, use the same default storage class for the root disk of your VMs, because to create FlexClones, the source and the target PVC protocols must match.
Conclusion:
We have seen the relationship between the Storage Profile parameters and Trident. We have seen the behavior of Trident when Trident Storage Class and Volume Snapshot class are set as default. We have seen that when the following best practices are followed, you can fully harness the benefits that Trident and NetApp storage offer, ensuring a robust and efficient OpenShift Virtualization setup.
- Ensure Trident is installed first and set the Trident storage class as the default in your cluster before installing OpenShift Virtualization. This foundational step is crucial for seamless integration and optimal performance.
- Create and configure the Trident VolumeSnapshotClass as the default. This setup leverages NetApp Clone Technology for fast and efficient VM operations by setting the CloneStrategy parameter in StorageProfile to snapshot.
- When creating VMs, use the same storage class for the boot disk as used for the golden image.
