Using NVMe/TCP to consume ONTAP storage for your modern containerized apps on VMware

[banusundhar]

As you are moving along in your journey towards modernization, you may be making many changes to your infrastructures and to the workloads deployed on them. Some of you are modernizing your apps by moving towards a microservices based architecture, while still running them on your trusted hypervisor. You are presented with many options to run your Kubernetes (k8s) based workloads. As an example, if you are using VMware environments, you can deploy vanilla Kubernetes clusters on VMware to run your modernized apps. Or you can choose to deploy your apps on VMware Tanzu that enables you to run modern applications on Kubernetes-based infrastructure in a supported manner across on-premises, cloud and edge infrastructure. Yet another option is to use Google Distributed Cloud Virtual. Red Hat OpenShift is another platform that not only allows you to run your container apps but also the legacy apps on VMs and manage both on a single platform using the same management interface.

Regardless of which option you choose for your workloads, ONTAP storage provides a great solution for the persistent storage needs of your stateful container-based workloads. You can get the same data management capabilities that you are so accustomed to, and use the skillsets that you already have, to consume ONTAP storage. Astra Trident, implemented to meet the industry-standard Container Storage Interface (CSI) provides dynamic storage orchestration services for the Kubernetes workloads.

NVMe/TCP is the next generation storage protocol that uses Ethernet networks to extend the performance of NVMe SSDs. Like iSCSI, NVMe/TCP can use any Ethernet NICs and switches, making it simpler and less expensive to deploy and maintain. Since

it can be deployed on any TCP network, it can be used on-premises or in the cloud. Chance Bingen has written a very detailed blog about how to configure NVMe/TCP for VMFS Datastores on VMware backed by ONTAP storage. The VMFS Datastores can then be consumed by workloads running on the VMs in VMware. But if your workloads are running as containers, then the storage paradigm changes for the workloads. Several articles have been written to show details of creating ONTAP NAS volumes or ONTAP block volumes using iSCSI for Kubernetes workloads. This blog will provide details about consuming ONTAP storage using NVMe/TCP protocol.

For the purposes of this blog, I have a Kubernetes cluster as shown below with a master node and worker nodes.

I also have an ONTAP system that will serve as the backend storage for the apps on the k8s cluster.

 

Pre-requisite to use NVMe/TCP to consume block storage provided by ONTAP

Note: For writing this blog, I used Astra Trident version 24.02.

1. An administrator should install and configure ONTAP with a storage VM(SVM) for NVMe. Details are here in the ONTAP documentation.
2. A cluster administrator should deploy a k8s based cluster and integrate it with the ONTAP storage using Trident:
   1. Deploy a cluster with any of the following container engines and orchestrators
      • Anthos On-Prem (VMware) and Anthos on bare metal 1.16
      • Kubernetes 1.23 - 1.29
      • OpenShift 4.10 - 4.15
   2. Once you have the cluster, install Astra Trident on the cluster using the procedures given here. Astra Trident is a Container Storage Interface (CSI) compliant dynamic storage orchestrator that natively integrates with Kubernetes. Astra Trident runs as a single Controller Pod plus a Node Pod on each worker node in the cluster. It enables consumption and management of storage resources across all popular NetApp storage platforms, in the public cloud or on premises, including ONTAP (AFF, FAS, Select, Cloud, Amazon FSx for NetApp ONTAP), Element software (NetApp HCI, SolidFire), Azure NetApp Files service, and Cloud Volumes Service on Google Cloud.

Once the cluster is ready, NVMe/TCP protocol needs to be enabled on the worker nodes of the cluster for communications with the storage using that protocol. To do this, I created a daemon set as shown belowOnce Astra Trident is installed and NVMe/TCP tools are installed, the cluster administrator can create a TridentBackendConfig object that contains information for Trident to communicate with the ONTAP storage system and to provision volumes from it.  A TridentBackend object that defines the relationship between Astra Trident and ONTAP storage is automatically created when a TridentBackendConfig object is created.

Here is a sample yaml file for the TridentBackendConfig for NVMe/TCP.

The above manifest specifies “ontap-san” as the storage driver and sanType as nvme. Using the storage driver, the backend object will provision an NVMe volume. The other fields in the config provide the SVM name, SVM IP addresses, and login credentials that Trident needs for provisioning storage on ONTAP. The username here is “vsadmin”, indicating that the backend is using an SVM administrator account that has privileges scoped just to that SVM, rather than a cluster administrator account that has cluster-wide privileges. The associated password value is not encrypted, so keep your backend manifest files safe, since they contain sensitive information.

Note: Instead of password-based authentication, you can do certificate-based authentication for the communication with ONTAP.  Review the procedures here to enable certificate authentication.

Create a backend configuration object with the above manifest.        

Here is the backend object that is automatically created.

Since the backend object is purely a trident construct, let us look at the details of the trident backend object for NVMe using tridentctl command. Towards the end of the details, you can see that the volumes: [] list is empty indicating that it is not managing any volumes  yet.

Now the administrator can proceed to create a StorageClass (SC) object to represent the NVMe/TCP storage option that will be made available for the Kubernetes cluster. If the  SC name is descriptive enough, cluster users and application developers can pick the appropriate SC when creating a persistent volume claim (PVC) without having to look into the internal configuration details of the SC.

Here is the yaml used to create the storage class.

Use Kubectl to create the storage class object using the above yaml

We can now describe the SC to get the details.

Using NVMe/TCP to consume block storage provided by ONTAP

Now let us look at what an application developer would do to use the NVMe storage in ONTAP that has become available for use via the storage class.

The application developer creates a persistent volume claim (pvc), that in turn will create a persistent volume (pv). The volume attributes will be based on the storage class used that in turn will use the appropriate backend to communicate with ONTAP to create the volume as requested.

To show this in action, I deployed a sample application as shown in this tutorial https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/ with modifications to the pvc as follows: As you see from the table below, ontap-san driver can support RWX access mode using NVMe/TCP only when configured as a raw block device. The PVC configuration below shows volumeMode: Block and accessMode: ReadWriteMany. You could use accessMode ReadWriteOnce as well in which case you will need to include the volumeMounts in the manifest to mount the file system to the pod.

 

I deployed the application with the command as shown

With the use of  the Storage Class in the PVC, a PV with the name specified in the manifest is created. The pods come to a running state with the attached to the NVMe block storage as specified in the yaml.

Let us now look at the Trident backend Object to see if it reflects the new volumes that are created. In the screenshot below that has been abbreviated, towards the end, in the volumes[] list,  you can see that it now shows that it manages 2 volumes that it has just created.

Look at the pv object to find the details about the volume that is created. You can see the storage class that is used to create this volume. The volume attribute list contains internalName which is the name of the volume in ONTAP.

Now, let us look at the volume in ONTAP using the name of the volume in the internalName field. You can see that the volume is created in the svm that we specified in the trident backend configuration class.

List the sansvm SVM’s NVMe namespaces and subsystems. In ONTAP, an NVMe/TCP namespace is conceptually similar to an iSCSI LUN, while an NVMe/TCP subsystem is conceptually similar to an iSCSI igroup. You should expect that the command output in your lab will report different path, subsystem, and target NQN values than those present in the following example output.

Trident created a 20GB namespace named “namespace0” on the ONTAP volumes created for the application. That namespace is mapped to a named subsystem, and Trident populated that subsystem with the NVMe Qualified Name (NQN) of the worker node on which Kubernetes deployed the pod.

 

Summary

In this blog I have shown you how easy it is to consume ONTAP storage using NVME/TCP for the persistent storage needs of your modernized container apps. Astra Trident makes it easy to dynamically provision the volumes for the applications upon request using persistent volume claims. The application does not have to keep track of how to access the storage or any of the details about where the storage needs to be provisioned. It simply needs to request the amount of storage using the appropriate storage class and Trident takes care of everything else to create it and attach it to the application pod. The lifecycle of the pod is independent of the lifecycle of the storage. When a pod crashes, a new pod is created and is attached to the same volume, thus retaining the data for the application. For additional details about Astra Trident and all the drivers it supports, please see the Astra Trident documentation.