Authors: Banu Sundhar, NetApp
Mayur Shetty, Red Hat
Kubernetes has become the go-to platform for developers looking to deploy applications, and many of these deployments can greatly benefit from a persistent, shared storage layer. Red Hat OpenShift Service on AWS (ROSA) offers a managed OpenShift experience on AWS, created by Red Hat and supported collaboratively by both AWS and Red Hat. While ROSA clusters typically utilize locally-attached Amazon Elastic Block Store (EBS) volumes for data storage, many customers require a more robust solution—one that ensures data persistence and sharing across multiple containers, even across different Availability Zones (AZs). These customers seek a scalable storage solution that provides a consistent interface for managing workloads across both on-premises and cloud environments.
ROSA integrates seamlessly with Amazon FSx for NetApp ONTAP (FSxN), a fully managed, scalable shared storage service built on NetApp's renowned ONTAP file system. With FSxN, customers can leverage key features such as snapshots, FlexClones, cross-region replication with SnapMirror, and a highly available file server that supports seamless failover. The integration with NetApp Trident driver—a dynamic Container Storage Interface (CSI)—facilitates the management of Kubernetes Persistent Volume Claims (PVCs) on storage disks. This driver automates the on-demand provisioning of storage volumes across diverse deployment environments, making it simpler to scale and protect data for your applications. One key benefit of FSxN is that it is a true first party AWS offering just like EBS, meaning customers can retire their committed spend with AWS and get support directly from them as well.
In this blog, we will explore how to utilize FSxN as a persistent storage layer for applications running on ROSA. We’ll guide you through the installation of the NetApp Trident CSI driver on a ROSA cluster, the provisioning of an FSxN file system, and the deployment of a sample stateful application. Additionally, we’ll cover strategies for backing up and restoring your application data. With this integrated solution, you can establish a shared storage framework that effortlessly scales across AZs, simplifying the processes of scaling, protecting, and restoring your data using the Trident CSI driver.
Solution overview
This diagram shows the ROSA cluster deployed in multiple AZs. ROSA cluster’s master nodes, infrastructure nodes are in Red Hat’s VPC, while the worker nodes are in a VPC in the customer's account. We’ll create an FSxN file system within the same VPC and install the Trident provisioner in the ROSA cluster, allowing all the subnets of this VPC to connect to the file system.
Prerequisites:
- AWS account
- A Red Hat account
- IAM user with appropriate permissions to create and access ROSA cluster
- AWS CLI
- ROSA CLI
- OpenShift command-line interface (oc)
- Helm 3 documentation
- A HCP ROSA cluster
- Access to Red Hat OpenShift web console
1. Provision FSx for NetApp ONTAP
Create a multi-AZ FSx for NetApp ONTAP in the same VPC as the ROSA cluster.
There are several ways to do this. I am showing the creation of FSxN using a
CloudFormation Stack.
a. Clone the GitHub repository
# git clone https://github.com/aws-samples/rosa-fsx-netapp-ontap.git
b. Run the CloudFormation Stack
Run the command below by replacing the parameter values with your own values:
# cd rosa-fsx-netapp-ontap/fsx
# aws cloudformation create-stack \
--stack-name ROSA-FSXONTAP \
--template-body file://./FSxONTAP.yaml \
--region <region-name> \
--parameters \
ParameterKey=Subnet1ID,ParameterValue=[subnet1_ID] \
ParameterKey=Subnet2ID,ParameterValue=[subnet2_ID] \
ParameterKey=myVpc,ParameterValue=[VPC_ID] \
ParameterKey=FSxONTAPRouteTable,ParameterValue=[routetable1_ID,routetable2_ID] \
ParameterKey=FileSystemName,ParameterValue=ROSA-myFSxONTAP \
ParameterKey=ThroughputCapacity,ParameterValue=1024 \
ParameterKey=FSxAllowedCIDR,ParameterValue=[your_allowed_CIDR] \
ParameterKey=FsxAdminPassword,ParameterValue=[Define Admin password] \
ParameterKey=SvmAdminPassword,ParameterValue=[Define SVM password] \
--capabilities CAPABILITY_NAMED_IAM
Where :
region-name: same as the region where the ROSA cluster is deployed
subnet1_ID : id of the Preferred subnet for FSxN
subnet2_ID: id of the Standby subnet for FSxN
VPC_ID: id of the VPC where the ROSA cluster is deployed
routetable1_ID, routetable2_ID: ids of the route tables associated with the subnets chosen above
your_allowed_CIDR: allowed CIDR range for the FSx for ONTAP security groups ingress rules to
control access. You can use 0.0.0.0/0 or any appropriate CIDR to allow all
traffic to access the specific ports of FSx for ONTAP.
Define Admin password: A password to login to FSxN
Define SVM password: A password to login to SVM that will be created.
Verify that your file system and storage virtual machine (SVM) has been created using the Amazon FSx console, shown below:
2. Install and configure Trident CSI driver for the ROSA cluster
a. Add the Trident Helm repository
# helm repo add netapp-trident https://netapp.github.io/trident-helm-chart
b. Install trident using helm
Note that depending on the version you install, the version parameter will need to be changed in the following command: Refer to the documentation for the correct version number
# helm install trident netapp-trident/trident-operator --version 100.2406.0 --create-namespace --namespace trident
For additional methods of installing Trident, refer to the Trident documentation.
c. Verify that all Trident pods are in the running state
3. Configure the Trident CSI backend to use FSx for NetApp ONTAP (ONTAP NAS)
The Trident back-end configuration tells Trident how to communicate with the storage system (in this case, FSxN). For creating the backend, we will provide the credentials of the Storage Virtual machine to connect to, along with the cluster management and the NFS data interfaces. We will use the ontap-nas driver to provision storage volumes in FSxN file system.
a. First, create a secret for the SVM credentials using the following yaml
apiVersion: v1
kind: Secret
metadata:
name: backend-fsx-ontap-nas-secret
namespace: trident
type: Opaque
stringData:
username: vsadmin
password: <value provided for Define SVM password as a parameter to the Cloud Formation Stack>
Note: You can also retrieve the SVM password created for FSxN from the AWS Secrets Manager as shown below.
b. Next, add the secret for the SVM credentials to the ROSA cluster using the following command
# oc apply -f svm_secret.yaml
You can verify that the secret has been added in the trident namespace using the following command
# oc get secrets -n trident |grep backend-fsx-ontap-nas-secret
c. Next, create the backend object
For this, move into the fsx directory of your cloned Git repository. Open the file backend-ontap-nas.yaml. Replace the following:
managementLIF with the Management DNS name
dataLIF with the NFS DNS name of the Amazon FSx SVM and
svm with the SVM name. Create the backend object using the following command.
# oc apply -f backend-ontap-nas.yaml
Note: You can get the Management DNS name, NFS DNS name and the SVM name from the Amazon FSx Console as shown in the screenshot below:
d. Now, run the following command to verify that the backend object has been created and Phase is showing Bound and Status is Success
4. Create Storage Class
Now that the Trident backend is configured, you can create a Kubernetes storage class to use the backend. Storage class is a resource object made available to the cluster. It describes and classifies the type of storage that you can request for an application.
a. Review the file storage-class-csi-nas.yaml in the fsx folder.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: trident-csi
provisioner: csi.trident.netapp.io
parameters:
backendType: "ontap-nas"
fsType: "ext4"
allowVolumeExpansion: True
reclaimPolicy: Retain
b. Create Storage Class in ROSA cluster and verify that trident-csi storage class has been created.
This completes the installation of Trident CSI driver and its connectivity to FSxN file system. Now you can deploy a sample Postgresql stateful application on ROSA using file volumes on FSxN.
c. Verify that there are no PVCs and PVs created using the trident-csi storage class.
d. Verify that applications can create PV using Trident CSI.
Create a PVC using the pvc-trident.yaml file provided in the fsx folder.
pvc-trident.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: basic
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: trident-csi
You can issue the following commands to create a pvc and verify that it has been created.
5. Deploy a sample Postgresql stateful application
a. Use helm to install postgresql
# helm install postgresql bitnami/postgresql -n postgresql --create-namespace
b. Verify that the application pod is running, and a PVC and PV is created for the application.
c. Deploy a Postgresql client
Use the following command to get the password for the postgresql server that was installed.
# export POSTGRES_PASSWORD=$(kubectl get secret --namespace postgresql postgresql -o jsoata.postgres-password}" | base64 -d)
Use the following command to run a postgresql client and connect to the server using the password
# kubectl run postgresql-client --rm --tty -i --restart='Never' --namespace postgresql --image docker.io/bitnami/postgresql:16.2.0-debian-11-r1 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
> --command -- psql --host postgresql -U postgres -d postgres -p 5432
d. Create a database and a table. Create a schema for the table and insert 2 rows of data into the table.
6. Create a Snapshot of the app volume
a. Create a VolumeSnapshotClass
Save the following manifest in a file called volume-snapshot-class.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: fsx-snapclass
driver: csi.trident.netapp.io
deletionPolicy: Delete
Create a snapshot by using the above manifest.
b. Next, create a snapshot
Create a snapshot of the existing PVC by creating VolumeSnapshot to take a point-in-time copy of your Postgresql data. This creates an FSx snapshot that takes almost no space in the filesystem backend. Save the following manifest in a file called volume-snapshot.yaml:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgresql-volume-snap-01
spec:
volumeSnapshotClassName: fsx-snapclass
source:
persistentVolumeClaimName: data-postgresql-0
c. Create the volume snapshot and confirm that it is created
d. Delete the database to simulate the loss of data (data loss can happen due to a variety of reasons, here we are just simulating it by deleting the database)
7. Restore from Snapshot
a. Create a volume clone from the snapshot
To restore the volume to its previous state, you must create a new PVC based on the data in the snapshot you took. To do this, save the following manifest in a file named pvc-clone.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgresql-volume-clone
spec:
accessModes:
- ReadWriteOnce
storageClassName: trident-csi
resources:
requests:
storage: 8Gi
dataSource:
name: postgresql-volume-snap-01
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
Create a clone of the volume by creating a PVC using the snapshot as the source using the above manifest.
Apply the manifest and ensure that the clone is created.
b. Delete the original postgresql installation
c. Create a new postgresql application using the new clone PVC
# helm install postgresql bitnami/postgresql --set primary.persistence.enabled=true --set primary.persistence.existingClaim=postgresql-volume-clone -n postgresql
d. Verify that the application pod is in the running state
e. Verify that the pod uses the clone as its PVC
# oc describe pod/postgresql-0 -n postgresql
f. To validate that the database has been restored as expected, go back to the container console and show the existing databases
Conclusion
In this blog, we demonstrated the integration of FSx for NetApp ONTAP as a shared file system for a ROSA cluster with Hosted Control Plane using the NetApp Trident CSI driver. We also walked through deploying a stateful application across multi-AZ using FSxN file system. We then showed how you can use the rich NetApp snapshots feature to quickly and easily protect and restore your stateful application. Combining all the capabilities of FSxN with the sub-millisecond latencies and multi-AZ availability, FSxN is a great storage option for your containerized applications running in ROSA.
For more information about Trident, refer to the NetApp Trident documentation. See Red Hat OpenShift with NetApp under the Containers section in the NetApp Solutions documentation for additional information and videos. If you’d like to clean up the solution set up in this post, follow the instructi