Logging is an integral part of any software system. It is important that logs from software systems as well as from infrastructure are stored in persistent storage, so that they can be used as a tool for troubleshooting at any point when the system does not function as expected.
Customers who are already using ONTAP for persistent storage of their stateful containerized applications on OpenShift Container Platform need to look no further than using the same ONTAP system for log storage as well. For those who are evaluating the use of ONTAP, it is another added value that ONTAP storage brings to the table. In this blog, I will provide details about how ONTAP storage can be used for the logging subsystem of OpenShift Container Platform.
For workloads running on OpenShift Container Platform, the application logs originate from the application containers. A cluster administrator can deploy logging subsystem on a cluster and use it to collect and aggregate node system audit logs, application container logs and infrastructure logs.
For details about the logging architecture in OpenShift Container Platform, see the OpenShift documentation. Essentially, there are 3 main components: Collector, Log store, and Visualization. In the setup shown below, I used Vector as the collector of logs. Vector collects logs from each node, transforms the data and forwards it to the configured log store. The log store that I have used is LokiStack. The log forwarder will forward and store the log data for analysis in the log store. I have provided local filesystem using ontap-nas as the storage class as well as ONTAP S3 bucket for storing the logs. For visualization, I have selected OpenShift Container Platform web console UI. (Please note that Fluentd for collector is deprecated as of logging version 5.6, OpenShift Elasticsearch Operator for log store is deprecated as of logging version 5.4.3 and Kibana web console for visualization is deprecated as of OpenShift version 4.11).
Here are the details of the setup to store logs in ONTAP storage:
I configured an ONTAP System (9.12.1) with NetApp Astra Trident (23.07) as the backend persistent storage for apps on OpenShift Container Platform (4.13.13) on vSphere. I then created an object store in the same ONTAP System for long-term log storage. Refer to the ONTAP documentation for setting up Object storage in ONTAP.
I created a bucket and obtained the secret key for the bucket in ONTAP. I then used the secret key to create a secret in the OpenShift Cluster in which I will be installing the logging subsystem.
On my cluster, I already have ontap-nas and ontap-san backends created to provision volumes in ONTAP that can provide NFS and iSCSI storage. Refer to Astra Trident documentation for installing Trident and creating backend and storage class configurations. I then followed the OpenShift documentation to install the Loki Operator (5.8) and the OpenShift Logging Operator (5.8).
Next, I created LokiStack from the Loki Operator.
Fill out the form view as shown above and then go to the yaml view to modify it to ensure that it looks like the following:
For Loki deployment sizing, please consult the documentation.
Next step is to create the Cluster Logging instance from the Red Hat OpenShift Logging Operator. I selected vector for Collector Implementation, lokistack for the logstore and provided the name of the lokistack that I created in the previous step, and OCPconsole for visualization. I configured the retention policy for application, audit and infra logs. This policy is for the temporary storage and does not impact the retention for logs in object storage.
When you click on create, an instance of cluster logging will be created. You will see a message to refresh the browser to see the update to the web console. When you refresh the page, on the left side navigation menu, you will see Logs under Observe.
Now, let us go find all the resources created in ONTAP by LokiStack for its use.Before you can view the ONTAP volumes, you need to first find the volume names created by Trident.
From the command line on a VM from where you can access the cluster, you can look up the PVCs, and the PVs created by the LokiStack. These are created based on the storage class you configured earlier.
For each persistent volume (pv), you can find the corresponding volume name in ONTAP by using the following command. These are the volumes used for temporary storage by LokiStack.
You can login to the System Manager of the ONTAP system using its management LIF and view the volumes used by the LokiStack.
You can also view the Object Storage bucket that you configured to store the logs.
Additionally, you can use an S3 browser to view the logs stored in the S3 bucket.
Summary
Red Hat OpenShift is being widely used as a platform for container workloads. Customers are already using NetApp ONTAP for the persistent storage needs of their stateful applications on OpenShift Containers. In this blog, I have shown you how to integrate the same ONTAP storage system with the OpenShift Logging subsystem. This added value enables our customers to extend their storage system utilization to the RedHat ecosystem.