Tech ONTAP Blogs

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Tech ONTAP Blogs

Boot an SAP System from Red Hat OpenShift Virtualization with NetApp Solutions on Bare Metal Servers

Holgerz
NetApp
NetApp
10 hours ago
63 Views

Introduction

In complex enterprise environments, testing and migration scenarios from virtualized workload on physical hardware are often time-critical and resource-intensive. With Red Hat OpenShift Virtualization, NetApp Trident, and FlexClone technology, virtual machines running in Red Hat OpenShift can be seamlessly migrated to physical servers in seconds - ideal for performance testing, error analysis, or hardware benchmarks. To enable this flexibility, the VMs must have its root and hence its boot LUN on NetApp ONTAP and be connected via the iSCSI protocol.

This blog captures the following details:

Red Hat OpenShift

Red Hat® OpenShift® provides a unified hybrid cloud foundation for developing and scaling containerized applications. Trusted by thousands of organizations across industries, it powers business-critical workloads—whether modernizing existing applications for the cloud or creating new customer experiences. Backed by Red Hat, a leading contributor to Kubernetes, it combines innovation with proven reliability.

Red Hat OpenShift Virtualization

Red Hat® OpenShift® Virtualization, included with Red Hat OpenShift, offers a modern platform for running and deploying both new and existing virtual machine (VM) workloads. It enables organizations to easily migrate and manage traditional VMs on a reliable, consistent, and fully integrated hybrid cloud application platform. By streamlining VM migration, OpenShift Virtualization supports infrastructure modernization while leveraging the speed and simplicity of a cloud-native environment. It protects existing virtualization investments, embraces modern management practices, and serves as the cornerstone of Red Hat’s complete virtualization offering.

NetApp

NetApp is a global leader in innovative data management and storage solutions. NetApp is committed to helping companies maximize the value of their data – whether in the cloud, on-premises, or in hybrid environments.

 

With the NetApp ONTAP storage operating system, modern all-flash and hybrid storage systems, and integrated cloud data services, NetApp enables efficient, secure, and scalable data management. NetApp solutions deliver performance, reliability, and seamless integration into multi-cloud strategies.

 

NetApp helps customers worldwide accelerate their digital transformation – data-driven, innovative, and future-proof.

NetApp Trident

Trident is an open-source storage orchestrator from NetApp, specifically designed for container platforms such as Kubernetes and OpenShift. Trident integrates seamlessly with OpenShift and enables the automated provisioning and management of persistent storage, whether for containers, databases, or VMs. Trident is a CSI driver which comes at no extra cost and it is fully supported and maintained by NetApp.

 

All NetApp storage platforms are supported – including ONTAP (AFF, FAS, Select, Cloud Volumes ONTAP), Element OS, StorageGRID, and more. Trident supports both block (iSCSI, FC) and file (NFS, SMB) protocols, meeting the needs of DevOps teams as well as VM administrators who want to run traditional enterprise workloads such as databases and ERP systems on OCP VMs.

Prerequisites

To understand and successfully implement the content described in this document, it is assumed that the reader has a solid understanding of the following technologies:

  • Red Hat OpenShift together with Red Hat OpenShift Virtualizatio
  • NetApp Trident
  • Core NetApp ONTAP concepts like Storage Virtual Machines (SVMs), Snapshot and FlexClone technology
  • iSCSI and FC block protocol
  • NFS protocol

This document expects an understanding of these technologies and therefore does not provide an introductory explanation of them.

 

In order to follow the corresponding steps, you need:

NetApp Reference Architecture for running Applications and Databases inside VMs

Databases can be run in VMs in a variety of ways, especially with regard to the storage backend. Two common options are:

  • "in-guest" NFS mounts (i.e., NFS within the VM)
  • Logical Volumes on block storage (e.g., LVM on iSCSI/FC/virtual block device)

For application workload NetApp recommends to run them using “in-guest” NFS mounts. When using "in-guest" NFS mounts, the application and/or database runs in the VM, and one or more NFS volumes (from the NetApp storage system ) are mounted directly within the virtualized guest operating system. The following picture shows a high level architecture for an OpenShift VM, having an iSCSI root LUN for its operating system and two “in-guest” NFS mounts (/data and /log) for application file systems.

NFS "in guest mounts" with NetAppNFS "in guest mounts" with NetApp

 Let’s look at the “in-guest” NFS mounts from an operating system level. According to NetApps best practice, “in-guest” NFS mounts are mounted over a dedicated storage LAN configured with end-to-end jumbo frames for the ethernet protocol. To check if jumbo frames are configured properly end-to-end, you can use the following command "ping -M do  <IP address of NetApp system over which the NFS volumes are exported> -s 8960”

 

The screenshot shows the configuration of the storage LAN's network interface (eth2), as well as the two “in-guest” NFS mounts, /data and /log.

NFS "in guest mounts"NFS "in guest mounts"

 For different applications and databases running as “in-guest” NFS mounts, NetApp offers best practices documents which cover all aspects of configuring the NetApp storage system, parameterization of the operating system and the respective database to ensure the best possible performance for the application, i.e.:

From NetApp experience "in-guest" NFS mounts offer a lot of advantages compared to traditional file systems for applications which reside on block storage:

  • Flexibility and Simplicity:
    Mounts can be easily customized within the VM without requiring intervention at the hypervisor or storage level
  • Storage-level snapshots and backups:
    NetApp offers unique Snapshots based backups, restore and cloning mechanisms. These mechanisms are extremely fast (backup, restore and cloning within minutes independent of the volume size) and can be used directly for consistent backups when using appropriate application integrations
  • Scalability:
    Volumes can be resized (expanded and shrinked) dynamically, without application impact on the VM level
  • Reusability:
    An NFS share can be easily remounted on another VM or on a physical server for testing, error analysis, high availability or disaster recovery scenarios
  • Hypervisor independence:
    No special storage driver or integration required – works seamlessly on any hypervisor
  • Physical to virtual or virtual to physical:
    Applications can be moved from virtual machines to physical servers and vice versa without having the need for operating system reconfiguration (given the Linux kernel and TCP/IP parameters have been configured accordingly and the correct mount options are used)

 When moving applications from virtual machines to physical server (i.e. for performance and/or error analysis) it is very important to use identical kernel parameterization. This ensures that the application runs equally in both environments:

  • Exhibits identical behavior
  • Does not introduce new errors
  • Does not fail due to environmental differences

 The most reliable way to ensure consistent configuration between an operating system running in a virtual machine and one running on a physical server (for performance tests and error analysis) is to boot the physical server using the same root disk as the virtual machine.
With the combination of Red Hat OpenShift Virtualization, NetApp Trident, NetApp Snapshot and FlexClone technology - which enables the creation of Fiber Channel boot LUNs for physical servers from iSCSI boot LUNs used for virtual machines - a complex migration or troubleshooting scenario becomes a fast, secure, and repeatable process. This saves time, simplifies processes, and increases flexibility in dynamic IT landscapes.

Since, according to these best practices, all application data is stored on NetApp volumes mounted via NFS into the guest operating systems, it is only necessary for operators to create the iSCSI root LUN for VMs via the Container Storage Interface (CSI) using NetApp Trident (which is be described in detail in the subsequent chapters). The volumes for application data - whose configuration is static - are therefore created "traditionally" using the NetApp UI (System Manager), command line interface or automation tools such as Ansible.

Configuration

This chapter gives an NetApp Trident overview and describes the basic installation and configuration steps, so that NetApp Trident can be used as storage provisioner in Red Hat OpenShift.

Configure Red Hat OpenShift cluster for SAP workloads with Ansible

There are a number of settings required in order to configure an Red Hat OpenShift cluster for SAP workloads. There is an Ansible role named sap_hypervisor_node_preconfigure which automates the configuration steps. The major things done are:

 

  1. Set tuned profile to virtual-host.
  2. Install Operators:
    1. CNV operator,
    2. SRIOV operator,
    3. nmstate operator,
    4. Netapp Trident operator.
  3. Setup worker nodes by executing the following steps:
    1. Set kernel parameters,
    2. Enable CPU Manager,
    3. Configure network.
  4. Storage: Setup and configure Netapp Trident Storage.

The role sap_vm_provision should be used to create virtual machines incorporating the settings and tunings prepared by the above sap_hypervisor_node_preconfigure role.

 

There is a Knowledgebase article describing the usage of those roles in more in-depth detail.

NetApp Trident installation and configuration

Trident allows the deployment of both file-based and block-based Persistent Volume Claims (PVCs). In this blog, we will only consider the deployment of block-based PVCs as iSCSI LUNs, since only iSCSI LUNs can be mapped to physical servers as FC LUNs and thus booted from them.

 

Triden can be easily installed in OpenShift as an Operator-based installation which has the following advantages:

  • Trident is available as a Red Hat certified operator
  • Automated lifecycle management
  • Better integration with OpenShift (e.g., for upgrades)
  • Clearly defined states through Custom Resource Definitions (CRDs) 

The following picture shows high level the process of configuring Trident and the workflow when provisioning storage 

Trident high level architectureTrident high level architecture

Trident Installation

The complete documentation for installing and configuring NetApp Trident can be found here:

https://docs.netapp.com/us-en/trident/

In this chapter we therefore only give a brief overview of the necessary installation steps for NetApp Trident in RedHat OCP

  • In the OpenShift Web Console, navigate to the Administrator view and select Operators → OperatorHub.
  • Search for the Trident Operator
    Enter "Trident" in the search field.
  • Select the Trident Operator.
  • Start the installation
    • Select a namespace, e.g., trident (recommended).
    • Installation mode: All namespaces on the cluster (default).
    • Click Install.
  • Deploy the TridentOrchestrator
    • Wait until the operator is successfully installed (Succeeded status).
    • Click “Create TridentOrchestrator”
    • Modify Trident Orchestrator YAML Modify Trident operatorModify Trident operator
      Select the Kebab menu → “Edit TridentOrchestrator” and change
      nodePrep: []
      To
      nodePrep:
          - iscsi
      → This step eliminates the need to apply machine configurations for enabling iscsi and multipath on the worker no

Trident Configuration

After the installation, some additional steps must be performed to enable the automatic provisioning of PVCs in OpenShift:

  • Define the Trident backend (ONTAP, Azure NetApp Files, etc.)
  • Create a StorageClass that points to the Trident backend

A complete YAML file which creates a secret, configures an ONTAP backend and creates a default iSCSI storage class is attached in the chapter “Example Configuration Files”

Create an FC LUN Out of an iSCSI VM Root Disk and Boot Your Physical Server Using the Newly Created LUN

For simplicity, we created a virtual machine from an available CentOS Stream 9 OpenShift virtualization image. The configuration details are shown in the next screenshot.

We added two additional NICs:

  • nic-172
    This is our application data LAN
  • nic-192
    This is our storage LAN

After starting the VM, we assigned fixed IP addresses for nic-172 (172.30.20.34) and nic-192 (192.168.20.34) using “nmtui” (Network Manager Text User Interface). An example is shown for nic-192 which has the device name eth2 inside the VM

As a final step we mounted two NFS volumes into the VM. The resulting filesystem structure has already been shown in the section NetApp Reference Architecture for running Applications and Databases inside VMs

High Level Workflow

The workflow consists of the following steps, which are described in detail in the next chapter:

 

  1. Shut down the VM to ensure all NFS filesystems are properly unmounted, the iSCSI root disk which is going to be cloned does not have inconsistent filesystems and we do not have duplicate IP addresses when booting the physical server afterwards.
  2. Create a FlexClone from the iSCSI root volume from which the VM is booting
  3. Split the Clone from its original volume and rehost the volume to the SVM, from which the physical host does booting from FC LUNs
    Technically, splitting a clone from its original volume and rehosting it to another SVM is not required for the volume to function or be accessible. A cloned volume can be used as-is without splitting or rehosting, as long as the necessary permissions and mappings (e.g., LUN mappings, initiator groups) are correctly configured.
    However, in most enterprise environments, these steps are commonly performed because:
    1. Splitting the clone ensures it becomes a fully independent volume, avoiding dependency on the source volume
    2. Different SVMs (Storage Virtual Machines) are typically used to separate workloads or functional domains
    3. Physical hosts boots from FC LUNs presented by a specific SVM, to maintain clean logical separation
  4. Boot a spare physical host from the newly created FC LUN, adjust network configuration to the new device names, mount the NFS volumes and do your application testing
    Snapshot, Clone and Boot workflowSnapshot, Clone and Boot workflow

Detailed Workflow Steps

For creating a FlexClone, we have to identify the PVC name and the corresponding Trident LUN and its Volume (in this case pvc-f60162e2-a98f-4e74-9deb-097ca56127bd).

 

To identify the PVC, open the corresponding VM in OCP. In our case the VM is named “centos-stream9-test-3”

Virtual machine detailsVirtual machine details Then select “Configuration” and view the “Storage” details

Storage detailsStorage details The root disk is called “dv-centos-stream9-test-3-rootdisk-5r39h6”. Open the PVC details

Identify PVC of root diskIdentify PVC of root disk

In our case the PVC name is “pvc-f60162e2-a98f-4e74-9deb-097ca56127bd”

Shutdown the VM by pressing the “Stop” icon

Virtual machine power optionsVirtual machine power optionsImportant to know: From an NetApp ONTAP point of view, a LUN (independant if it is a iSCSI or FC LUN) is a file which resides inside a volume. This file is then presented as a block device either via iSCSI or FC protocol.

 

All NetApp Snapshot, SnapRestore and FlexClone operations are always done on Volume level. In NetApp System Manager we therefore have to identify the correct volume which stores the iSCSI LUN.

According to the YAML Trident Secret, Backend und (Default) Storage Class Configuration, the volume name has the prefix “trident_” so the correct volume name is “trident_pvc_f60162e2_a98f_4e74_9deb_097ca56127bd”, because “-” from PVC names are translated into “_” in ONTAP volume names. System Manager shows that a LUN with name “lun0” is stored on the volume “trident_pvc_f60162e2_a98f_4e74_9deb_097ca56127bd”.

iSCSI LUNs in NetApp System ManageriSCSI LUNs in NetApp System Manager

Select the volume from the “Volumes” menu and choose “Clone” from the Kebab menu.

Clone SubmenuClone Submenu If no Snapshot exists from the volume, or if you want to create a more up to date Snapshot choose “Add a snapshot”. If already a Snapshot exists, choose an existing Snapshot, provide a name for the clone volume and press "Clone"

Clone detailsClone details 

In our environment we need to rehost the newly cloned volume, because

  • Kubernetes and container data created via Trident are deployed on a dedicated SVM
  • FC boot in our case takes place in a different SVM

 To achieve this, you must do the following steps:

  • Clone Split
    Select the menu of the cloned volume and press “Split clone”
    Split Clone menuSplit Clone menu
    It is safe to select “Delete snapshots and split the clone” since no Snapshots exist on the newly created clone. Then press “Split”
    Split Clone confirmationSplit Clone confirmation
    Watch the progress bar
    Split Clone progressSplit Clone progress
  • Volume Rehost (CLI)
    • Log in to the Cluster as admin and rehost the volume
      grenada::> volume rehost -volume trident_clone_for_physical_server_boot -vserver svm-osbmc1 -destination-vserver svm-grenada-san

      Warning: Rehosting a volume from one Vserver to another Vserver does not change the security information about that volume.
               If the security domains of the Vservers are not identical, unwanted access might be permitted, and desired access might be denied. An attempt to rehost a volume will disassociate the volume
               from all volume policies and policy rules. The volume must be reconfigured after a successful or unsuccessful rehost operation.
      Do you want to continue? {y|n}: y
      [Job 621557] Job succeeded: Successful

      Info: volume is rehosted on the target Vserver.
      Set the desired volume configuration - such as the export policy and QoS policy - on the target Vserver.
  •  Map the newly created LUN as boot LUN to the physical server
    • If a boot LUN has already been assigned to the physical server via FC protocol, this assignment must first be unmapped
    • Map the newly created LUN as Boot LUN
      Choose the new Boot LUN and press “Edit”
      LUN mappingLUN mapping

       


      Mark the check box which maps the Boot LUN to the initiator group of the physical server
      LUN - Server mappingLUN - Server mapping
  • Turn on the server and watch the console to ensure the server boots from the new Boot LUN
    Bare metal server boot processBare metal server boot process
  • On our server with a Broadcom FC card, the new boot LUN is automatically detected. However, in other environments, you may need to enter the BIOS again to specify the new boot LUN.
    The new LUN is bootableThe new LUN is bootable
    Wait until the operating system has been booted
    Successfull startupSuccessfull startup
  • Adjust network configuration and mount application volumes via NFS protocol
    Edit the network connection of the physical server, because on the physical server, the NICs got new device names (i.e. eth2 changed to ens49f0np0). Assign the same IP addresses which you used on your virtual machine before, because on NetApp systems volume access is typically restricted to single IP addresses only. You can use     nmtui to configure the network interface.
     

    Network setupNetwork setup

    After you have edited the configuration of your network device, you can do a “mount -a”, because your /etc/fstab and hence filesystem configuration has been taken over when cloning the iSCSI LUN.
     

    Mount application filesystemsMount application filesystems

    Then you should see your application filesystems and you can start your application on your spare physical server and continue with error analysis or performance tests.

Conclusion

The here presented method shows how to boot a VM running on Red Hat OpenShift Virtualization as a bare metal system. This can be useful to debug and investigate potential issues found on that system. It especially helps clarify the question, whether the issue is related to Red Hat OpenShift Virtualization or not.

Appendix

Example Configuration Files

Trident Secret, Backend and (Default) Storage Class Configuration

Trident Secret

---

apiVersion: v1

kind: Secret

metadata:

  name: <secret name>

  namespace: <trident namespace>

type: Opaque

stringData:

  username: <username>

  password: <password>

Backend

---

apiVersion: trident.netapp.io/v1

kind: TridentBackendConfig

metadata:

  name: <backend name>

  namespace: <trident namespace>

spec:

  version: 1

  storageDriverName: ontap-san

  managementLIF: <SVM management LIF IP>

  svm: "<SVM name>"

  backendName: <backend name>

  igroupName: <igroup name>

  credentials:

    name: <secret name>

  storagePrefix: trident_

  defaults:

    snapshotPolicy: default-1weekly

    snapshotReserve: '20'

(Default) Storage Class Configuration

---

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

  name: trident-iscsi

  annotations:

    storageclass.kubernetes.io/is-default-class: "true"

    storageclass.kubevirt.io/is-default-virt-class: 'true'

provisioner: csi.trident.netapp.io

parameters:

  backendType: ontap-san

  csi.storage.k8s.io/fstype:: "ext4"

volumeBindingMode: Immediate

allowVolumeExpansion: true

reclaimPolicy: Delete

Authors

Holger Zecha (NetApp)
Nils Bauer (NetApp)
Prena Mohta (Red Hat) Doc Writer
Nils Koenig(Red Hat)

0 Kudos
All Community Forums
Public