Most ONTAP estates are watched two ways today. Active IQ Unified Manager gives the overall view, and it has done that job faithfully for a long time. But it is legacy now. It has no agentic capabilities, and it does not cover the newer platforms teams are buying—an ASA R2 controller simply does not appear there. Everything else happens in System Manager, which is a genuinely good interface for one cluster at a time. That is exactly the limitation: one cluster at a time.

So the estate-level question—which system needs me first?—either goes to a tool that cannot see all of it, or gets answered by opening System Manager repeatedly and holding the comparison in your head. And once you find the problem, seeing it and fixing it are still two different places.

That is the gap the NetApp Console local deployment closes.

Console local brings fleet management, observability and day-to-day operations into an on-premises control plane. That last point matters. In a regulated environment, a sovereign data center or a dark site, "send the telemetry to a SaaS service" may simply not be an option. Console local keeps the operational experience local while still giving teams a fleet-wide view.

Start with the fleet, not the cluster

Console local starts at the level the question is usually asked. It gives an operations team a common view across the ONTAP fleet—systems, capacity, performance and health—including the newer platforms that legacy monitoring never learned to see. The estate can be grouped the way the business is actually organised, so a fleet health view can show Dev, Finance, APAC and Prod side by side, each with its systems, storage VMs and volumes, used capacity, security compliance and open critical alerts. From there, an administrator can move from an estate-level signal down to a cluster, workload or volume without starting the investigation over somewhere else.

This sounds like a small change in navigation. In practice, it changes the morning routine. Instead of working through clusters one at a time, an administrator can answer a few basic questions in one place:

• Where is capacity growing faster than expected?
• Which systems are carrying active critical alerts?
• Is a latency problem isolated to one workload or visible across the fleet?
• Which issue should the team work on first?

Underneath, Console local draws on the full depth of NetApp Harvest metrics—so this is not a thinner view of the estate than administrators are used to. It is the same class of data, presented in the environment where the fleet is actually managed.

Dashboards that answer the team's questions

A fixed overview is useful, but an infrastructure lead and a storage administrator rarely need the same view.

Custom dashboards let teams arrange the signals that matter to them: capacity by cluster, latency trends, top workloads by IOPS, protection status and recent alerts, for example. Multiple visualisations make it possible to look at the same estate from different angles without exporting data into a spreadsheet first.

The first three charts—average latency, total throughput and total IOPS—also show the top 5 volumes. They are drawn as a single line, so a per-volume legend is not shown.

The practical value is not the chart itself. It is the ability to notice a relationship. A capacity warning beside a rising workload trend tells a more useful story than either signal in isolation. A protection panel beside recent infrastructure alerts helps an operator understand whether an incident is merely noisy or has started to put recovery objectives at risk.

For a deeper investigation, the Workload Analyzer narrows the view to an individual volume. Its guided flow brings together the event timeline, latency, throughput, resource utilisation, capacity and data protection. That gives an administrator a sensible path through a performance case instead of a blank query box and a wall of metrics.

Alerts with enough context to be useful

Most infrastructure products can produce alerts. The real test is whether an alert helps someone decide what to do next.

Console local provides a fleet-wide alert view with search, filtering and detailed information about the affected resource. Alerts follow a consistent model: three severities—Critical, Warning and Info—and six impact areas covering Availability, Capacity, Performance, Protection, Security and Configuration. That consistency becomes important as more services contribute signals to the same console.

Notifications also stay on premises. Teams can route them through email, webhooks or the Console notification bell, including in environments without internet connectivity. Rules can be tailored so the right group sees the right events rather than receiving every signal produced by the estate.

There is an equally important detail behind the scenes: alerts are expected to be edge-triggered, so a condition raises an alert when it becomes true rather than on every evaluation cycle. Services are also encouraged to resolve alerts automatically when the underlying condition clears. Both choices help keep the alert list closer to an operational queue and further away from a historical dumping ground.

From "something is wrong" to "here is a safe next step"

This is where the NetApp Console local deployment becomes more than a monitoring console.

Where a condition has a supported fix, Console local surfaces it as a recommended remediation rather than leaving the operator to go elsewhere and apply it by hand.

Choosing Fix it opens the action with the context needed to judge it: what Console local proposes to change, the current state of the object, the constraints that affect the result, and whatever input the operator must supply. The equivalent CLI command sits alongside it, for anyone who would rather run the change themselves or fold it into automation.

The panel is built around getting the decision right before it is applied. The initial set is deliberately narrow, covering common volume-management actions, and an auto-remediation setting exists for teams ready to let recommended fixes run on their own.

The loop also removes a hand-off. Nobody has to copy an object name into another console, rebuild the context there, and hope they are changing the right thing.

The bigger shift

All of this runs inside the customer's own environment. The control plane, the metrics, the alert history and the notification path stay on premises, which is what makes the NetApp Console local deployment workable for air-gapped sites, regulated organisations and anyone operating under data-sovereignty rules. Monitoring stops being one more thing that depends on an internet connection.

Running locally no longer means accepting less. An on-premises deployment gives fleet-wide monitoring, role-based access and local notification routing, and it covers the platforms an estate is buying today rather than the ones it bought a decade ago. It connects those metrics to inventory, alerts, workload context and management actions, so the work moves in one line: start with the fleet, narrow to the resource, understand the risk and—where a supported remediation exists—fix it without leaving the console.

The most useful observability platform is not the one with the most charts. It is the one that shortens the distance between a signal and a safe decision. That is the direction the NetApp Console local deployment is taking: from monitoring storage to operating the fleet.

Call to Action

If Console local is new to you, start with the launch announcement: Rethinking On-premises Storage Management at Scale. It covers what the NetApp Console local deployment is, why we built it, and everything it does beyond observability—fleet organization, storage classes and the Console AI assistant.

When you are ready to see it against your own estate:

• Download the NetApp Console local deployment OVA from the NetApp Support site and deploy it into a non-production VMware environment.
• Add a handful of clusters and group them the way your business is actually organised. One site and a named admin is enough to see whether the fleet view earns its place.
• Take one alert your team genuinely cares about and follow it all the way through to a recommended remediation.
• Talk to your NetApp account team about a fleet workshop or a proof of concept.

The fastest way to judge any observability tool is to give it a problem you already have and see how far it gets you. With Console local, that is a two-week exercise—not a two-quarter one.

Jointly authored by Iddo Avneri at lakeFS and Sathish Thyagarajan at NetApp

A Fortune 500 pharmaceutical company recently answered a question many organizations are asking: How do you modernize enterprise file data for AI without disrupting the storage platform you already trust?

The answer started with years of critical data and established workflows built around file systems. Linux users mounted NFS. Windows users expected SMB and Windows Explorer. Access controls were tied to Active Directory. When the company moved this environment from on-premises ONTAP to AWS, asking every team to adopt a new storage model wasn't a realistic migration plan.

At the same time, its analytics and AI stack increasingly expected S3. The company selected Amazon FSx for NetApp ONTAP because it could preserve the file access its users depended on while creating a path for S3-based applications. Amazon S3 access points for FSx for ONTAP can present file data through an S3 endpoint while that data remains available through NFS and SMB.

That solved the storage and protocol problem. It didn't solve what happened when multiple teams needed to change, validate, and publish the data feeding analytics and AI.

The missing layer above storage

Consider a data science team preparing clinical trial data for a new analysis. One engineer corrects source records. Another changes a transformation. A third adds derived features. Every change may be reasonable on its own, but nobody wants partially tested work appearing in a shared production dataset. The usual workaround is duplication. Teams create new directories, new buckets, or files with names such as trial-data-final-v2-reviewed. Those copies consume storage, drift from one another, and make it difficult to reconstruct the exact dataset behind a result. Storage snapshots remain important for infrastructure protection. Object-level versioning also has a role. But AI and data engineering teams need to manage a dataset as a coherent unit: isolate a change, test it, inspect the difference, and publish it only after approval.

lakeFS is the control plane for AI-ready data. It adds that workflow above the storage layer, sitting between data storage and the tools, users, and AI agents that consume the data. A team can create a zero-copy branch of a dataset, make changes without affecting production, commit an immutable version, compare it with the current production state, and merge it when the work is ready. The data stays on the customer's storage. The workflow changes, not the storage foundation.

A working reference environment on Amazon FSx for NetApp ONTAP

Customers can use the public lakeFS + NetApp ONTAP reference environment as a practical starting point for this architecture. It shows how lakeFS and Amazon FSx for NetApp ONTAP work together on AWS, with a deployable environment and a concrete data workflow to run against it.

The reference environment uses Amazon FSx for NetApp ONTAP, with a native ONTAP S3 object-store server and bucket configured inside the file system. lakeFS runs on an Amazon EC2 instance and uses the ONTAP S3 bucket as its backing storage. Every object written through lakeFS is physically stored on FSx for ONTAP.

The included workflow follows a simple ML feature-engineering example:

1. A baseline training dataset was committed to the main branch.
2. A data scientist, pipeline, or AI agent created an isolated branch for a proposed feature.
3. The updated dataset was written and committed on that branch without changing the shared version.
4. The team inspected the exact difference between the branch and main.
5. After validation, the branch was merged into main with its history intact.

No full copy of the original dataset was required when the branch was created. The branch initially referenced the existing data and stored only subsequent changes. The repository is a working reference, not a performance benchmark. It gives customers a repeatable way to use native ONTAP S3 on Amazon FSx for NetApp ONTAP as the storage foundation for lakeFS, with branching, commits, diffs, merges, and complete version history available to the data team. For an AI agent, the branch is an isolated data sandbox. The agent can start from an approved dataset version, write proposed outputs, and submit them to the same validation and review process used for human and pipeline changes. The input and output commits give the run exact data references, so it can be reproduced, audited, or debugged later.

One storage foundation, with clear responsibilities

The products solve different problems. NetApp ONTAP handles how data is stored, protected, and accessed. It supports the enterprise file protocols that many organizations already depend on, while also supporting S3-based workloads. On AWS, FSx for ONTAP gives customers a managed ONTAP environment without forcing established file users to redesign their workflows. lakeFS handles how AI and data teams change data. It provides isolation for development and testing, reproducible dataset versions, automated validation points, controlled promotion, rollback, and an audit trail of what changed. Those controls apply whether the change comes from a person, a pipeline, or an AI agent.

That division is useful inside large enterprises. Storage and infrastructure teams can keep the controls and operating model they trust. Data scientists and engineers get a workflow designed for experimentation and review. The organization doesn't need to maintain one storage estate for traditional applications and another simply because newer applications speak S3. Not every AI or analytics application is designed to work directly with S3. lakeFS Mount exposes a lakeFS branch through a local filesystem interface, so existing applications and AI agents can read and write data using familiar file APIs. Their changes remain isolated in the branch and can be validated, committed, and merged through lakeFS before reaching the shared version. FSx for ONTAP can support established file workloads and new data engineering and AI projects on the same storage foundation. lakeFS brings modern data workflows to that foundation.

Native ONTAP S3 and Amazon S3 access points

There are two related deployment patterns worth distinguishing. The reference environment uses a native ONTAP S3 bucket running on Amazon FSx for NetApp ONTAP. This is a good fit for a new lakeFS-managed namespace stored directly on ONTAP. Amazon S3 access points for FSx for ONTAP address a different requirement. They attach an S3 endpoint to an FSx for ONTAP volume, allowing S3 applications and AWS services to access file data without moving it out of the file system. NFS and SMB access can continue alongside the S3 path.

The right pattern depends on the workload. A team building a new governed data environment may choose native ONTAP S3 as the lakeFS backing store. A team with an existing file-based data estate may use an S3 access point to make that data available to S3 applications.

In either case, ownership boundaries need to be explicit. A namespace managed by lakeFS should be changed through lakeFS so its commit history remains authoritative. Multiprotocol support should not become a way to bypass validation or write directly around the versioned workflow.

What the pharmaceutical use case looks like in practice

For the pharmaceutical company, the attraction wasn't another storage endpoint. It was the ability to support conservative, file-based workflows and newer S3 applications on the same NetApp foundation. That creates a practical model for clinical and research data:

• Existing applications and users can retain the NFS or SMB access patterns they require.
• New analytics and AI workloads can access the storage through an S3-based path on AWS.
• lakeFS-managed datasets can be branched for experiments or transformations without copying the complete source.
• People, pipelines, and AI agents can work in isolated branches instead of writing directly to the shared version. Workloads that expect a filesystem can access those branches through lakeFS Mount.
• Data quality and policy checks can run before changes are merged into the shared version.
• Every committed data state has an immutable identifier and full history, so experiments, analyses, and agent runs can be tied to exact input and output data versions.

A researcher can return to the exact data used for an earlier analysis. A data engineer can show which records or files changed before approving a merge. An AI agent can propose a transformed dataset without changing the approved version. A platform team can give people, pipelines, and agents isolated workspaces without creating another uncontrolled collection of copied datasets. These are ordinary requirements in regulated data environments. The hard part has been satisfying them without disrupting the applications, access methods, and storage controls already in place.

A more practical cloud modernization path

Cloud migrations are often described as clean breaks: move the data, replace the interfaces, and retrain the users. Large data environments rarely cooperate with that plan. The pharmaceutical company's experience points to a more practical approach. Keep the storage model that critical users and applications depend on. Add S3 access where modern services require it. Then add lakeFS where people, pipelines, and AI agents need isolation, reproducibility, review, and governance across dataset changes. NetApp carries the established data estate into AWS. lakeFS gives new pipelines, experiments, and agent runs a controlled way to work with it. That may sound less dramatic than a clean-slate architecture. For a regulated enterprise, that's usually the point. Teams can modernize the data workflow without treating every established access pattern as technical debt.

Explore the solution architecture

lakeFS + NetApp ONTAP reference environment

Amazon S3 access points for Amazon FSx for NetApp ONTAP

NetApp ONTAP S3 multiprotocol support

lakeFS - the Control Plane for AI Ready Data

thanks

The core content of the exam remains unchanged and continues to cover the areas outlined below:

• Storage Platforms
  
• Core ONTAP
  
• ONTAP Storage
  
• Networking
• Storage Protocols and Connectivity
• Data Protection
  
• Security
• Performance
  

Plugin Output:

 Path              : /data/netapp-manageability-sdk-9.8P10/zedi/zexplore.jar

 Installed version : 2.19.0

 Fixed version     : 2.25.5

 Path              : /data/netapp-manageability-sdk-9.8P10/lib/java/classes/manageontap-9.8P10.jar

 Installed version : 2.19.0

 Fixed version     : 2.25.5

Description:

The version of Apache Log4j on the remote host is affected by a vulnerability:

• Improper encoding of non-finite floating-point values during MapMessage JSON serialization in Apache Log4j API produces output that is not valid JSON. When a MapMessage contains a non-finite IEEE 754 value (NaN, Infinity, or -Infinity), MapMessage.asJson() emits the corresponding bare token. RFC 8259 does not permit these tokens, so a conformant parser rejects the resulting document. An attacker who can supply a non-finite value can cause the affected layout to emit malformed JSON, which may corrupt the enclosing log record or disrupt downstream log ingestion and parsing. (CVE-2026-49844)

Note that Nessus has not tested this issue but has instead relied only on the application's self-reported version number.

Can someone please provide more details to remediate this issue and way to fix whether we should upgrade SDk to 9.8P12 and it iwll include latest updated .jar file?

The provisioning itself takes minutes. What takes longer is deciding where it should go: which cluster, which aggregate or SVM, whether it meets the storage class her architect defined, and whether anything else in the fleet would be a better fit. That decision means jumping between interfaces and drawing on experience, not because any one tool has failed her, but because no layer yet brings fleet context into the everyday work of running storage.

Sound familiar? This is the daily reality for teams managing on-premises ONTAP at scale, not because the storage isn't capable, but because the management experience has been fragmented within the estate. That gap is exactly why we built NetApp Console™ local deployment.

We're announcing the general availability of NetApp Console local deployment, a locally deployed, AI-driven autonomous control plane that brings fleet-scale management to your on-premises ONTAP estate for the first time as a unified Console experience.

This is a significant milestone for NetApp and for every customer running ONTAP at scale. Console local deployment delivers the management layer our on-premises customers have been asking for, fleet organization, unified observability, policy-governed provisioning through storage classes, and an AI-powered Console Assistant, all running inside your environment, on your terms. No constant cloud connection required. The same intelligent experience as NetApp Console in the cloud. One solution. Two delivery models.

Validated through our Early Access Program with customers across industries, Console local deployment is ready to change how teams organize, monitor, and act across their storage estates. Whether you manage ten systems or a hundred, this is the next chapter in how NetApp helps you take complete control of the NetApp Platform, locally, securely, and at fleet scale.

When Growth Outruns Your Tools

ONTAP was built to scale and your environment proves it. More systems, more sites, more workloads running on a platform you trust, supported by NetApp management tools that excel at the cluster and system level. Console local deployment extends that foundation with a fleet-scale layer designed for the estate you have today, and the one you're building toward.

For storage engineers, that means everyday work, such as provisioning, placement, troubleshooting, happens with fleet-wide context at your fingertips, so decisions are faster and better informed. For architects, it means the standards you define through storage classes and policies, reach every cluster consistently, without losing the depth of system-level management. For executives, it means a single, locally deployed view of capacity, health, and risk across your on-premises environment, built on the unified control plane and tools you already rely on.

Console local deployment doesn't change what ONTAP does. It unlocks what your team can do with it at scale.

A Different Kind of Control Plane

Console local deployment is designed to meet your storage management where you operate.

It runs locally, deployed as an OVA in your VMware environment. No constant cloud connection required. That matters for air-gapped sites, regulated industries, and any organization that needs management to stay inside its own infrastructure. Yet it delivers the same intelligent experience as Console in the cloud. One console, two delivery models, no retraining.

Think of it less as another tool and more as an autonomous control plane, a single place to organize your estate, understand what's happening across it, and act when something needs attention.

Organize the Estate, Not Just the Clusters

The story usually starts with fleet management. Instead of thinking cluster by cluster, teams begin grouping systems the way the business actually thinks, by region, by workload, by tenant, by business unit. A manufacturing site in Munich and a DR pair in Dublin can live in the same fleet. A system can belong to multiple fleets without being rediscovered or reconfigured.

For architects, this is where design intent starts to stick. Storage classes let senior admins define what "good" looks like, including performance, capacity, security, data protection, once. From that point on, every provisioning action, whether manual or AI-assisted, follows those rules. Engineers stop reinventing decisions they've made a hundred times. Executives get something they've wanted for years: consistency at scale.

See What's Happening Before It Becomes a Problem

Visibility comes next and for many teams, this is the moment things click.

Instead of piecing together health from scattered sources, Console local deployment brings monitoring, alerting, and analysis into the same place you manage the estate. Dashboards show what matters to your role.

When something looks off, the workload analyzer helps you drill down without hours of manual correlation. Alerts arrive with context. The audit trail is already there when compliance asks who changed what.

Intelligence That Respects Your Boundaries

The newest chapter in this story is the Console AI assisted ChatOps, and we know the first question is always the same: Is it safe?

We built it to be. Ask a question in plain language, what's wrong with a cluster, where to place a new volume, which storage class fits a workload, and the ChatOps responds in read-only mode by default. It can analyze and recommend without changing a thing. When you're ready for it to act, you switch modes explicitly. It uses your LLM, not ours. PromptGuard screens content before it reaches any AI. Every action respects the same RBAC permissions you already enforce. Everything is logged.

For engineers, that means less time on repetitive requests and faster answers when something breaks. For architects, it means AI that works within policy, not around it. For executives, it means the speed of intelligent automation without the compliance anxiety that usually comes with it.

Why This Moment Matters

The timing isn't accidental. Estates are growing while teams stay flat. Cloud providers have raised expectations for storage experiences, and on-premises environments need to keep up. Console local deployment brings it together: organize your estate, understand it, act on it, from one place that runs on your terms.

Call to action

If you're an executive, the conversation is about reducing risk, accelerating application delivery, and consolidating a sprawl of tools into one strategic platform. If you're an architect, it's about making standards real across a growing environment. If you're an engineer, it's about getting your Tuesday mornings back. Download NetApp Console local deployment from the NSS site now and unbox the opportunity to manage your Storage estate in the most advanced and smart way.

Talk to your NetApp team about a fleet workshop or proof of concept. See what it looks like when your on-premises estate is managed as one — not as fifty separate problems.

Take complete control of the NetApp Platform with Console autonomous operations.

Declare outcomes. Automate operations. Secure every action.

Excited to learn more? Read these blogs: Empower autonomous operations across your storage estate now and NetApp Console Autonomous Operations: From Managing Infrastructure to Declaring Outcomes

Want to kick the tires? Try guided demos, see videos, and access additional resources thru the Console Interactive Showcase.

Want to get educated? Take the Introduction to NetApp Console and NetApp Console Fundamentals courses

Ready to get started? Login into NetApp Console in SaaS.

That shift changes what customers should expect from infrastructure management. Traditional dashboards and per-array tools are no longer enough. IT teams need fleet-wide observability, predictive insight, governed automation, and secure autonomous action across the full estate. Storage management is moving through the same kind of evolution that compute observability has gone through over the last decade: from reactive monitoring to proactive insights, to AI-assisted operations, and now toward autonomous execution inside defined guardrails.

This is where NetApp Console comes in.

NetApp Console is the autonomous control plane for the NetApp Platform, delivering unified manageability, observability, agentic operations, and autonomous optimization across storage and data services. It is designed to help teams take complete control of the Platform through one secure, intelligent interface, across hybrid multicloud environments and across connected, restricted, and disconnected deployment models.

The autonomous estate era

We are entering the autonomous estate era. In this model, humans do not spend their time manually coordinating every step across tools, tickets, scripts, and clusters. Instead, humans declare the desired operational outcome, and autonomous operations help keep the estate aligned to that intent.

That distinction matters.

In a reactive model, the administrator does almost everything: detect the issue, interpret the signal, identify the remediation path, and execute the change. In a proactive model, telemetry and predictive alerts help the administrator understand what is likely to happen next, but the human still interprets and responds. In an autonomous model, the system can recommend or execute actions inside guardrails, but the human still approves. In the autonomous estate model, the human declares the operational outcome, with specific and clear guardrails, and named capabilities help hold that outcome true—and prove that they did.

For storage and data infrastructure, that means shifting from “manage this array” to “keep this fleet compliant,” “place this workload according to policy,” “optimize capacity before it becomes a constraint,” or “identify and remediate operational risk before it becomes downtime.” The goal is not automation for automation’s sake. The goal is governed, outcome-driven operations at fleet scale.

Why this matters now

Customer environments are becoming more distributed, more regulated, and more operationally complex. The attached messaging toolkit identifies three consistent pressure points: siloed management, operations that do not scale, and sovereignty gaps created by cloud-only operational models.

Siloed management creates fragmented visibility. Each tool owns a slice of the truth, but no single tool owns the whole operational picture. When teams rely on separate interfaces for system management, grid management, reporting, analytics, and data services, day-to-day operations become slower and more error-prone.

Fleet scale creates another challenge. Customers operating 50, 100, or 500 clusters cannot rely on per-cluster workflows for routine operations such as provisioning, upgrades, capacity management, policy enforcement, and health monitoring. Manual workflows that are acceptable for a few systems become operational gridlock when multiplied across hundreds of environments.

Sovereignty adds a third dimension. Regulated, air-gapped, and dark-site environments often cannot depend on cloud-only AIOps models. These estates still need modern observability, automation, AI-assisted operations, and governed remediation, but they require those capabilities to operate within strict security, compliance, and connectivity boundaries.

NetApp Console is designed to address these realities with one autonomous control plane for fleet operations.

What Console enables

At a technical level, Console brings together several operational capabilities that are increasingly essential for modern infrastructure teams.

First, it provides fleet-wide intelligent management. Console is designed to help teams operate distributed storage environments as one coordinated estate, with centralized oversight across storage and data services. The control plane supports intelligent fleet operations, centralized management, predictive analytics, and optimization across hybrid multicloud environments.

Second, it supports continuous observability and operational foresight. Rather than relying only on after-the-fact alerts or manual troubleshooting, Console emphasizes predictive analytics, health dashboards, alerts, workload analysis, and intelligent incident resolution. This helps teams move from reactive firefighting to proactive optimization.

Third, Console enables intent-based, policy-driven operations. Infrastructure management becomes more focused on declared intent: placement, provisioning, guardrails, compliance monitoring, and remediation workflows. This is foundational to autonomous operations because it allows teams to define the desired outcome and let the control plane and AI-driven ChatOps guide or execute the steps required to maintain that state.

Fourth, Console is built for operational productivity. Guided onboarding, automated system discovery, intuitive workflows, home page visibility, subscription and license monitoring, and consolidated administration all reduce the manual effort required to manage complex estates.

Finally, autonomous operations require trust, and trust requires security. Console embeds security into the control plane with identity-based access, least-privilege controls, secure credential management, REST API governance, compliance monitoring, anomaly detection, audit support, and customer-controlled support access. It also enables secure AI operations, allowing customers to bring their own LLMs, enabling a conversational ChatOps interface through an open LLM gateway to the models that organizations already trust. For autonomous operations to be useful in enterprise environments, every action must be authenticated, scoped, governed, and auditable.

One solution, multiple deployment realities

A key part of the Console strategy is deployment flexibility. The same autonomous control plane must support different operational realities, including cloud-connected SaaS environments and local deployments for air-gapped, dark-site, and sovereign environments. Console is one solution with two delivery models: Console in SaaS and Console local deployment.

Console in SaaS provides cloud-connected, NetApp-hosted experience with support for on-premises storage, hyperscaler and self-managed storage, and data services. Console local deployment is designed for environments that require local installation, no SaaS dependency on the critical path, full data sovereignty, and support for highly secure or disconnected operations.

This matters because autonomous operations cannot be limited only to cloud-connected estates. Some of the organizations that need operational automation the most, such as government, federal, defense, financial, healthcare, and regulated industries, also have the strictest requirements for isolation, sovereignty, and control.

The path forward

Console represents a shift from fragmented infrastructure administration to outcome-driven fleet operations. It gives teams one autonomous control plane for managing storage and data services, with the observability, automation, governance, and security required to operate at enterprise scale.

The message is simple: Declare outcomes. Automate operations. Secure every action.

As hybrid multicloud estates continue to expand, the winning operating model will not be more tools, more dashboards, or more manual coordination. It will be autonomous operations that allow humans to set intent, apply policy, preserve control, and trust the platform to help keep the estate aligned to the outcomes the business requires.

Take complete control of the NetApp Platform with Console autonomous operations.

Declare outcomes. Automate operations. Secure every action.

Excited to learn more? Read these blogs: Empower autonomous operations across your storage estate now and Rethinking On-premises Storage Management at Scale

Want to kick the tires? Try guided demos, see videos, and access additional resources thru the Console Interactive Showcase.

Want to get educated? Take the Introduction to NetApp Console and NetApp Console Fundamentals courses

Ready to get started? Login into NetApp Console in SaaS.

Why ONTAP-mode? 

While Flex Unified default mode provides excellent dynamic storage provisioning and management capabilities, many enterprise workloads demand deeper data management capabilities only available with GCNV Flex Unified ONTAP-mode. With the introduction of the Trident integration with ONTAP-mode, Trident can now interface with Google Cloud NetApp Volumes storage pools created with the ONTAP-mode option, and brings additional Trident features to your Kubernetes applications, including:

• Custom file names: Enhanced security and compliance auditing.
• SnapMirror Integration: Facilitating disaster recovery with volume replication.
• Clone across namespaces: Enabling consistent snapshots across multiple volumes.
• Volume Group snapshots: Enables ontap-mode consistency groups
• NFS v4.2 Support: Bringing updated protocol support for your workloads.

How to Set It Up

To get started with ONTAP-mode, ensure you are using Trident version 26.06.1 or later. This example is created on GKE using Trident Cloud Identity, although other Kubernetes distributions may be used with a secret. 

Step 1 - Create your GCNV Storage Pool

Ensure your Google Cloud NetApp Volumes storage pool is created with the ontap-mode option. Note that the storage pool must be created outside of Trident (e.g., via the Google Cloud console or gcloud CLI). A sample on the console is shown below.

The pool will come up.

Step 2. Create a GKE cluster and  Install Trident

In order to use cloud identity with Trident, you must first create a GCP service account with Google Cloud NetApp Volumes Admin role.  This service account will later be linked to the trident-controller kubernetes service account. You must also enable workload identity when creating your standard GKE cluster. More information on how to set this up is in Deploying cloud identity with Trident, GKE, and Google Cloud NetApp Volumes steps 1 and 2.

Install Trident using cloud identity. An example using helm is shown below. Replace <service account> and <project name> with your values.

~$helm install trident netapp-trident/trident-operator --version 100.2606.1 --create-namespace --namespace trident --set cloudProvider="GCP" --set cloudIdentity="'iam.gke.io/gcp-service-account: <service account>@<project name>.iam.gserviceaccount.com'"

Check to be sure Trident is up and running:

~$kubectl get pods -n trident
NAME                                  READY   STATUS    RESTARTS   AGE
trident-controller-67f7cd9bf8-2ffl5   6/6     Running   0          3m
trident-node-linux-2r82s              2/2     Running   0          2m59s
trident-node-linux-f8gnh              2/2     Running   0          2m59s
trident-node-linux-jwqzn              2/2     Running   0          2m59s
trident-operator-5cbf7f857-ktj2n      1/1     Running   0          3m26s

After the trident pods are up and running, you must bind the GCP service account to the trident controller, and annotate the trident controller service account. More information can be found in Deploying cloud identity with Trident, GKE, and Google Cloud NetApp Volumes Steps 4 and 5.

Step 3. Create a secret.

If you are using Trident Cloud Identity with GKE, you must create an empty secret as shown below.

empty_secret.yaml

 If you are not using Cloud Identity, the secret must contain credentials to create volumes on GCNV. Obtain a key from a GCP service account that has GoogleCloud NetApp Volumes Admin role assigned to it and information from the key is entered into the secret. A sample secret is located in the Trident documentation.

Apply the secret.

~$kubectl create -f empty_secret.yaml
secret/gcnv-ontap-empty created

~$kubectl get secret -n trident
NAME                            TYPE                 DATA   AGE
gcnv-ontap-empty                Opaque               0      8s
sh.helm.release.v1.trident.v1   helm.sh/release.v1   1      5m12s
trident-csi                     Opaque               6      4m45s
trident-encryption-keys         Opaque               1      4m45s

Step 4. Configure and apply the Trident Backend:

Create a TridentBackendConfig manifest. You will use the ontap-nas or ontap-san drivers depending on if you need file or block volumes. In this example, we will use ontap-nas to create an NFS volume in us-central1-a. Change the poolID to the name of the pool.

backend.yaml

apiVersion: trident.netapp.io/v1
kind: TridentBackendConfig
metadata:
  name: tbc-gcnv-ontap
  namespace: trident
spec:
  version: 1
  storageDriverName: ontap-nas # Or san, depending on your needs
  backendName: ontap-mode-backend
  useREST: true
  credentials:
     name: gcnv-ontap-empty
     type: secret
  gcnv:
     proxyURL: "https://netapp.googleapis.com"
     projectNumber: '<insert-project-number>'
     location: us-central1-a
     poolID: ontap-mode-pool 

~$kubectl create -f  backend.yaml -n trident 
tridentbackendconfig.trident.netapp.io/tbc-gcnv-ontap created

Step 5. Create Storage Classes:

Define a Kubernetes StorageClass that points to your new Trident backend. The backendType is the trident driver, ontap-nas.  If you are using san, it would be ontap-san. We use nas as an example below.

scontap.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gcnv-flex-k8s-nfs-ontap
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-nas" 
  trident.netapp.io/nasType: "nfs"
allowVolumeExpansion: true

~$kubectl apply -f scontap.yaml

storageclass.storage.k8s.io/gcnv-flex-k8s-nfs-ontap created

Step 6. Deploy PVC and Applications:

Deploy your applications using Persistent Volume Claims (PVCs) that reference this storage class. Trident will automatically handle the provisioning of volumes within your ontap-mode storage pool.

pvcnfs.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-nas 
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: gcnv-flex-k8s-nfs-ontap

~$kubectl create -f pvcnfs.yaml

persistentvolumeclaim/pvc-nas created

~$kubectl get pvc

NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS  VOLUMEATTRIBUTESCLASS   AGE

pvc-nas   Bound    pvc-8fc7d774-789a-4b00-bb01-59e57e96ef2a   10Gi       RWX            gcnv-flex-k8s-nfs-ontap   <unset>                 13s

You can then attach your application pods to the PVC for persistent storage.

What now?

The integration of ONTAP-mode into NetApp Trident brings enterprise-grade data management directly into your Kubernetes environments on Google Cloud NetApp Volumes. By bridging these capabilities, you can now leverage extra ONTAP-mode features in Kubernetes, all while maintaining the simplicity of dynamic storage provisioning within your Kubernetes clusters. This preview release marks a significiant step forward for workloads requiring robust, native ONTAP configuration and data control within the Google Cloud ecosystem. Try it out today!

Let’s get into it.

Granular Datastore Protection: Protect Only What Needs Protecting

The headline feature in 10.6 is Granular Datastore Protection (GDP), which shifts both SnapMirror active sync (SMas) and most traditional SnapMirror replication policies from only protecting entire host clusters down to individual VMFS datastores. Instead of protecting every datastore in a cluster whether it needs it or not, vCenter administrators can now choose exactly which datastores belong in a protection group — and add, remove, or delete them one at a time without disturbing the rest.

What is a protection group? It’s a vCenter-local representation of a consistency group (CG) in ONTAP 9 on AFF, ASA, FAS, and software-defined systems. We chose to use the term “protection group” in ONTAP tools to be aligned with VMware’s terminology. What this ultimately means is that you can now create and manage CGs directly in the vCenter UI.

So, what does this mean for new ONTAP tools users?

• Selective protection — mix protected and unprotected datastores freely within the same host cluster.
• Operational flexibility — modify protection groups without tearing down protection for an entire cluster.
• Multiple protection groups per cluster — replicate different datastores to different storage virtual machines (SVMs) or targets as business needs dictate.
• Incremental adoption — protect new datastores as they’re created, or bring existing ones into a group over time.

Ok, what if you are an existing user of ONTAP tools? It's simple — upgrades from ONTAP tools 10.4 and 10.5 seamlessly migrate existing host-cluster protection settings to the new model, no manual reconfiguration or import required.

GDP supports SnapMirror active sync's "Automated Failover Duplex (AFD)" policy across uniform and non-uniform vSphere Metro Storage Cluster (vMSC) topologies — giving architects far more precision when designing business continuity for mixed-criticality workloads. GDP can also use traditional asynchronous SnapMirror policies, though that would not be a vMSC configuration.

For a deep dive into GDP, check out the blog that Rajiv and I wrote about it: SnapMirror Active Sync + ONTAP tools for VMware vSphere Granular Datastore Protection (OTV10.6) - NetApp Community

A Lighter Footprint, Right Out of the Box

Not every environment needs a heavyweight appliance. ONTAP tools 10.6 introduces an Extra Small (XS) deployment size — just 6 vCPUs and 12 GB of RAM per node — and makes it the default Day-0 deployment profile for traditional provisioning and protection workloads. That’s a noticeable reduction from the previous Small footprint (9 vCPU / 18 GB), removing a resource barrier that had kept smaller environments and edge sites from adopting ONTAP tools.

Also new in this release, CPU hot add and memory hot plug are now enabled by default, so scaling up to small, medium, or large later requires no manual reconfiguration on fresh 10.6 deployments, nor do you have to shut down the appliance to reconfigure it if you forgot to do so during the initial deployment.

Stronger Security Visibility with Expanded Audit Logging

Security and compliance teams get a significant upgrade in 10.6. ONTAP tools now captures detailed audit logs (including full request and response bodies), discovery and cleanup jobs, and maintenance console operations — complete with username, timestamp, and correlation IDs for full traceability. Logs are retained for 30 days by default (configurable), protected by automated integrity checks that raise alerts on tampering or deletion, and can be forwarded to external systems like VCF Operations for Logsvia syslog. Enterprise security teams increasingly require this level of visibility. With ONTAP tools 10.6, it’s now built directly into the appliance.

Faster NFS Datastores with nconnect Support

NFS performance got a major boost with nconnect support back in 8.0u2. But it lacked any kind of GUI or API (I wrote a blog about it here: Automate nconnect and snapshot offload for NFS datastores and turn your VMware storage up to 11 - NetApp Community). It is now configurable directly during NFSv3 and NFSv4 datastore provisioning. By establishing multiple TCP connections over a single IP address, nconnect lets ESXi hosts distribute file operations across those connections in round-robin fashion — unlocking more of the available network bandwidth for NFS workloads without any change to the storage network topology. With nconnect, NFS datastores deliver SAN-level performance with NAS-level simplicity.

ONTAP tools 10.6 can even modify the maximum nconnect limit of your ESXi hosts from the default limit of 4 up to the maximum supported 8.

When it comes to nconnect integration, there is one caveat to be aware of: due to a limitation in the vCenter SDK, a plugin can’t modify nconnect settings after the datastore is mounted. However, starting with vSphere 9.0, you can change these settings in the datastore's host connectivity menu. Additionally, you can use the script that I shared earlier.

Cleaner Lifecycle Management

Even good old-fashioned housekeeping gets easier with ONTAP tools 10.6. A new force-delete flag for vCenter removal lets administrators clean up stale or unreachable vCenter entries — including automatic ONTAP tools plugin unregistration — without being blocked by lingering storage backend or datastore associations, provided no vVol datastores or active protection are involved.

Under the hood, ONTAP tools 10.6 also dynamically switches from ntpd to chronyd for time synchronization (with automatic migration on upgrade), re-implements and optimizes the authentication service for a smaller resource footprint and faster startup, and reorganizes log storage under a dedicated /var/log/otvng directory for cleaner troubleshooting and log rotation.

Availability

ONTAP Tools for VMware vSphere 10.6 launched on July 24, 2026, and supports ONTAP 9.16.1 through 9.19.1 alongside VMware Cloud Foundation (VCF) 5.2.1 through 9.0, and vSphere 8.0U3 through 9.0(the appliance can be deployed on ESXi 9.1). More version qualifications may be coming, so stay tuned for further updates. Direct upgrades are supported from ONTAP tools 10.4 and 10.5, with existing host-level protection settings automatically migrated to the new Granular Datastore Protection model.

In summary, whether you’re running a lean single-site deployment or a stretched, multi-site VMware environment protected by SnapMirror Active Sync, ONTAP tools 10.6 gives you finer control, a smaller footprint, and better visibility — all without compromising on the enterprise-grade protection NetApp customers count on.

ONTAP 9.19.1 expands Foreign LUN Import (FLI) by adding iSCSI backend support, giving more SAN teams a built-in, no-cost, ONTAP-native path to move block workloads from qualified third-party and ONTAP source arrays to modern NetApp storage. Since the first introduction in ONTAP 8.3, FLI has simplified LUN migrations by keeping data movement in the storage layer instead of pushing complexity to each host. Until now, many Ethernet-standardized and iSCSI-centric environments could not use that workflow without Fibre Channel (FC) connectivity. With ONTAP 9.19.1, FLI supports both FC-based and iSCSI-based migration designs across ONTAP SAN platforms, including AFF, ASA and FAS. This helps customers preserve existing network standards, reduce host-side effort, and bring more storage migration projects into a consistent, ONTAP-managed process. It removes the operational complexities when upgrading from legacy third-party storage from vendors such as Dell, HPE, HDS, and Pure Storage to ONTAP by making migration simpler, faster, built in, and less dependent on host-by-host cutover work.

For Ethernet-standardized SAN teams, this reduces host-side effort, simplifies migration sequencing, and improves operational control. Figure 1 shows FLI migrating one or more LUNs from a foreign source array to an ONTAP destination over either iSCSI or FC.

Figure 1) Foreign LUN Import network topology.

Problem Context

Many SAN environments are iSCSI-first and either do not have, or do not want to deploy, a Fibre Channel fabric. These teams still face the same migration pressures as FC environments:

• Move production LUNs from aging arrays on tight timelines.
• Minimize disruption to business-critical applications.
• Avoid host-by-host migration mechanics that expand risk and labor.

Customer Benefits

Customers gain greater migration flexibility with less operational friction. Instead of ruling out FLI because FC infrastructure is unavailable, teams can assess it based on workload fit and migration risk.

Expected outcomes in supported environments include:

• Accelerate migration programs by running storage-led LUN imports in parallel.
• Reduce host migration overhead and simplify change coordination across application teams.
• Preserve host-facing protocol flexibility because the backend import transport does not dictate the final host-access protocol after migration.
• Scale execution with governed parallel imports that account for latency, workload priority, and migration sequence.
• Improve cutover predictability through a repeatable LUN-level workflow.

A Practical Example

A mid-sized enterprise runs VMware and SQL Server on a third-party iSCSI SAN array and has no Fibre Channel infrastructure. The team must move dozens of production LUNs from an aging array to a faster, more capable NetApp system within a narrow maintenance window. With ONTAP 9.19.1, it can preserve its Ethernet SAN architecture, use iSCSI-based FLI backend connectivity, and migrate multiple LUNs through a consistent ONTAP workflow. This enables faster migration, simpler maintenance planning, and far less host-side effort than host-based copy methods.

Technical Breakdown

FLI remains a storage-led copy workflow. ONTAP uses iSCSI to discover and mount the foreign LUN, creates a same-size destination LUN, establishes the import relationship, and copies blocks in the background to keep the source and destination aligned during the import.

FLI operates at the LUN level rather than the volume level, so it applies whether customers use a strict 1:1 volume-to-LUN model or place multiple LUNs in the same volume.

Operational Guidance

During online imports, host I/O and import copy operations compete for backend resources, so plan concurrency conservatively and validate it before production waves. Treat the FLI concurrency limit as specific to a source-controller-to-ONTAP-destination-controller pair, not as a cluster-wide, SVM-wide, portal-wide, or universal planning value. Although FLI can support up to 32 concurrent imports for that pair, it’s not recommended to size migration waves to the maximum by default. NetApp typically finds the performance sweet spot between 8 and 16 concurrent imports. Start with 8 concurrent imports per validated source-destination controller pair, then adjust only after dry-run testing confirms acceptable application latency, import throughput, source-array utilization, destination-array utilization, and network behavior for the source array and workload mix. If multiple source or destination controllers are used, validate and plan each source-destination controller pair independently.

Run several dry migrations before scaling to verify the configuration and establish a realistic performance baseline:

1. Confirm that FLI is configured correctly and that the source and destination arrays can communicate.
2. Use dry runs to measure import performance and identify a concurrency level that balances throughput, migration duration, and application latency.

If latency affects production workloads, FLI can be paused and resumed from its last checkpoint.

Getting Started

Use this quick-start path to operationalize iSCSI-based FLI:

1. Confirm ONTAP 9.19.1 prerequisites before planning the migration wave. Verify interoperability for the required ONTAP platforms—FAS, AFF, ASA r1, and ASA r2—and confirm that the intended FLI transport, whether iSCSI-based, FC-based, or both, is supported across the source array, ONTAP target, and network or fabric configuration. Validate source-array qualification before proceeding.
2. Identify the workloads and migration candidates in scope.
3. Map all LUNs associated with each workload.
4. Build an import schedule by workload and LUN group, accounting for criticality, size, and maintenance windows.
5. Validate iSCSI backend pathing, initiator settings, and security controls.
6. Run dry tests on non-critical or test LUNs to verify configuration, measure latency and throughput, and confirm the procedure.
7. Set concurrent import counts based on application latency guardrails, not theoretical maximums.
8. Execute production waves with defined cutover criteria and pause/resume checkpoints.
9. Define success criteria for each wave—latency thresholds, copy time, and host validation checks—before scaling to larger batches.

For More Information

For more information about FLI, visit the FLI documentation hub.

https://docs.netapp.com/us-en/ontap-fli/index.html

Fibre Channel Direct Attach With NetApp Storage: Simpler SAN For The Right Environments

SAN complexity is often treated as a fixed cost of block storage. Fibre Channel Direct Attach, available with NetApp’s ONTAP 9.19.1, gives customers a simpler option when they do not need the scale and policy model of a full FC fabric. The result is a cleaner architecture, lower infrastructure cost, faster deployment, and more predictable host-to-array performance.

Just as important, FC Direct Attach can be treated as a valid SAN design point —not a compromise pattern. In edge, remote, cost-sensitive, and otherwise bounded environments, it delivers enterprise block outcomes without forcing a fabric-first design.

Design Boundaries

The most useful way to position FC Direct Attach is through environment fit. The model is best when host counts are known and bounded, ownership is clear, and expansion patterns are predictable.

Teams are generally better served by switch-attached fabric when they need higher fan-out, frequent topology changes, broad zoning policy flexibility, or long-range expansion options. In other words, this is not about good versus bad design —it is about matching SAN topology to workload scope.

Where FC Direct Attach Fits Best

Direct attach is especially useful in environments where simplicity and repeatability matter more than broad fan-out. Typical good-fit use cases include:

• Edge or ROBO deployments with predictable host growth
• Small clusters, typically in the two-to-four-host range
• Test and development environments
• Cost-constrained deployments that still require strong SAN behavior

Technical Value

At a technical level, FC Direct Attach enables per-port direct attach (DA) with one SCSI or NVMe LIF per DA port. This creates a flatter data path and reduces operational overhead tied to external switching layers.

Key technical benefits include:

• Simpler setup and troubleshooting with fewer moving parts
• Reduced TCO and administrative complexities from FC switch elimination
• Lower latency potential from fewer topology hops
• Dedicated bandwidth per host for stronger performance predictability
• Clearer fault isolation tied directly to host links and array ports

High availability remains part of the design. With multipath FC across redundant host and controller ports, teams can build resilient direct-attach connectivity even without an external fabric switch. This does not mirror every capability of large dual-fabric SAN designs, but it provides a credible HA model for bounded deployments.

NetApp ASA – Block Optimized, Simple, Reliable

NetApp ASA is designed for simplified SAN deployment where FC Direct Attach sharpens the value proposition for customers who need superior performance without switched fabric overhead. The economics are structural: fewer components, less cabling complexity, and lower day-2 management drag.

This also aligns with the purpose-built ASA block capabilities around simplicity (deployment, configuration, management, provisioning, automatic load balancing), reliability/resiliency, security and enterprise data services. The combination supports a clear message: simplify topology where it makes sense, while preserving enterprise block credibility.

Real-World implementation

Consider a global consumer goods manufacturer assessing storage standardization across more than 500 bottling and manufacturing facilities. Many sites already operate Fibre Channel direct connect without FC fabrics as a deliberate cost and simplicity choice. With FC Direct Attach in ONTAP 9.19.1, NetApp AFF, ASA and FAS storage can support those locations without adding a switch layer or forcing topology exceptions, enabling one block platform and one operational model across diverse site types.

The Takeaway

With Fibre Channel Direct Attach, you can keep the architecture straightforward while still leveraging the core strengths of FC: deterministic behavior, well-understood operational models, and consistent performance. It’s a strong fit for environments budgeted for cost that want clean connectivity, faster bring-up, simplicity and fewer dependencies.

Getting Started

Start by identifying direct-attach-fit locations: bounded host counts, predictable growth, and limited on-site SAN specialization.  Keep a dual-track architecture standard: direct attach where simplicity and cost savings are the value drivers, switched fabric where scale and policy needs justify it.

For More Information

Fibre Channel Direct Attach is supported on the following NetApp block products – our dedicated block solution (ASA), our Unified All Flash (AFF) and our Hybrid Flash (FAS) with ONTAP 9.19.1. 

https://docs.netapp.com/us-en/ontap/san-config/configure-fc-nvme-hosts-ha-pairs-reference.html

Databases are at the heart of many business‑critical applications and services. They underpin essential functions such as order processing, billing, inventory management, customer portals, analytics, and much more. When a database becomes unavailable, the consequences are immediate: users lose access to vital information, business operations are disrupted, and reputational damage can quickly ensue.

To ensure business continuity, organizations set strict recovery point objectives (RPO) and recovery time objectives (RTO) for their most critical databases. Having a backup alone isn’t enough; you must be able to restore swiftly to a recent point in time. Achieving these demanding recovery targets for self-managed databases on Amazon Web Services (AWS) becomes increasingly difficult as the environment expands in scale.

By using Amazon FSx for NetApp ONTAP as the backbone of your data infrastructure, you can optimize platforms such as SQL Server, Oracle DB, MongoDB, SAP HANA, and others. For many demanding workloads with strict SLAs, FSx for ONTAP is often the only service that enables these databases to run reliably and at scale on the AWS Cloud. In this post, I am going to explain how this is achieved.

Here’s what we’ll cover:

• Navigating database resilience challenges: Meeting strict objectives at scale on AWS
• How FSx for ONTAP optimizes resilience for self-managed databases
• What this means for your team: Benefits in practice
• Summary and next steps

Navigating database resilience challenges: Meeting strict objectives at scale on AWS

Protecting a single database is one thing, but the task changes dramatically when you’re responsible for a large-scale environment. A typical mission-critical deployment can involve hundreds of databases, several hundred terabytes or more of data, and multiple environments dedicated to development, testing, pre-production, and other purposes.

Within such a landscape, you must meet a range of requirements simultaneously: High availability to ensure that business-critical workloads withstand both component failures and disruptions to Availability Zones (AZs), strict backup recovery objectives, robust disaster recovery (DR) architecture to maintain business connectivity during major incidents, and diligent cost control given the significant expenses associated with database licences, compute resources, and storage capacity.

Traditional designs that rely solely on database-level replication and periodic backups to block storage or object storage often struggle to address all these demands at scale. As a result, backup windows can expand, restore processes might take longer, and disaster recovery deployments can become increasingly complex and costly to maintain.

How FSx for ONTAP optimizes resilience for self-managed databases

Let’s take a closer look at the standard architecture of self-managed databases on AWS and then discuss how FSx for ONTAP can optimize it. For this example, we will focus on a SQL Server.

Initial architecture: AOAG SQL Server running on Amazon EC2 and Amazon EBS

I recently worked with a financial customer managing hundreds of databases and over 500 TB of data across multiple environments, including development, test, pre-production, and others. This deployment required high availability across AZs, RPO of 10 minutes and RTO of 1 hour, and an effective DR site to protect against large-scale events and maintain business continuity. The following diagram illustrates the SQL Server architecture the customer chose, with Amazon Elastic Block Store (Amazon EBS)—a common solution for running a SQL Server on AWS:

In this design:

• Two Amazon Elastic Compute Cloud (Amazon EC2) instances running SQL Server are deployed in different AZs within an AWS Region.
• Each instance uses multiple Amazon EBS volumes for database files, log files, and other data.
• SQL Server replication, such as AOAG, keeps the databases synchronized between the instances.
• Amazon EBS snapshots are used for volume-level backups, with snapshot data stored in Amazon Simple Storage Service (Amazon S3).

This architecture delivers high availability by distributing SQL Server instances across two separate AZs, while ensuring data durability and recoverability through SQL-level replication and Amazon EBS snapshots stored in Amazon S3. It provides flexible scaling, clear separation of compute and storage, and protection against outages within the Region.

However, it has several implications:

• Licensing. AOAG often requires SQL Server Enterprise licensing, which can significantly increase costs when you have many cores and multiple replicas.
• Backup and restore times. As data volumes grow, the time required to create and restore from volume-level snapshots and copies increases. Meeting the 1-hour RTO across hundreds of databases becomes difficult.
• Storage and cost. Multiple Amazon EBS volumes, frequent snapshots, and long retention periods increase storage consumption.
• DR site. Building a DR site using this design typically involves additional infrastructure, extra replicas, more storage, and complex orchestration to meet the RPO and RTO targets.

At this stage, companies are seeking an alternative architecture to overcome these challenges and optimize resilience. In many cases, as was the case with this customer, the solution lies in the choice of storage service.

Optimized architecture: SQL Server FCI running on Amazon EC2 and FSx for ONTAP

Now let’s look at an architecture for a SQL Server failover cluster instance (FCI) using FSx for ONTAP as the underlying storage, which the abovementioned customer eventually moved to:

With this approach, backup and recovery workflows change. Instead of relying only on host-based backups, you use storage-level snapshots as the primary mechanism for protecting data. Creating a snapshot is a fast operation and restore workflows can be built around reverting or cloning from those snapshots.

Because FSx for ONTAP provides high availability across AZs, it can be used as a shared block storage for both Amazon EC2 instances. With this capability, the customer can move away from an AOAG architecture to FCI, while achieving the same or even better availability. For this architecture, a Standard SQL licence can be used, substantially reducing licensing costs.

Additionally, because this architecture requires only a single copy of data, it reduces storage capacity and costs and eliminates cross-AZ charges. And without the need for SQL-level replication, Amazon EC2 instance resources are freed up.

FSx for ONTAP backup operations are based on NetApp incremental Snapshot™ technology. Stored locally on the file system, These Snapshots are created rapidly, require minimal storage due to their space-efficient design, and enable quick recovery from logical failures. Their efficient use of space also keeps backup costs low and ensures performance remains unaffected during creation.

Thanks to an integrated free tool called NetApp SnapCenter® that synchronizes these Snapshot copies with the SQL Server, they become database-aware and require no log forwarding as part of the restoration process.

Adding disaster recovery to the mix

Now let’s add another layer of protection. To further safeguard against physical failures, the design extends to a secondary FSx for ONTAP file system, which can be air-gapped and deployed in another Region. Using NetApp SnapMirror®, data is replicated asynchronously and efficiently, ensuring the latest backups are always available for DR without the overhead of full data copies.

The following diagram shows the SQL Server DR architecture with FSx for ONTAP:

In this setup, the primary FSx for ONTAP file system replicates data incrementally and asynchronously to a secondary FSx for ONTAP file system in a separate Region, providing protection against site-wide or physical failures.

A disaster recovery SQL Server instance can swiftly attach to replicated volumes or space-efficient thin clones, enabling rapid failover and recovery. DR tests and failovers rely on these storage-level copies, eliminating the need for time-consuming full data restores.

This approach aligns easily with demanding recovery objectives, such as a 10-minute RPO and 1-hour RTO, because both replication and restoration are handled efficiently at the storage layer.

Outcomes of the revised architecture

Compared to the initial architecture, FSx for ONTAP streamlines resilience with high availability across AZs. Backup happens in seconds and restore in minutes, and an effective disaster recovery site is provided.

For the customer, this meant:

• Meeting its RPO and RTO requirements.
• Achieving high availability across AZs using a storage‑centric design.
• Reducing backup operations to seconds and restore operations to minutes by using application-aware, lightweight snapshots.
• Building an efficient DR site backed by storage replication.
• Transitioning to a SQL Server Failover Cluster Instance (FCI) design that provided an approximately 40% reduction in licensing and maintenance costs, along with additional savings on compute and storage.

What this means for your team: Benefits in practice

This strategy delivers significant benefits for teams managing their own databases on AWS.

Database administrators enjoy quicker and more reliable backup and restore processes, even when working with hundreds of databases. Regular testing of recovery and disaster recovery procedures becomes more straightforward, because snapshot-based clones allow for the rapid creation of temporary environments.

Storage administrators benefit from a solution built for efficient snapshots, thin clones, and replication—naturally supporting database protection. They can set storage policies once and apply them consistently across multiple databases and environments.

Cloud architects can implement an architecture that meets stringent RPO and RTO objectives, while optimizing licensing, compute, and storage expenses. Additionally, this approach makes it easier to scale as more databases, environments, and Regions are introduced.

In short, FSx for ONTAP empowers each team to focus on strategic improvements rather than routine maintenance, enhancing both productivity and resilience.

Summary and next steps

Protecting self‑managed databases on AWS at scale is challenging. You need high availability, fast recovery capabilities, and a robust DR strategy, while keeping an eye on cost and operational complexity.

By adopting FSx for ONTAP as the storage layer and optimizing the architecture, companies can meet strict SLA requirements, achieve high availability across AZs, build efficient DR sites, while gaining substantial savings on licensing costs and additional cost savings on compute and storage.

If you’re running self-managed databases on the AWS Cloud today, this architecture is worth evaluating. Start by assessing your current RPO and RTO targets, backup times, and disaster recovery design, and consider how FSx for ONTAP can help you protect your databases more efficiently.

To deepen your understanding and explore practical guidance:

• Watch the explainer demo video, showing 3.1 TB of SQL data backed up in just 9 seconds and restored in 8 minutes.
• Read about how FSx for ONTAP optimizes database performance and reduces costs by up to 50% and how to protect your SQL Server workloads using NetApp SnapCenter in our blogs.
• Check out the how-to guides and documentation to help you get started.

Are you still managing your SAN with a “if it isn’t broken, don’t fix it” mindset? It might be time to rethink that approach. According to a recent IDC Business Value Solution Brief, sponsored by NetApp, Interviewed companies reported a 49% faster identification of root causes for issues, thanks to unified dashboards and topology mapping from servers to storage. This report dives deep into the return on investment of observability and AI-powered analytics, showcasing how these advanced capabilities can streamline and enhance the process of upgrading SAN environments, turning  your infrastructure into a strategic asset rather than a constant headache.

According to a NetApp technology customer, “NetApp Data Infrastructure Insights has helped us pinpoint the root cause of issues faster because it provides us with a single dashboard. We can see everything in the infrastructure, including the topology from the server to the storage, and quickly identify where a failure has occurred. It is easy for us to recognize which server is failing and where it is connected to the SAN.”

Breaking down operational silos

One of the biggest challenges with legacy SAN environments is the lack of end-to-end visibility. Troubleshooting block workloads often leads to finger-pointing between virtualization admins, network admins, and storage admins, which delays resolution and drives up costs.

Data Infrastructure Insights tackles this problem head-on with its SAN Analyzer feature, providing a complete VM-to-LUN topology map. This feature breaks down silos, offering a unified view across heterogeneous environments and supporting both Fibre Channel and iSCSI. By overlaying performance metrics and alerts directly on the topology, all stakeholders gain a shared understanding of issues, leading to faster resolution times.

The result? A 49% quicker identification of root causes of issues, as highlighted in the IDC Business Value Solution Brief. That means less time troubleshooting and more time focusing on strategic initiatives.

Moving from reactive to proactive with AI

Downtime is the enemy of ROI, and traditional monitoring tools often flood teams with noisy alerts, leading to missed critical issues. Data Infrastructure Insights leverages AI to cut through the noise, using self-learning algorithms to detect anomalies before they cause disruptions. Whether it’s storage performance saturation, SAN port throughput issues, or hardware degradation, AI-powered insights help you catch problems early.

This proactive approach doesn’t just save time—it prevents costly service disruptions, ensuring mission-critical workloads stay online and your business avoids lost revenue. In fact, the IDC Business Value Solution Brief found that customers saved an average of $128,539 value in staff time after implementing Data Infrastructure Insights thanks to proactive alerting and faster troubleshooting.

De-risking change management

Infrastructure changes are one of the leading causes of IT incidents. A minor configuration update can ripple across the infrastructure, creating unexpected outages. Data Infrastructure Insights tackles this challenge with Change Analysis, which monitors your environment for changes and correlates them with alerts and performance issues.

This feature is invaluable both for daily operations—quickly identifying whether a recent change caused an issue—and for large-scale projects like SAN refreshes or migrations. By validating each step, you can reduce risk and ensure stability throughout the process.

Bridging the SAN skills gap

Advanced SAN expertise is in short supply, creating bottlenecks and over-reliance on specific individuals. Modern tools like Data Infrastructure Insights democratize this expertise, providing intuitive visualizations and actionable insights that empower generalists to handle complex tasks. This reduces your team’s dependency on SAN specialists, allowing senior engineers to focus on strategy while improving morale and efficiency.

Unlocking the full value of SAN modernization

The “if it isn’t broken” mentality may feel safe, but it’s costly in the long run. Modernizing your SAN management with tools like Data Infrastructure Insights gives you the clarity, control, and proactive capabilities needed to optimize your infrastructure investment. By reducing downtime, accelerating troubleshooting, and improving efficiency, you can unlock real ROI and transform your operations.

Ready to see the full picture?

Don’t let outdated strategies hold your infrastructure back—download the report and take the first step towards smarter, more efficient SAN management. Download the full IDC Business Value Solution Brief

* IDC Business Value Solution Brief, Sponsored by NetApp, The Business Value of NetApp Data Infrastructure Insights, #US54271626-BVSB, March 2026

NFS version 3 has been a gold standard for NFS applications for decades – starting all the way back in 1995, when it was first officially released. Its blend of performance and resiliency has made it difficult to consider a move to newer NFS versions for good reason.

However, NFSv3 is not without its limitations. Statelessness of the protocol, while great for performance and for minimizing disruptions on storage failover, is not so great for data consistency and lock management. An NFS server doesn’t really keep track of lock states, so if there is a failure, the NFS server may or may not release the locks, and the NFS client may not know whether a file is locked or not.

Security for NFSv3 is also a bit lacking. The protocol requires multiple open firewall ports to function properly and numeric IDs are sent in plaintext over the wire. Furthermore, NFS does not have robust ACL support, and does not include native file and folder auditing. As a result of these limitations, NFSv4 was created in 2003 via RFC-3530 (obsoleted in 2015 by RFC-7530).

What does NFSv4 bring to the equation?

NFSv4 delivers a number of improvements over NFSv3, including:

Integrated locking

Rather than splitting lock mechanisms into their own services (such as NLM and NSM) from the NFS protocol itself, which require their own network ports and can create stale lock issues in failover scenarios, NFSv4 integrates locking directly into the protocol stack. That way, the client and server always agree on the lock states. If a client or server has an extended failure, then locks will eventually expire on their own, which prevents stale lock issues.

Firewall friendly

NFSv4 integrates many of the services that NFSv3 separated out, such as mount, portmapping, locking, etc, into the protocol. As a result, only port 2049 (the NFS port) is required for use of NFSv4, which makes the protocol firewall friendly.

Granular access controls

NFSv3 supports basic mode bit permissions (such as RWX) with a limitation of owner, group, and everyone else as the only subjects to which permissions can be applied. These limitations make it difficult to create granular access controls for additional users and groups. NFSv4 attempts to improve upon that by implementing support for NFSv4 access control lists (ACLs). These ACLs provide a way to manage access not just for basic RWX across only a small subset of subjects, but also for a larger number of file operations (such as deletes, ownership changes, permission changes, etc) across up to 1024 owner and group objects. An added benefit in ONTAP environments is that NFS access controls can more closely match NTFS permissions, which makes multiprotocol NAS access across NFS and SMB to the same datasets more seamlessly integrated.

Name string/domain enforcement

NFSv3 users and groups send numeric IDs to the NFS server by default, which means that the NFS server then must translate that numeric ID into a human-readable name. NFSv4, on the other hand, will send name strings (ie, user@DOMAIN.COM) to the server, which not only helps prevent spoofing of numeric IDs, but also requires the client and server to negotiate the name strings with identical local representation of users and groups (either via local files or LDAP). If the name string translation fails, then the NFS server will squash the user or group to the anonymous user defined by the NFS client’s configuration. This helps prevent unintentional or unauthorized access to files and folders.

File and folder auditing

In addition to security Access Control Entries (ACEs), NFSv4 also supports the ability to add audit ACEs to files and folders to track and log specific file operations. For instance, if you want to see who is attempting to read or delete a file, audit ACEs can track that. Native NFSv3 supported no such capability.

Kerberos mandate

Kerberos authentication provides an optional extra layer of security for NFS access by requiring users to enter a username and password to collect a Kerberos ticket to initially access NFS shares. Kerberos can also be leveraged to further encrypt data traffic end-to-end (with a performance impact due to the encryption/decryption CPU work) for added security, and since all NFS functionality is integrated into the same port, all NFS traffic would be encrypted – including mounts and locks. While the NFSv4 RFC mandates Kerberos support by the NFS server, use of Kerberos is optional.

Additional features and functionality

While NFSv4 added a multitude of new functionality within the protocol stack, perhaps the biggest improvement the new version brought was the addition of new features and functionality for NFS in general.

These include:

• Parallel NFS (NFS v4.1)
• NFS session trunking (NFSv4.1)
• File delegations (NFSv4.1)
• Referrals (NFSv4.1)
• Stateless migration (NFSv4.1)
• Security labels (NFSv4.2)
• Extended attributes (NFSv4.2)

For a deeper look:

NFSv3 and NFSv4: What's the difference? - NetApp Community

Sounds great… why don’t more people use it?

While NFSv4.x has been around for over 20 years, there still hasn’t been widespread adoption of it for a few reasons.

• Complexity of identity management: Many environments don’t have a name service infrastructure in place to properly take advantage of the name string and Kerberos security requirements in NFSv4.x.
• Need for newer NFS clients: This concern is less pressing in today’s modern NFS environments, as the further we get from the initial release date of NFSv4. Almost all currently used OS’s include NFS clients with full NFsv4 support, but there are still legacy systems that may not have the necessary NFSv4.x packages. In fact, some applications still require use of older NFS versions.
• “If it ain’t broke, don’t fix it” mentality: Enterprise IT organizations are notoriously conservative in adoption of newer technologies – even ones that have been around for over 20 years. And if the current NFS version is working fine, why change?
• Performance concerns: Performance of a stateful protocol like NFSv4.x has lagged behind stateless NFSv3 for much of the past 20 years. In the past, the performance impact often outweighed the NFSv4.x benefits.

What is NetApp AFX?

NetApp AFX is the latest innovation from NetApp that introduces the use of disaggregated storage with NAS and object for high performance workloads, such as AI/ML, EDA, HPC and anything that requires independent scaling of capacity and compute, massive throughput, and a single capacity pool.

AFX does all of this while still running the same ONTAP images as unified ONTAP. That means you get all of the same ONTAP functionality and interoperability (such as SnapMirror and FlexCache) you’re accustomed to.

Since disaggregated ONTAP is a new way to consume ONTAP, it opens the door to additional performance gains, such as the changes made in ONTAP 9.18.1 to improve overall performance for NFSv4.x.

NFSv4.x improvements in ONTAP 9.18.1 using AFX

Some architectural changes to ONTAP have provided a much needed performance boost to NFS in general, and have made some serious inroads to improving NFSv4.x performance in general.

Below is a high-level summary of some of those changes.

Sequential read enhancement - NFSv4.1 30% better than NFSv3

ONTAP 9.18.1 introduces support for multipath IO with NFSv4.1. Rather than processing reads from the WAFL file system, MPIO shifts read operations into a network domain to be served in a multipath-safe manner. This approach reduces context switches, providing greater overall parallelism in sequential read traffic, as well as reducing the overhead from buffer management by bypassing WAFL.

Random read enhancement for FlexGroup volumes – NFSv4.1 within 7% of NFSv3

FlexGroup volumes are volumes that take many underlying constituent volumes and present them as a single unified namespace. In AFX, FlexGroup volumes have Advanced Capacity Balancing enabled by default, which will write files larger than 10GB across multiple constituent volumes as multipart files. Because of the remote location of these file parts, random reads traditionally have had a modest perf disadvantage with NFSv4.x (around 18% less than NFSv3). ONTAP 9.18.1 introduces support for cached IO for multipart reads with NFSv4.x to help address this.

Note: This change does not apply to FlexVol volumes.

Sequential writes - +10% improvement from previous releases

An improvement of how we replicate NVLOG data used for HA failover functionality increased overall sequential write performance for NetApp AFX systems.

Metadata operations – Within 15% performance of NFSv3 for EDA benchmarks

NFSv4.1 traditionally serializes all OPEN and CLOSE operations, with a cluster node processing them one at a time before they can be sent from the network to WAFL. ONTAP 9.18.1 introduces Concurrent Open Close (COC), which eliminates network serialization by changing how race conditions are resolved, which removes OPEN/CLOSE bottlenecks seen in prior releases.

All of these changes – along with the architecture changes brought about in AFX – have made it possible to improve the overall NFSv4.1 performance in ONTAP 9.18.1.

Sequential IO results

One of the areas where some modest performance improvements were seen was with sequential IO (ie, IO that is predictable and issued consecutively). In standard performance tests using fio, AFX running ONTAP 9.18.1 improved sequential read performance by nearly 30% and sequential write performance by 10%.

Metadata-heavy workload results

Even more impressive are the improvements with one of NFSv4.x’s top performance pain points – metadata. These are random IOs, usually in the 4K range, used to manage file owners and attributes, creating and listing files, and so on. Because of the statefulness of NFSv4.x, these types of operations tend to cost more in CPU and latency, which in turn reduces overall possible performance.

With the changes in AFX ONTAP 9.18.1, NFSv4.x performance for these types of workloads has improved substantially and has closed the gap on NFSv3 performance (within 15%).

Our performance engineering teams compared performance of standard AI image, EDA, and software build benchmarks and found massive gains from the previous ONTAP release.

So now what?

With the latest NFSv4.x performance improvements (along with the other benefits of NFSv4.x), it’s likely worth revisiting use of NFSv4.x in your high-performance NAS workloads. Either test it out with your applications or schedule a meeting with our Customer Proof of Concept (CPOC) labs through your NetApp sales representative. If you’d like to learn more about CPOC, check out this Tech ONTAP Podcast episode:

• Episode 403 - CPOC! by Tech ONTAP Podcast

If you’d like to learn more about NFSv4.x, these links can also help:

• Episode 119: NFSv4.x Deep Dive by Tech ONTAP Podcast
• NFSv3 and NFSv4: What's the difference? - NetApp Community
• NFS best practice and implementation guide | TR-4067

NetApp Image

• Overview
• NFS with Kerberos
• NFS over IPsec
• ONTAP enforcement of IPsec
• General IPsec storage information
• General IPsec client configuration
• IPsec with pre-shared key authentication
• IPsec with certificate authentication
• IPsec configuration notes for IPv6
• NFS over TLS
• NFS over TLS storage configuration
• NFS over TLS client configuration

Overview

NetApp is the most secure storage on the planet.  With that in mind, let's look at the available technology for encrypting NFS traffic over-the-wire with NetApp ONTAP.  The options include using NFS with Kerberos, running NFS over IPsec connections, and a nascent approach using NFS over TLS.  While the setup for NFS with Kerberos is well documented, examples of end-to-end configuration for IPsec and NFS over TLS with ONTAP are a bit harder to find, so we will cover those details in this article.

If you want a quick summary of these options, here are the key points:

• NFS with Kerberos provides the most security features, including an option for over-the-wire encryption, but comes with the most complexity and the biggest hit to performance. 
• NFS over IPsec is mature, fully supported, and comes in the middle concerning configuration complexity.  NetApp already offers support for IPsec hardware offload cards (as of 9.16.1) in our new platforms to reduce the performance impact.
• NFS over TLS is the newcomer.  The configuration complexity is the lowest, but it is not quite ready for production as of early 2025.  The performance hit with IPsec and NFS over TLS is similar, and both provide better performance than NFS with krb5p.

If you are looking for details on how to configure over-the-wire encryption using these technologies with NetApp storage, read on!

NFS with Kerberos

We will start with a quick discussion of using NFS with Kerberos.  This option has been available for many years and is well documented in TR-4616: NFS Kerberos in ONTAP.  If you cannot trust who has root access on your NFS clients, then using NFS with Kerberos allows you to make sure that the users on your NFS client are who they say they are.  NFS with Kerberos also provides for over-the-wire encryption, using the krb5p mount option.  I won't go into the configuration steps for NFS with krb5p in this blog, since it is covered very well in TR-4616.  While this is a great option for security, it also comes with the most complexity to configure and the largest performance hit of all the options for NFS over-the-wire encryption.  In addition, there is no roadmap for improved performance of NFS with krb5p via hardware offload.

NFS over IPsec

Next up is running NFS over IPsec connections.  ONTAP has supported IPsec for years, since the 9.8 release!  In addition to pre-shared keys being supported since ONTAP 9.8, we have supported certificate authentication since 9.10.1.  More recently, we support IPsec hardware offload using ONTAP 9.16.1 with our newer platforms.  See the Evolution of the ONTAP IPsec Implementation for more details.

For the remainder of the IPsec section of this blog, I give some end-to-end examples of configuring IPsec with ONTAP and on the NFS client.  There are multiple approaches to configuring IPsec, and many configuration options with the popular IPsec client software strongSwan.  All traffic can be encrypted, or only certain ports.  Policies can be configured between specific IP addresses, or between entire subnets.  The approach I document in this article tries to keep the IPsec client configuration as generic as possible while allowing for simultaneous connections to multiple storage LIFs.

ONTAP enforcement of IPsec

When configuring NFS with Kerberos or using NFS over TLS, the NFS client can choose to mount an NFS export with the required options to enable these features or a standard mount can be performed with "sec=sys". This concept is different from IPsec in that once an IPsec security policy is defined and enabled for a LIF on the storage system, traffic is only allowed on that LIF via an IPsec connection.  

General IPsec storage information

Below are some storage side commands that are useful to be aware of for any type of IPsec configuration:

# Enable ipsec on your cluster
security ipsec config modify -is-enabled true

# Check for active IPsec connections
security ipsec show-ikesa -node <nodename>

# Check for IPsec related errors
event log show -event ipsec.*

General IPsec client configuration

To use IPSec with ONTAP we need IPsec client software that supports IKEv2.  In the Linux world, two popular software packages that fill this requirement are strongSwan and Libreswan.  For my examples, I will be using strongSwan.  To install strongSwan, run the following commands on a Red Hat compatible Linux distribution:

dnf install epel-release
dnf install strongswan
systemctl enable strongswan
systemctl start strongswan

Here are some notes about strongSwan that will apply to all of the IPsec configurations we show in this article:

• The strongSwan client configuration file path is /etc/strongswan/swanctl/swanctl.conf
• Debugging logging on the IPsec client can be seen with "grep charon /var/log/messages"

IPsec with pre-shared key authentication

Configuration with a pre-shared key is straightforward.  In the following example, we configure a storage system policy for each of two LIFs on the SVM and use a subnet definition for the IPsec clients:

::> security ipsec policy create -vserver svm1 -name lif1_psk -local-ip-subnets 192.168.0.201/32 -remote-ip-subnets 192.168.0.0/24
{Enter your passphrase - I used netapp123netapp123netapp123}
::> security ipsec policy create -vserver svm1 -name lif2_psk -local-ip-subnets 192.168.0.202/32 -remote-ip-subnets 192.168.0.0/24
{Enter your passphrase - I used netapp123netapp123netapp123}

On the client, we use the following swanctl.conf configuration with a connection definition for each storage LIF and the pre-shared key:

connections {
    svm1_lif1 {
        children {
            svm1_lif1 {
                esp_proposals = aes256gcm16
                mode = transport
                start_action = trap
                # Disabling replay_window required with ONTAP IPSec HW offload.
                replay_window = 0
                # Connection from local subnet to lif1.
                local_ts = 192.168.0.0/24
                remote_ts = 192.168.0.201/32
            }
        }
        # Connection from local subnet to lif2.
        local_addrs = 192.168.0.0/24
        remote_addrs = 192.168.0.201/32
        proposals = aes256-sha384-ecp384
        local {
            auth = psk
        }
        remote {
            auth = psk
        }
    }
    svm1_lif2 {
        children {
            svm1_lif2 {
                esp_proposals = aes256gcm16
                mode = transport
                start_action = trap
                # Disabling replay_window required with ONTAP IPSec HW offload.
                replay_window = 0
                # Connection from local subnet to lif2.
                local_ts = 192.168.0.0/24
                remote_ts = 192.168.0.202/32
            }
        }
        # Connection from local subnet to lif2.
        local_addrs = 192.168.0.0/24
        remote_addrs = 192.168.0.202/32
        proposals = aes256-sha384-ecp384
        local {
            auth = psk
        }
        remote {
            auth = psk
        }
    }
}
secrets {
    ike-svm1 {
        # This must match the pre-shared key entered for storage policy.
        secret = netapp123netapp123netapp123
        # The ike id must match the storage policy remote-ip-subnets.
        id = 192.168.0.0/24
    }
}

Now run "swanctl --load-all" and then "swanctl --list-conns" to make sure your connections are properly loaded.  If all is well, you should be able to successfully ping the storage LIFs from the IPsec client,

IPsec with certificate authentication

Configuration with certificate authentication is a bit more involved due to the requirement for creating and configuring certificates both on the storage LIFs and the IPsec clients.  If you are familiar with PKI, you can create and manage the certificates however you wish.  In this example I use the certificate management tools built into ONTAP.  Note that when copying a public key certificate or a private key, always include the "-----BEGIN ..." and "-----END ..." lines.  First, we will create a certificate authority (CA), add that CA to the local ONTAP ipsec configuration, and copy the CA public key certificate to the IPsec client strongSwan configuration directory:

security certificate create -vserver svm1 -type root-ca -common-name svm1 -cert-name svm1_ipsec_ca
security ipsec ca-certificate add -vserver svm1 -ca-certs svm1_ipsec_ca
security certificate show -vserver svm1 -cert-name svm1_ipsec_ca -instance
{Copy the public key certificate to a file on the IPsec client at /etc/strongswan/swanctl/x509ca/svm1_ipsec_ca.pem}

Next, we will create the certificate to use for our storage LIF.  Repeat the following steps for each storage LIF that is part of your IPsec policy, making sure to use the proper FQDN and IP address when generating the certificate signing request:

security certificate generate-csr -common-name svm1_lif1.demo.netapp.com -algorithm RSA -hash-function SHA256 -ipaddr 192.168.0.201 -dns-name svm1_lif1.demo.netapp.com
{Save each certificate request and private key}
security certificate sign -vserver svm1 -ca svm1 -ca-serial 1821A2A76A371FFC -expire-days 360
{Enter each certificate to sign and get signed certificate}
security certificate install -type server -vserver svm1
{Enter each signed certificate and private key}

We also need a certificate to use for the Linux IPsec client.  We will again use the ONTAP certificate management functionality to create the certificate, and then place the public key certificate and private key in the correct location for strongSwan on the IPsec client:

security certificate generate-csr -common-name linux1.demo.netapp.com -algorithm RSA -hash-function SHA256 -ipaddr 192.168.0.61
{Save certificate request and private key}
security certificate sign -vserver svm1 -ca svm1 -ca-serial 1821A2A76A371FFC -expire-days 360
{Enter certificate to sign and get signed certificate}
{copy the signed public key certificate to /etc/strongswan/swanctl/x509/linux1.demo.netapp.com.pem}
{copy the private key to /etc/strongswan/swanctl/private/linux1.demo.netapp.com.key}

OK!  With all the certificates in place, we once again configure a storage system policy for each of two LIFs on the SVM and use a subnet definition for the IPsec clients:

security ipsec policy create -vserver svm1 -name lif1_pki -local-ip-subnets 192.168.0.201/32 -remote-ip-subnets 192.168.0.0/24 -auth-method PKI -cert-name svm1_lif1.demo.netapp.com -local-identity CN=svm1_lif1.demo.netapp.com -remote-identity ANYTHING
security ipsec policy create -vserver svm1 -name lif2_pki -local-ip-subnets 192.168.0.202/32 -remote-ip-subnets 192.168.0.0/24 -auth-method PKI -cert-name svm1_lif2.demo.netapp.com -local-identity CN=svm1_lif2.demo.netapp.com -remote-identity ANYTHING

Finally, we have our swanctl.conf on the IPsec client:

connections {
    svm1_lif1 {
        children {
            svm1_lif1 {
                esp_proposals = aes256gcm16
                mode = transport
                start_action = trap
                # Disabling replay_window required with ONTAP IPSec HW offload.
                replay_window = 0
                # Connection from local subnet to lif1.
                local_ts = 192.168.0.0/24
                remote_ts = 192.168.0.201/32
            }
        }
        # Connection from local subnet to lif1.
        local_addrs = 192.168.0.0/24
        remote_addrs = 192.168.0.201/32
        proposals = aes256-sha384-ecp384
        local {
          auth = pubkey
          id = "CN=linux1.demo.netapp.com"
          certs = linux1.demo.netapp.com.pem
        }
        remote {
          auth = pubkey
          id = "CN=svm1_lif1.demo.netapp.com"
        }
    }
    svm1_lif2 {
        children {
            svm1_lif2 {
                esp_proposals = aes256gcm16
                mode = transport
                start_action = trap
                # Disabling replay_window required with ONTAP IPSec HW offload.
                replay_window = 0
                # Connection from local subnet to lif2.
                local_ts = 192.168.0.0/24
                remote_ts = 192.168.0.202/32
            }
        }
        # Connection from local subnet to lif2.
        local_addrs = 192.168.0.0/24
        remote_addrs = 192.168.0.202/32
        proposals = aes256-sha384-ecp384
        local {
          auth = pubkey
          id = "CN=linux1.demo.netapp.com"
          certs = linux1.demo.netapp.com.pem
        }
        remote {
          auth = pubkey
          id = "CN=svm1_lif2.demo.netapp.com"
        }
    }
}

Now run "swanctl --load-all" and then "swanctl --list-conns" to make sure your connections are properly loaded.  If all is well, you should be able to successfully ping the storage LIFs from the IPsec client.

IPsec configuration notes for IPv6

I've also tested IPsec with IPv6, and ran into a couple of strongSwan configuration requirements to make this work.  The first issue is that we need to bypass IPv6 NDP NS and NA traffic in the swanctl.conf as shown in this snippet of the configuration file:

{Add this at the start of the connections block}
    # Bypass IPv6 NDP NS and NA traffic.
    # As per https://docs.strongswan.org/docs/5.9/config/IPv6Ndp.html
    ndp {
        children {
            ns {
                mode = pass
                start_action = trap
                local_ts = ::/0[ipv6-icmp/135]
                remote_ts = ::/0[ipv6-icmp/135]
            }
            na {
                mode = pass
                start_action = trap
                local_ts = ::/0[ipv6-icmp/136]
                remote_ts = ::/0[ipv6-icmp/136]
            }
        }
    }

One more change that I found from my testing with IPv6 is that simple IP address endpoints (without subnet notation) must be given in the local_addrs and remote_addrs files of swanctl.conf, as shown in this snippet:

        local_addrs = fd12:1234:5678:1::2
        remote_addrs = fd12:1234:5678:1::4

Finally, note that ONTAP IPsec HW offload is not currently supported for IPv6.

NFS over TLS

Finally, we will showcase the use of NFS over TLS.  The NFS over TLS feature was introduced in ONTAP 9.15.1 as a Tech Preview.  This means that it is not yet ready for production, and full GA support for this feature in ONTAP will come in a future release.  You should also be aware that this is a very new feature on the Linux NFS client side as well, as it first shows up in RHEL as a Technology Preview in RHEL 9.4.

NFS over TLS storage configuration

Let's look at the configuration steps to get NFS over TLS running.  NFS over TLS relies on certificates, and we will need to configure a unique certificate for each LIF in the SVM that will handle NFS over TLS traffic.  If you are familiar with PKI, you can create and manage the certificates however you wish.  In this example, I will simplify the setup by using the certificate management tools built into ONTAP.  First, we will create a certificate authority (CA) in the SVM and display the public key certificate of the CA to be used later on our NFS client.  Note that when copying a public key certificate or a private key, always include the "-----BEGIN ..." and "-----END ..." lines.

security certificate create -vserver svm1 -type root-ca -common-name svm1_ca
security certificate show -vserver svm1 -common-name svm1_ca -type root-ca -instance

Next, we will create a certificate for one of our SVM LIFs and configure the LIF to use this certificate for NFS over TLS:

security certificate generate-csr -common-name svm1_lif1.demo.netapp.com -algorithm RSA -hash-function SHA256 -ipaddr 192.168.0.61 -dns-name svm1_lif1.demo.netapp.com,svm1_lif1,svm1
{Save certificate request and private key}
security certificate sign -ca svm1_ca -vserver svm1 -ca-serial 181F8F545154B6BA -expire-days 364
{Enter certificate to sign and get signed certificate}
security certificate install -vserver svm1 -type server -cert-name svm1_lif1.demo.netapp.com
{Enter signed certificate and private key}
nfs tls interface enable -vserver svm1 -lif svm1_lif1 -certificate-name svm1_lif1.demo.netapp.com

The above commands will be required for each LIF on which you want to run NFS over TLS traffic.  Also, a few notes are in order about the security certificate generate-csr command above:

• The -common-name argument must be the FQDN of the LIF IP address.
• You must use the correct LIF IP address in the -ipaddr argument.
• You should configure the -dns-name argument to include any hostnames that may be used in NFS mounts from the client.  This includes configurations where you may use multi-homed DNS entries so that multiple LIF IP addresses are returned for a single hostname.

NFS over TLS client configuration

Now, on the Linux NFS client, we have to first install the ktls package.  In my example, I'm using a Red Hat 9.4 compatible client:

dnf install ktls-utils
systemctl enable tlshd
systemctl start tlshd

Next, remember that CA public key certificate I told you to make a copy of from the output of the "security certificate show -vserver svm1 -common-name svm1_ca -type root-ca -instance" command?  We need to place it on the Linux client and run a command as follows:

vi /etc/pki/ca-trust/source/anchors/svm1_ca.pem
{Paste the CA public certificate that was used to sign LIF certificates on storage side}
update-ca-trust extract

Now we should be able to mount an NFS export using TLS!  Here is the syntax to mount with NFSv3 over TLS:

mount -o nfsvers=3,xprtsec=tls svm1_lif1:/vol1 /vol1

That's it!  If you are using IPv6, the procedure for NFS over TLS works the same way - substitute IPv6 IP addresses as required when creating your certificates. 

I hope you found all of these details useful!  Hit me in the comments if you have any questions on these technologies or how the configuration works.

AI-assisted coding with GitHub Copilot has transformed how millions of developers write code. Yet, when provisioning the infrastructure their applications depend on, developers still face time-consuming manual workflows that break focus and slow delivery.

That’s because, while infrastructure-as-code (IaC) automates many tasks, it still demands researching configurations, writing complex templates, and switching between tools. What if developers could instead ask for the infrastructure they need, and let an AI handle the rest?

There’s a new way for Amazon FSx for NetApp ONTAP (FSx for ONTAP) users to do just that: The FSx for ONTAP Visual Studio Code (VS Code) extension. Now available in the Visual Studio Marketplace, this extension brings enterprise-grade storage provisioning directly into VS Code. 

In this post, we introduce the FSx for ONTAP VS Code extension. We see how application developers, DevOps engineers, and platform engineers can use it to generate optimized infrastructure templates to deploy and manage FSx for ONTAP file systems and volumes—all with the help of an AI assistant that’s built directly into the developer interface, where the code lives.

Here’s what we cover:

• The efficiency gap between development and provisioning in cloud storage
• What is the FSx for ONTAP VS Code extension?
• Core capabilities of the FSx for ONTAP VS Code extension
• Key benefits of the FSx for ONTAP VS Code extension
• Common operations using the FSx for ONTAP VS Code extension
• Getting started
• Deploy or modify FSx for ONTAP infrastructure
• Launch ONTAP CLI or AWS CLI terminal sessions
• Get AI-powered insights through @fsx-ontap
• Try out the FSx for ONTAP VS Code extension today

The efficiency gap between development and provisioning in cloud storage

Modern software development often spans multiple layers, from writing code to provisioning infrastructure and managing data. To manage these effectively, developers and DevOps engineers must frequently move across integrated development environments (IDEs), the AWS Console, and documentation. This fragmented workflow creates four key challenges:

• Constant context switching disrupts productivity: Moving between tools forces mental reloading, turning simple tasks into time-consuming exercises.

• Manual processes introduce configuration risk: Handcrafting IaC templates and copying connection strings or credentials can lead to deployment errors that require hours of debugging.

• Provisioning cycles stretch from hours to days: Manual configuration and coordination between tools and teams delay deployments and slow delivery velocity.

• Teams lack visibility into costs and best practices: There are no embedded insights or recommendations for costs and optimizations in this process, so tracking spending and governance becomes another manual task.

The result: Developers lose momentum, platform engineers have more tickets to manage, and innovation slows before code even reaches production.

What is the FSx for ONTAP VS Code extension?

The FSx for ONTAP VS Code extension brings FSx for ONTAP operations directly into VS Code, connecting GitHub Copilot with your AWS environment through a unified, AI-assisted experience.

The extension provides full visibility and insights into your FSx for ONTAP infrastructure using a conversational AI interface directly in your developer environment. It integrates with the ONTAP® CLI, FSx APIs, and Amazon CloudWatch APIs.

At the heart of the extension is the @fsx-ontap chat participant. By simply typing @fsx-ontap in the GitHub Copilot Chat, you can use natural language prompts to ask context-based questions about your file systems, retrieve metrics, check backups, or request intelligent recommendations—all powered by real-time data from FSx for ONTAP and CloudWatch APIs.

Together, these capabilities support IaC automation, resource management, and AI-driven operations within a single, developer-first interface.

Core capabilities of the FSx for ONTAP VS Code extension 

The FSx for ONTAP VS Code extension simplifies how developers and platform engineers use and manage FSx for ONTAP, unlocking more automation, visibility, and control.

• Seamless onboarding and resource navigation: After installation, you can authenticate with existing AWS credentials or select an AWS profile configured through the AWS CLI. The extension automatically discovers all FSx for ONTAP resources—file systems, storage virtual machines (SVMs), and volumes—and presents them in a structured navigation panel for immediate visibility.

• IaC generation and deployment: Generate AWS CloudFormation or Terraform templates from your prompts or selections. Parameters such as capacity, throughput, and deployment type are pre-populated according to AWS and NetApp® best practices to streamline configurations. You can also review and customize templates for your specific requirements before deployment.

• Direct CLI access for advanced customization: For hands-on control, you can open a terminal session directly from the extension to access the ONTAP CLI or AWS CLI. The extension supports connections through Amazon Elastic Compute Cloud (Amazon EC2) Instance Connect or standard SSH for more secure, in-context access to the underlying file system. With this capability, you can perform advanced configurations and troubleshooting without leaving VS Code.

• AI-powered Q&A with @fsx-ontap : The integrated @fsx-ontap chat participant brings conversational AI into the IDE. You can type questions such as:

• List all volumes in my development Region.
• Show Snapshot copies for fsx-prod.
• How much storage is available on fsx-west?

The chat participant retrieves live data from FSx for ONTAP and Amazon CloudWatch APIs, providing actionable insights in real time. Include /filesystem in your question to access ONTAP commands for deeper insights beyond the AWS Console.

• Built-in best practices and smart recommendations: The extension surfaces guidance aligned with AWS Well-Architected Framework principles and NetApp best practice recommendations for cost, performance, reliability, and security. It highlights configuration options, flags potential missteps, and suggests optimizations that you can apply before deployment.

Note: Always review and validate AI-generated templates, recommendations, and responses against your specific environment and requirements.

Key benefits of the FSx for ONTAP VS Code extension 

The capabilities of the FSx for ONTAP VS Code extension deliver value at every level, from individual developers accelerating daily workflows to organizations standardizing infrastructure operations at scale.

For developers and DevOps engineers managing FSx for ONTAP, the extension supports:

• Faster provisioning: Generate and deploy infrastructure directly from your IDE with AI-guided templates.

• Unified workflow: Manage file systems, volumes, and configurations from a single workspace without switching between the AWS Console, command-line tools, and documentation.

• Improved quality: Real-time visibility into system metrics and configurations combined with built-in guidance and instant access to troubleshooting commands through @fsx-ontap helps maintain best practices.

These individual improvements compound at the organizational level, supporting:

• Cost optimization: AI-driven insights recommend optimal storage configurations, data tiering, and performance settings to help control spending and improve efficiency.

• Standardization and repeatability: Consistent templates and embedded validation align every deployment with AWS and NetApp best practices for security, reliability, and governance.

• Operational reliability: On-demand reporting through Amazon CloudWatch supports proactive health management and minimizes downtime.

Whether you're provisioning a single volume or managing FSx for ONTAP across multiple teams and accounts, the extension brings speed, consistency, and intelligence to every interaction, making storage operations a seamless part of the development workflow.

Common operations using the FSx for ONTAP VS Code extension

Let’s take a look at installation and some of the common operations that you can carry out using the FSx for ONTAP VS Code extension.

Getting started

In the VS Code Marketplace, search for “NetApp FSx ONTAP” and then install the extension. Select the NetApp icon in the activity bar to begin.

The following screen shows the welcome page for the NetApp FSx for ONTAP VS Code extension.

From the welcome page, select Connect to AWS to choose your AWS credential profile (from those configured through aws configure) and select your preferred Region(s) using the globe icon.

Once configured, the extension automatically displays a hierarchical view of the entire FSx for ONTAP environment in the sidebar.

You can now view and manage your FSx for ONTAP file systems directly from your IDE. 

Deploy or modify FSx for ONTAP infrastructure

You can select from pre-built Terraform or CloudFormation templates that embed AWS and NetApp best practices. You can review, customize, and deploy directly from your IDE with full visibility into the process.

Here you can see the process of selecting the pre-built Terraform template to quickly deploy FSx for ONTAP infrastructure.

Launch ONTAP CLI or AWS CLI terminal sessions

You can open terminal sessions from the extension to access the ONTAP CLI or AWS CLI for advanced management operations.

The following screen shows the in-IDE terminal session connecting directly to your file system for advanced configuration. You can connect using SSH or through Amazon EC2 Instance Connect for more secure, in-context access to your file systems.

Get AI-powered insights through @fsx-ontap

You can access the @fsx-ontap chat participant in the Copilot Chat to get AI-based insights using natural language.

The following screen shows how @fsx-ontap provides real-time, contextual resource information. With the chat participant, you can use natural language to query resources, retrieve metrics, and perform operational tasks and get responses you can understand.

For more advanced use cases, explore the source code and documentation in the fsx-ontap-vscode GitHub repository to learn about extensibility, issue reporting, and contribution guidelines.

Try out the FSx for ONTAP VS Code extension today

The FSx for ONTAP VS Code extension simplifies how developers and platform teams interact with storage on AWS. Combining automation, AI assistance, and best practices within VS Code turns complex infrastructure tasks into intuitive, conversational operations.

For teams managing FSx for ONTAP on AWS, this isn't just added convenience, it's a productivity enhancer that supports faster deployment while maintaining quality, security, and cost efficiency.

Ready to explore the difference?

• Install from the Visual Studio Code Marketplace.
• Explore the open-source project on GitHub.
• Learn more about Amazon FSx for NetApp ONTAP.

Understanding ARP/AI Functionality

The ARP/AI operates by analyzing file characteristics without accessing actual file contents, ensuring data privacy while identifying suspicious encrypted files through probabilistic modeling.

1. It monitors saved data for signals such as data entropy, encrypted file content and headers to detect potential ransomware without opening files 
2. Upon detection, ARP/AI can automate responses like taking an immutable snapshot, evidence creation, alert generation, and with additional services like user blocking using Ransomware Protection Services in NetApp Console.

Data Safety and AI Model Design

The AI model is purpose-built for ransomware detection, with no command line access or use for other tasks. It assigns probabilities to modified files indicating the likelihood of malicious encryption, ensuring no file content is used for training or decision-making.

Evolution of ARP/AI

Since its initial introduction in ONTAP 9.10.1 with basic detection capabilities, ARP/AI has undergone significant advancements. The introduction of learning modes  in ONTAP 9.10.1, pretrained model and automatic model  updates introduced in ONTAP 9.16.1, have continued to enhance its functionality.

Model Training Process

ARP/AI models are trained in a lab sandbox using clean files from open datasets, exposed to known malware to generate corrupted files.. Along See Figure 1 for a visual explanation of the process.

Figure 1 – Generating the AI Model Package for ONTAP
To ensure data safety, ARP/AI creates snapshots every four hours and additional snapshots upon ransomware detection. These snapshots allow rollback to pre-infection states and are marked for deletion after specified durations, which are adjustable by users. See Figure 2 for an explanation of how this works.

Figure 2 – How ARP/AI uses Snapshots to protect data in ONTAP 9.17.1

ARP/AI for SAN Environments

Introduced in ONTAP 9.17.1, ARP/AI for SAN detects ransomware at the volume level using data entropy. It reports volumes infected by ransomware without pinpointing specific files inside complex storage like VMDKs, enabling rollback of entire volumes. This capability is now also available in NetApp’s new ASA r2 storage systems.

In Figure 3 below, the image shows that NFS/CIFS volumes detects ransomware at the file level. In the example it shows regular files (.docx, .jpg, etc) and VMware VMDK virtual disk files. If there is ransomware within the virtual disk, ARP/AI would point you at the VMDK but is unable to “see inside” the VMDK to detect the ransomware infected file itself.

ARP/AI for SAN will show you which volume contains ransomware. In the example, there is a LUN on the SAN volume, and, in this example, it is formatted with VMware’s VMFS file system.

On that file system are two virtual machines and their files. If ransomware got into one of the virtual machines, ARP/AI would detect that the whole VOLUME is suspect. This is because ARP/AI for SAN detects ransomware at the block level. ONTAP doesn’t have visibility into both the VMFS file system and the VMDK’s that reside in it.

Note that the VMFS file system is only being used as an example. ARP/AI for SAN is agnostic to the contents of the SAN volume because it is looking for ransomware at the volume/block level, not at the LUN or filesystem level. This means that the virtual machine could be running on a different hypervisor with a different filesystem.

In either case, File or SAN, you are notified that ransomware exists, and you can respond automatically via a snapshot and/or additional methods that may be unique for your environment.

Figure 3 – ARP/AI for Block vs NFS/CIFS

Beginning with ONTAP 9.17.1, ARP requires an evaluation period to determine if entropy levels for SAN volume workloads are suitable for ransomware protection. After ARP is enabled on a SAN volume, it monitors data continuously during an evaluation period to determine an optimal encryption threshold. ARP distinguishes between suitable and unsuitable workloads in the evaluated SAN volume and, if the workloads are determined to be suitable for protection, automatically sets an encryption threshold based on evaluation period statistics.

User Interface and Thresholds

Enabling ARP/AI is straightforward via the The AI triggers on multiple signals, detecting abnormal entropy spikes and suspected ransomware, with thresholds that automatically adjust 

This is best covered in the documentation.

Wrap Up

To wrap things up, ARP/AI for ONTAP 9.17.1 is a great improvement in your security posture. Now it’s not just for file shares but for your SAN workloads as well! You are alerted immediately in ONTAP or via SYSLOG to your SIEM that ransomware has been found and a Ransomware discovery snapshot has been taken.

It’s been my personal belief for many years that the more security we can push down into the infrastructure at the closest level possible to the data is the best security. ARP/AI for ONTAP is another great feature that brings that capability to your storage.

Oh, and did I mention that it’s free? That’s right, it’s built in (not bolted on like some solutions!) and part of ONTAP. Give it a try, let us know what you think. It’s easy to enable (or disable if you really must) and will give you piece of mind should the bad actors decide you are their next target.

Your infrastructure teams face mounting pressures daily, and managing modern data infrastructure has never been more complex, or more critical. NetApp customers understand the high stakes: maintaining resilient operations while facing mounting demands on data center storage, governance, and business continuity.

That’s why we’re thrilled to introduce Infrastructure Health, the powerful new addition to Data Infrastructure Insights, designed to give you a smarter, unified approach to risk visibility, governance, and proactive issue resolution so you can prevent problems before they impact your operations.

Real-Time Intelligence for Your Infrastructure

Infrastructure Health delivers a real-time overview of your entire system's health, detects anomalies as they emerge, and provides actionable insights to ensure peak performance across your environment.

At its core, this feature consolidates key metrics and trends across storage and fabric infrastructure components, creating a unified view that eliminates blind spots. Built in monitors work continuously in the background, identifying patterns and potential issues before they escalate into service-impacting problems. It codifies enterprise policies across multi-vendor storage estates, turning scattered alerts into clear, prioritized scorecards to provide a true understanding of storage and fabric health at a glance.

What’s in it for you?

Comprehensive Health Dashboard: Infrastructure Health serves as your command center, presenting critical system metrics in an intuitive, easy-to-navigate interface. This centralized view eliminates the need to jump between multiple monitoring tools, saving time and reducing the risk of missing important indicators.

ML-Powered Anomaly Detection: Advanced algorithms analyze historical patterns and current performance metrics to identify deviations that might indicate emerging issues. This proactive approach allows your team to address problems during maintenance windows rather than during critical business hours.

Actionable Insights and Recommendations: The system doesn't just alert you to problems—it provides specific, actionable recommendations based on best practices and your environment's unique characteristics. These prescriptive insights help guide decision-making and reduce troubleshooting time.

Amplifying your monitoring impact

Effective monitoring is key to driving meaningful change. Strong governance is an essential foundation to ensure your infrastructure remains compliant, audit-ready, and aligned with business priorities. With Infrastructure Health, you can gain deeper insights to deliver:

Proactive Issue Resolution

Gone are the days of firefighting infrastructure problems. Infrastructure Health enables early detection of performance degradations, allowing your team to resolve issues before they impact end users or critical services. This shift from reactive to proactive management significantly improves service reliability, delivers unified risk visibility, enables proactive issue resolution, and enhances business resilience by reducing unplanned downtime and associated revenue loss.

For existing Data Infrastructure Insights customers, this enhancement builds upon the existing monitoring capabilities, adding a layer of intelligence that makes your current investment even more valuable. New users will immediately benefit from this proactive approach, experiencing faster time-to-value as they implement Data Infrastructure Insights in their environments.

Improved Operational Efficiency

Time is your most valuable resource, and Infrastructure Health helps you use it more effectively. By streamlining monitoring processes and providing clear, prioritized alerts, your team spends less time troubleshooting and more time on strategic initiatives that drive business value. Centralized monitoring minimizes alert fatigue and boosts IT team productivity by accelerating mean time to resolution (MTTR). Holistic infrastructure health insights support data-driven decisions to optimize resource allocation and lower operational costs.

The unified dashboard reduces context switching between tools, while intelligent filtering ensures you focus on the most critical issues first. This efficiency gain translates directly into cost savings and improved team productivity. Enhanced compliance and audit readiness are built in through comprehensive risk tracking to mitigate potential fines and reputational damage.

Unified Visibility Across Your Environment

One of the most significant advantages of Infrastructure Health is its ability to provide comprehensive visibility across multiple categories and vendor platforms. Whether you're managing NetApp storage systems alongside third-party solutions or operating in a purely NetApp environment, this unified view eliminates silos. The hierarchical views organize information by region and data center, supporting governance and executive visibility. Unified risk scores per location make it easy to drill down from high-level overviews to specific component details. This flexibility ensures that both executive stakeholders and technical teams get the information they need in the format that makes most sense for their role.

Seamless Integration

Infrastructure Health integrates seamlessly with your existing Data Infrastructure Insights deployment. For current customers, the feature activates through a simple configuration process that leverages your existing data sources and monitoring infrastructure. This approach minimizes implementation time and ensures continuity of service.

Scalable Design

Built with enterprise scalability in mind, Infrastructure Health handles environments of any size, from single data centers to global, multi-region deployments. The system automatically adjusts its monitoring scope and analysis depth based on your environment's complexity and requirements. This proactive approach to infrastructure monitoring aligns with broader industry trends toward predictive maintenance and AI-driven operations.

Take the Next Step

Whether you're an existing Data Infrastructure Insights customer looking to enhance your monitoring capabilities or a NetApp customer exploring comprehensive infrastructure monitoring solutions, Infrastructure Health offers immediate value and long-term strategic benefits.

Ready to transform your infrastructure monitoring approach? Schedule a personalized demo to see Infrastructure Health in action.

In today’s fast-paced IT landscape, managing complex environments with growing storage footprints, multiple sites, and diverse workloads have become increasingly challenging. Organizations face fragmentation, inefficiency, higher operational risks, and slower response times due to the proliferation of management tools, each with its own login, workflow, and learning curve. Enter NetApp Console – the game-changing solution designed to address these challenges head-on.

One Console. Simplified Operations. Security First.

NetApp® Console is built on a restructured and enhanced foundation, encompassing platform investments, core services, enterprise readiness, security, administration and functionality. It is now the intuitive, intelligent, highly secure, and compliant single point of control for seamless management for your NetApp intelligent data infrastructure. With its reimagined user interface and experience, managing your NetApp Data Services and NetApp storage and has never been more intuitive, smarter, and insightful.

Centralized Management Interface

NetApp Console offers seamless management of storage and Data Services across your NetApp systems. With one secure login and one intuitive interface, it centralizes control, enabling administrators to deploy, discover, configure, monitor, and maintain systems, wherever your data lives.  The integrated NetApp manageability tools, that you know and love, improve agility with centralized access and usage. You can orchestrate NetApp ONTAP protection and resilience by activating and managing Data Services.

NetApp Console storage management

 Intelligent, Simplified Operations

Manual, repetitive processes waste time and increase the risk of mistakes. Console replaces them with intelligent, guided workflows that accelerate administration, simplifies those processes, and reduces errors and risks.  From initial onboarding assistant that gets you operational quickly—covering account association, agent deployment, and marketplace connectivity—to ongoing management, Console is designed for speed and simplicity. The Console agent’s simplified deployment makes it easy to secure connectivity between environments while providing centralized administration and monitoring. Easily manage your NetApp systems and enable Data Services with drag and drop functionality and wizard driven configurations.

The new home page delivers a consolidated view of insightful dashboards, quick access to recent activity, links to demos, videos, and important features and Data Services, and information on updates and resources. The dashboards provide real-time visibility into performance, alerts, protection, and resource utilization—identifying any issues or concerns so you can proactively address them before they impact business continuity. Intuitive automated workflows guide provisioning, monitoring, and optimization of your storage and Data Services, reducing manual effort, risk, and complexity. Built-in licenses and subscription administration gives you real-time visibility into usage patterns, helping you track costs, identify trends, optimize resource allocation, and manage billing.

NetApp Console home page

Security-First Foundation

Rising threats and compliance requirements demand stronger security. NetApp Console is zero-trust and compliance ready, and enforces identity verified access control, least-privilege permissions, credential protection, and incorporates an advanced multi-tenancy model. Multi-factor authentication and federated identity ensure that only verified users gain access, no exceptions. Console is purpose-built and certified to meet strict compliance and governance requirements—ready for HIPAA, SOC 2, Texas state and local mandates, and more.

Fine-grained access controls and policy-driven boundaries allow external partners to provide real-time secure collaboration and support without compromising visibility, authority, or compliance. Read-only modes give safe, non-destructive visibility for those who need to see but not change. Real-time monitoring and anomaly detection help identify unusual activity early, while comprehensive activity auditing strengthens governance and compliance readiness.

NetApp Console identity and access

Why It Matters

NetApp Console is built for the way you work today—and ready for how you’ll scale tomorrow. Whether managing a single ONTAP cluster or dozens of systems across multiple sites, the NetApp Console provides you with centralized, intelligent, and secure management experience.

Conclusion

NetApp Console is more than just a control tool; it is a transformative management plane that centralizes, simplifies, and secures your IT operations. Embrace the future of IT management with NetApp Console.

One Console. Simplified Operations. Security First.

Take command of your NetApp Data Services and storage with NetApp Console.

Excited to learn more? Read the solution brief.

Want to kick the tires? Take your own Console demo journey with these guided demos.

Ready to get started? Login into NetApp Console.

In today’s data-driven world, organizations are shifting from siloed data stacks toward intelligent, composable data platforms that support real-time analytics and AI workloads. By leveraging Spice.ai data engines, NetApp ONTAP S3 Lakehouse, and Azure NetApp Filers by Instaclustr,  we can connect various data sources like Dremio, PostgreSQL, and S3 data to build a modern, performant, and cost-efficient architecture for data applications.

Seeing is believing,  to watch a live demo and ask more questions, visit Will Stowe at NetApp Insight AI booth this week.

In this post, we’ll walk through how Spice.ai connects and queries multiple NetApp data sources, combines data, integrates LLM’s like GPT-4, and accelerates locally to present data and answers to end users in real-time.

Spice.ai —The ETL-less SQL Query Federation & Acceleration Platform

Spice.ai provides an intelligent data layer that combines SQL query federation & acceleration, hybrid search & retrieval, and LLM inference in a high-performance, lightweight runtime. It enables developers to query data across multiple sources with sub-second latency, cache frequently used data automatically, and integrate vector search and machine learning without complex pipelines.  With Spice.ai, you can:

• Connect to and query across multiple data sources (Postgres, S3, Dremio, etc.)
• Accelerate queries locally alongside the application - providing faster query times by eliminating network latency
• Run vector, keyword, and full-text search in the same SQL query
• Serve local or hosted LLM models 

Dremio — The Query Acceleration Engine

Dremio acts as a data Lakehouse query platform, enabling self-service analytics without needing to move or copy data. 

It connects seamlessly to object storage and databases, providing:

• High-performance SQL queries on data lakes
• Semantic layers for business-friendly data models
• Native Apache Iceberg support for modern table formats

Using Dremio, you can expose NetApp S3 data as tables and join them with structured data in Postgres - without ETL

PostgreSQL — The Structured Backbone

PostgreSQL remains the go-to relational database for structured transactional data. Whether you’re storing user profiles, product catalogs, or reference tables, Postgres provides robust SQL capabilities and easy integration with both Dremio and Spice.ai.

Quick easy beyond believe

Getting started with Spice.ai is incredibly straightforward. Here’s how you can set up and begin querying your data in just a few steps:

1. Get started with Spice.ai in just 30 seconds: Visit Spice.ai Documentation to get started quickly.  

2. Configure with a powerful and easy yaml file: Below is an example.

3. Run your spice.ai setup using the command: spice run

4. Check SQL connection: spice sql; show tables

5. Use gtp-4o with spice chat to get your question answered with data and summaries: spice chat

By combining the power of Spice.ai, Dremio, PostgreSQL, and NetApp ONTAP S3 Lakehouse, you can build a modern data platform that supports real-time analytics and AI workloads. This architecture is not only performant and scalable but also cost-efficient, making it ideal for today's data-centric applications.

Get started today and transform your data infrastructure into a powerful, intelligent platform that drives business insights and innovation.

If you’re running Microsoft Hyper-V, you’ve probably felt the pain of juggling many siloed tools, chasing down performance issues, and never quite having the full picture of how your VMs, hosts, and storage all fit together. It’s all too easy to get stuck context-switching between Hyper-V Manager, Perfmon, and your storage array’s UI or CLI, just trying to figure out where the real problem is. That leads to slow troubleshooting, finger-pointing, and sometimes overprovisioning “just in case.” 

This is where NetApp Data Infrastructure Insights (DII) really changes the game. DII gives you true end-to-end visibility, proactive alerting, and AI-powered troubleshooting across your entire Hyper-V environment, from the VM, through the host, all the way to your NetApp storage. 

Let’s take a look at how DII helps Hyper-V and storage teams break down silos, solve problems faster, and make the most of your resources. 

End-to-End Hyper-V Observability 

You can’t fix what you can’t see. DII consolidates and normalizes metrics from your Hyper-V hosts, VMs, datastores, and your NetApp storage backend (ONTAP, ASA, AFF, or FAS), giving you a clear, unified view of your entire service path. If you’re running fibre channel, DII even maps out your FC fabric, configuration, performance, and with alerts included. 

All this data comes together in a single topology model, so you can actually see how a VM’s performance flows through the host (and fabric) to the storage LUN or volume. 

With DII you get: 

• Unified topology maps that show VMs, Hyper-V hosts, datastores (on NAS or iSCSI/FC), and NetApp volume details, all in one place, thanks to the new VM and SAN Analyzer features. 

• At-a-glance performance and capacity trends for every VM, host, and volume (IOPS, throughput, latency, capacity, and more). 

• The ability to explore VM and host counters (CPU, capacity, disk and IP throughput, etc.) right alongside storage metrics, in a visual, intuitive way you just can’t get from native tools. 

Why does this matter? 

When you can visually trace an application’s performance from the VM guest all the way back to the storage volume in a single view, you cut out the guesswork. Troubleshooting gets faster, and both Hyper-V and storage teams are finally working from the same playbook. 

Scenario: Pinpointing a Latency Spike in Minutes 

1. A Hyper-V admin receives an alert that a workload on a VM is running slowly.  
2. Instead of reacting by scrambling between Windows Performance Monitor, Hyper-V Manager, and array GUIs, they proactively open the DII alert for the VM’s high disk latency.  
3. The timeline view of the alert shows the latency spike (55 ms).  

4. Through use of the VM Analyzer the admin reviews the service path showing end to end, from VM to volume for this VM.  

5. And by focusing on the Host for the VM, the topology map highlights that the VM shares its HyperV host with 30 other VMs and 3 other datastores. 

 6. The admin drills into the HyperV host CPU utilization and confirms the jump above 95% at the same time indicating CPU saturation from the multiple VMs. 

Within minutes, the admin identified that a high host CPU load, and not the storage array, is causing the slowdown. They coordinate with the virtualization team who confirms 30 high CPU workloads had been inadvertently launched at that time and agree to reschedule or throttle the workloads, and watch DII confirm that the VM latency returns to normal and the alert resolves. 

Proactive Monitoring and Change Analysis 

Let’s face it, being reactive is exhausting and usually means you’re one step behind. With DII, you can set up Monitors for your Hyper-V hosts and VMs, define thresholds for any metric that matters (CPU, memory, storage latency, you name it), and get alerted the moment something’s off. 

But it doesn’t stop there. DII’s Anomaly Detection and Change Analysis features let you see exactly what changed, whether it was an OS update, a policy tweak, or a storage config change, that triggered the alert. 

Here’s what’s in it for Hyper-V teams: 

• AI-driven anomaly detection flags inconsistencies before users even notice. 

• Automated change correlation ties metric shifts to real-world changes, which means no more hunting for what happened. 

• Prebuilt Monitor templates for all the critical counters (host CPU, memory, disk IOPS/latency, IP throughput) let you hit the ground running. 

Why does this matter?

It means no more bouncing between tools, no more endless log-diving, and way less time spent figuring out who needs to fix what. You get the context you need, right when you need it, and both virtualization and storage admins can actually work together to solve problems before they become tickets. 

Combined Benefits for Virtualization and Storage Teams 

When Hyper-V admins and NetApp storage managers are on the same page with DII, everybody wins. Here’s what that looks like: 

• Faster MTTR: AI-powered alerts, unified topology, and change analysis mean you’re solving problems in minutes, not days. 

• Smarter capacity planning: Trending and forecasting tools help you right-size hosts and volumes before things get tight. 

• Better teamwork: Shared dashboards and clear analysis take the blame game out of troubleshooting and speed up decisions. 

• Real business value: Proactive monitoring keeps users happy, maintains your SLAs, and helps you avoid costly overprovisioning. 

Curious to see DII in action for your Hyper-V environment? 

Check out the NetApp Data Infrastructure Insights product page and request a personalized demo to see how DII can help you turn complex Hyper-V and storage operations into something a whole lot simpler and more efficient. 

I'm excited to kick off a new blog series called Back to Basics (B2B). The goal is to revisit fundamental concepts that often slip through the cracks because the world is going too fast, life happens, and you have to keep all the plates spinning without breaking any of them. 

Mounting an NFS share might seem like a straightforward task - but have you ever wondered what really happens behind the scenes when you do it on a Linux client? To answer this question, we'll take a closer look at the NFS protocol and use some powerful tools to peek under the hood from the client's perspective. Let's dive in!

Network File System (NFS)

sudo modprobe sunrpc
sudo modprobe nfs
sudo modprobe nfsv4
sudo modprobe nfs_layout_nfsv41_files

echo 1 | sudo tee /sys/kernel/tracing/events/sunrpc/enabled
echo 1 | sudo tee /sys/kernel/tracing/events/nfs/enabled
echo 1 | sudo tee /sys/kernel/tracing/events/nfs4/enabled

cat /sys/kernel/tracing/trace_pipe

sudo mount -t nfs4 -o vers=4.1,rsize=65536,wsize=65536,proto=tcp 10.26.128.221:/vol_cp_5node /cp

           <...>-3331    [012] .....   949.293271: nfs_mount_option: option source
           <...>-3331    [012] .....   949.293276: nfs_mount_option: option vers
           <...>-3331    [012] .....   949.293277: nfs_mount_assign: option vers=4.1
           <...>-3331    [012] .....   949.293278: nfs_mount_option: option rsize
           <...>-3331    [012] .....   949.293279: nfs_mount_option: option wsize
           <...>-3331    [012] .....   949.293280: nfs_mount_option: option proto
           <...>-3331    [012] .....   949.293280: nfs_mount_assign: option proto=tcp
           <...>-3331    [012] .....   949.293282: nfs_mount_option: option addr
           <...>-3331    [012] .....   949.293282: nfs_mount_assign: option addr=10.26.128.221
       mount.nfs-3331    [012] .....   949.293283: nfs_mount_option: option clientaddr
       mount.nfs-3331    [012] .....   949.293283: nfs_mount_assign: option clientaddr=10.26.56.192
       mount.nfs-3331    [012] .....   949.293286: nfs_mount_path: path='/vol_cp_5node'

       mount.nfs-3331    [012] .....   949.293390: xprt_create: peer=[10.26.128.221]:2049 state=BOUND
       mount.nfs-3331    [012] .....   949.293405: rpc_clnt_new: client=00000000 peer=[10.26.128.221]:2049 program=nfs server=10.26.128.221 xprtsec=none flags=DISCRTRY|INFINITE_SLOTS|NO_RETRANS_TIMEOUT
       mount.nfs-3331    [012] .....   949.293407: rpc_task_begin: task:00000001@00000000 flags=NULLCREDS|DYNAMIC|SOFT|SOFTCONN|CRED_NOREF runstate=ACTIVE status=0 action=0x0
       mount.nfs-3331    [012] .....   949.294496: rpc_connect_status: task:00000001@00000000 status=-11

       mount.nfs-3331    [012] .....   949.295698: rpc_request: task:00000002@00000000 nfsv4 EXCHANGE_ID (sync)

       mount.nfs-3331    [012] .....   949.296538: rpc_stats_latency: task:00000002@00000000 xid=0x734ec2e0 nfsv4 EXCHANGE_ID backlog=11 rtt=719 execute=841
       mount.nfs-3331    [012] .....   949.296541: nfs4_exchange_id: error=0 (OK) dstaddr=10.26.128.221

   10.26.128.221-m-3333    [001] .....   949.297657: rpc_request: task:00000004@00000000 nfsv4 CREATE_SESSION (sync)
   10.26.128.221-m-3333    [001] .....   949.298389: rpc_stats_latency: task:00000004@00000000 xid=0x754ec2e0 nfsv4 CREATE_SESSION backlog=10 rtt=583 execute=731

   10.26.128.221-m-3333    [001] .....   949.298393: nfs4_create_session: error=0 (OK) dstaddr=10.26.128.221
   10.26.128.221-m-3333    [001] .....   949.299115: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=1 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()

       mount.nfs-3331    [012] .....   949.299259: rpc_request: task:00000001@00000001 nfsv4 SECINFO_NO_NAME (sync)
       mount.nfs-3331    [012] .....   949.300992: rpc_stats_latency: task:00000001@00000001 xid=0x774ec2e0 nfsv4 SECINFO_NO_NAME backlog=718 rtt=1612 execute=2444
       mount.nfs-3331    [012] .....   949.305147: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=3 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.305148: rpc_request: task:00000002@00000001 nfsv4 LOOKUP_ROOT (sync)
       mount.nfs-3331    [012] .....   949.306163: rpc_stats_latency: task:00000002@00000001 xid=0x784ec2e0 nfsv4 LOOKUP_ROOT backlog=10 rtt=782 execute=1020
       mount.nfs-3331    [012] .....   949.306164: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=3 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.306168: nfs4_lookup_root: error=0 (OK) fileid=00:00:64 fhandle=0x507c44a0 valid=TYPE|MODE|NLINK|OWNER|GROUP|SIZE|FSID|FILEID|ATIME|MTIME|CTIME|CHANGE|0x400200

       mount.nfs-3331    [012] .....   949.306786: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=5 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.306786: rpc_request: task:00000004@00000001 nfsv4 FSINFO (sync)
       mount.nfs-3331    [012] .....   949.307693: rpc_stats_latency: task:00000004@00000001 xid=0x7a4ec2e0 nfsv4 FSINFO backlog=3 rtt=897 execute=907
       mount.nfs-3331    [012] .....   949.307693: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=5 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.307694: nfs4_fsinfo: error=0 (OK) fileid=00:00:64 fhandle=0x507c44a0 valid=TYPE|MODE|NLINK|OWNER|GROUP|SIZE|FSID|FILEID|ATIME|MTIME|CTIME|CHANGE|0x400200
       mount.nfs-3331    [012] .....   949.307696: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=6 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.307697: rpc_request: task:00000005@00000001 nfsv4 SERVER_CAPS (sync)
       mount.nfs-3331    [012] .....   949.308249: rpc_stats_latency: task:00000005@00000001 xid=0x7b4ec2e0 nfsv4 SERVER_CAPS backlog=3 rtt=452 execute=552
       mount.nfs-3331    [012] .....   949.308249: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=6 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.309855: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=10 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.309855: rpc_request: task:00000009@00000001 nfsv4 GETATTR (sync)
       mount.nfs-3331    [012] .....   949.310555: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=10 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 () 
       mount.nfs-3331    [012] .....   949.310557: nfs4_getattr: error=0 (OK) fileid=00:2f:64 fhandle=0x507c44a0 valid=TYPE|MODE|NLINK|OWNER|GROUP|SIZE|FSID|FILEID|ATIME|MTIME|CTIME|CHANGE|0x400200

       mount.nfs-3331    [012] .....   949.310575: nfs_access_enter: fileid=00:2f:64 fhandle=0x507c44a0 version=1881467492116116952
       mount.nfs-3331    [012] .....   949.310577: nfs_access_exit: error=-10 (CHILD) fileid=00:2f:64 fhandle=0x507c44a0 type=4 (DIR) version=1881467492116116952 size=4096 cache_validity=0x0 () nfs_flags=0x0 () mask=0x81 permitted=0xffffffff
       mount.nfs-3331    [012] .....   949.310577: nfs_access_enter: fileid=00:2f:64 fhandle=0x507c44a0 version=1881467492116116952
       mount.nfs-3331    [012] .....   949.310580: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=11 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.310580: rpc_request: task:0000000a@00000001 nfsv4 ACCESS (sync)
       mount.nfs-3331    [012] .....   949.311414: rpc_stats_latency: task:0000000a@00000001 xid=0x804ec2e0 nfsv4 ACCESS backlog=3 rtt=688 execute=834
       mount.nfs-3331    [012] .....   949.311415: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=11 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.311421: nfs4_access: error=0 (OK) fileid=00:2f:64 fhandle=0x507c44a0
       mount.nfs-3331    [012] .....   949.311422: nfs_access_exit: error=0 (OK) fileid=00:2f:64 fhandle=0x507c44a0 type=4 (DIR) version=1881467492116116952 size=4096 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET) mask=0x1 permitted=0x7
       mount.nfs-3331    [012] .....   949.311428: nfs_lookup_enter: flags=0x5 (FOLLOW|AUTOMOUNT) name=00:2f:64/vol_cp_5node fileid=0
       mount.nfs-3331    [012] .....   949.311429: nfs4_setup_sequence: session=0x87e0d01f slot_nr=0 seq_nr=12 highest_used_slotid=0
       mount.nfs-3331    [012] .....   949.312312: nfs4_sequence_done: error=0 (OK) session=0x87e0d01f slot_nr=0 seq_nr=12 highest_slotid=63 target_highest_slotid=63 status_flags=0x0 ()
       mount.nfs-3331    [012] .....   949.312317: nfs4_lookup: error=0 (OK) name=00:2f:64/vol_cp_5node
       mount.nfs-3331    [012] .....   949.312320: nfs_lookup_exit: error=0 (OK) flags=0x5 (FOLLOW|AUTOMOUNT) name=00:2f:64/vol_cp_5node fileid=105

       mount.nfs-3331    [012] .....   949.315199: nfs_mount_option: option source
       mount.nfs-3331    [012] .....   949.315201: nfs_mount_path: path='/vol_cp_5node'
       mount.nfs-3331    [012] .....   949.316737: rpc_clnt_shutdown: client=00000002
       mount.nfs-3331    [012] .....   949.316738: rpc_clnt_release: client=00000002
    kworker/12:1-450     [012] .....   949.316760: rpc_clnt_free: client=00000002
       mount.nfs-3331    [012] .....   949.328981: rpc_clnt_shutdown: client=00000001
       mount.nfs-3331    [012] .....   949.328990: rpc_clnt_release: client=00000001
    kworker/12:1-450     [012] .....   949.329007: rpc_clnt_free: client=00000001

For organizations adapting to changes in virtualization licensing while optimizing total cost of ownership (TCO), OpenShift Virtualization combined with NetApp ONTAP offers a powerful, future-ready solution. This presents a compelling alternative option for virtualization. Built on KubeVirt, an open-source technology, it enables seamless management of virtual machines (VMs) alongside containers within a unified Kubernetes environment. This integration allows IT teams to consolidate workloads on a single platform, streamlining operations and accelerating cloud-native transformation. NetApp ONTAP stands out as the most versatile data platform for virtualization, on-premises or in the cloud. It supports all major hypervisors in on-premises environment and offers native data services across leading public clouds. ONTAP’s integration empowers organizations to:

• Seamlessly adapt to licensing changes
• Choose the optimal deployment model for each workload
• Maintain consistent data management across hybrid and multi-cloud environments

This flexibility positions ONTAP as a strategic enabler for organizations navigating the evolving virtualization landscape.

What is OpenShift virtualization

Key Capabilities

• KVM-based VMs run in containers, managed as Pods
• Fully scheduled, deployed, and managed by Kubernetes
• Intuitive VM lifecycle management via the OpenShift web console
• Integrated with Kubernetes-native persistent storage and orchestration

Storage & Data Protection with NetApp Trident

• Trident CSI: Dynamic storage provisioning over NFS, FC, iSCSI, NVMe/TCP
• Granular VM-level operations: snapshots, clones, volume expansion

Trident Protect:

• Crash-consistent VM backups to S3-compatible storage
• Disaster recovery with replication, failover, and failback

Beyond the powerful storage features like snapshots, clones, and efficiency, NetApp ONTAP plays a key role in streamlining VM migration to OpenShift Virtualization. Depending on the number of VM disks and their size, the available migration options will take some time. This is where ONTAP along with cloning capability via Shift toolkit make the migration seamless. Call it the virtual machine migration offloader to storage.

Why Shift toolkit

Shift toolkit combines high-fidelity format conversion with native storage cloning and efficiency to convert a VMDK to RAW and handles it at scale & speed. The metadata only nature of FlexClone means the conversion can be done in seconds to minutes (for example, 2 TB in seconds). This allows customers to run VM conversions with minimal storage overhead and near-zero downtime. To emphasize the point, there is no data copy in here, meaning the conversion happens instantly. This saves time and compute resources. This makes ONTAP a strategic enabler for large-scale migrations involving tens of thousands of VMs, delivering efficiency without disruption.

While VMKFStools cloning with copy offload is a supported method, it still requires data movement across volumes, even though the process is offloaded to the storage layer. In contrast, the Shift Toolkit significantly outperforms XCOPY offload. It acts as a VM migration offloader that eliminates the need for actual data copying. The Shift Toolkit API enables intra-volume copies, where both the source and destination reside within the same volume. By leveraging NetApp’s proprietary deduplication technology, the destination file typically occupies only a few kilobytes for metadata, allowing clone operations to complete almost instantly.

Importantly, this approach avoids reliance on deprecated protocols within the VMware ecosystem. Instead, it uses supported, traditional datastores to enable fast and seamless VM migration.

This blog covers how the NetApp Shift Toolkit brings the same seamless migration experience to Red Hat OpenShift Virtualization as it does for hypervisors like Hyper-V and provides a step-by-step guide on transitioning to OpenShift Virtualization using Shift Toolkit’s conversion APIs. Looking ahead, future releases of the toolkit will offer enhanced capabilities and deeper integration to further streamline the migration journey.

Prerequisites

• OpenShift cluster with OpenShift Virtualization operator and NetApp Trident CSI driver installed.
• Shift toolkit v2.2 installed
  • No additional configuration required. Since only Shift toolkit API is used in this case, there is no need to configure Shift toolkit resource groups or blueprint.

• Administrator level privileges on the OpenShift cluster
• A Linux instance (this can be an existing management instance as well) with tridentctl and OC command line tool installed.
• kubeconfig exported or OC login shd be executed to connect to the cluster.
• Download the script named - Shift VM to OpenShift from Shift toolkit script block and copy the zip file to this Linux instance.
• Unzip the zip using unzip <filename> (unzip 'shift-vm-to-openshift.zip')
• Ensure Python3 is installed (dnf install python3)
• Install openjdk 8 or later version (yum install java-1.8.0-openjdk)
• Run pip install -r requirements.txt (need internet)

VM requirements:

• Ensure the VMDKs for a VM are placed on individual volumes. For instance, consider a VM with 3 disks. Each of the disk should be placed on its individual volume (mapping datastore to PVC construct). This needs to be done manually using storage vmotion.
• The script runs against RHEL 8 and 9, SLES 12 and Windows 2019/2022

Note: The script doesn’t assign Network now. This needs to be manually assigned. Will be covered in a later update.

PrepareVM requirements (for injecting Virtio drivers):

• PrepareVM script is embedded in the zip file. Use powershell to execute the script from the Shift toolkit VM.
• Ensure vmware.powercli module is installed (install-module VMware.PowerCLI -scope AllUsers -force -SkipPublisherCheck -AllowClobber)
• For Windows VM, mount the virtio iso to the VM. The virtio driver can be downloaded from here.

Preparing the VM

Shift toolkit runs the VirtIO driver, qemu-agent installation and fstab update script. To accomplish the same, Shift toolkit uses powercli to connect to the guest VMs (Linux or windows) and updates the VirtIO drivers ensuring the necessary drivers are available once the VM boots up on OpenShift side.

The script for Linux and Windows can be found in the zip extract. The syntax to use is as follows:

.\prepare-vm.ps1 -vCenter "IP or FQDN" -vCenterUser "administrator@domain.com" -vCenterPass "password" -vmName "vm name" -guestUser "user" -guestPass "password" -osFlavor "linux" -localWindowsScriptPath "prepare-vm-win.ps1" -localLinuxScriptPath "prepare-vm-linux.sh"

Note: The script remains the same for Linux or windows distro, just need to ensure the right parameters are passed.

Discovering VMs from vCenter and volumes on ONTAP storage

In order to start the process, update the json file with the following details:

• ONTAP Cluster
  • This can be a SVM, its management IP and vsadmin credentials
  • Splitclone parameter can be set to “False” if the clone volume doesn’t need to be detached.
• vCenter
  • Min RBAC rights to access the vCenter to discover the VMs and associated VMDK files.
• Shift toolkit
  • The Shift toolkit VM IP and the credentials to access the same. If using the default credentials, it will be admin/admin
• Trident storage class
  • Should be NFS backend and ensure the right version is updated
• OpenShift
  • Specify the project name (default is used as an example)

Note: Keep the rest of the values as default

High level workflow:

Once the pre-reqs are met, simply execute python3 main.py which will do the orchestration.

Below sections cover what steps are triggered by Shift toolkit APIs to convert the VMDK ad create a VM on the OpenShift side.

Convert VMDK

Once the json file is updated, the script will automatically find the VMDKs associated with each VM including the primary boot disk. The conversion happens in seconds compared to other approaches.

Note: if there are multiple VMDK files, each VMDK will be converted.

Upload the RAW image to OpenShift Virtualization Cluster

With the virtual machine disk image converted to RAW format, script uploads the RAW file to OpenShift cluster and import them individually using trident.

Create Virtual Machine

Script generates a “YAML” for each VM depending on the OS.

Booting VM for the first time

Depending on the Virtual Machine OS, Shift toolkit will auto assign the VM boot option, along with the storage controller interface. In case of Linux distros, VirtIO is used. And for Windows, Shift toolkit starts with SATA interface and then auto installs VirtIO drivers and then changes the interface to VirtIO by modifying the YAML.

Note: Once the VM boots up, ensure to assign the network manually for this iteration.

All the above is covered by running python3 main.py

Fast, Simple VM Migration with ONTAP and Shift Toolkit

In this blog demo, a RHEL VM with 3 fully populated disks totalling 1.5 TB was migrated to OpenShift Virtualization in under 5 minutes. The VMDK to RAW conversion completed in seconds - thanks to ONTAP and Shift Toolkit.

This showcases a streamlined, low-impact approach to rehoming VMs using ONTAP storage. And if you're using block protocols, storage class migration is your go-to solution. This is just the beginning, stay tuned for deeper protocol level integration ahead.

Planning a Move to OpenShift Virtualization? Shift Toolkit Has You Covered.

If you are planning to migrate to OpenShift virtualization or already in the process, give Shift toolkit script a try. It’s completely free. Reach out to your NetApp team to register your interest and explore upcoming integrations that streamline operations.

• SnapMirror active sync and the active-active data center
• RPO=0
• RTO=0
• SnapMirror active sync architecture
• SAN design with SnapMirror active sync
• Uniform access
• Uniform access with local proximity
• Nonuniform access
• Uniform access with ASA
• SnapMirror active sync vs MetroCluster
• SnapMirror active sync setup
• Storage Virtual Machines (SVM)
• Provision storage
• Create the volumes
• Create the LUNS
• Define the CG
• Establish replication
• Create destination volumes
• Initialize replication
• Define the igroup
• Map the LUNs
• Oracle Configuration
• Device Names
• ASM Configuration
• Database Creation
• Failure scenarios
• Preferred sites
• Oracle and css_critical
• SnapMirror active sync preferred site
• Loss of SnapMirror replication connectivity
• Site A
• Site B
• Loss of Oracle RAC replication
• Complete loss of replication network
• Storage system failure
• Cut the power to the mediator
• Restoring services
• SnapMirror active sync failover
• Summary

SnapMirror active sync and the active-active data center

I've been beating up on SnapMirror active sync in the lab for about a year now, and I have to say this is the coolest feature I’ve seen added to ONTAP in years . In particular, I'm talking about SnapMirror active sync running in symmetric active-active mode. The value isn’t in the feature itself, it’s in the sorts of solutions it allows you to create. The foundation of SnapMirror active sync is SnapMirror synchronous, and it's been around much longer and is well-proven, but I only started using the active-active capabilities a year ago.

There are a number of ways to configure SnapMirror active sync, but the one I want to focus on here is an active-active application environment. You can build clustered environments including Oracle RAC, VMware, and Windows Server Failover Clusters (WSFC)  where the application cluster is distributed across two sites and IO response times are symmetric. You get RPO=0 through synchronous cross-site mirroring and RTO=0 through built-in ONTAP automation and the fact that storage resources are available on both sites, all the time.

This post should be useful for anyone looking at synchronous replication solutions, but I’m going to use Oracle RAC to illustrate SnapMirror active sync functionality because you can examine the RTO=0 and RPO=0 characteristics of the overall solution from the application layer all the way to the storage layer. The same database is operational at both sites, all the time. A disaster requires breaking a mirror, but you don’t really fail anything over because the services were running at the surviving site all along.

The examples used might be Oracle RAC, but the principles are universal. This includes concepts like "preferred site", heartbeats, tiebreaking, and timeouts. Even if you know next to nothing about Oracle RAC, read on.

Before diving into the details of how SnapMirror active sync (SM-as) works, I’d like to explain what SnapMirror active sync (SM-as) is for.

RPO=0

If you work in IT, you’ve definitely heard about the Recovery Point Objective. How much data can you afford to lose if certain events happen? If you're looking at synchronous replication, that means RPO=0, but that's more nuanced than it seems. When I see a requirement for RPO=0, I usually divide it into two categories:

• RPO for common problems
• RPO for disaster scenarios

RPO=0 for normal problems is fairly easy to accomplish. For example, if you have a database you should expect to need to recover it occasionally. Maybe a patch went wrong and caused corruption or someone deleted the wrong file. All you usually need is RAID-protected storage and a basic backup plan and you can restore the database, replay logs, and fix the problem with no loss of data. You should expect to have RPO=0 recoverability for these types of common problems, and if you choose ONTAP we can make the procedures easier, faster, and less expensive. 

If you want RPO=0 for disaster scenarios, things get more complicated. As an example, what if you have a malicious insider who decided to destroy your data and its backups? That requires a very different approach using the right technology with the right solution. We have some cool things to offer in that space.

SnapMirror active sync is about RPO=0 for disaster scenarios that lead to site loss. It might be permanent site loss because of a fire or maybe it's a temporary loss because of a power outage. Perhaps it's more isolated, like a power surge that destroys a storage array. SM-as addresses these scenarios with synchronous mirroring. It's not just RPO=0 replication either, it's a self-healing, integrated, intelligent RPO=0 replication solution.

RTO=0

A requirement for RPO=0 often comes with a requirement for Recovery Time Objective (RTO) of zero as well. The RTO is the amount of time you can be without a service, and overall it's an easily understood SLA, with one exception, which is RTO=0. What does that mean?

The term "RTO=0" across the whole IT industry. I continue to insist there’s no such thing as RTO=0. That would require instantaneously recognizing a problem exists and then instantly correcting it to restore service. That isn’t possible. From a storage perspective, there’s no way to know whether a 10 millisecond wait for a LUN to respond is because the storage system is merely busy with other IO or the data center itself fell into a black hole and no longer exists.

You can’t have RTO=0, but you can have a near-zero RTO from the perspective of IT operations. That exact definition of “near-zero” will depend on business needs. If an RTO is low enough that operations aren’t significantly affected, that’s what the industry calls RTO=0. It means “nondisruptive”.

Not all of the RTO is determined solely by storage availability. For example, a typical ONTAP controller failover completes in around 2-4 seconds, and during this time there will be a pause in SAN IO. The OS’s, however, often have much higher timeouts. ONTAP can fail over in 2 seconds and be fully ready to serve data again, but sometimes the host will wait 15 seconds before retrying an IO and you can’t change that. OS vendors have optimized SAN  behavior over the years, but it's not perfect. Those rare but real and occasionally lengthy IO timeouts are probably happening from time to time in your SAN already. Sometimes an FC frame will go missing because a cable is just slightly damaged and there will be a brief performance blip. It shouldn’t cause real-world problems, though. Nothing should crash.

Since SM-as failover operations should be nondisruptive to ongoing IT operations, including disasters, I’ll call SM-as an RTO=0 storage solution since that's the way the RTO term is used in the industry. 

You still have to think about the application layer, though. For example, you could distribute a VMware cluster across sites and replicate the underlying storage with SM-as, but failure of a given site would require VMware HA to detect the loss of a given VM and restart it on the surviving site. That takes some time. Does that requirement meet your RTO? If you can’t start a VM quickly enough, you can consider containerization. One of its benefits is a container can be activated nearly instantaneously. There’s no OS boot sequence. That might be an option to improve the big-picture RTO. Finally, if you’re using a truly active-active application like Oracle RAC, you can proactively have the same database running at both sites all the time. The right solution depends on the RTO and the behavior of the entire solution during failover events.

From a storage perspective, achieving that near-zero RTO is a lot more complicated than you’d think. How does one site know the other site is alive? What should happen if a write cannot be replicated? How does a storage system know the difference between a replication partner that is permanently offline versus a partner that is only temporarily unreachable? What should happen to LUNs that are being accessed by a host but are no longer guaranteed to be able to serve the most recent copy of data? What happens when replication is restored?

On to the diagrams…

SnapMirror active sync architecture

Let’s start with the basics of how SM-as replication works.

Here’s what this shows:

• These are two different storage clusters. One of them is Cluster1, and the other is Cluster2. (I'll explain the jfs_as1 and jfs_as2 labels a little later…)
• It might look like there are six LUNs but it’s logically only three.
• The reason is there are six different LUN images, but the LUN1 on each jfs_as1 is functionally the same as LUN1 on jfs_as2, the LUN2's are the same, and the LUN3's are the same.
• From a host point of view, this looks like ordinary SAN multipathing. LUN1 and its data is available and accessible on either cluster.
• IO behavior and IO performance are symmetric.
  • If you perform a read of LUN1 from the cluster on site A, the read will be serviced by the local copy of the data on site A
  • If you perform a read of LUN1 from the cluster on site B, the read will be serviced by the local copy of the data on site B
  • Performing a write of LUN1 will require replication of the write to the opposite site before the write is acknowledged.

Now let's look at failover. There’s another component involved – the mediator.

The mediator is required for safely automating failover. Ideally, it would be placed on an independent 3^rd site, but it can still function for most needs if it’s placed on site A or site B. The mediator is not really a tiebreaker, although that’s effectively the function it provides. It's not taking any actions; it’s providing an alternate communication channel to the storage systems.

The #1 challenge with automated failover is the split-brain problem, and that problem arises if your two sites lose connectivity with each other. What should happen? You don’t want to have two different sites activate themselves as the surviving copy of the data, but how can a single site tell the difference between actual loss of the opposite site and an inability to communicate with the opposite site?

This is where the mediator enters the picture. If placed on a 3^rd site, and each site has a separate network connection to that site, then you have an additional path for each site to validate the health of the other. Look at the picture above again and consider the following scenarios.

• What happens if the mediator fails or is unreachable from one or both sites?
  • The two clusters can still communicate with each other over the same link used for replication services.
  • Data is still served with RPO=0 protection
• What happens if Site A fails?
  • Site B will see both of the communication channels to the partner storage system go down.
  • Site B will take over data services, but without RPO=0 mirroring
• What happens if Site B fails?
  • Site A will see both of the communication channels to the partner storage system go down.
  • Site A will take over data services, but without RPO=0 mirroring

There is one other scenario to consider: Loss of the data replication link. If the replication link between sites is lost, RPO=0 mirroring will obviously be impossible. What should happen then?

This is controlled by the preferred site status. In an SM-as relationship, one of the sites is secondary to the other. This makes no difference in normal operations, and all data access is symmetric, but if replication is interrupted then the tie will have to be broken to resume operations. The results is that the preferred site will continue operations without mirroring and the secondary will halt IO processing until replication is reestablished. More on this topic below…

SAN design with SnapMirror active sync

As explained above, SM-as functionally provides the same LUNs on two different clusters. That doesn’t mean you necessarily want your hosts to access their LUNs across all available paths. You might, or you might not. There are two basic options called uniform and non-uniform access.

Uniform access

Let’s say you had a single host, and you wanted RPO=0, RTO=0 data protection. You’d probably want to configure SM-as with uniform access, which means the LUNs would be visible to your host from both clusters.

Obviously, this will mean that complete loss of site A would also result in the loss of your host and its applications, but you would have extremely high availability storage services for your data. This is probably a less common use case for SM-as, but it does happen. Not all applications can be clustered. Sometimes all you can do is provide ultra-available storage and accept that you’ll need to find alternate servers on a remote site in the event of disaster.

Another option with uniform access is in a full mesh networked cluster.

If you use SM-as with uniform access in a cluster as shown above, any host should always have access to a usable copy of the data. Even if one of the storage systems failed or storage replication connectivity was lost, the hosts would continue working. From a host multipath point of view, it all looks like a single storage system. A failure of any one component would result in some of the SAN paths disappearing off the network, but otherwise everything would continue working as usual.

Uniform access with local proximity

Another feature of SM-as is the option to configure the storage systems know where the hosts are located. When you map the LUNs to a given host, you can indicate whether or not they are local to the storage system.

In normal operation, all IO is local IO. Reads and writes are serviced from the local storage array. Write IO will, of course, need to be replicated by the local controller to the remote system before being acknowledged, but all read IO will be serviced locally and will not incur extra latency by traversing the SAN.

The only time the nonoptimized paths will be used is when all active/optimized paths are lost. For example, if the entire array on site A lost power, the hosts at site A would still be operational, although they would be experiencing higher read latency.

Note: There are redundant paths through the local cluster that are not shown on these diagrams for the sake of simplicity. ONTAP storage systems are HA themselves, so a controller failure should not result in site failure. It should merely result in a change in which local paths are used by an affected host.

Nonuniform access

Nonuniform access means each host has access to only a subset of available ports.

The primary benefit to this approach is SAN simplicity: you remove the need to stretch a SAN over the network. Some users don't have dark fiber between sites or lack the infrastructure to tunnel FC SAN traffic over an IP network. In some cases, the additional latency overhead of an application accessing storage across sites on a regular basis would be unacceptable, rendering the improved availability of uniform access unnecessary.

The disadvantage to nonuniform access is that certain failure scenarios, including loss of the replication link, will result in half of your hosts losing access to storage. Applications that run as single instances, such as a non-clustered database that is inherently only running on a single host at any given mount, would fail if local storage connectivity were lost. The data would still be protected, but the database server would no longer have access. It would need to be restarted on a remote site, preferably through an automated process. For example, VMware HA can detect an all-paths-down situation and restart a VM on another server where paths are available. In contrast, a clustered application such as Oracle RAC can deliver the same services that are constantly running at both sites. Losing a site with Oracle RAC doesn’t mean loss of the application service as a whole. Instances are still available and running at the surviving site. 

Uniform access with ASA

The diagrams above showed path prioritization with AFF storage systems. NetApp ASA systems provide active-active multipathing, including with the use of SM-as. Consider the following diagram:

An ASA configuration with non-uniform access would work largely the same as it would with AFF. With uniform access, IO would be crossing the WAN. This may or may not be desirable. If the two sites were 100 meters apart with fiber connectivity there should be no detectable additional latency crossing the WAN, but if the sites were a long distance apart then read performance would suffer on both sites. In contrast, with AFF those WAN-crossing paths would only be used if there were no local paths available and read performance would be better.

ASA with SM-as in a low-latency configuration offers two interesting benefits. First, it essentially doubles the performance for any single host because IO can be serviced by twice as many controllers using twice as many paths. Second, it offers extreme availability because an entire storage system could be lost without interrupting host access.

SnapMirror active sync vs MetroCluster

Those of you that know NetApp probably know SM-as' cousin, MetroCluster. SM-as is similar to NetApp MetroCluster in overall functionality, but there are important differences in the way in which RPO=0 replication is implemented and how it is managed.

• A MetroCluster configuration is more like one integrated cluster with nodes distributed across sites. SM-as behaves like two otherwise independent clusters that are cooperating in serving data from specified RPO=0 synchronously replicated LUNs.
• The data in a MetroCluster configuration is only accessible from one particular site at any given time. A second copy of the data is present on the opposite site, but the data is passive. It cannot be accessed without a storage system failover.
• MetroCluster and SM-as mirroring occur at different levels. MetroCluster mirroring is performed at the RAID layer. The low-level data is stored in a mirrored format using SyncMirror. The use of mirroring is virtually invisible up at the LUN, volume, and protocol layers.
• In contrast, SM-as mirroring occurs at the protocol layer. The two clusters are overall independent clusters. Once the two copies of data are in sync, the two clusters only need to mirror writes. When a write occurs on one cluster, it is replicated to the other cluster. The write is only acknowledged to the host when the write has completed on both sites. Other than this protocol splitting behavior, the two clusters are otherwise normal ONTAP clusters.
• The sweet spot for MetroCluster is large-scale replication. You can replicate an entire array with RPO=0 and near-zero RTO. This simplifies the failover process because there is only one "thing" to fail over, and it scales extremely well in terms of capacity and IOPS.
• The first sweet spot for SM-as is granular replication. Sometimes you don’t want to replicate all data on a storage system as a single unit, or you need to be able to selectively fail over certain workloads.
• The second sweet spot for SM-as is for active-active operations, where you want fully usable copies of data to be available on two different clusters located in two different locations with identical performance characteristics and, if desired, no requirement to stretch the SAN across sites. You can have your applications already running on both sites, which reduces the overall RTO during failover operations.

SnapMirror active sync setup

Although the following sections focus on an Oracle RAC configuration, the concepts are applicable to most any SnapMirror active sync deployment. Using Oracle RAC as an example is especially helpful because Oracle RAC is inherently complicated. If you understand how to configure and manage Oracle RAC on SM-as, including all the nuances, requirements, and especially the benefits ONTAP delivers for such a configuration, you’ll be able to use SM-as for any workload.

Additionally, I’ll be using the CLI to configure ONTAP, but you can also use the SystemManager GUI, and I want to say that I think it's outstanding. The UI team did great work with SnapMirror active sync. I’m using the CLI mostly because I personally prefer a CLI, but also because it's easier to explain what's happening with ONTAP if I can go step-by-step with the CLI. The GUI automates multiple operations in a single step.

Finally, I'll assume a mediator is already properly configured.

Storage Virtual Machines (SVM)

One item of ONTAP terminology you need to understand is the SVM, also called a vserver. ONTAP storage clusters are built for multitenancy. When you first install a system, it doesn't provide storage services in the same way that a new VMware server can't run applications. You need to create a VM.

An ONTAP SVM (again, also called a vserver, especially at the CLI) is not an actual VM running under a hypervisor, but conceptually it's the same thing. It's a virtual storage system with its own network addresses, security configuration, user accounts, and so forth. Once you create an SVM, you then provision your file shares, LUNs, S3 buckets, and other storage resources under the control of that SVM.

You'll see the use of -vserver in the examples below. I have two clusters, Cluster1 and Cluster2. Cluster1 includes an SVM called jfs_as1, which will be participating in a SnapMirror active sync relationship with SVM jfs_as2, located on Cluster2.

Provision storage

Once the SVM is defined, the next step is to create some volumes and LUNs. Remember – with ONTAP, a volume is not a LUN. A volume is a management object for storing data, including LUNs. We usually combine related LUNs in a single volume. The volume layout is usually designed to simplify management of those LUNs.

Confused? Here’s a diagram of my basic volume and LUN layout.

There is no true best practice for any database layout. The best option depends on what you want to do with the data. In this case, there are three volumes:

• One volume for the Oracle Grid management and quorum LUNs.
• One volume for the Oracle datafiles.
• One volume for the Oracle logs, and this includes redo logs, archive logs, and the controlfiles.
• LUN count for datafile IO of 8 LUNs. This isn't an ONTAP limitation, it's about the host OS. You need multiple LUNs to maximum SAN performance through the host IO stack. 4-8 LUNs is usuall required. The LUN count for logs and other sequentially accessed files is less important.

This design is all about manageability. For example, if you need to restore a database, the quickest way to do it would be to revert the state of the datafiles and then replay logs to the desired restore point. If you store the datafiles in a single volume, you can revert the state of that one volume to an earlier snapshot, but you need to ensure the log files are separated from datafiles. Otherwise, reverting the state of the datafiles would result in loss of archive log data that is critical for RPO=0 data recovery.

Likewise, this design allows you to manage performance more easily. You simply place a QoS limit on the datafile volume, rather than individual LUNs. As an aside, never place a QoS limit on a database transaction log because of the bursty nature of log IO. The average IO for transaction logs is usually fairly low, but the IO occurs in short bursts. If QoS engages during those bursts, the result will be significant performance damage. QoS is for datafiles only.

Create the volumes

I created my three volumes as follows.

Cluster1::> vol create -vserver jfs_as1 -volume jfsAA_grid_siteA -snapshot-policy none -percent-snapshot-space 0 -size 256g -space-guarantee
Cluster1::> vol create -vserver jfs_as1 -volume jfsAA_oradata_siteA -snapshot-policy none -percent-snapshot-space 0 -size 1t -space-guarantee
Cluster1::> vol create -vserver jfs_as1 -volume jfsAA_logs_siteA -snapshot-policy none -percent-snapshot-space 0 -size 500g -space-guarantee none

There’s a lot of different options when creating a volume. If you use SystemManager, you’ll get volumes with default behavior that is close to universally appropriate, but when using the CLI you might need to look at all the available options.

In my case, I wanted to create volumes for the grid, datafile, and log LUNs that include the following attributes:

• Disabling scheduled snapshots. Scheduled snapshots can provide powerful on-box data protection, but the schedule, retention policy, and naming conventions need to be based on the SLAs. For now, I’d rather just disable snapshots to ensure no snapshots are unknowingly created.
• Set the snapshot reserve to 0%. There is no reason to reserve snapshot space in a SAN environment.
• Set space guarantees to none. Space guarantees would reserve the volume’s capacity on the system, which is almost always wasteful with a database. Most databases compress well, so reserving the full size of the volume would be unnecessary.
• I want space efficiency settings to be enabled, but this is default behavior and does not require special arguments.

Create the LUNS

The next step is to create the required LUNs inside the volumes. I’ll be using the defaults, plus disabling space reservations. This will result in LUNs only consuming the space actually required by that LUN within the volume.

Cluster1::> lun create -vserver jfs_as1 /vol/jfsAA_grid_siteA/lun0 -size 64g -ostype linux -space-reserve disabled
Cluster1::> lun create -vserver jfs_as1 /vol/jfsAA_grid_siteA/lun1 -size 64g -ostype linux -space-reserve disabled
…
Cluster1::> lun create -vserver jfs_as1 /vol/jfsAA_logs_siteA/lun0 -size 64g -ostype linux -space-reserve disabled
Cluster1::> lun create -vserver jfs_as1 /vol/jfsAA_logs_siteA/lun1 -size 64g -ostype linux -space-reserve disabled

Define the CG

There are now three volumes of data: Oracle datafiles LUNs, Oracle log LUNs, and RAC cluster resource LUNs. While three separate volumes deliver optimal manageability, synchronous replication requires a single container. This RAC environment needs to be replicated and kept in sync as a unified whole. If site failure occurs, all resources at the surviving site need to be consistent with one another. We need a consistency group.

As mentioned above, with ONTAP, a volume is a not a LUN, it’s just a management container. If all LUNs in a given dataset were placed on a single volume, you can create snapshots, clone, restore, or replicate that single volume as a unit. In other words, a volume in ONTAP is natively a consistency group.

In many SM-as use cases, placing all the LUNs of a given application in a single volume might be all you need to meet your data protection requirements. Sometimes your requirements are more complicated. You might need to separate an application into multiple volumes based on manageability requirements, but also want to manage the application as a unit. That’s why we created ONTAP Consistency Groups, and you can read more about them here and here.

I can define a consistency group for my three volumes at the CLI by providing a CG name and the current volumes.

Cluster1::> consistency-group create -vserver jfs_as1 -consistency-group jfsAA -volumes jfsAA_oradata_siteA,jfsAA_logs_siteA,jfsAA_grid_siteA

The result is I've created a CG called jfsAA (I used the letters AA to denote the active-active configuration I’m building) based on the current grid, datafile, and log volumes and the LUNs they contain. Also, note that I used a suffix of _siteA so I can more easily keep track of while cluster hosts which volumes and the data within those volumes. More on that below…

Establish replication

There are now volumes of LUNs on site A, but replication isn’t operational yet. The first step to prepare for replication is to create volumes on site B. As mentioned previously, the SystemManager UI automates all of the setup work, but it’s easier to explain each step in the automated sequence of events using the CLI

Create destination volumes

Before I can replicate data, I need a place to host the replicas.

Cluster2::> vol create -vserver jfs_as2 -volume jfsAA_grid_siteB -snapshot-policy none -percent-snapshot-space 0 -size 256g -type DP -space-guarantee none
Cluster2::> vol create -vserver jfs_as2 -volume jfsAA_oradata_siteB -snapshot-policy none -percent-snapshot-space 0 -size 1t -type DP -space-guarantee none
Cluster2::> vol create -vserver jfs_as2 -volume jfsAA_logs_siteB -snapshot-policy none -percent-snapshot-space 0 -size 500g -type DP -space-guarantee none

The commands above created three volumes on site B using the same settings as used as site A, except they are a type “DP”, which means a data protection volume. This identifies a volume that will be joined to a replication relationship. No LUNs are being provisioned. They will be automatically created once replication is initialized and the content of the volumes on Cluster2 are synchronized from the source volumes on Cluster1.

Initialize replication

The following command creates an SnapMirror active sync relationship in active-active mode. This command is run on the uninitialized controller. SnapMirror is designed as a pull technology. For example, asynchronous SnapMirror updates pull new data from the source. It is not pushed from the source to a destination.

Cluster2::> snapmirror create -source-path jfs_as1:/cg/jfsAA -destination-path jfs_as2:/cg/jfsAA -cg-item-mappings jfsAA_grid_siteA:@jfsAA_grid_siteB,jfsAA_oradata_siteA:@jfsAA_oradata_siteB,jfsAA_logs_siteA:@jfsAA_logs_siteB -policy AutomatedFailOverDuplex
Operation succeeded: SnapMirror create for the relationship with destination "jfs_as2:/cg/jfsAA".

This operation makes a lot more sense if you see it in the GUI, but I can break down the command. Here’s what it does:

snapmirror create

That's self-evident. It's creating a snapmirror relationship.

-source-path jfs_as1:/cg/jfsAA

Use a source of the SVM called jfs_as1 and the consistency group called jfsAA. You'll see this syntax elsewhere in the ONTAP CLI. A consistency group is denoted as [svm name]:/cg/[cg name].

-destination-path jfs_as2:/cg/jfsAA

Replicate that source CG called jfsAA to the SVM called jfs_as2 and use the same CG name of jfsAA

-cg-item-mappings \
jfsAA_grid_siteA:@jfsAA_grid_siteB, \
jfsAA_oradata_siteA:@jfsAA_oradata_siteB,\
jfsAA_logs_siteA:@jfsAA_log_siteB

This section controls the mapping of volumes to volumes. The syntax is [source volume]:@[destination volume]. I've mapped the source grid volume to the destination grid volume, oradata to oradata, and logs to logs. The only difference in the name is the suffix. I used a siteA and siteB suffix for the volumes to avoid user errors. If someone is performing management activities on the UI, using either SystemManager or the CLI, they should be readily able to tell whether they're working on the site A or site B system based on the suffix of the volume names.

-policy AutomatedFailOverDuplex

The final argument specifies a snapmirror relationship of type AutomatedFailoverDuplex, which means bidirectional synchronous replication with automated failure detection. The relationship now exists, but it's not yet initialized. This requires the following command on the second cluster.

Cluster2::> snapmirror initialize -destination-path jfs_as2:/cg/jfsAA
Operation is queued: SnapMirror initialize of destination "jfs_as2:/cg/jfsAA".

I can check the status as follows, and the key is looking for a Relationship Status of InSync and Healthy being true.

Cluster2::> snapmirror show -vserver jfs_as2 -destination-path jfs_as2:/cg/jfsAA
                            Source Path: jfs_as1:/cg/jfsAA
                       Destination Path: jfs_as2:/cg/jfsAA
                      Relationship Type: XDP
                Relationship Group Type: consistencygroup
                      SnapMirror Policy: AutomatedFailOverDuplex
                           Mirror State: Snapmirrored
                    Relationship Status: InSync
                                Healthy: true

Define the igroup

Before I make the LUNs available, I need to define the initiator group (igroup). I’m building an Oracle RAC cluster, which means two different hosts will be accessing the same LUNs. I’m using iSCSI, but FC works the same. It just uses WWNs rather than iSCSI initiator IDs.

First, I'll create the igroup on the site A system. Any LUNs mapped to this igroup will be available via iSCSI to hosts using the specified initiator.

Cluster1::> igroup create -vserver jfs_as1 -igroup jfsAA -ostype linux -initiator iqn.1994-05.com.redhat:a8ee93358a32

Next, I'll enter Advanced mode and associate the local initiator name with the local SVM. This is how ONTAP controls path priority.  Any host with a WWN or iSCSI initiator listed in igroup will be able to access LUNs mapped to that igroup, but the path priorities would not be optimal. I want paths originating on site A to only be advertised as optimal paths to the hosts located on site A.

Cluster1::> set advanced
Cluster1::*> igroup initiator add-proximal-vserver -vserver jfs_as1 iqn.1994-05.com.redhat:a8ee93358a32 -proximal-vservers jfs_as1

I then repeat the process on site B, using the WWN or iSCSI initiator of the host on site B.

Cluster2::> igroup create -vserver jfs_as2  -igroup jfsAA -ostype linux -initiator iqn.1994-05.com.redhat:5214562dfc56
Cluster1::*> set advanced
Cluster2::*> igroup initiator add-proximal-vserver -vserver jfs_as2 iqn.1994-05.com.redhat:5214562dfc56 -proximal-vservers jfs_as2

I didn't really need to add the proximal-vserver information. Read on to understand why not.

Map the LUNs

Next, I map the LUNs to the igroup so the hosts will be able to access them. Although I used the same igroup name on each site, these igroups are located on different SVMs.

Cluster1::> lun map -vserver jfs_as1 /vol/jfsAA_grid_siteA/lun0  -igroup jfsAA
Cluster1::> lun map -vserver jfs_as1 /vol/jfsAA_grid_siteA/lun1  -igroup jfsAA
…
Cluster1::> lun map -vserver jfs_as1 /vol/jfsAA_oradata_siteA/lun6 -igroup jfsAA
Cluster1::> lun map -vserver jfs_as1 /vol/jfsAA_oradata_siteA/lun7 -igroup jfsAA

Cluster2::> lun map -vserver jfs_as2 /vol/jfsAA_grid_siteB/lun0  -igroup jfsAA
Cluster2::> lun map -vserver jfs_as2 /vol/jfsAA_grid_siteB/lun1  -igroup jfsAA
…
Cluster2::> lun map -vserver jfs_as2 /vol/jfsAA_oradata_siteB/lun6 -igroup jfsAA
Cluster2::> lun map -vserver jfs_as2 /vol/jfsAA_oradata_siteB/lun7 -igroup jfsAA

Oracle Configuration

From this point, setup is exactly like any other Oracle RAC server. Functionally, this is like a 2-site Oracle Extended RAC cluster, except there's no need for configuring ASM failgroups. The replication services are built-in to the storage system.

Device Names

There are multiple ways to control device names with Oracle, but my personal preference is using udev rules and multipath aliases. It takes more up-front work, but I have more control over the exact naming conventions to be used.

The multipath.conf file looks like this on each RAC node:

[root@jfs12 ~]# cat /etc/multipath.conf
multipaths {
    multipath {
        wwid  3600a0980383041327a2b55676c547247
        alias grid0
    }
    multipath {
        wwid  3600a0980383041327a2b55676c547248
        alias grid1
    }
    multipath {
        wwid  3600a0980383041327a2b55676c547249
        alias grid2
    }
    …
    }
    multipath {
        wwid  3600a0980383041334a3f55676c69734d
        alias logs5
    }
    multipath {
        wwid  3600a0980383041334a3f55676c69734e
        alias logs6
    }
    multipath {
        wwid  3600a0980383041334a3f55676c69734f
        alias logs7
    }
}

Then I have the following udev rule:

[root@jfs12 ~]# cat /etc/udev/rules.d/99-asm.rules
ENV{DM_NAME}=="grid*", GROUP:="asmadmin", OWNER:="grid", MODE:="660"
ENV{DM_NAME}=="oradata*", GROUP:="asmadmin", OWNER:="grid", MODE:="660"
ENV{DM_NAME}=="logs*", GROUP:="asmadmin", OWNER:="grid", MODE:="660"

The result is clear device names that are automatically assigned the correct user and group permissions. You can even see the devices in /dev/mapper.

[root@jfs12 ~]# ls /dev/mapper
control  grid1  logs0  logs2  logs4  logs6  oradata0  oradata2  oradata4  oradata6  vg00-root

ASM Configuration

Unlike typical Extended RAC, I created the ASM diskgroups with external redundancy, which means no mirroring by ASM itself. Replication services are provided by the storage system, not RAC. I created the following ASM diskgroups:

[root@jfs12 ~]# /grid/bin/asmcmd ls
DBF/
GRID/
LOGS/

From this point on, the installation process is exactly like any other Oracle RAC installation.

Database Creation

I used the following database layout:

[root@jfs12 ~]# /grid/bin/asmcmd ls DBF/NTAP
124D13D9FE3FFF29E06370AAC00A260E/
124DA9A2A3BB954AE06370AAC00A7624/
DATAFILE/
NTAPpdb1/
PARAMETERFILE/
PASSWORD/
TEMPFILE/
pdbseed/
sysaux01.dbf
system01.dbf
temp01.dbf
undotbs01.dbf
undotbs02.dbf
users01.dbf

[root@jfs12 ~]# /grid/bin/asmcmd ls LOGS/NTAP
ARCHIVELOG/
CONTROLFILE/
ONLINELOG/
control01.ctl
control02.ctl
redo01.log
redo02.log
redo03.log
redo04.log

There is a reason for this, but it's not connected to the use of SnapMirror active sync. See the section above called "Provision storage" for an explanation.

Failure scenarios

The following diagram shows the Oracle database and storage configuration as it exists at the hardware level (mediator not shown), and it is using a non-uniform network configuration. There is no SAN connectivity across sites.

I wrote above that I didn't need to configure host proximity. The reason is I'm using a nonuniform configuration. I have not stretched the SAN across sites. The only paths available to hosts are local paths. There's no reason to add host proximity settings because hosts will never be able to see storages paths to an opposite site. I included the instructions for configuring host proximity for readers who may be using uniform configurations and need to know how proximity settings are controlled.

Several of the scenarios described below that resulted in loss of database services would not have happened with a uniform network configuration. There is no cross-site SAN connectivity, so anything that results in the loss of active paths on a given site will mean there were no paths remaining at all. In a uniform network configuration, each site would be able to use alternate paths on the opposite site. The reason a non-uniform configuration was chosen for these tests was to illustrate that it is possible to have active-active RPO=0 replication without extending the SAN. It's a more demanding use case, and while it does have some limitations it also has benefits, such as a simpler SAN architecture.

There's another way to look at the storage architecture. The existence of replication is essentially invisible to the Oracle database and the RAC cluster. Various failure scenarios might result in disruption to certain database servers or the loss of certain paths, but as far as the database is concerned, there's just one set of LUNs. Logically, it looks like this:

Preferred sites

The configuration is symmetric, with one exception that is connected to split-brain management.

The question you need to ask is this - what happens if the replication link is lost and neither site has quorum? What do you want to happen? This question applies to both the Oracle RAC and the ONTAP behavior. If changes cannot be replicated across sites, and you want to resume operations, one of the sites will have to survive and the other site will have to become unavailable.

Oracle and css_critical

In the case of Oracle RAC, the default behavior is that one of the nodes in a RAC cluster that consists of an even number of servers will be deemed more important than the other nodes. The site with that higher priority node will survive site isolation while the nodes on the other site will evict. The prioritization is based on multiple factors, but you can also control this behavior using the css_critical setting.

My environment has two nodes, jfs12 and jfs13. The current settings for css_critical are as follows:

[root@jfs12 ~]# /grid/bin/crsctl get server css_critical
CRS-5092: Current value of the server attribute CSS_CRITICAL is no.

[root@jfs13 trace]# /grid/bin/crsctl get server css_critical
CRS-5092: Current value of the server attribute CSS_CRITICAL is no.

I want the site with jfs12 to be the preferred site, so I changed this value to yes on a site A node and restart services.

[root@jfs12 ~]# /grid/bin/crsctl set server css_critical yes
CRS-4416: Server attribute 'CSS_CRITICAL' successfully changed. Restart Oracle High Availability Services for new value to take effect.

[root@jfs12 ~]# /grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'jfs12'
CRS-2673: Attempting to stop 'ora.crsd' on 'jfs12'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'jfs12'
CRS-2673: Attempting to stop 'ora.ntap.ntappdb1.pdb' on 'jfs12'
…

CRS-2673: Attempting to stop 'ora.gipcd' on 'jfs12'
CRS-2677: Stop of 'ora.gipcd' on 'jfs12' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'jfs12' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@jfs12 ~]# /grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

SnapMirror active sync preferred site

At any given moment, SnapMirror active sync will consider one site the "source" and the other the "destination". This implies a one-way replication relationship, but that's not what's happening. This is how the preferred site is determined. If the replication link is lost, the LUN paths on the source will continue to serve data while the LUN paths on the destination will become unavailable until replication is reestablished and enters a synchronous state. The paths will then resume serving data.

My current configuration has site A as the preferred site for both Oracle and ONTAP. This can be viewed via SystemManager:

or at the CLI:

Cluster2::> snapmirror show -destination-path jfs_as2:/cg/jfsAA
                            Source Path: jfs_as1:/cg/jfsAA
                       Destination Path: jfs_as2:/cg/jfsAA
                      Relationship Type: XDP
                Relationship Group Type: consistencygroup
                    SnapMirror Schedule: -
                 SnapMirror Policy Type: automated-failover-duplex
                      SnapMirror Policy: AutomatedFailOverDuplex
                           Mirror State: Snapmirrored
                    Relationship Status: InSync

The key is that the source is the SVM on cluster1. As mentioned above, the terms "source" and "destination" don't describe the flow of replicated data. Both sites can process a write and replicate it to the opposite site. In effect, both clusters are sources and destinations. The effect of designating one cluster as a source simply controls which cluster survives as a read-write storage system if the replication link is lost.

Loss of SnapMirror replication connectivity

If I cut the SM-as replication link, write IO cannot be completed because it would be impossible for a cluster to replicate changes to the opposite site. Here's what will happen with an Oracle RAC environment.

Site A

The result on site A of a replication link failure will be an approximately 15 second pause in write IO processing as ONTAP attempts to replicate writes before it determines that the replication link is genuinely inoperable. After the 15 seconds elapses, the ONTAP cluster on site A resumes read and write IO processing. The SAN paths will not change, and the LUNs will remain online.

Site B

Since site B is not the SnapMirror active sync preferred site, its LUN paths will become unavailable after about 15 seconds.

The replication link was cut at the timestamp 15:19:44. The first warning from Oracle RAC arrives 100 seconds later as the 200 second timeout (controlled by the Oracle RAC parameter disktimeout) approaches.

2024-09-10 15:21:24.702 [ONMD(2792)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 99340 milliseconds.
2024-09-10 15:22:14.706 [ONMD(2792)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 49330 milliseconds.
2024-09-10 15:22:44.708 [ONMD(2792)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 19330 milliseconds.
2024-09-10 15:23:04.710 [ONMD(2792)]CRS-1604: CSSD voting file is offline: /dev/mapper/grid2; details at (:CSSNM00058:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc.
2024-09-10 15:23:04.710 [ONMD(2792)]CRS-1606: The number of voting files available, 0, is less than the minimum number of voting files required, 1, resulting in CSSD termination to ensure data integrity; details at (:CSSNM00018:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc
2024-09-10 15:23:04.716 [ONMD(2792)]CRS-1699: The CSS daemon is terminating due to a fatal error from thread: clssnmvDiskPingMonitorThread; Details at (:CSSSC00012:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc
2024-09-10 15:23:04.731 [OCSSD(2794)]CRS-1652: Starting clean up of CRSD resources.

Once the 200 second voting disk timeout has been reached, this Oracle RAC node will evict itself from the cluster and reboot.

Loss of Oracle RAC replication

Loss of the Oracle RAC replication link will produce a similar result, except the timeouts will be shorter by default. An Oracle RAC node will wait 200 seconds after loss of storage connectivity before evicting, but it will only wait 30 seconds after loss of the RAC network heartbeat.

The CRS messages are similar to those shown below.  You can see the 30 second timeout lapse. Since css_critical was set on jfs12, located on site A, that will be the site to survive and jfs13 on site B will be evicted.

2024-09-12 10:56:44.047 [ONMD(3528)]CRS-1611: Network communication with node jfs13 (2) has been missing for 75% of the timeout interval.  If this persists, removal of this node from cluster will occur in 6.980 seconds
2024-09-12 10:56:48.048 [ONMD(3528)]CRS-1610: Network communication with node jfs13 (2) has been missing for 90% of the timeout interval.  If this persists, removal of this node from cluster will occur in 2.980 seconds
2024-09-12 10:56:51.031 [ONMD(3528)]CRS-1607: Node jfs13 is being evicted in cluster incarnation 621599354; details at (:CSSNM00007:) in /gridbase/diag/crs/jfs12/crs/trace/onmd.trc.
2024-09-12 10:56:52.390 [CRSD(6668)]CRS-7503: The Oracle Grid Infrastructure process 'crsd' observed communication issues between node 'jfs12' and node 'jfs13', interface list of local node 'jfs12' is '192.168.30.1:33194;', interface list of remote node 'jfs13' is '192.168.30.2:33621;'.
2024-09-12 10:56:55.683 [ONMD(3528)]CRS-1601: CSSD Reconfiguration complete. Active nodes are jfs12 .
2024-09-12 10:56:55.722 [CRSD(6668)]CRS-5504: Node down event reported for node 'jfs13'.
2024-09-12 10:56:57.222 [CRSD(6668)]CRS-2773: Server 'jfs13' has been removed from pool 'Generic'.
2024-09-12 10:56:57.224 [CRSD(6668)]CRS-2773: Server 'jfs13' has been removed from pool 'ora.NTAP'.

Complete loss of replication network

Oracle RAC split-brain detection has a dependency on the Oracle RAC storage heartbeat. If loss of site-to-site connectivity results in simultaneous loss of both the RAC network heartbeat and storage replication services, the result is the RAC sites will not be able to communicate cross-site via either the RAC interconnect or the RAC voting disks. The result in an even-numbed set of nodes may be eviction of both sites under default settings. The exact behavior will depend on the sequence of events and the timing of the RAC network and disk heartbeat polls.

The risk of a 2-site outage can be addressed in two ways. First, using an odd number of RAC nodes, preferably by use of a 3^rd site tiebreaker. If a 3^rd site is unavailable, the tiebreaker instance could be placed on one site. This means that loss of site-to-site connectivity will cause LUN paths to go down on one site, but one of the RAC sites will still have quorum and will not evict.

If a 3^rd site is not available, this problem can be addressed by adjusting the misscount parameter on the RAC cluster. Under the defaults, the RAC network heartbeat timeout is 30 seconds. This normally is used by RAC to identify failed RAC nodes and remove them from the cluster. It also has a connection to the voting disk heartbeat.

If, for example, the conduit carrying intersite traffic for both Oracle RAC and storage replication services is cut by a backhoe, the 30 second misscount countdown will begin. If the RAC preferred site node cannot reestablish contact with the opposite site within 30 seconds, and it also cannot use the voting disks to confirm the opposite site is down within that same 30 second window, then the preferred site nodes will also evict. The result is a full database outage.

Depending on when the misscount polling occurs, 30 seconds may not be enough time for SnapMirror active sync to time out and allow storage on the preferred site to resume services before the 30 second window expires. This 30 second window can be increased.

[root@jfs12 ~]# /grid/bin/crsctl set css misscount 100
CRS-4684: Successful set of parameter misscount to 100 for Cluster Synchronization Services.

This value allows the storage system on the preferred site to resume operations before the misscount timeout expires. The result will then be eviction only of the nodes at the site where the LUN paths were removed. Example below:

2024-09-12 09:50:59.352 [ONMD(681360)]CRS-1612: Network communication with node jfs13 (2) has been missing for 50% of the timeout interval.  If this persists, removal of this node from cluster will occur in 49.570 seconds
2024-09-12 09:51:10.082 [CRSD(682669)]CRS-7503: The Oracle Grid Infrastructure process 'crsd' observed communication issues between node 'jfs12' and node 'jfs13', interface list of local node 'jfs12' is '192.168.30.1:46039;', interface list of remote node 'jfs13' is '192.168.30.2:42037;'.
2024-09-12 09:51:24.356 [ONMD(681360)]CRS-1611: Network communication with node jfs13 (2) has been missing for 75% of the timeout interval.  If this persists, removal of this node from cluster will occur in 24.560 seconds
2024-09-12 09:51:39.359 [ONMD(681360)]CRS-1610: Network communication with node jfs13 (2) has been missing for 90% of the timeout interval.  If this persists, removal of this node from cluster will occur in 9.560 seconds
2024-09-12 09:51:47.527 [OHASD(680884)]CRS-8011: reboot advisory message from host: jfs13, component: cssagent, with time stamp: L-2024-09-12-09:51:47.451
2024-09-12 09:51:47.527 [OHASD(680884)]CRS-8013: reboot advisory message text: oracssdagent is about to reboot this node due to unknown reason as it did not receive local heartbeats for 10470 ms amount of time
2024-09-12 09:51:48.925 [ONMD(681360)]CRS-1632: Node jfs13 is being removed from the cluster in cluster incarnation 621596607

Note: Oracle Support strongly discourages altering with the misscount or disktimeout parameters to solve configuration problems. Changing these parameters can, however, be warranted and unavoidable in many cases, including SAN booting, virtualized, and storage replication configurations. If, for example, you had stability problems with a SAN or IP network that was resulting in RAC evictions you should fix the underlying problem and not charge the values of the misscount or disktimeout. Changing timeouts to address configuration errors is masking a problem, not solving a problem. Changing these parameters to properly configure a RAC environment based on design aspects of the underlying infrastructure is different and is consistent with Oracle support statements. With SAN booting, it is common to adjust misscount all the way up to 200 to match disktimeout.

Storage system failure

The result of a storage system failure is nearly identical to the result of losing the replication link. The surviving site should experience a roughly 15 second IO pause on writes. Once that 15 second period elapses, IO will resume on that site as usual.

The Oracle RAC node on the failed site will lose storage services and enter the same 200 second disktimeout countdown before eviction and subsequent reboot.

2024-09-11 13:44:38.613 [ONMD(3629)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 99750 milliseconds.
2024-09-11 13:44:51.202 [ORAAGENT(5437)]CRS-5011: Check of resource "NTAP" failed: details at "(:CLSN00007:)" in "/gridbase/diag/crs/jfs13/crs/trace/crsd_oraagent_oracle.trc"
2024-09-11 13:44:51.798 [ORAAGENT(75914)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 75914
2024-09-11 13:45:28.626 [ONMD(3629)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 49730 milliseconds.
2024-09-11 13:45:33.339 [ORAAGENT(76328)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 76328
2024-09-11 13:45:58.629 [ONMD(3629)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 19730 milliseconds.
2024-09-11 13:46:18.630 [ONMD(3629)]CRS-1604: CSSD voting file is offline: /dev/mapper/grid2; details at (:CSSNM00058:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc.
2024-09-11 13:46:18.631 [ONMD(3629)]CRS-1606: The number of voting files available, 0, is less than the minimum number of voting files required, 1, resulting in CSSD termination to ensure data integrity; details at (:CSSNM00018:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc
2024-09-11 13:46:18.638 [ONMD(3629)]CRS-1699: The CSS daemon is terminating due to a fatal error from thread: clssnmvDiskPingMonitorThread; Details at (:CSSSC00012:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc
2024-09-11 13:46:18.651 [OCSSD(3631)]CRS-1652: Starting clean up of CRSD resources.

The SAN path state on the RAC node that has lost storage services looks like this:

oradata7 (3600a0980383041334a3f55676c697347) dm-20 NETAPP,LUN C-Mode
size=128G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| `- 34:0:0:18 sdam 66:96  failed faulty running
`-+- policy='service-time 0' prio=0 status=enabled
  `- 33:0:0:18 sdaj 66:48  failed faulty running

The linux host detected the loss of the paths much quicker than 200 seconds, but from a database perspective the client connections to the host on the failed site will still be frozen for 200 seconds under the default Oracle RAC settings. Full database operations will only resume after the eviction is completed.

Meanwhile, the Oracle RAC node on the opposite site will record the loss of the other RAC node. It otherwise continues to operate as usual.

2024-09-11 13:46:34.152 [ONMD(3547)]CRS-1612: Network communication with node jfs13 (2) has been missing for 50% of the timeout interval.  If this persists, removal of this node from cluster will occur in 14.020 seconds
2024-09-11 13:46:41.154 [ONMD(3547)]CRS-1611: Network communication with node jfs13 (2) has been missing for 75% of the timeout interval.  If this persists, removal of this node from cluster will occur in 7.010 seconds
2024-09-11 13:46:46.155 [ONMD(3547)]CRS-1610: Network communication with node jfs13 (2) has been missing for 90% of the timeout interval.  If this persists, removal of this node from cluster will occur in 2.010 seconds
2024-09-11 13:46:46.470 [OHASD(1705)]CRS-8011: reboot advisory message from host: jfs13, component: cssmonit, with time stamp: L-2024-09-11-13:46:46.404
2024-09-11 13:46:46.471 [OHASD(1705)]CRS-8013: reboot advisory message text: At this point node has lost voting file majority access and oracssdmonitor is rebooting the node due to unknown reason as it did not receive local hearbeats for 28180 ms amount of time
2024-09-11 13:46:48.173 [ONMD(3547)]CRS-1632: Node jfs13 is being removed from the cluster in cluster incarnation 621516934

Cut the power to the mediator

The mediator service does not directly control storage operations. It functions as an alternate control path between clusters. It exists primarily to automate failover without the risk of a split-brain scenario. In normal operation, each cluster is replicating changes to its partner, and each cluster therefore can verify that the partner cluster is online and serving data. If the replication link failed, replication would cease.

The reason a mediator is required for safe automated operations is because it would otherwise be impossible for a storage clusters to be able to determine whether loss of bidirectional communication was the result of a network outage or actual storage failure.

The mediator provides an alternate path for each cluster to verify the health of its partner. The scenarios are as follows:

• If a cluster can contact its partner directly, replication services are operational. No action required.
• If a preferred site cannot contact its partner directly or via the mediator, it will assume the partner is either actually unavailable or was isolated and has taken its LUN paths offline. The preferred site will then proceed to release the RPO=0 state and continue processing both read and write IO.
• If a non-preferred site cannot contact its partner directly, but can contact it via the mediator, it will take its paths offline and await the return of the replication connection.
• If a non-preferred site cannot contact its partner directly or via an operational mediator, it will assume the partner is either actually unavailable or was isolated and has taken its LUN paths offline. The non-preferred site will then proceed to release the RPO=0 state and continue processing both read and write IO. It will assume the role of the replication source and will become the new preferred site.

If the mediator is wholly unavailable:

• Failure of replication services for any reason will result in the preferred site releasing the RPO=0 state and resuming read and write IO processing. The non-preferred site will take its paths offline.
• Failure of the preferred site will result in an outage because the non-preferred site will be unable to verify that the opposite site is truly offline and therefore it would not be safe for the nonpreferred site to resume services.

Restoring services

It's difficult to demonstrate what happens when power is restored after a failure or a replication link failure is repaired. SnapMirror active sync will automatically detect the presence of a faulty replication relationship and bring it back to an RPO=0 state. Once synchronous replication is reestablished, the paths will come online again.

In many cases, clustered applications will automatically detect the return of failed paths, and those applications will also come back online. In other cases, a host-level SAN scan may be required, or applications may need to be brought back online manually. It depends on the application and how it's configured, and in general such tasks can be easily automated. ONTAP itself is self-healing and should not require any user intervention to resume RPO=0 storage operations.

Another feature of SnapMirror active sync that is difficult to demonstrate is the speed of recovery. Unlike many competitors, ONTAP replication uses a pointer-based mechanism to track changes. The result is resynchronization after replication interruption is extremely fast. ONTAP is able to identify changed blocks and efficiently and quickly ship them to the remote system, even if replication was interrupted for days, weeks, or months. Many competitors use a slower and less efficient extent-based approach which requires reading and transferring much more data than merely those blocks which have changed. In some cases, the entire mirror must be rebuilt.

In addition, ONTAP's unique pointer-based technology preserves an intact copy of your data during resynchronization. The copy would be out of date, but it would exist. Resynchronization with many competing technologies results in a copy that is corrupt until the resynchronization is complete. This leaves a customer exposed to significant data loss if the last remaining copy of the data is damaged. 

This is one of many examples of how ONTAP features not only deliver results, but also deliver a superior end result because ONTAP just works better. 

SnapMirror active sync failover

Changing the preferred site requires a simple operation. IO will pause for a second or two as authority over replication behavior switches between clusters, but IO is otherwise unaffected.

GUI example:

Example of changing it back via the CLI:

Cluster2::> snapmirror failover start -destination-path jfs_as2:/cg/jfsAA
[Job 9575] Job is queued: SnapMirror failover for destination "jfs_as2:/cg/jfsAA                      ".

Cluster2::> snapmirror failover show
Source    Destination                                          Error
Path      Path        Type     Status    start-time end-time   Reason
-------- -----------  -------- --------- ---------- ---------- ----------
jfs_as1:/cg/jfsAA
         jfs_as2:/cg/jfsAA
                      planned  completed 9/11/2024  9/11/2024
                                         09:29:22   09:29:32

The new destination path can be verified as follows:

Cluster1::> snapmirror show -destination-path jfs_as1:/cg/jfsAA
                            Source Path: jfs_as2:/cg/jfsAA
                       Destination Path: jfs_as1:/cg/jfsAA
                      Relationship Type: XDP
                Relationship Group Type: consistencygroup
                 SnapMirror Policy Type: automated-failover-duplex
                      SnapMirror Policy: AutomatedFailOverDuplex
                           Mirror State: Snapmirrored
                    Relationship Status: InSync

Summary

The examples above are designed to illustrate SnapMirror active sync functionality. As with most enterprise IT projects, there are few true best practices. The right configuration depends on business needs including RPO, RTO, and SLAs. With respect to disaster recovery, it also depends on which disaster scenarios are more likely than others to occur.

The next posts planned will include use of an Oracle RAC tiebreaker, use of uniform networking, and quantifying application IO resumption times. The foundation remains the same - SnapMirror active sync operating in active-active mode, an integrated RPO=0/RTO=0 SAN replication technology that is reliable, self-healing, easily configured, and easily managed.

If you want to learn more, you can visit the official ONTAP documentation for SnapMirror active sync, and if you'd really like to see it in action I recommend getting in touch with Neto and the rest of the CPOC (Customer Proof Of Concept) team. They've got all the cool hardware to simulate your real-world workloads. 

IT Operations teams are managing complex infrastructures with limited visibility, dealing with an overwhelming number of alerts, and reactively troubleshooting while dealing with constant context switching to get needed information, causing delays in incident response. New features in NetApp Data Infrastructure Insights make it easier for teams to deal with these challenges. With the launch of VM Analyzer and landing page enhancements, Data Infrastructure Insights enables storage infrastructure teams to simplify their workflows and accelerate troubleshooting. Here’s how this new feature and functionality transforms IT operations for teams while delivering value back to the business.  

VM Analyzer: Enhanced troubleshooting with application-centric insights  

Troubleshooting in virtual environments has long been hampered by complexity. Application dependencies across virtual machines, hosts, and storage layers often create a tangled web of metrics that make it difficult to pinpoint the root cause of issues. The new VM Analyzer feature in Data Infrastructure Insights can help you cut through the noise with unified topology views.  VM Analyzer isolates application-specific infrastructure and delivers AI-driven insights to pinpoint root causes and resolve performance issues faster.  

With VM Analyzer, teams can:  

• Quickly trace performance issues to their true source 
• Resolve incidents with prescriptive recommendations  
• Align VM and storage performance with mission-critical applications 

What’s new:  

• AI-driven metric correlation surfaces anomalies the moment they occur  
• Application-centric topology maps display VMs, hosts, storage paths, and network links in a single view  
• Automated change detection highlights configuration updates that may impact performance   

Why it matters: Modern environments shift quickly, and manual investigation across dashboards drains hours and delays fixes. VM Analyzer brings everything into one streamlined process to identify the true source of a problem so you can act immediately.  

Here’s an example scenario where VM Analyzer can help to quickly troubleshoot an issue:  

A storage manager needs to resolve a critical latency issue reported by the VM support team because a MySQL server VM is experiencing high disk latency. Without a tool like VM Analyzer, the storage team would likely spend hours combing through storage system metrics only to discover the problem doesn’t originate there. This is a common occurrence since storage teams are often the first to take blame for issues, and it not only wastes time but can lead to misdiagnoses and increased frustration for users and team members.  

With Data Infrastructure Insights, a storage manager can immediately dig into the issue through an alert and receive a detailed timeline that reveals the latency spike along with related metrics like IOPS and throughput. Utilizing VM Analyzer, the storage manager sees a full topology view of the VM’s infrastructure and dependencies. 

VM Analyzer correlates latency spikes with a timeline entry for a recent infrastructure change so the storage manager can quickly pinpoint the root cause of the issue. In this case, the VM’s memory allocation was significantly reduced just before the latency spike.  

With the problem identified, the storage manager can work with the virtual compute team to quickly restore the VM’s memory allocation and, within minutes, VM Analyzer will reflect the correction, showing that metrics have returned to normal levels. Perhaps just as valuable, this ability to confirm the resolution of an issue with precise data can ensure there are no lingering concerns.   

This scenario highlights the capability of VM Analyzer to correlate performance issues with infrastructure changes to eliminate blind troubleshooting and enable speedy diagnosis and resolution. Watch the demo video to see this scenario in action: 

VM Analyzer key benefits:  

• Faster root-cause analysis: Instantly see when IOPS, throughput, or latency deviate—and why.  

• Reduced mean time to resolution (MTTR): With AI-backed alerts and change correlation, troubleshoot in minutes, not days.  

• Team alignment: Shared topology views give both storage and compute teams the same data, reducing finger-pointing and wasted effort.  

Landing page enhancements to streamline your workflows  

Even the most advanced tools need the right interface to support users effectively. The latest updates to landing pages within Data Infrastructure Insights focus on maximizing screen real estate and reducing context switching to ensure that you have smoother workflows during investigations and troubleshooting.  

This new design keeps all the functionality and detail that storage teams have come to rely on from Data Infrastructure Insights while making it more concise and easier to consume. The new layout helps minimize distraction from context switching and saves teams time when every second counts.  

What you get from the improved interface:  

• A cleaner, more intuitive layout tailored for efficiency 

• Reduced context switching for faster problem resolution  

• Maximized usability without compromising functionality 

Combined Benefits for Your ITOps  

The introduction of VM Analyzer and enhanced user interface capabilities mark a significant step forward in simplifying storage operations. Together, these features and enhancements offer a cohesive experience that empowers storage operations teams to achieve more with less effort. With faster troubleshooting and streamlined monitoring, teams can escape the constant fire drills and manual processes and boost overall efficiency. Early issue detection paired with recommended actions minimizes downtime and prevents costly disruptions. By aligning data-driven recommendations with application and business priorities, Data Infrastructure Insights strengthens IT Operations with precise insights and proactive monitoring.  

Request a personalized demo to see how Data Infrastructure Insights can unlock the next level of operational excellence for your organization.