SnapMirror active sync and the active-active data center   I've been beating up on SnapMirror active sync in the lab for about a year now, and I have to say this is the coolest feature I’ve seen added to ONTAP in years . In particular, I'm talking about SnapMirror active sync running in symmetric active-active mode. The value isn’t in the feature itself, it’s in the sorts of solutions it allows you to create. The foundation of SnapMirror active sync is SnapMirror synchronous, and it's been around much longer and is well-proven, but I only started using the active-active capabilities a year ago.   There are a number of ways to configure SnapMirror active sync, but the one I want to focus on here is an active-active application environment. You can build clustered environments including Oracle RAC, VMware, and Windows Server Failover Clusters (WSFC)  where the application cluster is distributed across two sites and IO response times are symmetric. You get RPO=0 through synchronous cross-site mirroring and RTO=0 through built-in ONTAP automation and the fact that storage resources are available on both sites, all the time.   This post should be useful for anyone looking at synchronous replication solutions, but I’m going to use Oracle RAC to illustrate SnapMirror active sync functionality because you can examine the RTO=0 and RPO=0 characteristics of the overall solution from the application layer all the way to the storage layer. The same database is operational at both sites, all the time. A disaster requires breaking a mirror, but you don’t really fail anything over because the services were running at the surviving site all along.   The examples used might be Oracle RAC, but the principles are universal. This includes concepts like "preferred site", heartbeats, tiebreaking, and timeouts. Even if you know next to nothing about Oracle RAC, read on.   Before diving into the details of how SnapMirror active sync (SM-as) works, I’d like to explain what SnapMirror active sync (SM-as) is for.   RPO=0   If you work in IT, you’ve definitely heard about the Recovery Point Objective. How much data can you afford to lose if certain events happen? If you're looking at synchronous replication, that means RPO=0, but that's more nuanced than it seems. When I see a requirement for RPO=0, I usually divide it into two categories:   RPO for common problems RPO for disaster scenarios   RPO=0 for normal problems is fairly easy to accomplish. For example, if you have a database you should expect to need to recover it occasionally. Maybe a patch went wrong and caused corruption or someone deleted the wrong file. All you usually need is RAID-protected storage and a basic backup plan and you can restore the database, replay logs, and fix the problem with no loss of data. You should expect to have RPO=0 recoverability for these types of common problems, and if you choose ONTAP we can make the procedures easier, faster, and less expensive.    If you want RPO=0 for disaster scenarios, things get more complicated. As an example, what if you have a malicious insider who decided to destroy your data and its backups? That requires a very different approach using the right technology with the right solution. We have some cool things to offer in that space.   SnapMirror active sync is about RPO=0 for disaster scenarios that lead to site loss. It might be permanent site loss because of a fire or maybe it's a temporary loss because of a power outage. Perhaps it's more isolated, like a power surge that destroys a storage array. SM-as addresses these scenarios with synchronous mirroring. It's not just RPO=0 replication either, it's a self-healing, integrated, intelligent RPO=0 replication solution.   RTO=0   A requirement for RPO=0 often comes with a requirement for Recovery Time Objective (RTO) of zero as well. The RTO is the amount of time you can be without a service, and overall it's an easily understood SLA, with one exception, which is RTO=0. What does that mean?   The term "RTO=0" across the whole IT industry. I continue to insist there’s no such thing as RTO=0. That would require instantaneously recognizing a problem exists and then instantly correcting it to restore service. That isn’t possible. From a storage perspective, there’s no way to know whether a 10 millisecond wait for a LUN to respond is because the storage system is merely busy with other IO or the data center itself fell into a black hole and no longer exists.   You can’t have RTO=0, but you can have a near-zero RTO from the perspective of IT operations. That exact definition of “near-zero” will depend on business needs. If an RTO is low enough that operations aren’t significantly affected, that’s what the industry calls RTO=0. It means “nondisruptive”.   Not all of the RTO is determined solely by storage availability. For example, a typical ONTAP controller failover completes in around 2-4 seconds, and during this time there will be a pause in SAN IO. The OS’s, however, often have much higher timeouts. ONTAP can fail over in 2 seconds and be fully ready to serve data again, but sometimes the host will wait 15 seconds before retrying an IO and you can’t change that. OS vendors have optimized SAN  behavior over the years, but it's not perfect. Those rare but real and occasionally lengthy IO timeouts are probably happening from time to time in your SAN already. Sometimes an FC frame will go missing because a cable is just slightly damaged and there will be a brief performance blip. It shouldn’t cause real-world problems, though. Nothing should crash.   Since SM-as failover operations should be nondisruptive to ongoing IT operations, including disasters, I’ll call SM-as an RTO=0 storage solution since that's the way the RTO term is used in the industry.    You still have to think about the application layer, though. For example, you could distribute a VMware cluster across sites and replicate the underlying storage with SM-as, but failure of a given site would require VMware HA to detect the loss of a given VM and restart it on the surviving site. That takes some time. Does that requirement meet your RTO? If you can’t start a VM quickly enough, you can consider containerization. One of its benefits is a container can be activated nearly instantaneously. There’s no OS boot sequence. That might be an option to improve the big-picture RTO. Finally, if you’re using a truly active-active application like Oracle RAC, you can proactively have the same database running at both sites all the time. The right solution depends on the RTO and the behavior of the entire solution during failover events.   From a storage perspective, achieving that near-zero RTO is a lot more complicated than you’d think. How does one site know the other site is alive? What should happen if a write cannot be replicated? How does a storage system know the difference between a replication partner that is permanently offline versus a partner that is only temporarily unreachable? What should happen to LUNs that are being accessed by a host but are no longer guaranteed to be able to serve the most recent copy of data? What happens when replication is restored?   On to the diagrams…   SnapMirror active sync architecture   Let’s start with the basics of how SM-as replication works.   Here’s what this shows:   These are two different storage clusters. One of them is Cluster1, and the other is Cluster2. (I'll explain the jfs_as1 and jfs_as2 labels a little later…) It might look like there are six LUNs but it’s logically only three. The reason is there are six different LUN images, but the LUN1 on each jfs_as1 is functionally the same as LUN1 on jfs_as2, the LUN2's are the same, and the LUN3's are the same. From a host point of view, this looks like ordinary SAN multipathing. LUN1 and its data is available and accessible on either cluster. IO behavior and IO performance are symmetric. If you perform a read of LUN1 from the cluster on site A, the read will be serviced by the local copy of the data on site A If you perform a read of LUN1 from the cluster on site B, the read will be serviced by the local copy of the data on site B Performing a write of LUN1 will require replication of the write to the opposite site before the write is acknowledged.   Now let's look at failover. There’s another component involved – the mediator.       The mediator is required for safely automating failover. Ideally, it would be placed on an independent 3 rd site, but it can still function for most needs if it’s placed on site A or site B. The mediator is not really a tiebreaker, although that’s effectively the function it provides. It's not taking any actions; it’s providing an alternate communication channel to the storage systems.   The #1 challenge with automated failover is the split-brain problem, and that problem arises if your two sites lose connectivity with each other. What should happen? You don’t want to have two different sites activate themselves as the surviving copy of the data, but how can a single site tell the difference between actual loss of the opposite site and an inability to communicate with the opposite site?   This is where the mediator enters the picture. If placed on a 3 rd site, and each site has a separate network connection to that site, then you have an additional path for each site to validate the health of the other. Look at the picture above again and consider the following scenarios.   What happens if the mediator fails or is unreachable from one or both sites? The two clusters can still communicate with each other over the same link used for replication services. Data is still served with RPO=0 protection What happens if Site A fails? Site B will see both of the communication channels to the partner storage system go down. Site B will take over data services, but without RPO=0 mirroring What happens if Site B fails? Site A will see both of the communication channels to the partner storage system go down. Site A will take over data services, but without RPO=0 mirroring   There is one other scenario to consider: Loss of the data replication link. If the replication link between sites is lost, RPO=0 mirroring will obviously be impossible. What should happen then?   This is controlled by the preferred site status. In an SM-as relationship, one of the sites is secondary to the other. This makes no difference in normal operations, and all data access is symmetric, but if replication is interrupted then the tie will have to be broken to resume operations. The results is that the preferred site will continue operations without mirroring and the secondary will halt IO processing until replication is reestablished. More on this topic below…   SAN design with SnapMirror active sync   As explained above, SM-as functionally provides the same LUNs on two different clusters. That doesn’t mean you necessarily want your hosts to access their LUNs across all available paths. You might, or you might not. There are two basic options called uniform and non-uniform access.   Uniform access   Let’s say you had a single host, and you wanted RPO=0, RTO=0 data protection. You’d probably want to configure SM-as with uniform access, which means the LUNs would be visible to your host from both clusters.     Obviously, this will mean that complete loss of site A would also result in the loss of your host and its applications, but you would have extremely high availability storage services for your data. This is probably a less common use case for SM-as, but it does happen. Not all applications can be clustered. Sometimes all you can do is provide ultra-available storage and accept that you’ll need to find alternate servers on a remote site in the event of disaster.   Another option with uniform access is in a full mesh networked cluster.         If you use SM-as with uniform access in a cluster as shown above, any host should always have access to a usable copy of the data. Even if one of the storage systems failed or storage replication connectivity was lost, the hosts would continue working. From a host multipath point of view, it all looks like a single storage system. A failure of any one component would result in some of the SAN paths disappearing off the network, but otherwise everything would continue working as usual.   Uniform access with local proximity   Another feature of SM-as is the option to configure the storage systems know where the hosts are located. When you map the LUNs to a given host, you can indicate whether or not they are local to the storage system.       In normal operation, all IO is local IO. Reads and writes are serviced from the local storage array. Write IO will, of course, need to be replicated by the local controller to the remote system before being acknowledged, but all read IO will be serviced locally and will not incur extra latency by traversing the SAN.   The only time the nonoptimized paths will be used is when all active/optimized paths are lost. For example, if the entire array on site A lost power, the hosts at site A would still be operational, although they would be experiencing higher read latency.   Note: There are redundant paths through the local cluster that are not shown on these diagrams for the sake of simplicity. ONTAP storage systems are HA themselves, so a controller failure should not result in site failure. It should merely result in a change in which local paths are used by an affected host.   Nonuniform access   Nonuniform access means each host has access to only a subset of available ports.         The primary benefit to this approach is SAN simplicity: you remove the need to stretch a SAN over the network. Some users don't have dark fiber between sites or lack the infrastructure to tunnel FC SAN traffic over an IP network. In some cases, the additional latency overhead of an application accessing storage across sites on a regular basis would be unacceptable, rendering the improved availability of uniform access unnecessary.   The disadvantage to nonuniform access is that certain failure scenarios, including loss of the replication link, will result in half of your hosts losing access to storage. Applications that run as single instances, such as a non-clustered database that is inherently only running on a single host at any given mount, would fail if local storage connectivity were lost. The data would still be protected, but the database server would no longer have access. It would need to be restarted on a remote site, preferably through an automated process. For example, VMware HA can detect an all-paths-down situation and restart a VM on another server where paths are available. In contrast, a clustered application such as Oracle RAC can deliver the same services that are constantly running at both sites. Losing a site with Oracle RAC doesn’t mean loss of the application service as a whole. Instances are still available and running at the surviving site.    Uniform access with ASA   The diagrams above showed path prioritization with AFF storage systems. NetApp ASA systems provide active-active multipathing, including with the use of SM-as. Consider the following diagram:     An ASA configuration with non-uniform access would work largely the same as it would with AFF. With uniform access, IO would be crossing the WAN. This may or may not be desirable. If the two sites were 100 meters apart with fiber connectivity there should be no detectable additional latency crossing the WAN, but if the sites were a long distance apart then read performance would suffer on both sites. In contrast, with AFF those WAN-crossing paths would only be used if there were no local paths available and read performance would be better.   ASA with SM-as in a low-latency configuration offers two interesting benefits. First, it essentially doubles the performance for any single host because IO can be serviced by twice as many controllers using twice as many paths. Second, it offers extreme availability because an entire storage system could be lost without interrupting host access.   SnapMirror active sync vs MetroCluster   Those of you that know NetApp probably know SM-as' cousin, MetroCluster. SM-as is similar to NetApp MetroCluster in overall functionality, but there are important differences in the way in which RPO=0 replication is implemented and how it is managed.   A MetroCluster configuration is more like one integrated cluster with nodes distributed across sites. SM-as behaves like two otherwise independent clusters that are cooperating in serving data from specified RPO=0 synchronously replicated LUNs. The data in a MetroCluster configuration is only accessible from one particular site at any given time. A second copy of the data is present on the opposite site, but the data is passive. It cannot be accessed without a storage system failover. MetroCluster and SM-as mirroring occur at different levels. MetroCluster mirroring is performed at the RAID layer. The low-level data is stored in a mirrored format using SyncMirror. The use of mirroring is virtually invisible up at the LUN, volume, and protocol layers. In contrast, SM-as mirroring occurs at the protocol layer. The two clusters are overall independent clusters. Once the two copies of data are in sync, the two clusters only need to mirror writes. When a write occurs on one cluster, it is replicated to the other cluster. The write is only acknowledged to the host when the write has completed on both sites. Other than this protocol splitting behavior, the two clusters are otherwise normal ONTAP clusters. The sweet spot for MetroCluster is large-scale replication. You can replicate an entire array with RPO=0 and near-zero RTO. This simplifies the failover process because there is only one "thing" to fail over, and it scales extremely well in terms of capacity and IOPS. The first sweet spot for SM-as is granular replication. Sometimes you don’t want to replicate all data on a storage system as a single unit, or you need to be able to selectively fail over certain workloads. The second sweet spot for SM-as is for active-active operations, where you want fully usable copies of data to be available on two different clusters located in two different locations with identical performance characteristics and, if desired, no requirement to stretch the SAN across sites. You can have your applications already running on both sites, which reduces the overall RTO during failover operations.   SnapMirror active sync setup   Although the following sections focus on an Oracle RAC configuration, the concepts are applicable to most any SnapMirror active sync deployment. Using Oracle RAC as an example is especially helpful because Oracle RAC is inherently complicated. If you understand how to configure and manage Oracle RAC on SM-as, including all the nuances, requirements, and especially the benefits ONTAP delivers for such a configuration, you’ll be able to use SM-as for any workload.   Additionally, I’ll be using the CLI to configure ONTAP, but you can also use the SystemManager GUI, and I want to say that I think it's outstanding. The UI team did great work with SnapMirror active sync. I’m using the CLI mostly because I personally prefer a CLI, but also because it's easier to explain what's happening with ONTAP if I can go step-by-step with the CLI. The GUI automates multiple operations in a single step.   Finally, I'll assume a mediator is already properly configured.   Storage Virtual Machines (SVM)   One item of ONTAP terminology you need to understand is the SVM, also called a vserver. ONTAP storage clusters are built for multitenancy. When you first install a system, it doesn't provide storage services in the same way that a new VMware server can't run applications. You need to create a VM.   An ONTAP SVM (again, also called a vserver, especially at the CLI) is not an actual VM running under a hypervisor, but conceptually it's the same thing. It's a virtual storage system with its own network addresses, security configuration, user accounts, and so forth. Once you create an SVM, you then provision your file shares, LUNs, S3 buckets, and other storage resources under the control of that SVM.   You'll see the use of -vserver in the examples below. I have two clusters, Cluster1 and Cluster2. Cluster1 includes an SVM called jfs_as1, which will be participating in a SnapMirror active sync relationship with SVM jfs_as2, located on Cluster2.   Provision storage   Once the SVM is defined, the next step is to create some volumes and LUNs. Remember – with ONTAP, a volume is not a LUN. A volume is a management object for storing data, including LUNs. We usually combine related LUNs in a single volume. The volume layout is usually designed to simplify management of those LUNs.   Confused? Here’s a diagram of my basic volume and LUN layout.           There is no true best practice for any database layout. The best option depends on what you want to do with the data. In this case, there are three volumes:   One volume for the Oracle Grid management and quorum LUNs. One volume for the Oracle datafiles. One volume for the Oracle logs, and this includes redo logs, archive logs, and the controlfiles. LUN count for datafile IO of 8 LUNs. This isn't an ONTAP limitation, it's about the host OS. You need multiple LUNs to maximum SAN performance through the host IO stack. 4-8 LUNs is usuall required. The LUN count for logs and other sequentially accessed files is less important.   This design is all about manageability. For example, if you need to restore a database, the quickest way to do it would be to revert the state of the datafiles and then replay logs to the desired restore point. If you store the datafiles in a single volume, you can revert the state of that one volume to an earlier snapshot, but you need to ensure the log files are separated from datafiles. Otherwise, reverting the state of the datafiles would result in loss of archive log data that is critical for RPO=0 data recovery.   Likewise, this design allows you to manage performance more easily. You simply place a QoS limit on the datafile volume, rather than individual LUNs. As an aside, never place a QoS limit on a database transaction log because of the bursty nature of log IO. The average IO for transaction logs is usually fairly low, but the IO occurs in short bursts. If QoS engages during those bursts, the result will be significant performance damage. QoS is for datafiles only.   Create the volumes   I created my three volumes as follows.   Cluster1::> vol create -vserver jfs_as1 -volume jfsAA_grid_siteA -snapshot-policy none -percent-snapshot-space 0 -size 256g -space-guarantee Cluster1::> vol create -vserver jfs_as1 -volume jfsAA_oradata_siteA -snapshot-policy none -percent-snapshot-space 0 -size 1t -space-guarantee Cluster1::> vol create -vserver jfs_as1 -volume jfsAA_logs_siteA -snapshot-policy none -percent-snapshot-space 0 -size 500g -space-guarantee none   There’s a lot of different options when creating a volume. If you use SystemManager, you’ll get volumes with default behavior that is close to universally appropriate, but when using the CLI you might need to look at all the available options.   In my case, I wanted to create volumes for the grid, datafile, and log LUNs that include the following attributes:   Disabling scheduled snapshots. Scheduled snapshots can provide powerful on-box data protection, but the schedule, retention policy, and naming conventions need to be based on the SLAs. For now, I’d rather just disable snapshots to ensure no snapshots are unknowingly created. Set the snapshot reserve to 0%. There is no reason to reserve snapshot space in a SAN environment. Set space guarantees to none. Space guarantees would reserve the volume’s capacity on the system, which is almost always wasteful with a database. Most databases compress well, so reserving the full size of the volume would be unnecessary. I want space efficiency settings to be enabled, but this is default behavior and does not require special arguments.   Create the LUNS   The next step is to create the required LUNs inside the volumes. I’ll be using the defaults, plus disabling space reservations. This will result in LUNs only consuming the space actually required by that LUN within the volume.   Cluster1::> lun create -vserver jfs_as1 /vol/jfsAA_grid_siteA/lun0 -size 64g -ostype linux -space-reserve disabled Cluster1::> lun create -vserver jfs_as1 /vol/jfsAA_grid_siteA/lun1 -size 64g -ostype linux -space-reserve disabled … Cluster1::> lun create -vserver jfs_as1 /vol/jfsAA_logs_siteA/lun0 -size 64g -ostype linux -space-reserve disabled Cluster1::> lun create -vserver jfs_as1 /vol/jfsAA_logs_siteA/lun1 -size 64g -ostype linux -space-reserve disabled   Define the CG   There are now three volumes of data: Oracle datafiles LUNs, Oracle log LUNs, and RAC cluster resource LUNs. While three separate volumes deliver optimal manageability, synchronous replication requires a single container. This RAC environment needs to be replicated and kept in sync as a unified whole. If site failure occurs, all resources at the surviving site need to be consistent with one another. We need a consistency group.   As mentioned above, with ONTAP, a volume is a not a LUN, it’s just a management container. If all LUNs in a given dataset were placed on a single volume, you can create snapshots, clone, restore, or replicate that single volume as a unit. In other words, a volume in ONTAP is natively a consistency group.   In many SM-as use cases, placing all the LUNs of a given application in a single volume might be all you need to meet your data protection requirements. Sometimes your requirements are more complicated. You might need to separate an application into multiple volumes based on manageability requirements, but also want to manage the application as a unit. That’s why we created ONTAP Consistency Groups, and you can read more about them here and here.   I can define a consistency group for my three volumes at the CLI by providing a CG name and the current volumes.   Cluster1::> consistency-group create -vserver jfs_as1 -consistency-group jfsAA -volumes jfsAA_oradata_siteA,jfsAA_logs_siteA,jfsAA_grid_siteA   The result is I've created a CG called jfsAA (I used the letters AA to denote the active-active configuration I’m building) based on the current grid, datafile, and log volumes and the LUNs they contain. Also, note that I used a suffix of _siteA so I can more easily keep track of while cluster hosts which volumes and the data within those volumes. More on that below…   Establish replication   There are now volumes of LUNs on site A, but replication isn’t operational yet. The first step to prepare for replication is to create volumes on site B. As mentioned previously, the SystemManager UI automates all of the setup work, but it’s easier to explain each step in the automated sequence of events using the CLI   Create destination volumes   Before I can replicate data, I need a place to host the replicas.   Cluster2::> vol create -vserver jfs_as2 -volume jfsAA_grid_siteB -snapshot-policy none -percent-snapshot-space 0 -size 256g -type DP -space-guarantee none Cluster2::> vol create -vserver jfs_as2 -volume jfsAA_oradata_siteB -snapshot-policy none -percent-snapshot-space 0 -size 1t -type DP -space-guarantee none Cluster2::> vol create -vserver jfs_as2 -volume jfsAA_logs_siteB -snapshot-policy none -percent-snapshot-space 0 -size 500g -type DP -space-guarantee none   The commands above created three volumes on site B using the same settings as used as site A, except they are a type “DP”, which means a data protection volume. This identifies a volume that will be joined to a replication relationship. No LUNs are being provisioned. They will be automatically created once replication is initialized and the content of the volumes on Cluster2 are synchronized from the source volumes on Cluster1.   Initialize replication   The following command creates an SnapMirror active sync relationship in active-active mode. This command is run on the uninitialized controller. SnapMirror is designed as a pull technology. For example, asynchronous SnapMirror updates pull new data from the source. It is not pushed from the source to a destination.   Cluster2::> snapmirror create -source-path jfs_as1:/cg/jfsAA -destination-path jfs_as2:/cg/jfsAA -cg-item-mappings jfsAA_grid_siteA:@jfsAA_grid_siteB,jfsAA_oradata_siteA:@jfsAA_oradata_siteB,jfsAA_logs_siteA:@jfsAA_logs_siteB -policy AutomatedFailOverDuplex Operation succeeded: SnapMirror create for the relationship with destination "jfs_as2:/cg/jfsAA".   This operation makes a lot more sense if you see it in the GUI, but I can break down the command. Here’s what it does:   snapmirror create   That's self-evident. It's creating a snapmirror relationship.   -source-path jfs_as1:/cg/jfsAA   Use a source of the SVM called jfs_as1 and the consistency group called jfsAA. You'll see this syntax elsewhere in the ONTAP CLI. A consistency group is denoted as [svm name]:/cg/[cg name].   -destination-path jfs_as2:/cg/jfsAA   Replicate that source CG called jfsAA to the SVM called jfs_as2 and use the same CG name of jfsAA   -cg-item-mappings \ jfsAA_grid_siteA:@jfsAA_grid_siteB, \ jfsAA_oradata_siteA:@jfsAA_oradata_siteB,\ jfsAA_logs_siteA:@jfsAA_log_siteB   This section controls the mapping of volumes to volumes. The syntax is [source volume]:@[destination volume]. I've mapped the source grid volume to the destination grid volume, oradata to oradata, and logs to logs. The only difference in the name is the suffix. I used a siteA and siteB suffix for the volumes to avoid user errors. If someone is performing management activities on the UI, using either SystemManager or the CLI, they should be readily able to tell whether they're working on the site A or site B system based on the suffix of the volume names.   -policy AutomatedFailOverDuplex   The final argument specifies a snapmirror relationship of type AutomatedFailoverDuplex, which means bidirectional synchronous replication with automated failure detection. The relationship now exists, but it's not yet initialized. This requires the following command on the second cluster.   Cluster2::> snapmirror initialize -destination-path jfs_as2:/cg/jfsAA Operation is queued: SnapMirror initialize of destination "jfs_as2:/cg/jfsAA".   I can check the status as follows, and the key is looking for a Relationship Status of InSync and Healthy being true.   Cluster2::> snapmirror show -vserver jfs_as2 -destination-path jfs_as2:/cg/jfsAA                             Source Path: jfs_as1:/cg/jfsAA                        Destination Path: jfs_as2:/cg/jfsAA                       Relationship Type: XDP                 Relationship Group Type: consistencygroup                       SnapMirror Policy: AutomatedFailOverDuplex                            Mirror State: Snapmirrored                     Relationship Status: InSync                                 Healthy: true Define the igroup   Before I make the LUNs available, I need to define the initiator group (igroup). I’m building an Oracle RAC cluster, which means two different hosts will be accessing the same LUNs. I’m using iSCSI, but FC works the same. It just uses WWNs rather than iSCSI initiator IDs.   First, I'll create the igroup on the site A system. Any LUNs mapped to this igroup will be available via iSCSI to hosts using the specified initiator.   Cluster1::> igroup create -vserver jfs_as1 -igroup jfsAA -ostype linux -initiator iqn.1994-05.com.redhat:a8ee93358a32   Next, I'll enter Advanced mode and associate the local initiator name with the local SVM. This is how ONTAP controls path priority.  Any host with a WWN or iSCSI initiator listed in igroup will be able to access LUNs mapped to that igroup, but the path priorities would not be optimal. I want paths originating on site A to only be advertised as optimal paths to the hosts located on site A.   Cluster1::> set advanced Cluster1::*> igroup initiator add-proximal-vserver -vserver jfs_as1 iqn.1994-05.com.redhat:a8ee93358a32 -proximal-vservers jfs_as1   I then repeat the process on site B, using the WWN or iSCSI initiator of the host on site B.   Cluster2::> igroup create -vserver jfs_as2  -igroup jfsAA -ostype linux -initiator iqn.1994-05.com.redhat:5214562dfc56 Cluster1::*> set advanced Cluster2::*> igroup initiator add-proximal-vserver -vserver jfs_as2 iqn.1994-05.com.redhat:5214562dfc56 -proximal-vservers jfs_as2   I didn't really need to add the proximal-vserver information. Read on to understand why not.   Map the LUNs   Next, I map the LUNs to the igroup so the hosts will be able to access them. Although I used the same igroup name on each site, these igroups are located on different SVMs.   Cluster1::> lun map -vserver jfs_as1 /vol/jfsAA_grid_siteA/lun0  -igroup jfsAA Cluster1::> lun map -vserver jfs_as1 /vol/jfsAA_grid_siteA/lun1  -igroup jfsAA … Cluster1::> lun map -vserver jfs_as1 /vol/jfsAA_oradata_siteA/lun6 -igroup jfsAA Cluster1::> lun map -vserver jfs_as1 /vol/jfsAA_oradata_siteA/lun7 -igroup jfsAA   Cluster2::> lun map -vserver jfs_as2 /vol/jfsAA_grid_siteB/lun0  -igroup jfsAA Cluster2::> lun map -vserver jfs_as2 /vol/jfsAA_grid_siteB/lun1  -igroup jfsAA … Cluster2::> lun map -vserver jfs_as2 /vol/jfsAA_oradata_siteB/lun6 -igroup jfsAA Cluster2::> lun map -vserver jfs_as2 /vol/jfsAA_oradata_siteB/lun7 -igroup jfsAA   Oracle Configuration   From this point, setup is exactly like any other Oracle RAC server. Functionally, this is like a 2-site Oracle Extended RAC cluster, except there's no need for configuring ASM failgroups. The replication services are built-in to the storage system.   Device Names   There are multiple ways to control device names with Oracle, but my personal preference is using udev rules and multipath aliases. It takes more up-front work, but I have more control over the exact naming conventions to be used.   The multipath.conf file looks like this on each RAC node:   [root@jfs12 ~]# cat /etc/multipath.conf multipaths {     multipath {         wwid  3600a0980383041327a2b55676c547247         alias grid0     }     multipath {         wwid  3600a0980383041327a2b55676c547248         alias grid1     }     multipath {         wwid  3600a0980383041327a2b55676c547249         alias grid2     }     …     }     multipath {         wwid  3600a0980383041334a3f55676c69734d         alias logs5     }     multipath {         wwid  3600a0980383041334a3f55676c69734e        alias logs6     }     multipath {         wwid  3600a0980383041334a3f55676c69734f         alias logs7     } }   Then I have the following udev rule:   [root@jfs12 ~]# cat /etc/udev/rules.d/99-asm.rules ENV{DM_NAME}=="grid*", GROUP:="asmadmin", OWNER:="grid", MODE:="660" ENV{DM_NAME}=="oradata*", GROUP:="asmadmin", OWNER:="grid", MODE:="660" ENV{DM_NAME}=="logs*", GROUP:="asmadmin", OWNER:="grid", MODE:="660"   The result is clear device names that are automatically assigned the correct user and group permissions. You can even see the devices in /dev/mapper.   [root@jfs12 ~]# ls /dev/mapper control  grid1  logs0  logs2  logs4  logs6  oradata0  oradata2  oradata4  oradata6  vg00-root   ASM Configuration   Unlike typical Extended RAC, I created the ASM diskgroups with external redundancy, which means no mirroring by ASM itself. Replication services are provided by the storage system, not RAC. I created the following ASM diskgroups:     [root@jfs12 ~]# /grid/bin/asmcmd ls DBF/ GRID/ LOGS/ From this point on, the installation process is exactly like any other Oracle RAC installation.   Database Creation   I used the following database layout:   [root@jfs12 ~]# /grid/bin/asmcmd ls DBF/NTAP 124D13D9FE3FFF29E06370AAC00A260E/ 124DA9A2A3BB954AE06370AAC00A7624/ DATAFILE/ NTAPpdb1/ PARAMETERFILE/ PASSWORD/ TEMPFILE/ pdbseed/ sysaux01.dbf system01.dbf temp01.dbf undotbs01.dbf undotbs02.dbf users01.dbf [root@jfs12 ~]# /grid/bin/asmcmd ls LOGS/NTAP ARCHIVELOG/ CONTROLFILE/ ONLINELOG/ control01.ctl control02.ctl redo01.log redo02.log redo03.log redo04.log   There is a reason for this, but it's not connected to the use of SnapMirror active sync. See the section above called "Provision storage" for an explanation.   Failure scenarios   The following diagram shows the Oracle database and storage configuration as it exists at the hardware level (mediator not shown), and it is using a non-uniform network configuration. There is no SAN connectivity across sites.         I wrote above that I didn't need to configure host proximity. The reason is I'm using a nonuniform configuration. I have not stretched the SAN across sites. The only paths available to hosts are local paths. There's no reason to add host proximity settings because hosts will never be able to see storages paths to an opposite site. I included the instructions for configuring host proximity for readers who may be using uniform configurations and need to know how proximity settings are controlled.   Several of the scenarios described below that resulted in loss of database services would not have happened with a uniform network configuration. There is no cross-site SAN connectivity, so anything that results in the loss of active paths on a given site will mean there were no paths remaining at all. In a uniform network configuration, each site would be able to use alternate paths on the opposite site. The reason a non-uniform configuration was chosen for these tests was to illustrate that it is possible to have active-active RPO=0 replication without extending the SAN. It's a more demanding use case, and while it does have some limitations it also has benefits, such as a simpler SAN architecture.   There's another way to look at the storage architecture. The existence of replication is essentially invisible to the Oracle database and the RAC cluster. Various failure scenarios might result in disruption to certain database servers or the loss of certain paths, but as far as the database is concerned, there's just one set of LUNs. Logically, it looks like this:       Preferred sites   The configuration is symmetric, with one exception that is connected to split-brain management.   The question you need to ask is this - what happens if the replication link is lost and neither site has quorum? What do you want to happen? This question applies to both the Oracle RAC and the ONTAP behavior. If changes cannot be replicated across sites, and you want to resume operations, one of the sites will have to survive and the other site will have to become unavailable.   Oracle and css_critical   In the case of Oracle RAC, the default behavior is that one of the nodes in a RAC cluster that consists of an even number of servers will be deemed more important than the other nodes. The site with that higher priority node will survive site isolation while the nodes on the other site will evict. The prioritization is based on multiple factors, but you can also control this behavior using the css_critical setting.   My environment has two nodes, jfs12 and jfs13. The current settings for css_critical are as follows:   [root@jfs12 ~]# /grid/bin/crsctl get server css_critical CRS-5092: Current value of the server attribute CSS_CRITICAL is no.   [root@jfs13 trace]# /grid/bin/crsctl get server css_critical CRS-5092: Current value of the server attribute CSS_CRITICAL is no.   I want the site with jfs12 to be the preferred site, so I changed this value to yes on a site A node and restart services.   [root@jfs12 ~]# /grid/bin/crsctl set server css_critical yes CRS-4416: Server attribute 'CSS_CRITICAL' successfully changed. Restart Oracle High Availability Services for new value to take effect. [root@jfs12 ~]# /grid/bin/crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'jfs12' CRS-2673: Attempting to stop 'ora.crsd' on 'jfs12' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'jfs12' CRS-2673: Attempting to stop 'ora.ntap.ntappdb1.pdb' on 'jfs12' … CRS-2673: Attempting to stop 'ora.gipcd' on 'jfs12' CRS-2677: Stop of 'ora.gipcd' on 'jfs12' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'jfs12' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@jfs12 ~]# /grid/bin/crsctl start crs CRS-4123: Oracle High Availability Services has been started.   SnapMirror active sync preferred site   At any given moment, SnapMirror active sync will consider one site the "source" and the other the "destination". This implies a one-way replication relationship, but that's not what's happening. This is how the preferred site is determined. If the replication link is lost, the LUN paths on the source will continue to serve data while the LUN paths on the destination will become unavailable until replication is reestablished and enters a synchronous state. The paths will then resume serving data.   My current configuration has site A as the preferred site for both Oracle and ONTAP. This can be viewed via SystemManager:         or at the CLI:   Cluster2::> snapmirror show -destination-path jfs_as2:/cg/jfsAA                             Source Path: jfs_as1:/cg/jfsAA                        Destination Path: jfs_as2:/cg/jfsAA                       Relationship Type: XDP                 Relationship Group Type: consistencygroup                     SnapMirror Schedule: -                  SnapMirror Policy Type: automated-failover-duplex                       SnapMirror Policy: AutomatedFailOverDuplex                            Mirror State: Snapmirrored                     Relationship Status: InSync   The key is that the source is the SVM on cluster1. As mentioned above, the terms "source" and "destination" don't describe the flow of replicated data. Both sites can process a write and replicate it to the opposite site. In effect, both clusters are sources and destinations. The effect of designating one cluster as a source simply controls which cluster survives as a read-write storage system if the replication link is lost.   Loss of SnapMirror replication connectivity   If I cut the SM-as replication link, write IO cannot be completed because it would be impossible for a cluster to replicate changes to the opposite site. Here's what will happen with an Oracle RAC environment.   Site A   The result on site A of a replication link failure will be an approximately 15 second pause in write IO processing as ONTAP attempts to replicate writes before it determines that the replication link is genuinely inoperable. After the 15 seconds elapses, the ONTAP cluster on site A resumes read and write IO processing. The SAN paths will not change, and the LUNs will remain online.   Site B   Since site B is not the SnapMirror active sync preferred site, its LUN paths will become unavailable after about 15 seconds.   The replication link was cut at the timestamp 15:19:44. The first warning from Oracle RAC arrives 100 seconds later as the 200 second timeout (controlled by the Oracle RAC parameter disktimeout) approaches.   2024-09-10 15:21:24.702 [ONMD(2792)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 99340 milliseconds. 2024-09-10 15:22:14.706 [ONMD(2792)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 49330 milliseconds. 2024-09-10 15:22:44.708 [ONMD(2792)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 19330 milliseconds. 2024-09-10 15:23:04.710 [ONMD(2792)]CRS-1604: CSSD voting file is offline: /dev/mapper/grid2; details at (:CSSNM00058:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc. 2024-09-10 15:23:04.710 [ONMD(2792)]CRS-1606: The number of voting files available, 0, is less than the minimum number of voting files required, 1, resulting in CSSD termination to ensure data integrity; details at (:CSSNM00018:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc 2024-09-10 15:23:04.716 [ONMD(2792)]CRS-1699: The CSS daemon is terminating due to a fatal error from thread: clssnmvDiskPingMonitorThread; Details at (:CSSSC00012:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc 2024-09-10 15:23:04.731 [OCSSD(2794)]CRS-1652: Starting clean up of CRSD resources. Once the 200 second voting disk timeout has been reached, this Oracle RAC node will evict itself from the cluster and reboot.   Loss of Oracle RAC replication   Loss of the Oracle RAC replication link will produce a similar result, except the timeouts will be shorter by default. An Oracle RAC node will wait 200 seconds after loss of storage connectivity before evicting, but it will only wait 30 seconds after loss of the RAC network heartbeat.   The CRS messages are similar to those shown below.  You can see the 30 second timeout lapse. Since css_critical was set on jfs12, located on site A, that will be the site to survive and jfs13 on site B will be evicted.   2024-09-12 10:56:44.047 [ONMD(3528)]CRS-1611: Network communication with node jfs13 (2) has been missing for 75% of the timeout interval.  If this persists, removal of this node from cluster will occur in 6.980 seconds 2024-09-12 10:56:48.048 [ONMD(3528)]CRS-1610: Network communication with node jfs13 (2) has been missing for 90% of the timeout interval.  If this persists, removal of this node from cluster will occur in 2.980 seconds 2024-09-12 10:56:51.031 [ONMD(3528)]CRS-1607: Node jfs13 is being evicted in cluster incarnation 621599354; details at (:CSSNM00007:) in /gridbase/diag/crs/jfs12/crs/trace/onmd.trc. 2024-09-12 10:56:52.390 [CRSD(6668)]CRS-7503: The Oracle Grid Infrastructure process 'crsd' observed communication issues between node 'jfs12' and node 'jfs13', interface list of local node 'jfs12' is '192.168.30.1:33194;', interface list of remote node 'jfs13' is '192.168.30.2:33621;'. 2024-09-12 10:56:55.683 [ONMD(3528)]CRS-1601: CSSD Reconfiguration complete. Active nodes are jfs12 . 2024-09-12 10:56:55.722 [CRSD(6668)]CRS-5504: Node down event reported for node 'jfs13'. 2024-09-12 10:56:57.222 [CRSD(6668)]CRS-2773: Server 'jfs13' has been removed from pool 'Generic'. 2024-09-12 10:56:57.224 [CRSD(6668)]CRS-2773: Server 'jfs13' has been removed from pool 'ora.NTAP'.   Complete loss of replication network   Oracle RAC split-brain detection has a dependency on the Oracle RAC storage heartbeat. If loss of site-to-site connectivity results in simultaneous loss of both the RAC network heartbeat and storage replication services, the result is the RAC sites will not be able to communicate cross-site via either the RAC interconnect or the RAC voting disks. The result in an even-numbed set of nodes may be eviction of both sites under default settings. The exact behavior will depend on the sequence of events and the timing of the RAC network and disk heartbeat polls.   The risk of a 2-site outage can be addressed in two ways. First, using an odd number of RAC nodes, preferably by use of a 3 rd site tiebreaker. If a 3 rd site is unavailable, the tiebreaker instance could be placed on one site. This means that loss of site-to-site connectivity will cause LUN paths to go down on one site, but one of the RAC sites will still have quorum and will not evict.   If a 3 rd site is not available, this problem can be addressed by adjusting the misscount parameter on the RAC cluster. Under the defaults, the RAC network heartbeat timeout is 30 seconds. This normally is used by RAC to identify failed RAC nodes and remove them from the cluster. It also has a connection to the voting disk heartbeat.   If, for example, the conduit carrying intersite traffic for both Oracle RAC and storage replication services is cut by a backhoe, the 30 second misscount countdown will begin. If the RAC preferred site node cannot reestablish contact with the opposite site within 30 seconds, and it also cannot use the voting disks to confirm the opposite site is down within that same 30 second window, then the preferred site nodes will also evict. The result is a full database outage.   Depending on when the misscount polling occurs, 30 seconds may not be enough time for SnapMirror active sync to time out and allow storage on the preferred site to resume services before the 30 second window expires. This 30 second window can be increased.   [root@jfs12 ~]# /grid/bin/crsctl set css misscount 100 CRS-4684: Successful set of parameter misscount to 100 for Cluster Synchronization Services.   This value allows the storage system on the preferred site to resume operations before the misscount timeout expires. The result will then be eviction only of the nodes at the site where the LUN paths were removed. Example below:   2024-09-12 09:50:59.352 [ONMD(681360)]CRS-1612: Network communication with node jfs13 (2) has been missing for 50% of the timeout interval.  If this persists, removal of this node from cluster will occur in 49.570 seconds 2024-09-12 09:51:10.082 [CRSD(682669)]CRS-7503: The Oracle Grid Infrastructure process 'crsd' observed communication issues between node 'jfs12' and node 'jfs13', interface list of local node 'jfs12' is '192.168.30.1:46039;', interface list of remote node 'jfs13' is '192.168.30.2:42037;'. 2024-09-12 09:51:24.356 [ONMD(681360)]CRS-1611: Network communication with node jfs13 (2) has been missing for 75% of the timeout interval.  If this persists, removal of this node from cluster will occur in 24.560 seconds 2024-09-12 09:51:39.359 [ONMD(681360)]CRS-1610: Network communication with node jfs13 (2) has been missing for 90% of the timeout interval.  If this persists, removal of this node from cluster will occur in 9.560 seconds 2024-09-12 09:51:47.527 [OHASD(680884)]CRS-8011: reboot advisory message from host: jfs13, component: cssagent, with time stamp: L-2024-09-12-09:51:47.451 2024-09-12 09:51:47.527 [OHASD(680884)]CRS-8013: reboot advisory message text: oracssdagent is about to reboot this node due to unknown reason as it did not receive local heartbeats for 10470 ms amount of time 2024-09-12 09:51:48.925 [ONMD(681360)]CRS-1632: Node jfs13 is being removed from the cluster in cluster incarnation 621596607   Note: Oracle Support strongly discourages altering with the misscount or disktimeout parameters to solve configuration problems. Changing these parameters can, however, be warranted and unavoidable in many cases, including SAN booting, virtualized, and storage replication configurations. If, for example, you had stability problems with a SAN or IP network that was resulting in RAC evictions you should fix the underlying problem and not charge the values of the misscount or disktimeout. Changing timeouts to address configuration errors is masking a problem, not solving a problem. Changing these parameters to properly configure a RAC environment based on design aspects of the underlying infrastructure is different and is consistent with Oracle support statements. With SAN booting, it is common to adjust misscount all the way up to 200 to match disktimeout.   Storage system failure   The result of a storage system failure is nearly identical to the result of losing the replication link. The surviving site should experience a roughly 15 second IO pause on writes. Once that 15 second period elapses, IO will resume on that site as usual.   The Oracle RAC node on the failed site will lose storage services and enter the same 200 second disktimeout countdown before eviction and subsequent reboot.   2024-09-11 13:44:38.613 [ONMD(3629)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 99750 milliseconds. 2024-09-11 13:44:51.202 [ORAAGENT(5437)]CRS-5011: Check of resource "NTAP" failed: details at "(:CLSN00007:)" in "/gridbase/diag/crs/jfs13/crs/trace/crsd_oraagent_oracle.trc" 2024-09-11 13:44:51.798 [ORAAGENT(75914)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 75914 2024-09-11 13:45:28.626 [ONMD(3629)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 49730 milliseconds. 2024-09-11 13:45:33.339 [ORAAGENT(76328)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 76328 2024-09-11 13:45:58.629 [ONMD(3629)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/mapper/grid2 will be considered not functional in 19730 milliseconds. 2024-09-11 13:46:18.630 [ONMD(3629)]CRS-1604: CSSD voting file is offline: /dev/mapper/grid2; details at (:CSSNM00058:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc. 2024-09-11 13:46:18.631 [ONMD(3629)]CRS-1606: The number of voting files available, 0, is less than the minimum number of voting files required, 1, resulting in CSSD termination to ensure data integrity; details at (:CSSNM00018:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc 2024-09-11 13:46:18.638 [ONMD(3629)]CRS-1699: The CSS daemon is terminating due to a fatal error from thread: clssnmvDiskPingMonitorThread; Details at (:CSSSC00012:) in /gridbase/diag/crs/jfs13/crs/trace/onmd.trc 2024-09-11 13:46:18.651 [OCSSD(3631)]CRS-1652: Starting clean up of CRSD resources.   The SAN path state on the RAC node that has lost storage services looks like this:   oradata7 (3600a0980383041334a3f55676c697347) dm-20 NETAPP,LUN C-Mode size=128G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=0 status=enabled | `- 34:0:0:18 sdam 66:96  failed faulty running `-+- policy='service-time 0' prio=0 status=enabled   `- 33:0:0:18 sdaj 66:48  failed faulty running   The linux host detected the loss of the paths much quicker than 200 seconds, but from a database perspective the client connections to the host on the failed site will still be frozen for 200 seconds under the default Oracle RAC settings. Full database operations will only resume after the eviction is completed.   Meanwhile, the Oracle RAC node on the opposite site will record the loss of the other RAC node. It otherwise continues to operate as usual.   2024-09-11 13:46:34.152 [ONMD(3547)]CRS-1612: Network communication with node jfs13 (2) has been missing for 50% of the timeout interval.  If this persists, removal of this node from cluster will occur in 14.020 seconds 2024-09-11 13:46:41.154 [ONMD(3547)]CRS-1611: Network communication with node jfs13 (2) has been missing for 75% of the timeout interval.  If this persists, removal of this node from cluster will occur in 7.010 seconds 2024-09-11 13:46:46.155 [ONMD(3547)]CRS-1610: Network communication with node jfs13 (2) has been missing for 90% of the timeout interval.  If this persists, removal of this node from cluster will occur in 2.010 seconds 2024-09-11 13:46:46.470 [OHASD(1705)]CRS-8011: reboot advisory message from host: jfs13, component: cssmonit, with time stamp: L-2024-09-11-13:46:46.404 2024-09-11 13:46:46.471 [OHASD(1705)]CRS-8013: reboot advisory message text: At this point node has lost voting file majority access and oracssdmonitor is rebooting the node due to unknown reason as it did not receive local hearbeats for 28180 ms amount of time 2024-09-11 13:46:48.173 [ONMD(3547)]CRS-1632: Node jfs13 is being removed from the cluster in cluster incarnation 621516934   Cut the power to the mediator   The mediator service does not directly control storage operations. It functions as an alternate control path between clusters. It exists primarily to automate failover without the risk of a split-brain scenario. In normal operation, each cluster is replicating changes to its partner, and each cluster therefore can verify that the partner cluster is online and serving data. If the replication link failed, replication would cease.   The reason a mediator is required for safe automated operations is because it would otherwise be impossible for a storage clusters to be able to determine whether loss of bidirectional communication was the result of a network outage or actual storage failure.   The mediator provides an alternate path for each cluster to verify the health of its partner. The scenarios are as follows:   If a cluster can contact its partner directly, replication services are operational. No action required. If a preferred site cannot contact its partner directly or via the mediator, it will assume the partner is either actually unavailable or was isolated and has taken its LUN paths offline. The preferred site will then proceed to release the RPO=0 state and continue processing both read and write IO. If a non-preferred site cannot contact its partner directly, but can contact it via the mediator, it will take its paths offline and await the return of the replication connection. If a non-preferred site cannot contact its partner directly or via an operational mediator, it will assume the partner is either actually unavailable or was isolated and has taken its LUN paths offline. The non-preferred site will then proceed to release the RPO=0 state and continue processing both read and write IO. It will assume the role of the replication source and will become the new preferred site.   If the mediator is wholly unavailable:   Failure of replication services for any reason will result in the preferred site releasing the RPO=0 state and resuming read and write IO processing. The non-preferred site will take its paths offline. Failure of the preferred site will result in an outage because the non-preferred site will be unable to verify that the opposite site is truly offline and therefore it would not be safe for the nonpreferred site to resume services.   Restoring services   It's difficult to demonstrate what happens when power is restored after a failure or a replication link failure is repaired. SnapMirror active sync will automatically detect the presence of a faulty replication relationship and bring it back to an RPO=0 state. Once synchronous replication is reestablished, the paths will come online again.   In many cases, clustered applications will automatically detect the return of failed paths, and those applications will also come back online. In other cases, a host-level SAN scan may be required, or applications may need to be brought back online manually. It depends on the application and how it's configured, and in general such tasks can be easily automated. ONTAP itself is self-healing and should not require any user intervention to resume RPO=0 storage operations.   Another feature of SnapMirror active sync that is difficult to demonstrate is the speed of recovery. Unlike many competitors, ONTAP replication uses a pointer-based mechanism to track changes. The result is resynchronization after replication interruption is extremely fast. ONTAP is able to identify changed blocks and efficiently and quickly ship them to the remote system, even if replication was interrupted for days, weeks, or months. Many competitors use a slower and less efficient extent-based approach which requires reading and transferring much more data than merely those blocks which have changed. In some cases, the entire mirror must be rebuilt.   In addition, ONTAP's unique pointer-based technology preserves an intact copy of your data during resynchronization. The copy would be out of date, but it would exist. Resynchronization with many competing technologies results in a copy that is corrupt until the resynchronization is complete. This leaves a customer exposed to significant data loss if the last remaining copy of the data is damaged.    This is one of many examples of how ONTAP features not only deliver results, but also deliver a superior end result because ONTAP just works better.  SnapMirror active sync failover   Changing the preferred site requires a simple operation. IO will pause for a second or two as authority over replication behavior switches between clusters, but IO is otherwise unaffected.   GUI example:         Example of changing it back via the CLI:   Cluster2::> snapmirror failover start -destination-path jfs_as2:/cg/jfsAA [Job 9575] Job is queued: SnapMirror failover for destination "jfs_as2:/cg/jfsAA                      ". Cluster2::> snapmirror failover show Source    Destination                                          Error Path      Path        Type     Status    start-time end-time   Reason -------- -----------  -------- --------- ---------- ---------- ---------- jfs_as1:/cg/jfsAA          jfs_as2:/cg/jfsAA                       planned  completed 9/11/2024  9/11/2024                                          09:29:22   09:29:32   The new destination path can be verified as follows:     Cluster1::> snapmirror show -destination-path jfs_as1:/cg/jfsAA                             Source Path: jfs_as2:/cg/jfsAA                        Destination Path: jfs_as1:/cg/jfsAA                       Relationship Type: XDP                 Relationship Group Type: consistencygroup                  SnapMirror Policy Type: automated-failover-duplex                       SnapMirror Policy: AutomatedFailOverDuplex                            Mirror State: Snapmirrored                     Relationship Status: InSync   Summary   The examples above are designed to illustrate SnapMirror active sync functionality. As with most enterprise IT projects, there are few true best practices. The right configuration depends on business needs including RPO, RTO, and SLAs. With respect to disaster recovery, it also depends on which disaster scenarios are more likely than others to occur.   The next posts planned will include use of an Oracle RAC tiebreaker, use of uniform networking, and quantifying application IO resumption times. The foundation remains the same - SnapMirror active sync operating in active-active mode, an integrated RPO=0/RTO=0 SAN replication technology that is reliable, self-healing, easily configured, and easily managed.   If you want to learn more, you can visit the official ONTAP documentation for SnapMirror active sync, and if you'd really like to see it in action I recommend getting in touch with Neto and the rest of the CPOC (Customer Proof Of Concept) team. They've got all the cool hardware to simulate your real-world workloads.    ... View more

A-Series and C-Series   I've seen an extraordinary amount of interest in the C-Series systems for all sorts of workloads. I'm the Big Scary Enterprise Workload guy, which means I want proof before I recommend anything to a customer. So, I reached out to the Workload Engineering team and got some realistic test data that demonstrates what it can do.   If you want to real all the low-level details about the A-Series and C-Series architectures and their performance characteristics, there's a couple of hyperlinks below.   Before I get into the numbers, I wanted recap my personal view of "What is C-Series?,  or to phrase it better…   Why is C-Series?   For a long time, solid-state storage arrays just used "Flash". For a brief time, it appeared the solid state market was going to divide into Flash media and 3DCrosspoint media, which was a much faster solid state technology, but 3DXP couldn't break out of its niche.   Instead, we've seen the commercial Flash market divide into SLC/TLC drives and QLC drives. Without going into details, SLC/TLC are the faster and more durable options aimed at high-speed storage applications, whereas QLC is somewhat slower and less durable* but also less expensive and are aimed at capacity-centric, less IO-intensive storage applications.   *Note: Don't get misled on this topic. The durability difference between TLC and QLC might be important if you're purchasing a drive for your personal laptop, but the durabilty is essentially identical when drives are inserted into ONTAP.  ONTAP RAID technology is still there protecting data against media failures. Furthermore, ONTAP WAFL technology distributes inbound write data to free blocks across multiple drives. This minimizes overwrites of the individual cells within the drive, which maximizes the drive's useful life. Also, NetApp support agreements that cover drive failures also include drive replacement for SSDs that have exhausted their write cycles.   The result of the market changes is that NetApp now offers the A-Series for high-speed, latency sensitive databases or IO-intensive VMware estates, while the C-Series is for less latency-sensitive and more capacity-centric workloads.   That's easy enough to understand in principle, but it's not enough for DBAs, virtualization admins, and storage admins to make decisions. They want to see the numbers…   The Numbers       What makes this graph so compelling is its simplicity. The IOPS capabilities are comparable. Yes, the C-Series is saturating a touch quicker than A-Series, but it's very close. As expected, the C-Series is showing higher latency, but it's very consistent and also much closer to A-Series performance than it is to hybrid/spinning disk solutions. The workload is an Oracle database issuing a roughly 80%/20% random-read, random-write split. We graph total IOPS against read latency because read latency is normally the most important factor with real-world workloads.   The reason we use Oracle databases is twofold. First, we've got thousands and thousands of controllers servicing Oracle databases, so it's an important market for us. Second, Oracle databases are an especially brutal, latency sensitive workloads with lots of interdependencies between different IO operations. If anything is wrong with storage behavior, you should see it in the charts. You can also extrapolate these results to lots of different workloads beyond databases.   We are also graphing the latency as seen from the database itself. The timing is based on the elapsed time between the application submitting an IO, and the IO being reported as complete. The means we're measuring the entire storage path from the OS, through the network, into the storage system, back across the network to the OS, and up to the application layer.   I'd also like to point out that the performance of the A-Series is amazing. 900K IOPS without even breaching the 1ms latency runs circles around the competition, but I've posted about that before. This post is focusing on C-Series.   Note: These tests all used ONTAP 9.13.1, which includes some significant performance improvements for both A-Series and C-Series.   Write latency   Obviously write latency is also important, but the A-Series and C-Series both use the same write logic from the point of view of the host. Write operations commit to the mirrored, nonvolatile NVRAM journal. Once the data is in NVRAM, the write is acknowledged and host continues. The write to the drive layer comes much later.   Want proof?     This is a graph of total IOPS versus the write latency. Note that the latency is reported in microseconds.   You can see your workload's write latency is mostly unaffected by the choice of A-Series or C-Series. As the IOPS increase toward the saturation point, the write latency on C-Series increases more quickly than A-Series as a result of the somewhat slower media in use, but keep this in perspective. Most real-world workloads run as expected so long as write latency remains below 500µs. Even 1ms of write latency is not necessarily a problem, even with databases.   Caching   These tests used 10TB of data within the database (the database itself was larger, but we're accessing 10TB during the test runs). This means the test results above do include some cache hits on the controller, which reflects how storage is used in the real world. There will be some benefit from caching, but nearly all IO in these tests is being serviced from the actual drives.   We also run these tests with storage efficiency features enabled, using a working set with a reasonable level of compressibility. If you use unrealistic test data, you can get outrageously unreasonable amounts of caching that skew the results in the same way that running a test with a tiny working set that is unrealistically cacheable can skew results.   The reason I want to point this out is that customers considering C-Series need to understand that not all IO latency is affected. The higher latencies only appear with read operations that actually require a disk IO. Reads that can be serviced by onboard cache should be measurable in microseconds, as is true with the A-Series. This is important because all workloads, especially databases, include hot blocks of data that require ultra-low latency. Access times for cached blocks should be essentially identical between A-Series and C-Series.   Sequential IO   Sequential IO is much less affected by the drive type than random IO. The reason is sequential IO involves both readahead and larger blocks. That means the storage system can start performing read IO to the drives before the host even requests the data, and there are much fewer (but larger) IO operations happening on the backend drives.   On the whole, if you're doing sequential IO you should see comparable performance with A-Series, C-Series and even old FAS arrays if they have enough drives.   We compared A-Series to C-Series and saw a peak sequential read throughput of about 30GB/sec with the A-Series and about 27GB/sec with the C-Series. These numbers were using synthetic tools. It's difficult to perform a truly representative sequential IO tests from a database because of the configuration requirements. You'd need an enormous number of FC adapters to run a single A800 or C800 controller to its limit, and it's difficult to get a database to try to read 30GB/sec in the first place.   As a practical matter, few workloads are purely sequential IO, but tasks such as Oracle RMAN backups or database full table scans should perform about the same on both A-Series and C-Series. The limiting factor should normally be the available network bandwidth, not the storage controller.   Summary: A-Series or C-Series?   It's all ONTAP, it's just about the media. If you have a workload that genuinely requires consistent latency down in the 100's of µs range, then choose A-Series. If your workload can accept 2ms (or so) of read latency (and remember, cache hits and write IO latency is much faster) then look at C-Series.   As a general principle, think about whether your workload is about computational speed or is about the end-user experience. A bank performing end-of-day account reconciliation probably needs the ultra-low latency of the A-Series. In contrast, a CRM database is usually about the end users. If you're updating customer records, you probably don't care if it takes an extra 2ms to retrieve a block of data that contains the customer contact information.   You can also build mixed clusters and tier your databases between A-Series and C-Series as warranted. It's all still ONTAP, and you can nondisruptively move your workloads between controllers.   Finally, the C-Series is an especially good option for upgrading legacy spinning drive and hybrid arrays. Spinning drive latencies are typically around 8-10ms, which means C-Series is about 4X to 5X faster in terms of latency. If you're looking at raw IOPS, there's no comparison. A spinning drive saturates around 120 IOPS/drive. You would need about 6000 spinning drives to reach the 800,000 IOPS delivered by just 24 QLC drives as shown in these tests. It's a huge improvement in terms of performance, power/cooling requirements, and costs, but at a lower price point than arrays using SLC or TLC drives.   If you want to learn more, we published the following two technical reports today:   Oracle Performance on AFF A-Series and C-Series Virtualized Oracle Performance on AFF A-Series and C-Series   Bonus Chart:   If you're wondering how A-Series and C-Series compare under virtualization, here it is. It's not easy building a truly optimized 4-node virtualized RAC environment, and we probably could have tuned this better to reduce the overhead from ESX and VMDK's, but the results are still outstanding and more importantly consistent with the bare metal tests. The latency is higher with C-Series, but the IOPS levels are comparable to A-Series.   We also did this test with FCP, not NVMe/FC, because most VMware customers are using traditional FCP.  The protocol change is the primary reason for the lower maximum IOPS level.         ... View more

How many LUNs do I need? (notes on Consistency Groups in ONTAP)   This post is part 2 of 2. In the prior post, I explained consistency groups and how they exist in ONTAP in multiple forms. I’ll now explain the connection between consistency groups and performance, then show you some basic performance envelopes of individual LUNs and volumes.   Those two things may not seem connected at first, but they are. It all comes down to LUNs.   LUNs are accessed using the SCSI protocol, which has been around for over 40 years and is showing its age. The tech industry has worked miracles improving LUN technology over the years, but there are a lot of limits related to host OSs, drivers, HBAs, storage systems, and drives that limit the performance of a single LUN.   The end result is this – sometimes you need more than one LUN to host your dataset in order to get optimum performance. If you want to take advantage of the advanced features of a modern storage array, you’ll need to manage those multiple LUNs together, as a unit. You’ll need a consistency group.   My previous post explained how ONTAP delivers consistency group management. This post explains how you figure out just how many LUNs you might need in that group, and how to ensure you have the simplest, most easily managed configuration.   Note: There are a lot of performance numbers shown below. They do NOT represent maximum controller performance. I did not do any tuning at all beyond implementing basic best practices. I created some basic storage configurations, configured a database, and ran some tests to illustrate my point. That's it.   ONTAP < 9.9.1   First, let’s go back a number of years to the ONTAP versions prior to the 9.9.1 release. There were some performance boundaries and requirements that contributed to the need for multiple LUNs in order to get optimum performance.   Test Procedure   To compare performance with differing LUN/volume layouts, I wrote a script that built one Oracle database each, using the following configurations:   1 LUN in a single volume 2 LUNs in a single volume 4 LUNs in a single volume 8 LUNs in a single volume 16 LUNs in a single volume   16 LUNs in a single volume 16 LUNs across 2 volumes 16 LUNs across 4 volumes 16 LUNs across 8 volumes 16 LUNs across 16 volumes   Warning: I normally ban anyone from using the term “IOPS” in my presence without providing a definition, because “IOPS” has a lot of different meanings. What’s the block size? Sequential or random ratio? Read/write mix? Measured from where? What’s my latency cutoff? All that matters.   In the graphs below, IOPS refers to random reads, using 8K blocks, as measured from the Oracle database. Most tests used 100% reads.   I used SLOB2 for driving the workload. The results shown below are not the theoretical storage maximums, they're the result of a complicated test using an actual Oracle database where a lot of IO has interdependencies on other IO. If you used a synthetic tool like fio, you’d see higher IOPS.   The question was -  “How many LUNs do I need?” These tests used *one* volume. Multiple LUNs, but one volume. Let’s say you have a database. How many LUNs did you need in that LVM or Oracle ASM volume group to support your workload? What is the expected performance? Here’s the answer to that question when using a single AFF8080 controller prior to ONTAP 9.9.1.     There are three important takeaways from that test:   A single LUN hit the wall at about 35K IOPS. A single volume hit the wall at about 115K IOPS The sweet spot for LUN count in a single volume was about 8, but there was some benefit going all the way to 16.   To rephrase that:   If you had a single workload that didn’t need more than 35K IOPS, just drop it on a single LUN. If you had a single workload that didn’t need more than 115K IOPS, just drop it on a single volume, but distribute it across 8 LUNs. If you had more than 115K IOPS, you would have needed more than one volume.   That’s all <9.9.1 performance data, so let see what improved in 9.9.1 and how it erased a lot of those prior limitations and vastly simplified consistency group architectures.   ONTAP >= 9.9.1   Threading is important for modern storage arrays, because they are primarily used to support multiple workloads. On occasion, we’ll see a single database hungry enough to consume the full performance capabilities of an A900 system, but usually we see dozens of databases hosted per storage system.   We have to strike a balance between providing good performance to individual workloads while also supporting lots of independent workloads in a predictable, manageable way. Without naming names, there are some competitors out there whose products offer impressive performance with a single workload but suffer badly from poor and unpredictable performance with multiple workloads. One database starts stepping on another, different LUNs outcompete others for attention from the storage OS, and things get bad. One of the ways storage systems manage multiple workloads is through threading, where work is divided into queues that can be processed in parallel.   ONTAP 9.9.1 included many improvements to internal threading. In prior versions, SAN IO was essentially being serviced in per-volume queues. Normally, this was not a problem. Controllers would be handling multiple workloads running with a lot of parallelism, all the queues stayed busy all the time, and it was easy for customers to reach platform maximum performance.   Most of my work is in the database space, and we’d often have the One Big Huge Giant Database challenge. I’ve architected systems where a single database ate the maximum capabilities of 12, yes twelve controllers. If you only have one workload, it can be difficult trying to create a configuration that ensures all those threads are busy and processing IO all the time. You had to be careful to avoid having one REALLY busy queue, while others would be idle. The result is leaving potential performance on the table, and you would not get maximum controller performance.   Those concerns are 99% gone as of 9.9.1. There are still threaded operations, of course, but overall, the queues that led to those performance concerns don’t exist anymore. ONTAP services SAN IO more like a general pool of FC operations, spread across all the CPUs all the time.   To illustrate, let’s start with the same set of tests I showed for <9.9.1, with a single volume and varying numbers of LUN in the diskgroup:       I see four important takeaways here:   A single LUN yields about 4X more IOPS than before. A single LUN not only delivers 4X more IOPS, but the latency is also about 40% lower. A single volume (with 8 LUNs) yields about 2X more IOPS A single volume (with 8 LUNs) delivers 2X more IOPS and with 40% lower latency.   That might seem simple, but there are a lot of implications to those four points. Here are some of the things you need to understand.   ONTAP is only part of the picture   The graph above does show that two LUNs are faster than one LUN, but it doesn’t say why. It’s not really ONTAP that is the limiting factor, it’s the SCSI protocol itself. Even if ONTAP was infinitely fast with FC LUNs that delivered 0µs of latency, it can’t service IO it hasn’t received.   You also have to think about the host-side limits. Hosts also have queues, like per-LUN queues, and per-path queues, and HBA queues. You still need some parallelism up at the host level to get maximum performance.   In the tests above, you can see incremental improvements in performance as we bring more LUNs into play. I’m sure some of the benefits are a result of ONTAP parallelizing work better, but that’s only a small part of it. Most of the benefits flow from having more LUNs driven by the OS itself.   The reason I wanted to explain this is because we have a lot of support cases about performance that aren’t exactly complaints, but are instead more like “Why isn’t my database faster than it is?” There’s always a bottleneck somewhere. If there wasn’t, all storage operations would complete in 0 microseconds, database queries would complete in 0 milliseconds, and servers would boot in 0 seconds.   We often discover that whatever the performance bottleneck might be, it ain’t ONTAP. The performance counters show the controller is nowhere near any limits, and in many cases, ONTAP is outright bored. The limit is usually up at the host. In my experience, the #1 cause of SAN performance complaints is an insufficient number of LUNs at the host OS layer. We therefore advise the customer to add more LUNs, so they can increase the parallelism though the host storage stack.   Yes, LUNS simply got faster   A lot of customers had single-LUN workloads that suddenly became a lot faster, because they updated to 9.9.1 or higher. Maybe it was a boot LUN that got faster and now patching is peppier. Maybe there was an application on a single LUN that included an embedded database, and now that application is suddenly a lot more responsive.   A volume of LUNs got faster too   Previously, I maxed out SAN IOPS in a single volume at about 110K IOPS. The limit roughly doubled to 240K IOPS in 9.9.1. That’s a big increase. IO-intensive workloads that previously required multiple volumes can be consolidated to a single volume. That means simpler management. You can create a single snapshot, clone a single volume, set a single QoS policy, or configure a single SnapMirror replication relationship.   Even if you don’t need the extra IOPS, you still get better performance   The latency dropped, too. Even a smaller database that only required 25K IOPS and was happily running on a single volume prior to 9.9.1 should see noticeably improved performance, because the response times of those individual 25K IOPS got better. Application response times get better, queries complete faster, and end users get happier.   How Many Volumes Do I Need?   I’d like to start by saying there is no best practice suggesting the use of one LUN per volume. I don’t know for sure where this idea originated, but I think it came from a very old performance benchmark whitepaper that included a 1:1 LUN:Volume ratio.   As mentioned above, it used to be important to distribute a workload across volumes in some cases, but it mostly only applied to single-workload configuration. If we were setting up a 10-node Oracle RAC cluster, and we wanted to push performance to the limit, and we wanted to get every possible IOP with the lowest possible latency, then we’d need perhaps 16 volumes per controller. There were often only a small number of LUNs on the system as a whole, so we may have used a 1:1 LUN:Volume ratio.   We didn’t HAVE to do that, and it’s in no way a best practice. We often just wanted to squeeze out a few extra percentage points of performance.   Also, don’t forget that there’s no value in unneeded performance. Configure what you need. If you only need 80K IOPS, do yourself a favor and configure a 2-LUN or perhaps 4-LUN diskgroup. It’s not hard to create more LUNs if you need them, but why do that? Why create unnecessary storage objects to manage? Why clutter up the output of commands like “lun show” with extra items that aren’t providing value? I often use the post office as an analogy – a 200MPH vehicle is faster than a 100MPH vehicle, but the neighborhood postal carrier won’t get any benefit from that extra performance.   If you have an unusual management need where one-LUN-per-volume makes more sense, that’s fine, but you have more things to manage, too. Look at the big picture and decide what’s best for you.   Want proof that multiple volumes don’t help? Check this out.       It’s the same line! In this example, I created a 16-LUN volume group and compared performance between configurations where those 16 LUNs were in a single volume, 2 volumes, 4, 8, and 16. There’s literally no difference, nor should there be. As mentioned above, ONTAP SAN processing as of 9.9.1 does not care if the underlying LUNs were located in different volumes. The FC IO was processed as a common pool of FC IO operations.   Things get a little different when you introduce writes, because there still is some queuing behavior related to writes that may be important to you.   Write IO processing   If you have heavy write IO, you might want more than one volume. The graphs below illustrate the basic concepts, but these are synthetic tests. In the real world, especially with databases, you get different patterns of IO interdependencies.   For example, picture a banking database used to support online banking activity by customers. That will be mostly concurrent activity where a little extra latency doesn’t matter. If you need to withdraw money at the ATM, would you care if it took 2.001 seconds rather than the usual 2 seconds?   In contrast, if you have a banking database used for end-of-day processing, you have dependencies. Read #352 might only occur after read #351 has completed. A small increase in latency can have a ripple effect on the overall workload.   The graphs below show what happens when one IO depends on a prior IO and latency increases. It’s also a borderline worst-case scenario.   First, let’s look at a repeat of my first 9.9.1 test, but this time I’m doing 70% reads and 30% writes. What happens?     The maximum measured IOPS dropped. Why? The reason is that writes are more expensive to complete than reads for a storage array. Obviously, platform maximums will be reduced as write IO becomes a larger and larger percentage of the IO, but this is just one volume. I’m nowhere near controller maximums. Performance remains awesome. I’m at about 150us latency for most of the curve, and even at 100K IOPS, I’m only at 300us of latency. That’s great, but it is slower than the 100% read IOPS test.   What you’re seeing is the result of read IOPS getting held back by the write IOPS. There were more IOPS available to my database from this volume, but they weren’t consumed, because my database was waiting on write IO to complete. The result is that the total IOPS dropped quite a bit.   Multi-Volume write IOPS   Here’s what happens when I spread these LUNs across two volumes.        Looks weird, doesn’t it? Why would 2 volumes be 2X as fast as a single volume, and why would 2, 4, 8, and 16 volumes perform about the same?   The reason is that ONTAP is establishing queues for writes. If I want to maximize write IOPS, I’m going to need more queues, which will require more volumes. The exact behavior can change between configurations and platforms, so there’s no true best practice here. I’m just calling out the potential need to spread your database across more than one volume.   Key takeaways:   If I have 16 LUNs, there is literally no benefit to splitting them amongst multiple volumes with a 100% read workload. Look at that earlier graph. The datasets all graphed as a single line. Two volumes with a 70% read workload showed a big improvement going from 1 volume to 2, but then nothing further. That’s because, in my configuration, there are two queues for write processing within ONTAP. Two volumes are no different than 3 or 4 or 5 in terms of keeping those queues busy.   I also want to repeat – the graphs are the worst-case scenario. A real database workload shouldn’t be affected nearly as much, because reads and writes should be largely decoupled from one another. In my test, there are about two reads for each write with limited parallelization, and those reads do not happen until the write completes. That does happen with real-world database workloads, but very rarely. For the most part, real database read operations do not have to wait for writes to complete.   Summary   To recap:   If you’re using ONTAP <9.9.1 with FC SAN, upgrade. We’ve observed LUNs deliver 4X more IOPS at 40% lower latency.   Once you get to ONTAP 9.9.1 (or higher):   A single LUN is good for around 100K IOPS on higher-end controllers. That’s not an ONTAP limit, it’s an “all things considered” limit that is a result of ONTAP limits, host limits, network limits, typical IO sizes, etc. I’ve seen much, much better results in certain configurations, especially ESX. I’m only suggesting 100K as a rule-of-thumb. For a single workload, a 4-LUN volume group on a single volume can hit 200K with no real tuning effort. More LUNs in that volume are desirable in some cases (especially with AIX due to its known host FC behavior), but it’s probably not worth the effort for typical SAN workloads. If you know you’ve got a very, very write-heavy workload, you might want to split your workload into two volumes. If you’re that concerned about IOPS, you probably did that anyway, simply because you probably chose to distribute your LUNs across controllers. That’s a common practice – split each workload evenly across all controllers to achieve maximum performance, as well as guaranteed even loading across the entire cluster.   Lastly, don’t lose perspective.   It’s nice to have an AFF system with huge IOPS capabilities for the sake of consolidating lots of workloads, but I find admins obsess too much about individual workloads and targeting hypothetical performance levels that offer no real benefits.   I look at a lot of performance stats, and virtually every application and database workload I see plainly shows no storage performance bottleneck whatsoever. The performance limits are almost universally the SQL code, the application code, available raw network bandwidth, or Oracle RAC cluster contention. Storage is usually less than 5% of the problem. The spinning-disk days of spending your way out of performance problems are over.   Storage performance sizing should be about determining actual performance requirements, and then architecting the simplest, most manageable solution possible. The SAN improvements introduced in ONTAP 9.9.1 noticeably improve manageability as well as performance.     ... View more

Consistency Groups in ONTAP   There’s a good reason you should care about CGs – it’s about manageability.   If you have an important application like a database, it probably involves multiple LUNs or multiple filesystems. How do you want to manage this data? Do you want to manage 20 LUNs on an individual basis, or would you prefer just to manage the dataset as a single unit?   This post is part 1 of a 2. First, I will explain what we mean when we talk about consistency groups (CGs) within ONTAP.   Part II covers the performance aspect of consistency groups, including real numbers on how your volume and LUN layout affects (and does not affect) performance. It will also answer the universal database storage question, “How many LUNs do I need?” Part II will be of particular interest to long-time NetApp users who might still be adhering to out-of-date best practices surrounding performance.   Volumes vs LUNs   If you’re relatively new to NetApp, there’s a key concept worth emphasizing – volumes are not LUNs.   Other vendors use those two terms synonymously. We don’t. A Flexible Volume, also known as a FlexVol, or usually just a “volume,” is just a management container. It’s not a LUN. You put data, including NFS/SMB files, LUNs, and even S3 objects, inside of a volume. Yes, it does have attributes such as size, but that’s really just accounting. For example, if you create a 1TB volume, you’ve set an upper limit on whatever data you choose to put inside that volume, but you haven’t actually allocated space on the drives.   This sometimes leads to confusion. When we talk about creating 5 volumes, we don’t mean 5 LUNs. Sometimes customers think that they create one volume and then one LUN within that volume. You can certainly do that if you want, but there’s no requirement for a 1:1 mapping of volume to LUN. The result of this confusion is that we sometimes see administrators and architects designing unnecessarily complicated storage layouts. A volume is not a LUN.   Okay then, what is a volume?   If you go back about eighteen years, an ONTAP volume mapped to specific drives in a storage controller, but that’s ancient history now.   Today, volumes are there mostly for your administrative convenience. For example, if you have a database with a set of 10 LUNs, and you want to limit the performance for the database using a specific quality of service (QoS) policy, you can place those 10 LUNs in a single volume and slap that QoS policy on the volume. No need to do math to figure out per-LUN QoS limits. No need to apply QoS policies to each LUN individually. You could choose to do that, but if you want the database to have a 100K IOPS QoS limit, why not just apply the QoS limit to the volume itself? Then you can create whatever number of LUNs that are required for the workload.   Volume-level management   Volumes are also related to fundamental ONTAP operations, such as snapshots, cloning, and replication. You don’t selectively decide which LUN to snapshot or replicate, you just place those LUNs into a single volume and create a snapshot of the volume, or you set a replication policy for the volume. You’re managing volumes, irrespective of what data is in those volumes.   It also simplifies how you expand the storage footprint of an application. For example, if you add LUNs to that application in the future, just create the new LUNs within the same volume. They will automatically be included in the next replication update, the snapshot schedule will apply to all the LUNs, including the new ones, and the volume-level QoS policy will now apply to IO on all the LUNs, including the new ones.   You can selectively clone individual LUNs if you like, but most cloning workflows operate on datasets, not individual LUNs. If you have an LVM with 20 LUNs, wouldn’t you rather just clone them as a single unit than perform 20 individual cloning operations? Why not put the 20 LUNs in a single volume and then clone the whole volume in a single step?   Conceptually, this makes ONTAP more complicated, because you need to understand that volume abstraction layer, but if you look at real-world needs, volumes make life easier. ONTAP customers don’t buy arrays for just a single LUN, they use them for multiple workloads with LUN counts going into the 10’s of thousands.   There’s also another important term for a “volume” that you don’t often hear from NetApp. The term is “consistency group,” and you need to understand it if you want maximum manageability of your data.   What’s a Consistency Group?    In the storage world, a consistency group (CG) refers to the management of multiple storage objects as a single unit. For example, if you have a database, you might provision 8 LUNs, configure it as a single logical volume, and create the database. (The term CG is most often used when discussing SAN architectures, but it can apply to files as well.)   What if you want to use array-level replication to protect that database? You can’t just set up 8 individual LUN replication relationships. That won’t work, because the replicated data won’t be internally consistent across volumes. You need to ensure that all 8 replicas of the source LUN are consistent with one another, or the database will be corrupt.   This is only one aspect of CG data management. CGs are implemented in ONTAP in multiple ways. This shouldn’t be surprising – an ONTAP system can do a lot of different things. The need to manage datasets in a consistent manner requires different approaches depending on the chosen NetApp storage system architecture and which ONTAP feature we’re talking about.   Consistency Groups – ONTAP Volumes   The most basic consistency group is a volume. A volume hosting multiple LUNs is intrinsically a consistency group. I can’t tell you how many times I’ve had to explain this important concept to customers as well as NetApp colleagues simply because we’ve historically never used the term “consistency group.”   Here’s why a volume is a consistency group:   If you have a dataset and you put the dataset components (LUNs or files) into a single ONTAP volume, you can then create snapshots and clones, perform restorations, and replicate the data in that volume as a single consistent unit. A volume is a consistency group. I wish we could update every reference to volumes across all the ONTAP documentation in order to explain this concept, because if you understand it, it dramatically simplifies storage management.    Now, there are times where you can’t put the entire dataset in a single volume. For example, most databases use at least two volumes, one for datafiles and one for logs. You need to be able to restore the datafiles to an earlier point in time without affecting the logs. You might need some of that log data to roll the database forward to the desired point in time. Furthermore, the retention times for datafile backups might differ from log backups.   We have a solution for that, too, but first let’s talk about MetroCluster.   Consistency Groups & MetroCluster   While regular ol’ ONTAP volumes are indeed consistency groups, they’re not the only implementation of CGs in ONTAP. The need for data consistency appears in many forms. SyncMirrored aggregates are another type of CG that applies to MetroCluster.   MetroCluster is a screaming fast architecture, providing RPO=0 synchronous mirroring, mostly used for large-scale replication projects. If you have a single dataset that needs to be replicated to another site, MetroCluster probably isn’t the right choice. There would probably be simpler options.   If, however, you’re building an RPO=0 data center infrastructure, MetroCluster is awesome, because you’re essentially doing RPO=0 at the storage system layer. Since we’re replicating everything, we can do replication at the lowest level – right down at the RAID layer. The storage system doesn’t know or care about where changes are coming from, it just replicates each little write-to-drives to two different locations. It’s very streamlined, which means it’s faster and makes failovers easier to execute and manage, because you’re failing over “the storage system” in its entirety, not individual LUNs.   Here's a question, though. What if I have 20 interdependent applications and databases and datasets? If a backhoe cuts the connection between sites, is all that data at the remote site still consistent and usable? I don’t want one database to be ahead in time from another. I need all the data to be consistent.   As mentioned before, the individual volumes are all CGs unto themselves, but there’s another layer of CG, too – the SyncMirror aggregate itself. All the data on a single replicated MetroCluster aggregate makes up a CG. The constituent volumes are consistent with one another. That’s a key requirement to ensure that some of the disaster edge cases, such as rolling disasters, still yield a surviving site that has usable, consistent data and can be used for rapid data center failover. In other words, a MetroCluster aggregate is a consistency group, with respect to all the data on that aggregate, which guarantees data consistency in the event of sudden site loss.   Consistency Groups & API’s    Let’s go back to the idea of a volume as a consistency group. It works well for many situations, but what if you need to place your data in more than one volume? For example, what if you have four ONTAP controllers and want to load up all of them evenly with IO? You’ll have four volumes. You need consistent management of all four volumes.   We can handle that, too. We have yet another  consistency group capability that we implement at the API level. We did this about 20 years ago, originally for Oracle ASM diskgroups. Those were the days of spinning drives, and we had some customers with huge Oracle databases that were both capacity-hungry and IOPS-hungry to the point they required multiple storage systems.   How do you get a snapshot of a set of 1000 LUNs spread across 12 different storage systems? The answer is “quite easily,” and this was literally my second project as a NetApp employee. You use our consistency group API’s. Specifically, you’d make an API call for “cg-start” targeting all volumes across various systems, then call “cg-commit” on all those storage systems. If all those cg-commit API calls report a success, you know you have a consistent set of snapshots that could be used for cloning, replication, or restoration.   You can do this with a few lines of scripting, and we have multiple management products, including SnapCenter, that make use of those APIs to perform data consistent operations.    These APIs are also part of the reason everyone, including NetApp personnel, often forget that an ONTAP volume is a consistency group. We had those APIs that had the letters “CG” in them, and everyone subconsciously started to think that this must be the ONLY way to work with consistency groups within ONTAP. That’s incorrect; the cg-start/cg-commit API calls are merely one way ONTAP delivers consistency group-based management.   Consistency Groups & SM-BC    SnapMirror Business Continuity (SM-BC) is similar to MetroCluster but provides more granularity. MetroCluster is probably the best solution if you need to replicate all or nearly all the data on your storage system, but sometimes you only want to replicate a small subset of total data.   SM-BC almost didn’t need to support any sort of “consistency group” feature. We could have scoped that feature to just single volumes. Each individual volume could have been replicated and able to be failed over as a single entity.   However, what if you needed a business continuity plan for three databases, one application server, and all four boot LUNs? Sure, you might be able to put all that data into a single volume, but it’s likely that your overall data protection, performance, monitoring, and management needs would require the use of more than one volume.   Here’s how that affects data consistency with SM-BC.  Say you’ve provisioned four volumes. The key is that a business continuity plan requires all 4 of those volumes entering and exiting a consistent replication state as a single unit.   We don’t want to have a situation where the storage system is recovering from an interruption in site-to-site connectivity with one volume in an RPO=0 state, while the other three volumes are still synchronizing. A failure at that moment would leave you with mismatched volumes at the destination site. One of them would be later in time than others. That’s why we base your SM-BC relationships on CGs. ONTAP ensures those included volumes enter and exit an RPO=0 state as a single unit.   Native ONTAP Consistency Groups   Finally, ONTAP also allows you to configure advanced consistency groups within ONTAP itself. The results are similar to what you’d get with the API calls I mentioned above, except now you don’t have to install extra software like SnapCenter or write a script.   Here’s an example of how you might use ONTAP Consistency Groups:       In this example, I have an Oracle database with datafiles distributed across 4 volumes located on 4 different controllers. I often do that to ensure my IO load is guaranteed to be evenly distributed across all controllers in the entire cluster. I also have my logs in 3 different volumes, plus I have a volume for my Oracle binaries.   The point of the ONTAP Consistency Group feature is to enable users to manage applications and application components, and not worry about LUNs and individual volumes. Once I add this CG (which is composed of two child CGs), I can do things like schedule snapshots for the application itself. The result is a CG snapshot of the entire application. I can now use those snapshots for cloning, restoration or replication.   I can also work at a more granular level. For example, I could do a traditional Oracle hot backup procedure as follows:    “alter database begin backup;”  POST /application/consistency-groups/(Datafiles)/snapshots “alter database end backup;” “alter database archive log current;” POST /application/consistency-groups/(Logs)/snapshots   The result of that is a set of volume snapshots, one of the datafiles and one of the logs, which are recoverable using a standard Oracle backup procedure.   Specifically, the datafiles were in backup mode when a snapshot of the first CG was taken. That’s the starting point for a restoration. I then removed the database from backup mode and forced a log switch before making the API call to create a snapshot of the log CG. The snapshot of the log CG now contains the required logs for making that datafile snapshot consistent.   (Note: You don’t really have to place an Oracle database in backup mode since 12cR1, but most DBA’s are more comfortable with that additional step)   Those two sets of snapshots constitute a restorable, clonable, usable backup. I’m not operating on LUNs or filesystems; I’m making API calls against CGs. It’s application-centric management. There’s no need to change my automation strategy as the application evolves over time and I add new volumes or LUNs, because I’m just operating on named CGs.  It even works the same with SAN and file-based storage.   We’ve got all sorts of idea of how to keep expanding this vision of application-centric storage management, so keep checking in with us. ... View more