Cassandra on FSxN (NetApp-ONTAP) for Enterprise Workloads
- What is ONTAP
- What is FSxN
- Cassandra
- Cassandra Installations & Internals
- Benchmarking EBS vs FSxN
- Cost saving details EBS vs FSxN
- Conclusion
What is ONTAP
NetApp's ONTAP is a robust shared storage solution, renowned for its advanced features and capabilities. It facilitates high-speed data access across all nodes, ensuring optimal performance and efficiency.
ONTAP's versatility is highlighted by its support for multiple protocols, enabling both file and block sharing. This flexibility allows it to cater to a variety of storage needs and use cases.
The software excels in data management, offering sophisticated tools for data storage, organization, and access. These features streamline the management process, making it easier to control and manipulate data.
ONTAP also provides comprehensive data migration management tools. These tools simplify the process of moving data, reducing the time and effort required.
In terms of data protection, ONTAP offers a robust system management solution. This includes features designed to safeguard data, ensuring it remains secure and intact.
Finally, ONTAP's Auto Support feature enhances its management capabilities. This automated system provides proactive support, helping to identify and resolve issues before they can impact operations.
In summary, NetApp's ONTAP is a comprehensive shared storage solution, offering a wide range of advanced features designed to optimize data access, management, and protection.
Further links for more knowledge: https://library.netapp.com/ecmdocs/ECMP1368862/html/GUID-C6A665D4-EB47-485E-B2C9-29405DEE4510.html
What is FSxN
Amazon FSx for NetApp ONTAP is a sophisticated storage service offered by AWS. It allows users to seamlessly create and manage NetApp ONTAP file systems within the AWS Cloud environment. This service combines the robust features, high performance, versatile capabilities, and familiar APIs (application programming interfaces) of NetApp file systems with the dynamic scalability, operational agility, and streamlined simplicity of a fully managed AWS service.
Two key features of Amazon FSx for NetApp ONTAP that significantly contribute to cost reduction are:
- Elastic Capacity Pool Tiering: This intelligent feature enables automatic migration of infrequently accessed data to more cost-effective storage tiers. It is an efficient way to optimize storage costs, particularly when dealing with data that does not require frequent or rapid access.
- Compression/Deduplication: This feature optimizes storage utilization by eliminating redundant data (through deduplication) and reducing the size of the data (through compression). This can lead to substantial cost savings, especially in scenarios where there is a high degree of data redundancy, or the data is highly compressible.
These features offer a strategic advantage for businesses or individuals aiming to optimize their cloud storage expenditure. However, it is important to note that the actual cost savings will be influenced by specific use cases and the nature of the data stored. Do you have any specific queries or require further details about Amazon FSx for NetApp ONTAP? Please refer further links for more knowledge: https://aws.amazon.com/fsx/netapp-ontap/
Cassandra
Cassandra is a free, open-source, distributed, wide-column store, NoSQL database management system. It is designed to manage large volumes of data across numerous commodity servers, ensuring high availability without a single point of failure. In its peer-to-peer architecture, all nodes share equal responsibility.
Cassandra is an AP system from the CAP (Consistency, Availability, & Partition Tolerance) theorem principle, which means it prioritizes availability and partition tolerance over consistency. Here is what that means:
- Consistency: Ideally, all nodes or servers in the cluster should have identical data. This ensures that any query to the system will return the same copy of data, regardless of which node or server responds to the request.
- Availability: The system should always respond to a request, even if the data returned is not the most recent or immediately consistent across the system.
- Partition Tolerance: The system should continue to function even if some nodes fail or become unreachable for any reason.
A brief history of Cassandra:
- Apache Cassandra was initially developed at Facebook to power their Inbox Search feature. The developers were Avinash Lakshman, one of the authors of Amazon's Dynamo, and Prashant Malik.
- The project was released as open source on Google code in July 2008.
- In March 2009, it became an Apache Incubator project and graduated to a top-level project on February 17, 2010.
- As of October 27, 2023, the current stable release is version 4.1.3
Cassandra Installations & Internals
BareMetal: For the most recent release of Cassandra, please always refer to the official download page at the following URL: https://cassandra.apache.org/_/download.html. This will ensure you have access to the latest and most stable version of the software.
Once you have selected and downloaded the appropriate distribution, you may need guidance for the installation process. For the most up-to-date installation instructions, please visit the official Cassandra documentation at https://cassandra.apache.org/doc/stable/cassandra/getting_started/installing.html
FSxN (ONTAP) details at https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/how-it-works-fsx-ontap.html
Kubernetes: For implementing stateful applications in Kubernetes, and for using the FSxN-ONTAP filesystem as a storage class to claim Persistent Volume Claims (PVCs) to Persistent Volumes (PVs) on FSxN-ONTAP file system storage, please refer to the following resources: These resources will provide you with comprehensive guidance on the topics.
- Kubernetes' guide on stateful applications: https://kubernetes.io/docs/tutorials/stateful-application/cassandra/
- Documentation on Trident plug-ins and operators: https://docs.netapp.com/us-en/trident/index.html
These resources will provide you with the necessary steps to successfully install and run Cassandra on FSxN (ONTAP). Please do not hesitate to reach out if you encounter any issues or have further questions at ng-netappdataplatform-team@netapp.com
Cassandra Replication Strategies:
Cassandra ensures high availability and fault tolerance by replicating data across multiple nodes in a cluster. The replication strategy dictates the placement of these replicas within the cluster. Each node in the cluster is responsible for the data within its assigned token range and serves as a replica for a different data range. If a node fails, its replica can handle queries for that data range.
The Replication Factor (RF) indicates the number of copies of each row. RF=1 means a single copy of each row exists on one node, while RF=2 means two copies of each row exist, each on a different node. All replicas hold equal importance; there is no primary or master replica. In production environments, an RF of 3 is typically preferred. However, the RF should never exceed the number of nodes in the cluster.
Cassandra offers two replication strategies:
- SimpleStrategy: Suitable for a single data center and one rack. If you plan to expand beyond one data center, use NetworkTopologyStrategy.
- NetworkTopologyStrategy: Recommended for most deployments as it facilitates expansion to multiple data centers when necessary.
The DURABLE_WRITES option controls whether data is written into the commit log. By default, it is set to TRUE. Setting it to FALSE bypasses the commit log when writing data to the keyspace, which can risk data loss in case of crashes or failures. It is generally recommended to leave this option as TRUE.
Each keyspace has its own replication strategy, which must be defined at the time of keyspace creation.
Cassandra's Quorum and Consistency Level (CL) details:
Quorum is a concept that refers to the minimum number of members required to conduct business.
In Apache Cassandra, the QUORUM consistency level dictates that a write operation must be logged in the commit log and memtable of a quorum of replica nodes spanning all data centers. This operation only returns the record once a quorum of replicas, from all data centers involved, has acknowledged the write. The calculation for determining the quorum is given by the formula (sum of replication factors from all data centers / 2) + 1. This means that more than half of the replica nodes across all data centers need to confirm the write operation for it to be considered successful. This approach ensures a balance between data consistency and availability, as it requires acknowledgment from a majority of nodes but not all.
LOCAL_QUORUM is calculated based on the replication factor of the same data center as the coordinator node. This means that even if the cluster spans multiple data centers, the quorum is calculated only with local replica nodes. This level ensures strong consistency and avoids the latency of inter-datacenter communication.
EACH_QUORUM is calculated based on the replication factor of each data center as the coordinator node. This means that if the cluster spans multiple data centers, the quorum is calculated with all data center replica nodes. This level ensures strong consistency and requires a quorum of replica nodes in each data center to respond to the coordinator node for the read or write request to succeed. However, it is not supported for read operations.
The reliability of read and write operations depends on the consistency level used to verify each operation. Here, R represents the consistency level of read operations, W represents the consistency level of write operations, and N represents the number of replicas (RF). Strong consistency can be guaranteed when R + W > N, while eventual consistency occurs if R + W <= N.
Coordinators in Cassandra are nodes that manage client requests. They receive a request, hash the partition key (or row key) to a token, and forward the request to the corresponding node in the cluster. Every node can serve as a coordinator. The choice of coordinator is typically determined by an algorithm considering 'network distance', optimizing for efficient communication. When data is inserted, the partition key's hash value dictates the node to which the data row is sent.
The consistency level determines how many replicas must confirm the success of the read or write operation. The choice of consistency level can affect the durability, consistency, and availability of the system. Here's simple example along with CL numbers:
Assuming a cluster has 3 data centers, each with a Replication Factor (RF) of 3, the following consistency levels apply:
- ALL: Requires all 9 replicas to confirm for the write operation to succeed. If any node is down, the write operation fails.
- EACH_QUORUM: Requires at least 2 nodes in each data center to confirm for the write operation to succeed. This means 6 nodes must be confirmed.
- QUORUM: Requires confirmation from 5 nodes from any data center for the write operation to succeed.
- LOCAL_QUORUM: Requires confirmation from 2 nodes from the local data center for the write operation to succeed.
- ONE: Requires confirmation from 1 node from any data center for the write operation to succeed.
- TWO: Requires confirmation from 2 nodes from any data center for the write operation to succeed.
- THREE: Requires confirmation from 3 nodes from any data center for the write operation to succeed.
- LOCAL_ONE: Requires confirmation from 1 node from the local data center for the write operation to succeed.
- ANY: Only needs to successfully write a hinted handoff, which is typically a mechanism used to ensure a write is retried by the database. This consistency level is not recommended or preferred due to its potential for data inconsistency.
Remember, the choice of consistency level depends on your application's specific requirements for data consistency, availability, and partition tolerance. Furthermore, READ & WRITE Consistency Level details at https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlConfigConsistency.html
The number of nodes is less important than the RF, which determines how many nodes have a copy of the data. Strong consistency, also known as immediate consistency, is a key feature of Cassandra. It offers the flexibility to choose tunable data consistency for each query or operation, supporting various consistency levels for both write and read operations as per your use case demands.
Benchmarking EBS vs FSxN
In our evaluation of Amazon EBS GP3 volumes and FSxN ONTAP volumes, we found no significant performance difference. However, for large-scale, long-term data storage in Cassandra, FSxN-ONTAP emerged as a superior option. This is due to ONTAP's robust features, including high availability (HA), security, reliability, tiering, compaction/compression, and deduplication, among others.
Based on our benchmark evaluations and observations, FSxN-ONTAP storage can result in significant cost savings for enterprise-level production Cassandra clusters. For a detailed breakdown of these savings, please refer to our cost details section. Our tests involved 10 million records for WRITE and READ operations, and 1 million records for MIXED (Read & Write) operations. We used the Cassandra-stress native tool for benchmarking, with the following configuration and version details:
- Tests were conducted on both EBS and FSxN-ONTAP.
- The Consistency Level (CL) was set to QUORUM.
- The Keyspace Replication Factor (RF) was set to 3.
- Both Simple and NetworkTopology strategies were used.
We believe that these findings will provide valuable insights into your decision-making process.
The benchmarking results for Write(10Million), Read(10Million), and Mixed(1M) operations are presented in the following graph. Upon close examination, it is evident that the "mean" and "median" latencies for both EBS and FSxN-ONTAP are nearly identical. Moreover, the other latency metrics are also closely aligned, except for the maximum latency.
This discrepancy in maximum latency is expected, as it is common for a small fraction of transactions (one in thousands or millions) to experience delays or reach their maximum limit, prompting the client to retry these specific transactions. However, our primary concern should be the "mean" and "99th percentile" latencies. Any significant deviation in these metrics would warrant further investigation and potentially indicate performance issues. As per our evaluation and observation there is NO significant difference of these key metrics on EBS(GP3) vs FSxN (ONTAP).
Cost saving details EBS vs FSxN
The benchmarking environment under discussion is a 5-node cluster, with the cost details provided below. It is important to note that expanding your cluster can result in cost savings, especially for real-world applications such as production systems.
If you have some experience with Cassandra, you know that a Cassandra cluster with 10 or fewer nodes is typically used for non-production purposes, such as development or quick proof-of-concept projects. Medium to large scale production clusters, depending on your use case, should ideally have a ring of 20, 30, 40, 60, or 10X nodes in a data center (DC), or multiple rings in different DCs. This setup allows for horizontal scalability, high availability, and parallelism.
However, as you increase the number of nodes in your cluster, the cost of Elastic Block Store (EBS) will also rise. This is because EBS disks require a one-to-one mapping with compute nodes. On the other hand, FSxN-ONTAP is a shared filesystem that can be used across all compute nodes, resulting in a more stable cost, except for compute cost.
For instance, when your cluster size reaches 20 nodes, you will notice that the cost of EBS surpasses that of FSxN-ONTAP. It is worth noting that a 20-node Cassandra production cluster is considered small scale. Therefore, consider the implications for larger and enterprise clusters where the Cassandra cluster size can reach hundreds of nodes.
Another significant advantage of FSxN-ONTAP is that it is a shared filesystem that remains available indefinitely and can be attached to any number of new nodes at any time. Unlike EBS disks, which require a one-to-one mapping and are tightly coupled with compute nodes.
For reference on cost, please visit the following links, and below chart details for your quick reference on cost savings
Conclusion
- Utilizing Cassandra on FSxN (ONTAP) presents a highly advantageous strategy for medium to large-scale enterprise use cases. This method not only ensures data safety and high availability but also offers additional data management capabilities inherent to ONTAP.
- Significant cost savings become evident when the cluster size surpasses 20 nodes. Those experienced with production Cassandra cluster sizes and data modeling will recognize the potential for savings in this scenario. Even small-scale production workloads could have at least 20 nodes in a ring, indicating the potential for cost efficiency.
- A key benefit of FSxN (ONTAP) is the elimination of data movement from one node to another in the event of node failure. Unlike EBS, which uses dedicated storage tightly coupled with compute nodes, FSxN (ONTAP) employs shared storage. This feature minimizes downtime and bolsters overall system resilience.
- With FSxN (ONTAP), there is no compromise on performance. It offers flexible options for storage and network to meet your specific needs, ensuring optimal system operation.
