Modern application architectures have been using event streaming for many years now. The open-source Apache Kafka project, and the commercially supported Confluent platform, are undeniable leaders in this space. Does an enterprise-grade storage provider such as NetApp have any value to add to these types of shared-nothing architectures? Let's take a look!
Why Centralized Storage for Kafka?
I will begin with the "why". Given the distributed, replicated, and fault-tolerant nature of Kafka data, local disk is certainly a valid storage choice. This does bring some challenges to be aware of, however, especially when running Kafka at a large scale. To name a few:
- Capacity management - Can you accurately predict how much disk space will be required for your Kafka partitions on each broker? You may end up with stranded capacity, or not enough capacity.
- Resource scaling - Do you want to add both compute and storage each time you are short on one of them? Would independent scaling be advantageous?
- Performance issues - Node failures or disk failures will always result in additional CPU and network load as data is being rebuilt if all of your data is stored on local disk.
Most of this looks very familiar to anyone who has considered the pros and cons of managing centralized storage versus local storage. In addition, the performance overhead during recovery after HW failure in shared-nothing architectures is a common concern.
Now that we have identified some of the pain points with local storage, I will consider how centralized storage from NetApp can solve these issues. You can use the typical block-based storage protocols supported by NetApp (FCP, iSCSI, NVMe/FC, NVMe/TCP) to continue with local disk semantics, but get the capacity management benefits of centralized storage. All of these will still suffer from overhead during a Kafka broker failure, however. With that in mind, I will walk through the options that we have with NFS and NetApp S3 compatible object storage solutions.
Kafka on NFS Testing
Recently, NetApp announced NFSv4 enhancements in both ONTAP and the Linux kernel to support running Kafka over NFS, as described in these blogs:
- Simplify Apache Kafka operations with Confluent Platform and NetApp ONTAP
- ONTAP is ready for streaming applications
I took some time to test this out, including the reassignment of partitions between NFSv4.1 mounted Kafka log directories. This is the command I used to move my partition between NFS mounts /kafka1 and /kafka2:
# cat reassignment-file.json
{"version":1,"partitions":[{"topic":"myevents1","partition":0,"replicas":[0],"log_dirs":["/kafka2/logs"]}]}
# bin/kafka-reassign-partitions.sh -reassignment-json-file reassignment-file.json --bootstrap-server localhost:9092 --execute
Here is the error I saw in the Kafka logs as I moved a Kafka partition between NFS mountpoints without the new configuration requirements in place.
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting dir for myevents1-0 in dir /kafka1/logs
Caused by: java.nio.file.FileSystemException: /kafka1/logs/myevents1-0.b5fce3d70ee0489bb21adcfc74347310-delete/.nfs000000000000007400000008: Device or resource busy
Running the same reassignment commands with the fixes described below in place results in a clean and successful partition movement as shown in the Kafka logs below:
[2023-02-10 20:49:54,795] INFO Deleted log for partition myevents1-0 in /kafka1/logs/myevents1-0.e3d8ff1f3d464cba89ba653e8cae1ec5-delete. (kafka.log.LogManager)
Kafka on NFS Requirements
Here are the requirements to make Kafka work reliably over NFS:
- ONTAP 9.12.1 or higher with the is-preserve-unlink-enabled volume option set to true
- At least RHEL 8.7 or RHEL 9.1, and the NFS client mounting with NFSv4.1 or NFSv4.2
- Kafka data must go into a directory or qtree within the volume, not the root of the volume, due to ONTAP bug 1513528
I know some will discount NFS due to a bad experience with performance in the past. It is 2023 folks, NetApp All-Flash storage systems can deliver many GB/s of throughput with sub-1ms latencies when running NFS over 40Gb or 100GB ethernet!
Confluent Storage Tiering and NetApp S3 Solutions
I will also point out that Confluent has something that Apache Kafka does not - support for tiered storage. Confluent Kafka allows topics to move older events to a S3 compatible object storage tier, as described in the Confluent Tiered Storage documentation. This helps address both capacity management and performance concerns during broker failures, and NetApp has support for this with both ONTAP S3 and StorageGrid. Oh, and by the way, performance in this configuration can be great as well, as demonstrated in this blog about Extreme S3 Performance with Confluent Tiered Storage and ONTAP.
Containerized Kafka
To wrap up, I will also remind everyone that NetApp has great support for containerized applications, including Kafka. Note that support for the ONTAP NAS backends used via Trident will need to catch up to the NFS requirements for Kafka that were documented earlier in this post. If you have an interest in running containerized Kafka with NFS connected PVCs, please voice your requirements in this Trident issue.
