In the ever-changing landscape of software development, we recognized a need to evolve the architecture of NetApp Astra™. The main impetuses for this evolution were to better align with modern development practices and to meet growing customer demand for ecosystem automation and Kubernetes-native tooling integrations. We saw an opportunity to integrate more seamlessly with the broad ecosystem of tools and platforms that our customers use.
Options and constraints
Architectural changes often bring a sense of freedom and creativity. Given the opportunity to reshape the structure of our system, we are eager to apply the latest technologies, insights from recent books, and ideas from the latest blogs. Paradoxically, too much freedom can sometimes stifle creativity rather than drive it, leading to paralysis. Constraints, however, can lead to focused, creative solutions.
As we considered the evolution of the Astra architecture, we faced many options, along with a few built-in constraints. We needed to maintain the functionality that our customers expect. We already had a working product, so the feature set was well defined. The solution had to be scalable, secure, and flexible enough to adapt to changing requirements and new problem spaces. Rather than inventing a new way of communicating and running operations, we leveraged Kubernetes and its controller model to define the interface (custom resources [CRs]), the overall lifecycle of the interaction (reconcile loops), and the storage mechanism (etcd with CRs).
Even after creating these constraints, we still faced many decisions. Efficiency was a key consideration, not just in the processes themselves, but also in coordination and communication of both the code and the developers. Our aim was to maintain low coupling between the controllers.
We diagrammed the application data management operations needed, based on what the Astra APIs required. We then broke down these operations into lower-level components, with certain components chosen as candidates for reuse and made into CRs. Understanding the types of operations needed was crucial. Starting from that point, we developed a deeper understanding of what a snapshot is compared to a backup, for example. We used diagrams to visualize the building blocks. Also, we had to make decisions about when to make something its own controller, and when to create branching logic within a controller.
Custom resources as building blocks
One of the most powerful aspects of the new architecture is the use of lower-level custom resources, or primitives, as building blocks for our system. These primitives represent the system’s fundamental operations. By defining operations as reusable CRs, we created a flexible foundation upon which we can quickly build and deploy new features.
This approach has several advantages. First, it allows us to develop new features in a modular way by combining and reusing primitives. This method speeds up development time and ensures consistency across different features, because they are all built from the same basic components. Second, it makes the system more adaptable to changing requirements. As we encounter new use cases or challenges, we can create new combinations of primitives to meet these needs, without having to redesign or reimplement our core operations. Finally, it promotes a deeper understanding of the system by our development team. By working with these lower-level primitives, developers gain a detailed knowledge of the system's operations and capabilities, which can lead to more innovative solutions and improvements.
Evolving higher-level workflows
From the lower-level primitives, Astra also constructs higher-level workflows. These workflows represent sequences of operations that achieve a specific goal, such as cloning an application. By building these workflows on top of our primitives, we can create complex processes that are still easy to understand and manage. Each step in the workflow corresponds to a primitive operation, so we can trust that the primitives are well tested and that they behave as expected. By combining primitives in different ways, we can tailor our workflows to the exact requirements of a task, providing a high degree of flexibility and adaptability.
These higher-level workflows represent another layer of abstraction in our system. They allow us to harness the power and flexibility of our lower-level primitives, while providing a simple, intuitive interface for managing complex processes. This balance between simplicity and power is a key aspect of our architectural approach with Astra.
We’re excited to leverage these primitives to rapidly develop and deploy new features for Astra. We believe that this approach will allow us to better serve our customers by quickly adapting to their needs and delivering reliable, high-quality features. In the journey of software development, the balance between freedom and constraints is a delicate one. Embracing constraints, as we did in the evolution of the Astra architecture, can result in focused, creative solutions.
Learning, adapting, and improving are a journey
As we move forward, we know that some of these design choices will be refactored and changed. That is part of the process and the beauty of software development. It's a constant journey of learning, adapting, and improving. But one thing remains clear: Embracing constraints does not limit our creativity, it drives it. It gives us a focused lens through which we can examine problems and deliver solutions that meet the needs of our customers and the demands of an ever-evolving technological landscape.
In the end, our journey with the Astra architecture is a testament to the power of constraints in driving innovation and efficiency in software development. We are proud to share this story, and we hope that it inspires others to see constraints not as limitations, but as opportunities for creative problem solving and growth.
