Spinning up new database environments is essential to supporting your organization and streamlining development processes. Database copies are needed for development and QA, testing, system updates, training and education. But the vital task of spinning up database copies comes with a range of challenges: impact on the performance of the production system, high storage costs, lengthy provisioning, and inconsistent data.
Amazon FSx for NetApp ONTAP (FSx for ONTAP) allows you to shorten the time needed to spin up writable database copy from hours to minutes and reduce storage space and costs, all with virtually no impact on the production environment.
In this post, we go into depth about the challenges in creating database copies and show how FSx for ONTAP can help you overcome them.
Read on as we cover:
- The challenges of creating new database environments
- Spinning up database copies with FSx for ONTAP
- How does it work?
- Use NetApp BlueXP workload factory to simplify operations
- The bottom line: Speed up development, cut costs
The challenges of creating new database environments
Database copies are essential to business processes but creating them can be challenging. Let’s take a deeper look at the main challenges.
Overhead on production system
In most cases, database spin-up is performed with a backup and restore process. This is a slow method and utilizes tools and processes in use by the live production environment. Furthermore, traditional backup and restore methodologies require heavy computational and network resources and can negatively impact the performance of the main database or relevant applications. Solving this by adding more compute power is costly, in most cases.
Storage costs
With traditional backup and restore methods, every new environment requires a full copy of the data. For example, if you need to support 500 ancillary environments, you will need 500 copies of the data, and this means 500 times the storage cost.
Delays
Creating a full copy of a large database can take a considerable amount of time, sometimes many days. This impacts productivity and delays crucial processes. For example, delays in testing and providing feedback to dev can slow the product cycle and ultimately hurt time to market.
Data consistency
Maintaining consistency between the original database and its copies can be complex. Changes made to the main database need to be replicated to the copies efficiently to avoid data discrepancies.
Spinning up database copies with FSx for ONTAP
FSx for ONTAP and its thin cloning method allow you to create copies of your production database in minutes rather than hours or days, in a cost- and space-efficient manner. This means that teams that need database copies get them almost instantly and can maintain high levels of productivity while storage costs are kept to a minimum. Furthermore, the database clones are fully writable so that the teams using them can replicate whatever production scenarios are needed.
No hit to performance
Database spin-up with FSx for ONTAP has virtually no impact on the performance of the production environment. This is because rather than copying the actual data and adding a load on the storage layer, FSx for ONTAP uses the NetApp® Snapshot™ technology which only copies metadata. This means that the stress put on the production system, even for large-scale databases, is almost negligible.
Provide development team with quick access with instantaneous database environment creation
Database copies are created in just minutes, rather than hours, or, in some cases, days. This means that no extra time is added onto your CI/CD pipeline, and time to market for rollout of new features and new versions is not affected by spin-up. Furthermore, the teams waiting for database copies get what they need immediately and take no hit to their productivity.
Although clones act as an independent, writable environment, they rely on the source clone, and therefore, provide a consistent, point-in-time view across the environments.
Reduce costs by adding minimal storage capacity
Instead of adding full data capacity for each additional environment, which would require alternative storage services, when using FSx for ONTAP the initial database copy is a set of pointers and other metadata and uses just a few megabytes of storage space. As updates are done on the copy, only the new data is stored, and the metadata is used for all unchanged blocks. Therefore, storage costs are kept to a minimum.
FSx for ONTAP and its thin cloning feature allow you to create copies of your production database, instantaneously, in a cost- and space-efficient manner. Furthermore, these clones are fully writable so that the teams using them can replicate whatever production scenarios are needed.
How does it work?
NetApp FlexClone® uses NetApp snapshots to capture the state of the source volume at a specific point in time. The clone references the snapshot's metadata and shares data blocks with the source. Moreover, this clone is fully writable. When changes are made, the new data is stored only on the clone’s volume and not on the source’s volume. Let’s take a deeper look.
How snapshots work
Unlike traditional copy methods that involve duplicating every data block, snapshots in FSx for ONTAP contain only pointers to the source volume’s data, not actual data. This means that clones are created almost instantaneously.
How clones access data
When applications running in a non-production environment read from the clone, the actual data is accessed from the source. This approach means that initially clones use only a very small amount of storage space.
How clones perform updates
When an update is done to the clone, the changes are stored on the clone’s volume and have no impact on the source volume.
This means that the cloned volume is fully writable, because no changes are required to the source volume used by production.
All this means that you can provide your teams with the database clones you need without the common pitfalls.
Use NetApp BlueXP workload factory to simplify operations
FSx for ONTAP provides an intelligent infrastructure for spinning up database copies and that’s a great start. The clones can be created and managed through a variety of tools including NetApp ONTAP® CLI, NetApp API and NetApp SnapCenter®. BlueXP ™️ workload factory can help to streamline the process of creating and managing clones even further.
Workload factory is a free-of-charge optimization and automation service that uses industry best practices to deploy, operate and protect key workloads using FSx for ONTAP.
Workload factory automates and simplifies the process of creating clones (referred to as sandboxes in workload factory) for use in isolated environments.
Workload factory uses Codebox to generate Infrastructure-as-code snippets to include in existing operational flows. The code is automatically generated in the format required for an API request, Amazon CloudFormation, or Terraform.
You can also use workload factory to manage and track clones with a consolidated view of the clones/sandboxes you have created. At a glance, you can track details such as the clone’s age, associated tags, time last used, and more.
How workload factory automates your spin-up workload
Workload factory automates and orchestrates the provisioning and deployment of a variety of isolated environments. In our case, we use it to spin up database copies.
Workload factory guides you through a few simple steps: (1) select the source database, (2) set the destination parameters, (3) name the new sandbox, (4) choose a mount point, and (5) assign a tag.
Keeping Track of Your Sandboxes
The workload factory dashboard provides visibility to all your sandbox deployments. This allows you to easily and, at-a-glance, track sandbox configurations, usage and overall health.
To learn more, read our blog Enhancing database operations with BlueXP workload factory for AWS , and to try it for free, get started with BlueXP workload factory now.
The bottom line: Speed up development, cut costs
Spinning up database clones is an essential need for your organization; however, it poses many challenges for those responsible for the task. You must quickly provide a full, writable copy of the production database. While doing so you must verify that there is no negative impact on the production system such as draining of compute resources. Of course, you are also supposed to maintain low costs for the extra storage needed for database copies. And finally, the users of the database clones need them to be both updateable with fresh data from production and writable for their test data.
With its unique method of creating cloned volumes, FSx for ONTAP solves these problems. You can spin up writable database clones instantaneously, keep storage space and costs to a minimum and verify that there is no negative impact on the production environment.
To learn how to lower your AWS database costs with FSx for ONTAP read our blog, or request a meeting with an FSx for ONTAP specialist.