Cloud Backup: Search and Restore (Preview Feature)
What is Cloud Backup “Search and Restore”?
Cloud Backup has introduced a new “Search and Restore” feature which enables users to search for a specific file from all backups stored on Amazon S3 cloud storage and then perform a restore. It provides users with a quick and focused search for files across all the backups in the cloud and gives options to narrow searches using multiple search criteria. The implementation has been done based on SnapDiff v3 with the full file system representation. The Search and Restore” solution has been developed using native cloud technology, which is fast, lean, and cost-optimized.
How different is it from Cloud Backup “Browse and Restore”
Cloud Backup already provides you the option to “Browse and Restore” while doing a volume level or single/multiple file restore from backups stored on the cloud.This existing service is quite different from the new “Search and Restore” feature. For restoring individual files to on-premises ONTAP or Cloud Volumes ONTAP, in addition to Cloud Manager Connector, a Cloud Restore instance needs to be deployed by Cloud Manager on the same subnet. This instance is used only to create the catalog for “Browse and Restore”. The instance is turned off if not used to save the costs, and it is not be deployed if you don’t attempt to restore individual files.
Features of Cloud Backup “Search and Restore”
- Searches based on
- File Name
- File Type
- File Creation Time
- File Size
- Working Environment
- Local and Cloud Backups
- Searches based on Volume name
- Handles both Local and Cloud backups
- Handles newly created, deleted, and renamed directories.
- Handles newly created, deleted, and renamed files.
- Restore only supported on Cloud backups.
- No separate Cloud Restore instance is required
How does Cloud Backup “Search and Restore” Work?
Cloud Backup Search & Restore:
- Uses a combination of ONTAP SnapDiff v3 SDK technology along with BigData tools like AWS Athena, AWS Glue, and S3.
- Uses a dedicated container that runs on the Cloud Manager instance to provide indexed catalog functionalities. The catalog container can be easily decoupled to run on more powerful machines, if required.
- Enables Snapdiff v3 on the Storage VM in your Working Environment. It also provisions a new S3 bucket, AWS Athena interactive query service, and AWS Glue serverless data integration service to co-ordinate with the new Catalog feature.
- Runs scheduled tasks periodically on the catalog container. It scans for changes in the backups incrementally on working environments enabled for indexing, and then updates AWS S3 Storage with changes and snapshot details.
- Provides better performance by compressing, partitioning, and storing catalog data in a columnar format in S3 buckets. We use parquet format for data storage. Data is partitioned by Working Environment Id, Volume ID, and Snapshot ID.
- Uses AWS Athena to provide efficient search capabilities to browse data within S3 storage
What do I need before I get started with Cloud Backup “Search and Restore”: Prerequisites
You can start using the Search & Restore feature if you already have a Cloud Manager Connector deployed and Cloud Backup is enabled on the Working Environment. Make sure that your Working Environment (On-prem ONTAP or Cloud Volumes ONTAP) version is 9.8 and above.
- ONTAP requirements
- ONTAP version 9.8 or above
- NFS protocol enabled
- SnapDiff RPC Server should be activated on the indexed SVMs. This is automated by Cloud Manager.
- Online Storage VM Data-LIF
- Cloud permissions that have to be manually added for existing Connectors
Athena:
"athena:StartQueryExecution",
"athena:GetQueryResults",
"athena:GetQueryExecution"
For Glue:
"glue:GetDatabase",
"glue:GetTable",
"glue:CreateTable",
"glue:CreateDatabase",
"glue:GetPartitions",
"glue:BatchCreatePartition",
"glue:BatchDeletePartition"
S3:
"s3:DeleteObject"
"s3:PutObject"
"Resource": ["arn:aws:s3:::netapp-backup-*"]
3. If you are new to Cloud Manager, complete the following steps to get started with “Search & Restore”.
- A valid cloud account with required admin rights to deploy EC2 instances and create IAM roles.
- Make sure that you have subscribed to the NetApp Cloud Central Account.
- Log in into Cloud Manager.
- Deploy a Cloud Manager Connector.
- Deploy a Cloud Volumes ONTAP Working Environment.or discover an onpremises ONTAP system (version 9.8 and above)
- Choose the “Canvas” page on the Cloud Manager UI.
- Select the required Working Environment.
- On the righthand side, you will be able to see the list of services.
- If the “Backup and Restore” service is not enabled, click on Enable.
- Now the “Backup and Restore” wizard will open.
- Provide the required Account name and the region.
- Create a backup policy with the required backup requirements.
- Choose the volume, or volumes, that need to be backed up.
- Click Submit so that the backup job is started.
For more information, visit the Cloud Manager documentation.
Enabling “Search & Restore” on the Working Environment
To get started with the Search & Restore feature, make sure that Cloud Backup is enabled.
- On the Cloud Manager Canvas, choose the required Working Environment.
- On the right-hand side of the Cloud Manager UI, all the services applicable to the Working Environment will be listed.
- Click on the “Backup and Restore”, 3 dots UI drop down, and choose “Restore Dashboard”. Now you will be able to see the Restore window as shown below.
- For the first time using Backup & Restore you need to enable Indexed Catalog. Click on “Enable Indexing for Working Environments” as shown in the figure below.
5. Now you will see all the Working Environments and you can go ahead and enable Indexing on the Working Environment that you would like. Click “Enable Indexing” as shown in the figure.
6. Once the Indexed Catalog is enabled, appropriate services and databases will be created at the backend to start the cataloging features. Enabling Indexed Cataloging will result in the following:
- The service will enable SnapDiff v3 on the Storage VM in your Working Environment.
- It will provision AWS Athena interactive query service and AWS Glue serverless data integration service to run the Catalog service.
7. Once the services are provisioned, the Indexed Catalog service for the Working Environment will be shown as “Active”.
How do I use the Indexed Catalog UI
Now let's go ahead and check out the new UI that has been introduced for the “Search and Restore” feature and understand its functionality.
- The new “Restore Dashboard” UI, will provide the option to “Browse & Restore” and “Search & Restore”. Click on “Search & Restore”.
2. To search for a file or a list of files having a common name, input the name of the file in the search bar and choose from where you would like to restore. You can search from “All Resources”, “Files”, or “Volumes”. To filter further, choose the “Filter by” options to filter by file types, file-creation dates, specific working environments, file sizes, and backup location.
3. Clicking on “View All Backups” will show all the backups both on the Cloud and Primary (local) which contains the particular file. Note that you will only be able to restore the selected file from the cloud backups. The “Restore” button on the Primary (local) backup copies will be grayed out as it's not supported as yet.
4. To restore a file, select the Restore location. You can Restore to the Original Location or Restore to an Alternate location.
5. Click Restore to complete the file restore process.
Try It Out!
Now that you’ve read about our new Cloud Backup: Search and restore feature, we hope it entices you to go ahead try out this new preview feature .
If you would like more information, contact us [Bezalel, Aviram (Aviram.Bezalel@netapp.com), John Andathethu, Jacob (Jacob.A@netapp.com)] and we would be happy to answer your questions.
