- Indexed catalog or “Search and Restore” On-Premise connector Implementation Modes
- Features of Cloud Backup “Search and Restore”
- On-premise Connector Deployment
- Enabling “Search & Restore” on the Working Environment
- How do I use the Indexed Catalog UI
- Try It Out!
Cloud Backup has now added “Search and Restore” support for On-Premise Connector (with internet access), which means you can now manage Indexed Catalog feature on on-premise, AWS, and Azure through the on-premise connector to search for a specific file from all backups stored on StorageGRID bucket, AWS S3 and Azure Blob storage and then perform a required restore. It provides users with a quick and focused search for files across all the backups in the StorageGRID bucket and gives options to narrow searches using multiple search criteria. The implementation has been done based on SnapDiff v3 with the full file system representation.
Indexed catalog or “Search and Restore” On-Premise connector Implementation Modes
The Indexed catalog or “Search and Restore” On-Premise connector implementation can be done in 3 modes
1) Implementing Indexed catalog on-premise on the snapshots backed up to StorageGRID Bucket using On-prem connector
In this deployment mode, the On-premise connector will be deployed on a VM/Bare-metal within the data center and Cloud Backup will backup the data from the On-Prem ONTAP System to On-Prem StorageGRID. In this mode, the Indexed Cataloging feature utilizes DuckDB for Querying. The cataloging service continually scans backups for incremental changes in working environments that are enabled for indexing. It then updates DuckDB with both the changes and snapshot details. The SnapDiff response from ONTAP is converted to parquet format, it is stored in DuckDB. The cataloging service runs scheduled indexing tasks at regular intervals. Whenever a query is executed, the results are fetched from DuckDB tables and temporarily stored in Level DB.
2) Implementing Indexed catalog on AWS(using Athena and Glue) on the snapshots backed up to AWS S3 Bucket using On-prem connector
In this deployment mode, the On-premise connector will be deployed on a VM/Bare-metal within the data center and Cloud Backup will backup the data from the On-Prem ONTAP System to AWS S3 object store. When Indexed cataloging is enabled on the Cloud Backup UI, appropriate services (Athena and Glue )will be deployed to start the cataloging service for the data stored in AWS S3 Object store
3) Implementing Indexed catalog on Azure(using Synapse) on the snapshots backed up to Azure Blob using On-prem connector
In this deployment mode, the On-premise connector will be deployed on a VM/Bare-metal within the data center and Cloud Backup will backup the data from the On-Prem ONTAP System to the Azure Blob S3 object store. When Indexed cataloging is enabled on the Cloud Backup UI, appropriate services (Synapse )will be deployed to start the cataloging service for the data stored in the Azure Blob S3 object store.
Features of Cloud Backup “Search and Restore”
Searches can be done based on the following criteria
- File Name
- File Type
- File Creation Time
- File Size
- Working Environment
- Local and Cloud Backups
- Searches based on the Volume name
- Handles both Local and Cloud backups
- Handles newly created, deleted, and renamed directories.
- Handles newly created, deleted, and renamed files.
- Restore only supported on Cloud backups.
- No separate Cloud Restore instance is required
On-premise Connector Deployment
To deploy on-premise BlueXP connector, you would need to download the installer from NetApp Support Site and install the Connector software on an existing Linux host in your network or in the cloud.
Please check out the VM host requirements for installing the On-Premise Connector
1) Download the Cloud Manager software from the NetApp Support Site, and then copy it to the Linux host.
2) Assign permissions to run the script.
chmod +x OnCommandCloudManager-V3.9.19.sh
3) Run the installation script.If you have a proxy server, you will need to enter the command parameters as shown below. The installer doesn’t prompt you to provide information about a proxy.
./OnCommandCloudManager-V3.9.19.sh [silent] [proxy=ipaddress] [proxyport=port] [proxyuser=user_name] [proxypwd=password]
4) Open a web browser and enter the following URL:
https://ipaddress
ipaddress can be localhost, a private IP address, or a public IP address, depending on the configuration of the host. For example, if the Connector is in the public cloud without a public IP address, you must enter a private IP address from a host that has a connection to the Connector host.
Sign up at NetApp Cloud Central or log in.
5) Once you log in make sure to select the appropriate NetApp account and give in a BlueXP Connector name.
6) This will redirect and open NetApp BlueXP page.
7) Now go ahead and add/discover your On-prem Cluster on the BlueXP canvas page and start backing up your data using Cloud Backup.
Once your data is backed up to the cloud , you can go ahead and start enabling Indexed catalog or Search and Restore on the Working Environment.
Enabling “Search & Restore” on the Working Environment
To get started with the Search & Restore feature, make sure that Cloud Backup is enabled.
- On the Cloud Manager Canvas, choose the required Working Environment.
- On the right-hand side of the Cloud Manager UI, all the services applicable to the Working Environment will be listed.
- Click on the “Backup and Restore”, 3 dots UI drop down, and choose “Restore Dashboard”. Now you will be able to see the Restore window as shown below.
- For the first time using Backup & Restore you need to enable Indexed Catalog. Click on “Enable Indexing for Working Environments” as shown in the figure below.
5. Now you will see all the Working Environments and you can go ahead and enable Indexing on the Working Environment that you would like. Click “Enable Indexing” as shown in the figure.
6. Once the Indexed Catalog is enabled, appropriate services and databases will be created at the backend to start the cataloging features. Enabling Indexed Cataloging will result in the following:
- The service will enable SnapDiff v3 on the Storage VM in your Working Environment.
- It will provision AWS Athena interactive query service and AWS Glue server-less data integration service to run the Catalog service.
7. Once the services are provisioned, the Indexed Catalog service for the Working Environment will be shown as “Active”.
How do I use the Indexed Catalog UI
Now let's go ahead and check out the new UI that has been introduced for the “Search and Restore” feature and understand its functionality.
- The new “Restore Dashboard” UI, will provide the option to “Browse & Restore” and “Search & Restore”. Click on “Search & Restore”.
2. To search for a file or a list of files having a common name, input the name of the file in the search bar and choose from where you would like to restore. You can search from “All Resources”, “Files”, or “Volumes”. To filter further, choose the “Filter by” options to filter by file types, file-creation dates, specific working environments, file sizes, and backup location.
3. Clicking on “View All Backups” will show all the backups both on the Cloud and Primary (local) which contains the particular file. Note that you will only be able to restore the selected file from the cloud backups. The “Restore” button on the Primary (local) backup copies will be grayed out as it's not supported as yet. Click on the "Restore" button on the available cloud backup.
4. Check the restore folder information, select the destination type and Click on Restore to complete the file restore process.
Try It Out!
Now that you’ve read about our Cloud Backup: Search and restore feature support for On-Premise connector, we hope it entices you to go ahead try out this new preview feature .
If you would like more information, contact us [Bezalel, Aviram (Aviram.Bezalel@netapp.com), John Andathethu, Jacob (Jacob.A@netapp.com)] and we would be happy to answer your questions.