- Setup
- Cloud credential rotation
- Bucket credential rotation
- Cluster kubeconfig rotation
- Conclusion
- Take advantage of NetApp’s continuing innovation
By Michael Haigh and Patric Uebele (@PatricU), Technical Marketing Engineers at NetApp
In today’s digital world, data breaches and ransomware attacks are a constant threat to all organizations, making it crucial to implement robust security measures to prevent your organization from becoming the latest headline. One such measure is regular credential rotation for both public cloud and on-premises infrastructure, because frequently changing passwords and access keys reduces the risk of compromised credentials and unauthorized access to sensitive data. However, manual credential rotation can be time consuming and error prone, which is why it's essential to have a fully or partially automated system in place.
NetApp® Astra™ Control provides application-aware data protection, mobility, and disaster recovery for any workload running on any Kubernetes distribution. It’s available both as a fully managed service (Astra Control Service; ACS) and as self-managed software (Astra Control Center; ACC). It enables administrators to easily protect, back up, migrate, and create working clones of Kubernetes applications, through either its UI or robust APIs.
When performing regular cloud provider or Kubernetes cluster credential rotation of the clouds and clusters protected with Astra Control, it's necessary to also update the same credentials stored securely in Astra Control. With the NetApp Astra Control Toolkit this is a very simple process that can be easily integrated with your existing automation tools through the associated Python package actoolkit. Read on to find out how!
Setup
There are several ways to install and use the Astra Control Toolkit, as detailed in the installation section of the main readme on GitHub. In this blog we use the Administrator persona, which uses a prepared Docker image; this is the simplest setup. However, if you’re already using the toolkit, then feel free to stick to your preferred method and skip ahead to the next section.
Regardless of your method of installation, the toolkit relies on a config.yaml credential file to authenticate into your Astra Control environment. This file should have the following syntax:
headers:
Authorization: Bearer <Bearer-Token-From-API-Access-Page>
uid: <Account-ID-From-API-Access-Page>
astra_project: <Shortname-or-FQDN>
verifySSL: <True-or-False>
Open your favorite text editor on your local workstation and copy and paste the above template into the editor.
To gather the necessary information to populate this template, log in to your Astra Control environment, click the user profile icon in the upper right corner, and then select API Access:
Figure 1: Astra Control user profile icon.
Copy the Account ID listed on the top of the page and replace the <Account-ID-From-API-Access-Page> placeholder with this value in your config.yaml file:
Figure 2: Astra Control Account ID.
Next, click the blue Generate API Token button on the same page. In the pop-up that appears, optionally give the token a descriptive name. Then click the Generate API T button:
Figure 3: Astra Control Generate API Token button.
Copy the API token value that appears and replace the <Bearer-Token-From-API-Access-Page> placeholder with this value in your config.yaml file. Be sure that the value is copied before you close the pop-up, because this value cannot be shown again (although it can be destroyed and re-created).
Figure 4: Astra Control generated API token.
For the astra_project value, replace the <Shortname-or-FQDN> placeholder with your Astra Control instance fully qualified domain name. If you’re using Astra Control Service, it’s probably astra.netapp.io; if you’re using Astra Control Center, it’s probably an internal FQDN or IP.
Finally, for the verifySSL value, replace the <True-or-False> placeholder with either True or False, depending on your environment. For Astra Control Service, it should be True; for Astra Control Center, it may be either value.
If completed successfully, your config.yaml file should look something like this:
headers:
Authorization: Bearer ubuFUEafH6F5VlFTt7OLPvvpFQPuqxDvVVKFFLYAkfM=
uid: 12345678-abcd-4efg-1234-567890abcdef
astra_project: astra.netapp.io
verifySSL: True
The remaining step before we can get started with credential rotation is to launch the prepared Docker image. We’ll use a volume flag to mount our config.yaml file to one of the directories that the toolkit reads from. That way, we don’t have to continually re-create the credential file each time the container is launched:
docker run -it -v config.yaml:/etc/astra-toolkits/config.yaml netapp/astra-toolkits:2.6.2 /bin/bash
If everything up to this point has been done correctly, you should be able to run an actoolkit command (which is prepackaged in this container), such as list clouds:
~# actoolkit list clouds
+-----------------------------------+--------------------------------------+-------------+
| cloudName | cloudID | cloudType |
+===================================+======================================+=============+
| GCP Astra Control Customer Demo | a7b79c31-aeb3-4910-bd04-47c205887c88 | GCP |
+-----------------------------------+--------------------------------------+-------------+
| Azure Astra Control Customer Demo | 2cbd45bd-0723-4747-922f-e635577cf111 | Azure |
+-----------------------------------+--------------------------------------+-------------+
| AWS Astra Control Customer Demo | 17961c6f-b915-4650-924e-eee07f41e7fe | AWS |
+-----------------------------------+--------------------------------------+-------------+
| Azure Astra Dev QA | b950103c-a4c3-4f49-9f6a-697df2de9959 | Azure |
+-----------------------------------+--------------------------------------+-------------+
Cloud credential rotation
To showcase the rotation of a cloud key, we’ll use an Azure Kubernetes cluster, pu-aks-test-1, which is managed in Astra Control Service via a service principal. The steps will be the same for the other cloud types supported by Astra Control Service, Google Cloud and Amazon Web Services.
Figure 5: Managed AKS cluster in ACS.
A WordPress application is deployed on the cluster, which is already protected by Astra Control by regular backups to an Azure storage container in the Azure cloud instance.
Figure 6: Scheduled backups of the WordPress app.
The credentials of the Azure service principal of the Azure Astra Control Customer Demo cloud instance are stored in Astra Control Service. (We expire the credential “sp-accustomerdemo.accustomerdemo.onmicros.oft.com”.)
Figure 7: Azure cloud credential “sp-accustomerdemo.accustomerdemo.onmicrosoft.com” stored in ACS.
Now let’s delete the service principal’s client secret in the Azure Portal and create a new one:
Figure 8: Delete Azure client secret.
Figure 9: Create a new client secret for the service principal.
With the new secret value copied to the clipboard, we can update the password field in the JSON file that contains the output from the Azure CLI when we created the service principal (actual values are hidden):
~# vi sp-accustomerdemo-v5.json
{
"appId": "5b380608-29f9-4b16-aa48-37758cae604d",
"displayName": "sp-accustomerdemo.accustomerdemo.onmicrosoft.com",
"name": "5b380608-29f9-4b16-aa48-37758cae604d",
"password": "<NEWSECRET>",
"tenant": "<TENANT>",
"subscriptionId": "<SUBID>"
}
Shortly after deleting the original client secret on the Azure Portal, Astra Control marks the buckets backed by Azure storage as Unavailable, because it can no longer access them:
Figure 10: Buckets on Azure are unavailable.
Subsequent scheduled backups to these buckets will fail, and the application’s protection state will be changed from Fully Protected to Partially Protected, because the backups are not available to Astra Control and the application is “only” protected by its NetApp Snapshot™ copies.
Figure 11: Backups failing due to expired secret.
So we must update the service principal’s credential in Astra Control to continue protecting the applications. Thanks to the Astra Control Toolkit, this can be done in minutes.
First, let’s find out the Astra Control internal ID of the Azure cloud instance Azure Astra Control Customer Demo by listing all managed Azure clouds with a list clouds command:
~# actoolkit list clouds --cloudType Azure
+-----------------------------------+--------------------------------------+-------------+
| cloudName | cloudID | cloudType |
+===================================+======================================+=============+
| Azure Astra Control Customer Demo | 2cbd45bd-0723-4747-922f-e635577cf111 | Azure |
+-----------------------------------+--------------------------------------+-------------+
| Azure Astra Dev QA | b950103c-a4c3-4f49-9f6a-697df2de9959 | Azure |
+-----------------------------------+--------------------------------------+-------------+
Now we can easily update the credential for the Azure cloud with the new credential JSON file with an update cloud command:
~# actoolkit update cloud 2cbd45bd-0723-4747-922f-e635577cf111 \
--credentialPath ~/.azure/sp-accustomerdemo-v5.json | jq
{
"type": "application/astra-credential",
"version": "1.1",
"id": "cd611791-e674-4d5b-b09f-4eb1709fcfca",
"name": "astra-sa@Azure Astra Control Customer Demo",
"keyType": "generic",
"valid": "true",
"metadata": {
"creationTimestamp": "2023-03-02T15:49:53Z",
"modificationTimestamp": "2023-03-02T15:49:53Z",
"createdBy": "8247e00f-481b-444c-9171-afcfbe296e94",
"labels": [
{
"name": "astra.netapp.io/labels/read-only/credType",
"value": "service-account"
},
{
"name": "astra.netapp.io/labels/read-only/cloudName",
"value": "Azure"
}
]
}
}
{
"type": "application/astra-cloud",
"version": "1.0",
"id": "2cbd45bd-0723-4747-922f-e635577cf111",
"name": "Azure Astra Control Customer Demo",
"state": "running",
"stateUnready": [],
"cloudType": "Azure",
"credentialID": "cd611791-e674-4d5b-b09f-4eb1709fcfca",
"defaultBucketID": "5aa7d9b0-45bb-4236-92e5-86b8a30d5ea1",
"metadata": {
"labels": [],
"creationTimestamp": "2021-08-09T13:36:38Z",
"modificationTimestamp": "2023-03-02T15:49:54Z",
"createdBy": "8247e00f-481b-444c-9171-afcfbe296e94"
}
}
Next, we confirm that the new credential was successfully added to Astra Control:
~# actoolkit list credentials
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| credName | credID | credType | cloudName | clusterName |
+==================================================+======================================+=================+===================================+===============+
| sp-accustomerdemo.accustomerdemo.onmicrosoft.com | 0d70bc23-f796-4283-8c74-2154a36d3aa8 | service-account | Azure | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| ACSDemoServiceAccount-AWS | 4cfdcd5e-357c-4db3-8385-cf2a344a5301 | service-account | AWS | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| astra-sa@Azure Astra Control Customer Demo | cd611791-e674-4d5b-b09f-4eb1709fcfca | service-account | Azure | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| astra-sa@Azure-Astra-Dev-QA | e3623fdb-1519-4ba3-905a-8307c84b565d | service-account | Azure | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| kubeconfig | ecddd129-3d01-4670-9893-e7be61faf246 | kubeconfig | Azure Astra Control Customer Demo | pu-aks-test-1 |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| PU-AstraControlCustomerDemo | f5709a0b-fd4b-4b22-8708-75fb710b8ce2 | service-account | GCP | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
Bucket credential rotation
As the final step, we need to update the credential associated with the Azure buckets in Astra Control to reenable their use for backups. The list buckets command lets us easily find the bucketIDs of the affected Azure buckets:
~# actoolkit list buckets --provider azure
+--------------------------------------+--------------------------------+--------------------------------------+------------+---------+
| bucketID | name | credentialID | provider | state |
+======================================+================================+======================================+============+=========+
| d727f297-3a51-4446-8de6-2d2d0ba07dc2 | Azure LRS bucket EastUS | 0d70bc23-f796-4283-8c74-2154a36d3aa8 | azure | removed |
+--------------------------------------+--------------------------------+--------------------------------------+------------+---------+
| 796feb89-e874-4f59-8042-875b9d082aec | Azure North Europe GRS | 0d70bc23-f796-4283-8c74-2154a36d3aa8 | azure | removed |
+--------------------------------------+--------------------------------+--------------------------------------+------------+---------+
| 5aa7d9b0-45bb-4236-92e5-86b8a30d5ea1 | Azure West Europe GRS Standard | 0d70bc23-f796-4283-8c74-2154a36d3aa8 | azure | removed |
+--------------------------------------+--------------------------------+--------------------------------------+------------+---------+
With this information, we can update each bucket with the credentialID cd611791-e674-4d5b-b09f-4eb1709fcfca of the new credential. For the first bucket, the update bucket command is:
~# actoolkit update bucket d727f297-3a51-4446-8de6-2d2d0ba07dc2 \
--credentialID cd611791-e674-4d5b-b09f-4eb1709fcfca | jq
{
"type": "application/astra-bucket",
"version": "1.0",
"id": "d727f297-3a51-4446-8de6-2d2d0ba07dc2",
"name": "Azure LRS bucket EastUS",
"state": "removed",
"credentialID": "cd611791-e674-4d5b-b09f-4eb1709fcfca",
"provider": "azure",
"bucketParameters": {
"azure": {
"storageAccount": "acsbackupcustomerdemo",
"bucketName": "pu-acsbucket1"
}
},
"metadata": {
"creationTimestamp": "2022-02-21T13:59:39Z",
"modificationTimestamp": "2023-03-02T17:08:37Z",
"createdBy": "8247e00f-481b-444c-9171-afcfbe296e94"
}
}
After repeating the above command for the remaining two Azure buckets, all the buckets are again available:
~# actoolkit list buckets -p azure
+--------------------------------------+--------------------------------+--------------------------------------+------------+-----------+
| bucketID | name | credentialID | provider | state |
+======================================+================================+======================================+============+===========+
| d727f297-3a51-4446-8de6-2d2d0ba07dc2 | Azure LRS bucket EastUS | cd611791-e674-4d5b-b09f-4eb1709fcfca | azure | available |
+--------------------------------------+--------------------------------+--------------------------------------+------------+-----------+
| 796feb89-e874-4f59-8042-875b9d082aec | Azure North Europe GRS | cd611791-e674-4d5b-b09f-4eb1709fcfca | azure | available |
+--------------------------------------+--------------------------------+--------------------------------------+------------+-----------+
| 5aa7d9b0-45bb-4236-92e5-86b8a30d5ea1 | Azure West Europe GRS Standard | cd611791-e674-4d5b-b09f-4eb1709fcfca | azure | available |
+--------------------------------------+--------------------------------+--------------------------------------+------------+-----------+
The scheduled backups of the WordPress application will now work, and the previously created backups are again available, so the application’s protection state is back to Fully Protected:
Figure 12: Scheduled backups work again.
Finally, we can optionally delete the expired credential, since it is no longer used by any cloud or bucket. Let’s find its credentialID with a list credentials command:
~# actoolkit list credentials
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| credName | credID | credType | cloudName | clusterName |
+==================================================+======================================+=================+===================================+===============+
| sp-accustomerdemo.accustomerdemo.onmicrosoft.com | 0d70bc23-f796-4283-8c74-2154a36d3aa8 | service-account | Azure | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| ACSDemoServiceAccount-AWS | 4cfdcd5e-357c-4db3-8385-cf2a344a5301 | service-account | AWS | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| astra-sa@Azure-Astra-Dev-QA | e3623fdb-1519-4ba3-905a-8307c84b565d | service-account | Azure | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| kubeconfig | ecddd129-3d01-4670-9893-e7be61faf246 | kubeconfig | Azure Astra Control Customer Demo | pu-aks-test-1 |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
| PU-AstraControlCustomerDemo | f5709a0b-fd4b-4b22-8708-75fb710b8ce2 | service-account | GCP | N/A |
+--------------------------------------------------+--------------------------------------+-----------------+-----------------------------------+---------------+
Now we can remove it from Astra Control with a destroy credential command:
~# actoolkit destroy credential 0d70bc23-f796-4283-8c74-2154a36d3aa8
credential 0d70bc23-f796-4283-8c74-2154a36d3aa8 destroyed
Cluster kubeconfig rotation
Both Astra Control Service and Astra Control Center allow users to bring self-managed clusters under Astra Control management. These clusters are associated with the private cloud, and they are brought under management by uploading a cluster kubeconfig file, rather than hyperscaler cloud credentials.
Whether the token within the kubeconfig has unexpectedly expired, or you’re proactively rotating the associated kubeconfig, it’s easy to update the associated credential for a self-managed cluster. Here we have a Rancher Kubernetes cluster (although the process is the same for all self-managed clusters) that has gone into an Unavailable state due to an expired token:
Figure 13: Self-managed cluster unavailable.
Let’s head over to our terminal and gather the cluster UUID with a list clusters command:
~# actoolkit list clusters
+--------------+--------------------------------------+--------------+------------+----------------+----------------+------------------+
| clusterName | clusterID | clusterType | location | managedState | tridentState | tridentVersion |
+==============+======================================+==============+============+================+================+==================+
| rtp-rke-04 | cd7181e4-1931-4e15-8556-1b8b45a98aee | rke | | managed | unmanaged | 22.10.0 |
+--------------+--------------------------------------+--------------+------------+----------------+----------------+------------------+
We’ll then use that UUID in an update cluster command, which also requires passing in the updated kubeconfig file as an argument:
~# actoolkit update cluster cd7181e4-1931-4e15-8556-1b8b45a98aee \
--credentialPath ~/.kube/config | jq
{
"type": "application/astra-credential",
"version": "1.1",
"id": "ece5dd6f-089b-4de6-8f57-32e149450aa3",
"name": "rtp-rke-04",
"valid": "true",
"metadata": {
"creationTimestamp": "2023-03-15T16:52:23Z",
"modificationTimestamp": "2023-03-15T17:35:27Z",
"createdBy": "8146d293-d897-4e16-ab10-8dca934637ab"
}
}
After 5 to 10 minutes, refresh the UI. The cluster should now be in a Healthy state:
Figure 14: Self-managed cluster healthy.
Conclusion
Regular credential rotation is a critical component of any organization's security posture, and implementing semi- or fully automated credential rotation with the NetApp Astra Toolkit is a simple process. In this blog post we carried out the following steps:
- Configured a config.yaml credential file for the toolkit
- Launched a prepared Docker image, which contains the toolkit
- Created a new Azure service principal client secret
- Gathered the Azure cloud UUID
- Updated the Azure cloud to use the new client secret
- Gathered the Azure storage bucket UUIDs
- Updated the Azure storage buckets to use the new client secret
- Removed the expired client secret from Astra Control
- Gathered the self-managed cluster UUID in need of a new kubeconfig
- Updated the self-managed cluster to use the new kubeconfig
For next steps, learn more about the NetApp Astra family here, or view the NetApp Astra Toolkit documentation here.
Take advantage of NetApp’s continuing innovation
If you want to see for yourself how easy it is to protect persistent Kubernetes applications with Astra Control, by using either its UI or the powerful Astra Toolkit, apply for a free trial. Get started today!