Tech ONTAP Blogs

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Tech ONTAP Blogs

Introducing Trident protect role-based access control

PatricU
NetApp
NetApp
2 weeks ago
172 Views

In a recent series of blog posts, we introduced the advanced application data management (ADM) and protection capabilities of NetApp® Trident™ protect software for stateful Kubernetes applications and its newly introduced CLI. We discussed how its Kubernetes-native custom resources (CRs) can help to integrate application protection into your automation and deployment workflows using either manifests or the Trident protect CLI.

 

If your organization has many users or specific security needs, you can use the role-based access control (RBAC) features of Trident protect to gain more granular control over access to resources and namespaces.

 

This blog post walks through a basic setup of what a customer’s environment might look like in a tenant-based way.

Prerequisites

If you plan to follow this blog step by step, you need to have the following available:

  • A Kubernetes cluster with the latest versions of Trident and Trident protect installed, and its associated kubeconfig
  • A NetApp ONTAP® storage back end and Trident with configured storage back ends, storage classes, and volume snapshot classes
  • Two configured object storage buckets for storing backups and metadata information
  • A workstation with kubectl configured to use kubeconfig 
  • A workstation with Helm installed (or a different means of deploying a sample Kubernetes application)
  • The tridentctl-protect CLI of Trident protect installed on your workstation
  • Admin user permission to create service accounts and some Trident protect CRs

Background

Trident protect provides a set of custom resource definitions (CRDs) that allow users to protect their application’s lifecycles. Some of those CRDs include Applications, Snapshots, Backups, and AppMirrorRelationships, to name a few. The CRDs not deemed ADM-specific are AutoSupportBundle, AutoSupportBundleSchedules, and AppVaults. Although technically speaking, AppVault creation is a prequisite step to enabling the other ADM CRs, AppVault CRs are special in that they are created and managed by a “cluster admin” in the namespace that the Trident protect controller is installed in, typically trident-protect.

 

The RBAC design of Trident protect enables you to isolate tenant user activity while using Trident protect’s ADM capabilities. In effect, all ADM CRs are to be created inside the user’s application namespaces. The only CRs allowed in the trident-protect namespace are AppVaults, AutoSupportBundle, and AutoSupportBundleSchedules.

Spoiler

Note: Although all the ADM CRs are created inside the application namespace, their corresponding job pods (if applicable) are created and run inside the trident-protect namespace.

User initialization

In the example setup used throughout this blog, we have three players:

  • cluster-admin can see all namespaces, cluster-scoped resources, and so on. The cluster-admin will have access to the trident-protect namespace, in which we assume Trident protect was installed.
  • eng-user is a tenant user of the cluster and only has access to namespace eng-1; eng-user’s application (for example, NGINX) will go in this namespace.
  • mkt-user is another tenant user of the cluster and only has access to namespace mkt-1.

For this blog post, we will create service account (SA) users. First, we create the namespaces for the engineering and marketing users.

 

$ kubectl create ns eng-1
namespace/eng-1 created
$ kubectl create ns mkt-1
namespace/mkt-1 created

 

Now we create the service accounts for the eng-user and mkt-user in their respective namespaces.

 

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eng-user
  namespace: eng-1
EOF
serviceaccount/eng-user created
 
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mkt-user
  namespace: mkt-1
EOF
serviceaccount/mkt-user created

 

A service account secret is used to authenticate with the service accounts. It can easily be deleted and re-created if compromised. We create the service account secrets for our two users in the next step. (It’s important to include the annotation kubernetes.io/service-account.name: <USERNAME> and type: kubernetes.io/service-account-token. Kubernetes will fill in the secret data on its own.)

 

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: eng-user
  name: eng-user-secret
  namespace: eng-1
type: kubernetes.io/service-account-token
EOF
secret/eng-user-secret created

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: mkt-user
  name: mkt-user-secret
  namespace: mkt-1
type: kubernetes.io/service-account-token
EOF
secret/mkt-user-secret created

 

The Trident protect installation (Helm chart) comes with a predefined cluster role trident-protect-tenant-cluster-role that can be bound to a particular user in a specific namespace through role binding. This is a general view on the permission level that a tenant user requires to operate with Trident protect.

 

$ kubectl get clusterrole trident-protect-tenant-cluster-role -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    meta.helm.sh/release-name: trident-protect
    meta.helm.sh/release-namespace: trident-protect
  creationTimestamp: "2024-10-18T13:45:41Z"
  labels:
    app.kubernetes.io/instance: trident-protect
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: trident-protect
    app.kubernetes.io/version: 81635c2
    helm.sh/chart: trident-protect-24.10.0-preview.2
  name: trident-protect-tenant-cluster-role
  resourceVersion: "9353439"
  uid: 42f8102b-f64f-4362-9bb4-806915c8728c
rules:
- apiGroups:
  - protect.trident.netapp.io
  resources:
  - applications
  - appmirrorrelationships
  - appmirrorupdates
  - backupinplacerestores
  - backuprestores
  - backups
  - exechooks
  - exechooksruns
  - kopiavolumebackups
  - kopiavolumerestores
  - pvccopies
  - pvcerases
  - resourcebackups
  - resourcedeletes
  - resourcerestores
  - resticvolumebackups
  - resticvolumerestores
  - schedules
  - shutdownsnapshots
  - snapshotinplacerestores
  - snapshotrestores
  - snapshots
  verbs:
  - '*'

 

Spoiler

Note that the resources a tenant user can interact with are all the Trident protect CRDs except for AppVaults, AutoSupportBundle, and AutoSupportBundleSchedules. These are special CRDs and should be handled only by admins. AppVaults should be handled/provisioned only by cluster admins because their setup requires bucket login data in the form of secrets, something admins might not want to pass around to tenant users. AutoSupportBundles should be handled/provisioned only by cluster admins because they collect metrics, logs, and other things outlining the entire Trident protect controller operations. These could contain information about other tenant users, hence breaking the tenant isolation.

Now we bind the trident-protect-tenant-cluster-role to the engineering service account eng-user:

 

$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: engineering-ns-tenant-rolebinding
  namespace: eng-1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: trident-protect-tenant-cluster-role
subjects:
- kind: ServiceAccount
  name: eng-user
  namespace: eng-1
EOF
rolebinding.rbac.authorization.k8s.io/engineering-ns-tenant-rolebinding created

 

And do the same for the marketing service account mkt-user:

 

$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: marketing-ns-tenant-rolebinding
  namespace: mkt-1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: trident-protect-tenant-cluster-role
subjects:
- kind: ServiceAccount
  name: mkt-user
  namespace: mkt-1
EOF
rolebinding.rbac.authorization.k8s.io/marketing-ns-tenant-rolebinding created

 

For quickly testing the permissions, we use kubectl auth can-i. To confirm that the engineering user eng-user can access Trident protect resources only in the engineering namespace eng-1, run these two checks:

 

$ kubectl auth can-i --as=system:serviceaccount:eng-1:eng-user get applications.protect.trident.netapp.io -n eng-1
yes
 
$ kubectl auth can-i --as=system:serviceaccount:eng-1:eng-user get applications.protect.trident.netapp.io -n mkt-1
no

 

And in an analogous way for the marketing user mkt-user, run these two tests:

 

$ kubectl auth can-i --as=system:serviceaccount:mkt-1:mkt-user get applications.protect.trident.netapp.io -n mkt-1
yes
 
$ kubectl auth can-i --as=system:serviceaccount:mkt-1:mkt-user get applications.protect.trident.netapp.io -n eng-1
no

 

With the users and their correct permissions verified, we’re now ready to configure Trident protect and start testing data management operations with RBAC restrictions in place.

Configuring the AppVault CRs for storing backup data

To perform data management tasks such as backups and snapshots, the cluster admin needs to create two AppVault objects and grant access to individual users. So, for our demo setup, we (as cluster admin) create an AppVault CR for the eng-user and one for the mkt-user. The two AppVault resources will be backed by two Azure storage containers in two Azure storage accounts.

For the creation of the AppVault CRs, we put the account and container names and the access keys, together with the names for the secret and the AppVault CR, in environment variables, starting with the eng-user:

 

$ ACCOUNTNAME=putesteng
$ BUCKETNAME=eng-1
$ ACCOUNTKEY=<REDACTED>
$ SECRETNAME=appvault-for-enguser-only-secret
$ APPVAULT=appvault-for-enguser-only

 

First, we create the secret containing the access credentials for the eng-1 container in the trident-protect namespace:

 

$ kubectl -n trident-protect create secret generic $SECRETNAME --from-literal=accountName=$ACCOUNTNAME --from-literal=accountKey=$ACCOUNTKEY

 

Now we can create the corresponding AppVault CR appvault-for-enguser-only:

 

$ kubectl apply -f - <<EOF
apiVersion: protect.trident.netapp.io/v1
kind: AppVault
metadata:
  name: $APPVAULT
  namespace: trident-protect
spec:
  providerConfig:
    azure:
      accountName: $ACCOUNTNAME
      bucketName: $BUCKETNAME
      endpoint: core.windows.net
    gcp:
      bucketName: ""
      projectID: ""
    s3:
      bucketName: ""
      endpoint: ""
  providerCredentials:
    accountKey:
      valueFromSecret:
        key: accountKey
        name: $SECRETNAME
  providerType: Azure
EOF
appvault.protect.trident.netapp.io/appvault-for-enguser-only created

 

After some seconds, the newly created AppVault CR should be in the Available state:

 

$ tridentctl-protect get appvault
+---------------------------+----------+-----------+-------+-------+
|           NAME            | PROVIDER |   STATE   |  AGE  | ERROR |
+---------------------------+----------+-----------+-------+-------+
| appvault-for-enguser-only | Azure    | Available | 1m11s |       |
+---------------------------+----------+-----------+-------+-------+

 

To ensure that the above AppVault is usable only by the eng-user, we create the following role that we’ll bind to the eng-user. This role allows cluster admins to allow access to specific resources in a namespace instead of the whole lot of them in the entire namespace.

 

$ ROLENAME=eng-user-appvault-reader

$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: $ROLENAME
  namespace: trident-protect
rules:
- apiGroups:
  - protect.trident.netapp.io
  resourceNames:
  - $APPVAULT
  resources:
  - appvaults
  verbs:
  - get
EOF
role.rbac.authorization.k8s.io/eng-user-appvault-reader created

 

Spoiler

Note: This can be a ClusterRole kind as well. This is useful if you expect multiple different tenant users/SAs to bind to the same AppVault across separate namespaces.

Now we can create a RoleBinding object for our eng-user to be granted access to:

 

$ ROLEBINDINGNAME=eng-user-read-appvault-binding

$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eng-user-read-appvault-binding
  namespace: trident-protect
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: eng-user-appvault-reader
subjects:
- kind: ServiceAccount
  name: eng-user
  namespace: eng-1
EOF
rolebinding.rbac.authorization.k8s.io/eng-user-read-appvault-binding created

$ kubectl -n trident-protect get rolebindings
NAME                                          ROLE                                        AGE
eng-user-read-appvault-binding                Role/eng-user-appvault-reader               51s
trident-protect-job-copy                      Role/trident-protect-job-copy               45h
trident-protect-job-erase                     Role/trident-protect-job-erase              45h
trident-protect-leader-election-rolebinding   Role/trident-protect-leader-election-role   45h
trident-protect-manager-rolebinding           Role/trident-protect-manager-role           45h

 

With this Role/RoleBinding combination, the eng-user can’t list all the AppVaults in the trident-protect namespace, which we confirm with user impersonation:

 

$ kubectl get appvaults -n trident-protect --as=system:serviceaccount:eng-1:eng-user
Error from server (Forbidden): appvaults.protect.trident.netapp.io is forbidden: User "system:serviceaccount:eng-1:eng-user" cannot list resource "appvaults" in API group "protect.trident.netapp.io" in the namespace "trident-protect"

 

But the eng-user can get the AppVault specified in the role:

 

$ kubectl auth can-i --as=system:serviceaccount:eng-1:eng-user get appvaults.protect.trident.netapp.io/appvault-for-enguser-only -n trident-protect
yes
 
~$ kubectl get appvault appvault-for-enguser-only -n trident-protect --as=system:serviceaccount:eng-1:eng-user
NAME                        STATE       AGE
appvault-for-enguser-only   Available   19h

 

Our tenant eng-user is now properly siloed into the permitted namespaces, and eng-user has been given an AppVault to use for data management operations.

 

Now we repeat the previous steps for the mkt-user, first setting these variables to the corresponding values:

 

$ ACCOUNTNAME=putestmkt
$ BUCKETNAME=mkt-1
$ ACCOUNTKEY=<REDACTED>
$ SECRETNAME=appvault-for-mktuser-only-secret
$ APPVAULT=appvault-for-mktuser-only
$ ROLENAME=mkt-user-appvault-reader
$ ROLEBINDINGNAME=mkt-user-read-appvault-binding

 

Then we can create the Secret, AppVault, Role, and RoleBinding objects for the mkt-user like we did for the eng-user.

 

$ kubectl -n trident-protect create secret generic $SECRETNAME --from-literal=accountName=$ACCOUNTNAME --from-literal=accountKey=$ACCOUNTKEY
secret/appvault-for-mktuser-only-secret created

$ kubectl apply -f - <<EOF
apiVersion: protect.trident.netapp.io/v1
kind: AppVault
metadata:
  name: $APPVAULT
  namespace: trident-protect
spec:
  providerConfig:
    azure:
      accountName: $ACCOUNTNAME
      bucketName: $BUCKETNAME
      endpoint: core.windows.net
    gcp:
      bucketName: ""
      projectID: ""
    s3:
      bucketName: ""
      endpoint: ""
  providerCredentials:
    accountKey:
      valueFromSecret:
        key: accountKey
        name: $SECRETNAME
  providerType: Azure
EOF
appvault.protect.trident.netapp.io/appvault-for-mktuser-only created
 
$ tridentctl-protect get appvault
+---------------------------+----------+-----------+-------+-------+
|           NAME            | PROVIDER |   STATE   |  AGE  | ERROR |
+---------------------------+----------+-----------+-------+-------+
| appvault-for-enguser-only | Azure    | Available | 20h7m |       |
| appvault-for-mktuser-only | Azure    | Available | 1m18s |       |
+---------------------------+----------+-----------+-------+-------+

$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: $ROLENAME
  namespace: trident-protect
rules:
- apiGroups:
  - protect.trident.netapp.io
  resourceNames:
  - $APPVAULT
  resources:
  - appvaults
  verbs:
  - get
EOF
role.rbac.authorization.k8s.io/mkt-user-appvault-reader created

$ k apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: $ROLEBINDINGNAME
  namespace: trident-protect
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: mkt-user-appvault-reader
subjects:
- kind: ServiceAccount
  name: mkt-user
  namespace: mkt-1
EOF
rolebinding.rbac.authorization.k8s.io/mkt-user-read-appvault-binding created

 

The configured roles and role bindings are now:

 

$ kubectl -n tridet-protect get roles,rolebindings
NAME                                                                  CREATED AT
role.rbac.authorization.k8s.io/-appvault-reader                       2024-11-22T16:19:42Z
role.rbac.authorization.k8s.io/eng-user-appvault-reader               2024-11-22T16:48:35Z
role.rbac.authorization.k8s.io/mkt-user-appvault-reader               2024-11-22T16:48:41Z
role.rbac.authorization.k8s.io/trident-protect-job-copy               2024-11-12T14:26:28Z
role.rbac.authorization.k8s.io/trident-protect-job-erase              2024-11-12T14:26:28Z
role.rbac.authorization.k8s.io/trident-protect-leader-election-role   2024-11-12T14:26:28Z
role.rbac.authorization.k8s.io/trident-protect-manager-role           2024-11-12T14:26:28Z

NAME                                                                                ROLE                                        AGE
rolebinding.rbac.authorization.k8s.io/eng-user-read-appvault-binding                Role/eng-user-appvault-reader               26m
rolebinding.rbac.authorization.k8s.io/mkt-user-read-appvault-binding                Role/mkt-user-appvault-reader               31s
rolebinding.rbac.authorization.k8s.io/trident-protect-job-copy                      Role/trident-protect-job-copy               36h
rolebinding.rbac.authorization.k8s.io/trident-protect-job-erase                     Role/trident-protect-job-erase              36h
rolebinding.rbac.authorization.k8s.io/trident-protect-leader-election-rolebinding   Role/trident-protect-leader-election-role   36h
rolebinding.rbac.authorization.k8s.io/trident-protect-manager-rolebinding           Role/trident-protect-manager-role           36h

 

Now everything should be properly configured to test data management operations with our two tenant users, which we’ll do in the next section.

Testing data management operations—protect an application

Because the following tests of basic data management operations with our tenant users make heavy use of user impersonation commands, let’s first add some handy aliases to save us some typing effort:

 

$ alias k-mkt='kubectl --as system:serviceaccount:mkt-1:mkt-user'
$ alias k-eng='kubectl --as system:serviceaccount:eng-1:eng-user'

 

The tridentctl-protect CLI also supports user impersonation, so we also add aliases to execute tridentctl-protect commands as user eng-user and mkt-user:

 

$ alias pctl-mkt='tridentctl-protect --as system:serviceaccount:mkt-1:mkt-user'
$ alias pctl-eng='tridentctl-protect --as system:serviceaccount:eng-1:eng-user'

 

We’ll run most of the operations by the eng-user and the eng-user’s applications, but feel free to try with the mkt-user, too.

 

Let’s start with some basic permission tests by trying to create Trident protect applications in different namespaces.

  • The following command tests whether eng-user can create a Trident protect application in namespace eng-1 consisting of the eng-1 namespace. This works as expected:

 

$ pctl-eng create application eng-app --namespaces eng-1 -n eng-1
Application "eng-app" created.

 

  • The following command tests whether eng-user can create a Trident protect application in namespace mkt-1 consisting of the mkt-1 namespace. We expect this to fail:

 

$ pctl-eng create application mkt-app --namespaces mkt-1 -n mkt-1
2024/11/15 10:04:43 applications.protect.trident.netapp.io is forbidden: User "system:serviceaccount:eng-1:eng-user" cannot create resource "applications" in API group "protect.trident.netapp.io" in the namespace "mkt-1"

 

  • The following command tests whether eng-user can list their configured application in their namespace eng-1:

 

$ pctl-eng get applications -n eng-1
+---------+------------+-------+-----+
|  NAME   | NAMESPACES | STATE | AGE |
+---------+------------+-------+-----+
| eng-app | eng-1      | Ready | 52s |
+---------+------------+-------+-----+

 

  • User eng-user listing their configured applications across all namespaces:

 

$ pctl-eng get applications -A
2024/11/15 10:03:09 applications.protect.trident.netapp.io is forbidden: User "system:serviceaccount:eng-1:eng-user" cannot list resource "applications" in API group "protect.trident.netapp.io" at the cluster scope

 

This fails because the users eng-user and mkt-user don’t have permission to list resources in the protect.trident.netapp.io API group at the cluster scope.

Before we can deploy a “real” application into the user namespaces, we (as admin user) need to grant the tenant users the necessary permissions for “their” namespaces.

 

For the eng-user, we do this by, for example, applying this role/role binding combination in the eng-1 namespace:

 

$ kubectl apply -f - <<EOF
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: eng-user-role
  namespace: eng-1
rules:
  - apiGroups: ['*']
    resources: ['*']
    verbs: ['*']
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: eng-user-adminbinding
  namespace: eng-1
subjects:
  - kind: ServiceAccount
    name: eng-user
    namespace: eng-1
roleRef:
  kind: Role
  name: eng-user-role
  apiGroup: rbac.authorization.k8s.io
EOF
role.rbac.authorization.k8s.io/eng-user-role created
rolebinding.rbac.authorization.k8s.io/eng-user-adminbinding created

 

After applying the analog role/role binding combination for the mkt-user, we have these roles and role bindings in the two users’ namespaces:

 

$ kubectl -n eng-1 get roles,rolebindings
NAME                                           CREATED AT
role.rbac.authorization.k8s.io/eng-user-role   2024-11-22T16:48:42Z

NAME                                                                      ROLE                                              AGE
rolebinding.rbac.authorization.k8s.io/eng-user-adminbinding               Role/eng-user-role                                1d
rolebinding.rbac.authorization.k8s.io/engineering-ns-tenant-rolebinding   ClusterRole/trident-protect-tenant-cluster-role   1d

$ kubectl -n mkt-1 get roles,rolebindings
NAME                                           CREATED AT
role.rbac.authorization.k8s.io/mkt-user-role   2024-11-22T16:48:42Z

NAME                                                                      ROLE                                              AGE
rolebinding.rbac.authorization.k8s.io/engineering-ns-tenant-rolebinding   ClusterRole/trident-protect-tenant-cluster-role   1d
rolebinding.rbac.authorization.k8s.io/mkt-user-adminbinding               Role/mkt-user-role                                1d

 

Let’s create a sample application for the further testing in the eng-1 namespace now. As user eng-user, we deploy a simple NGINX application and confirm that the persistent volume claims (PVC) were bound, and the NGINX pod is up and running:

 

$ k-eng apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web-eng-1
  name: web-eng-1
  namespace: eng-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-eng-1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: web-eng-1
    spec:
      containers:
      - image: nginx:latest
        name: nginx
        resources: {}
        volumeMounts:
        - mountPath: /data
          name: data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: nginxdata
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginxdata
  namespace: eng-1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: netapp-anf-perf-premium
EOF
deployment.apps/web-eng-1 created
persistentvolumeclaim/nginxdata created

$ k-eng get all,pvc -n eng-1
NAME                             READY   STATUS    RESTARTS   AGE
pod/web-eng-1-6876bd76c5-52k9b   1/1     Running   0          5m33s
 
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web-eng-1   1/1     1            1           5m33s
 
NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/web-eng-1-6876bd76c5   1         1         1       5m33s
 
NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
persistentvolumeclaim/nginxdata   Bound    pvc-21695ab1-a618-4eb3-a2cc-578c75391716   100Gi      RWO            netapp-anf-perf-premium   5m33s

 

Now we can create the Trident protect application CR web-eng-1 as eng-user and confirm its successful creation:

 

$ pctl-eng create application web-eng-1 --namespaces eng-1 -n eng-1
Application "web-eng-1" created.

$ pctl-eng get application -n eng-1
+-----------+------------+-------+-----+
|   NAME    | NAMESPACES | STATE | AGE |
+-----------+------------+-------+-----+
| web-eng-1 | eng-1      | Ready | 30s |
+-----------+------------+-------+-----+

 

As user eng-user, we can now create a snapshot of our newly created application web-eng-1. We use the Trident protect CLI for this:

 

$ pctl-eng create snapshot --app web-eng-1 --appvault appvault-for-enguser-only -n eng-1
Snapshot "web-eng-1-sb34fn" created.

 

The snapshot completes in a couple of seconds, and we check for its success:

 

$ pctl-eng get snapshot -n eng-1
+------------------+-----------+-----------+------+-------+
|       NAME       |  APP REF  |   STATE   | AGE  | ERROR |
+------------------+-----------+-----------+------+-------+
| web-eng-1-sb34fn | web-eng-1 | Completed | 1m5s |       |
+------------------+-----------+-----------+------+-------+
 
$ k-eng get all,pvc,volumesnapshot -n eng-1
NAME                             READY   STATUS    RESTARTS   AGE
pod/web-eng-1-6876bd76c5-52k9b   1/1     Running   0          111m
 
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web-eng-1   1/1     1            1           111m
 
NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/web-eng-1-6876bd76c5   1         1         1       111m
 
NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
persistentvolumeclaim/nginxdata   Bound    pvc-21695ab1-a618-4eb3-a2cc-578c75391716   100Gi      RWO            netapp-anf-perf-premium   111m
 
NAME                                                                                                                            READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS           SNAPSHOTCONTENT                                    CREATIONTIME   AGE
volumesnapshot.snapshot.storage.k8s.io/snapshot-e2beb230-a2f6-4e38-8baa-8dc26b4fc8c1-pvc-21695ab1-a618-4eb3-a2cc-578c75391716   true         nginxdata                           100Gi         anf-trident-snapclass   snapcontent-cb24d70e-2ee7-4810-9ef2-5cee22a86678   79s            85s

 

When trying a snapshot operation with an AppVault CR we don’t have access to (like appvault-for-mktuser-only as user eng-user), the Trident protect admission webhook will block the snapshot creation:

 

$ pctl-eng create snapshot --app web-eng-1 --appvault appvault-for-mktuser-only -n eng-1
2024/11/15 16:54:05 admission webhook "vappvault-access.protect.trident.netapp.io" denied the request: user does not have access to AppVault 'appvault-for-mktuser-only': no adequate policy found for user. Access denied:

 

Creating a backup as eng-user works the same way; we can also watch its progress with kubectl:

 

$ pctl-eng create backup --app web-eng-1 --appvault appvault-for-enguser-only --data-mover Restic --reclaim-policy Delete -n eng-1
Backup "web-eng-1-wbdrt2" created.
 
$ k-eng get backup web-eng-1-wbdrt2 -n eng-1 -w
NAME               STATE     ERROR   AGE
web-eng-1-wbdrt2   Running           18s
web-eng-1-wbdrt2   Running           20s
web-eng-1-wbdrt2   Running           21s
web-eng-1-wbdrt2   Running           21s
web-eng-1-wbdrt2   Running           21s
web-eng-1-wbdrt2   Running           21s
web-eng-1-wbdrt2   Running           21s
web-eng-1-wbdrt2   Running           21s
web-eng-1-wbdrt2   Running           48s
web-eng-1-wbdrt2   Running           74s
web-eng-1-wbdrt2   Running           74s
web-eng-1-wbdrt2   Running           74s
web-eng-1-wbdrt2   Running           75s
web-eng-1-wbdrt2   Running           75s
web-eng-1-wbdrt2   Running           75s
web-eng-1-wbdrt2   Running           75s
web-eng-1-wbdrt2   Completed           75s
 
$ pctl-eng get backup -n eng-1
+------------------+-----------+-----------+-------+-------+
|       NAME       |  APP REF  |   STATE   |  AGE  | ERROR |
+------------------+-----------+-----------+-------+-------+
| web-eng-1-wbdrt2 | web-eng-1 | Completed | 5m12s |       |
+------------------+-----------+-----------+-------+-------+

 

To complete the protection of the eng-user’s application web-eng-1, we create a protection schedule with daily snapshots and backups, retaining the last two snapshots and backups:

 

$ pctl-eng create schedule --app web-eng-1 --appvault appvault-for-enguser-only --backup-retention 2 --snapshot-retention 2 --granularity Daily --hour 15 --minute 33 --data-mover Restic -n eng-1
Schedule "web-eng-1-5udukz" created.
 
$ pctl-eng get schedule -n eng-1
+------------------+-----------+----------------------+---------+-------+-----+-------+
|       NAME       |    APP    |       SCHEDULE       | ENABLED | STATE | AGE | ERROR |
+------------------+-----------+----------------------+---------+-------+-----+-------+
| web-eng-1-5udukz | web-eng-1 | Daily:hour=15,min=33 | true    |       | 21s |       |
+------------------+-----------+----------------------+---------+-------+-----+-------+

 

Application restores

With snapshots, backups, and a protection policy in place for the eng-user’s application web-eng-1, we can now go through the most common restore scenarios in an environment with RBAC in place.

 

We start with restoring from a snapshot into the original application namespace on the same cluster. We’ll pick a snapshot from the list of available snapshots:

$ pctl-eng get snapshot -n eng-1
+----------------------------+-----------+-----------+--------+-------+
|            NAME            |  APP REF  |   STATE   |  AGE   | ERROR |
+----------------------------+-----------+-----------+--------+-------+
| daily-4ac72-20241116153300 | web-eng-1 | Completed | 1d20h  |       |
| daily-4ac72-20241117153300 | web-eng-1 | Completed | 20h44m |       |
| web-eng-1-sb34fn           | web-eng-1 | Completed | 2d20h  |       |
+----------------------------+-----------+-----------+--------+-------+

We (as user eng-user) pick the snapshot web-eng-1-sb34fn and start a SnapshotInplaceRestore operation from it:

$ pctl-eng create sir --snapshot eng-1/web-eng-1-sb34fn -n eng-1
SnapshotInplaceRestore "web-eng-1-o33c98" created.

The restore completes in a short while, and checking in the application namespace, we confirm that the resources were restored successfully:

$ pctl-eng get sir -n eng-1
+------------------+---------------------------+-----------+-------+-------+
|       NAME       |         APPVAULT          |   STATE   |  AGE  | ERROR |
+------------------+---------------------------+-----------+-------+-------+
| web-eng-1-o33c98 | appvault-for-enguser-only | Completed | 5m58s |       |
+------------------+---------------------------+-----------+-------+-------+

$ k-eng get all,pvc,volumesnapshots -n eng-1
NAME                             READY   STATUS    RESTARTS   AGE
pod/web-eng-1-6876bd76c5-xtb8v   1/1     Running   0          4m39s
 
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web-eng-1   1/1     1            1           4m40s
 
NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/web-eng-1-6876bd76c5   1         1         1       4m40s
 
NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
persistentvolumeclaim/nginxdata   Bound    pvc-21695ab1-a618-4eb3-a2cc-578c75391716   100Gi      RWO            netapp-anf-perf-premium   4m44s
 
NAME                                                                                                                            READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS           SNAPSHOTCONTENT                                    CREATIONTIME   AGE
volumesnapshot.snapshot.storage.k8s.io/snapshot-539ae040-6e35-46ae-a4af-f106f931bd08-pvc-21695ab1-a618-4eb3-a2cc-578c75391716   true         nginxdata                           100Gi         anf-trident-snapclass   snapcontent-74c2a34e-0a96-4f9b-b558-11b9410a31f0   20h            20h
volumesnapshot.snapshot.storage.k8s.io/snapshot-9a92f0c8-8dd3-4749-a9f7-8904a11b4670-pvc-21695ab1-a618-4eb3-a2cc-578c75391716   true         nginxdata                           100Gi         anf-trident-snapclass   snapcontent-2c7ef6eb-5073-4912-acc4-f292f1d3c607   44h            44h
volumesnapshot.snapshot.storage.k8s.io/snapshot-e2beb230-a2f6-4e38-8baa-8dc26b4fc8c1-pvc-21695ab1-a618-4eb3-a2cc-578c75391716   true         nginxdata                           100Gi         anf-trident-snapclass   snapcontent-cb24d70e-2ee7-4810-9ef2-5cee22a86678   2d20h          2d20h

When restoring from backups, it’s worth noting that—as with our current setting—the tenant users don’t have access to the secret associated to their AppVault. A listing of the AppVault content with tridentctl-protect get appvaultcontent is not allowed for the tenant users:

$ pctl-eng get appvaultcontent appvault-for-enguser-only
2024/11/15 17:20:27 error creating AppVault from CR appvault-for-enguser-only (failed to resolve value for accountKey: unable to get secret trident-protect/appvault-for-enguser-only-secret: secrets "appvault-for-enguser-only-secret" is forbidden: User "system:serviceaccount:eng-1:eng-user" cannot get resource "secrets" in API group "" in the namespace "trident-protect")

It only works for a cluster admin:

$ pctl get appvaultcontent appvault-for-enguser-only --show-paths
+----------------+-----------+--------+------------------+---------------------------+--------------------------------------------------------------------------------------------------------------+
|    CLUSTER     |    APP    |  TYPE  |       NAME       |         TIMESTAMP         |                                                     PATH                                                     |
+----------------+-----------+--------+------------------+---------------------------+--------------------------------------------------------------------------------------------------------------+
| pu-aks-northeu | web-eng-1 | backup | web-eng-1-wbdrt2 | 2024-11-15 16:06:30 (UTC) | web-eng-1_f86d6ae8-1131-41c7-9c37-6284c9047c2e/backups/web-eng-1-wbdrt2_8de6353c-7edf-40a3-b831-a11712688642 |
+----------------+-----------+--------+------------------+---------------------------+--------------------------------------------------------------------------------------------------------------+

We’ll show you later in this blog post how this can be changed, if needed.

 

If the backup CRs are still available, the tenant user can still list the backups and start a BackupInplaceRestore operation to the original application namespace in the same way as for a restore from snapshot:

$ pctl-eng get backup -n eng-1
+----------------------------+-----------+-----------+--------+-------+
|            NAME            |  APP REF  |   STATE   |  AGE   | ERROR |
+----------------------------+-----------+-----------+--------+-------+
| daily-4ac72-20241116153300 | web-eng-1 | Completed | 1d20h  |       |
| daily-4ac72-20241117153300 | web-eng-1 | Completed | 20h59m |       |
| web-eng-1-wbdrt2           | web-eng-1 | Completed | 2d20h  |       |
+----------------------------+-----------+-----------+--------+-------+

$ pctl-eng create bir --backup eng-1/web-eng-1-wbdrt2 -n eng-1
2024/11/18 13:58:21 warning: error creating AppVault from CR appvault-for-enguser-only (failed to resolve value for accountKey: unable to get secret trident-protect/appvault-for-enguser-only-secret: secrets "appvault-for-enguser-only-secret" is forbidden: User "system:serviceaccount:eng-1:eng-user" cannot get resource "secrets" in API group "" in the namespace "trident-protect")
BackupInplaceRestore "web-eng-1-tiyzr0" created.

Note that the warning issued by the tridentctl-protect command can be safely ignored. We are performing a BackupInplaceRestore operation, and it is possible a namespace would be missing; that’s why the CLI tries to access the AppVault to figure out if any namespace would have to be created. But because our user doesn’t have that permission, the CLI just prints a warning and proceeds. The restore finishes after a couple of minutes:

$ pctl-eng get bir -n eng-1
+------------------+---------------------------+-----------+--------+-------+
|       NAME       |         APPVAULT          |   STATE   |  AGE   | ERROR |
+------------------+---------------------------+-----------+--------+-------+
| web-eng-1-tiyzr0 | appvault-for-enguser-only | Completed | 2m1s   |       |
+------------------+---------------------------+-----------+--------+-------+

When restoring to a new namespace, on the same or a different cluster, we need to consider that our tenant users don’t have permission to create new namespaces in our setup. Therefore, a cluster admin needs to create the destination namespaces and grant the tenant user the necessary permissions, before a tenant user can successfully restore into a new namespace, from a snapshot or backup.

Assuming the user eng-user needs to restore the application web-eng1 into the (new) namespace eng-1-restore, a cluster admin first needs to create the target namespace:

$ kubectl create ns eng-1-restore
namespace/eng-1-restore created

And the cluster admin needs to grant eng-user the required permission as we did for the original application namespace:

$ kubectl apply -f - <<EOF
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: eng-user-role
  namespace: eng-1-restore
rules:
  - apiGroups: ['*']
    resources: ['*']
    verbs: ['*']
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: eng-user-adminbinding
  namespace: eng-1-restore
subjects:
  - kind: ServiceAccount
    name: eng-user
    namespace: eng-1
roleRef:
  kind: Role
  name: eng-user-role
  apiGroup: rbac.authorization.k8s.io
EOF
role.rbac.authorization.k8s.io/eng-user-role created
rolebinding.rbac.authorization.k8s.io/eng-user-adminbinding created

Now eng-user can restore from a snapshot into the eng-1-restore namespace:

$ pctl-eng create snapshotrestore -n eng-1 --snapshot eng-1/web-eng-1-sb34fn --namespace-mapping eng-1:eng-1-restore -n eng-1-restore
SnapshotRestore "web-eng-1-1rl6mg" created.

$ pctl-eng get snapshotrestore -n eng-1-restore
+------------------+---------------------------+-----------+-------+-------+
|       NAME       |         APPVAULT          |   STATE   |  AGE  | ERROR |
+------------------+---------------------------+-----------+-------+-------+
| web-eng-1-1rl6mg | appvault-for-enguser-only | Completed | 1m54s |       |
+------------------+---------------------------+-----------+-------+-------+

To restore from a backup as eng-user, just follow the same steps and run pctl-eng create backuprestore.

Enable AppVault content listing for tenant users

As mentioned earlier, with the current settings, a tenant user doesn’t have access to the AppVault secret containing the access key to the bucket, so that user can’t list the AppVault content, which is needed, for example, to restore applications in a disaster recovery scenario when the backup CRs are not available anymore.

 

If we want to grant the eng-user access to the secret, we can modify the eng-user-appvault-reader role accordingly:

$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eng-user-appvault-reader
  namespace: trident-protect
rules:
- apiGroups:
  - protect.trident.netapp.io
  resourceNames:
  - appvault-for-enguser-only
  resources:
  - appvaults
  verbs:
  - get
- apiGroups:
  - ""
  resourceNames:
  - appvault-for-enguser-only-secret
  resources:
  - secrets
  verbs:
  - "*"
EOF
role.rbac.authorization.k8s.io/eng-user-appvault-reader configured

Now eng-user can list the AppVault content and doesn’t need to rely on an admin user for this anymore:

$ pctl-eng get appvaultcontent appvault-for-enguser-only
+----------------+-----------+--------+----------------------------+---------------------------+
|    CLUSTER     |    APP    |  TYPE  |            NAME            |         TIMESTAMP         |
+----------------+-----------+--------+----------------------------+---------------------------+
| pu-aks-northeu | web-eng-1 | backup | daily-4ac72-20241118153300 | 2024-11-18 15:35:05 (UTC) |
| pu-aks-northeu | web-eng-1 | backup | daily-4ac72-20241119153300 | 2024-11-19 15:35:04 (UTC) |
| pu-aks-northeu | web-eng-1 | backup | web-eng-1-wbdrt2           | 2024-11-15 16:06:30 (UTC) |
+----------------+-----------+--------+----------------------------+---------------------------+
Spoiler

Note that allowing tenant users to access secret data might compromise the integrity of the bucket. This document is mere guidance on what can be done.

Conclusion

In summary, we explained how the RBAC features of Trident protect help you to gain more granular control over access to Trident protect resources like backups. We also walked you through a multi-tenant example scenario with two users that can only access their resources and showed how applications can be protected and restored in such scenarios.

 

Announcements
All Community Forums
Public