Use every NetApp Volumes endpoint with Google Cluster Toolkit

okrause

Large file systems rarely serve one client at a time. EDA farms, AI training environments, HPC clusters, and general-purpose VM fleets can all place thousands of concurrent operations on shared storage. When a Google Cloud NetApp Volumes file system exposes multiple NFS endpoint IP addresses, mounting every client through one address leaves the other endpoints unused and concentrates connections on a single path.

Starting with Cluster Toolkit v1.104.0, this is solved with Google Cloud DNS. The integration creates a private Cloud DNS managed zone in your project and, for volumes Cluster Toolkit provisions, manages the A records automatically: it reads every NFS endpoint IP from the volume, publishes them as one round-robin A record, and passes the resulting hostname to clients as the NFS server address. Different clients can resolve the name to different endpoints while continuing to access the same data through the same NFS export.

This capability is especially important for Flex Unified large-capacity volumes, but it is not limited to them. Large-capacity Premium and Extreme volumes can also expose multiple endpoints, and the same DNS pattern works for pre-existing volumes and FlexCache.

One export, multiple endpoints

A multi-endpoint volume presents the same NFS export through several IP addresses. The endpoints do not contain separate copies of the file system: they are multiple paths into the same managed volume.

Figure: clients resolve one private hostname. Google Cloud DNS answers with every volume endpoint IP. All endpoints serve the same NFS export.

Without Cloud DNS, a blueprint needs one server address for the mount command. Selecting the first IP technically works, but all clients then use that endpoint. A Google Cloud DNS A record can represent every endpoint under a single name, for example homefs.storage.example.internal mapping to 10.10.0.10, 10.10.0.11, 10.10.0.12, and any additional endpoints the volume exposes.

The domain in this example is arbitrary. Cluster Toolkit uses Google Cloud DNS private zones; you choose the DNS suffix. Use any suffix that fits your naming standards, provided that the private zone is visible from every VPC in which clients need to mount the volume. gcnv.internal. is used in Cluster Toolkit examples, but it is not required or reserved.

Google Cloud DNS rotates the order of the A records it returns. Client resolvers and their caches influence the final distribution, so round-robin DNS is not a load balancer and does not promise a perfectly even split. In practice, it lets a large client population establish mounts through more than one volume endpoint without requiring separate blueprints or mount definitions for each IP.

What Cluster Toolkit automates with Google Cloud DNS

Cluster Toolkit does not run its own DNS service. It provisions and updates Google Cloud DNS resources in your project:

• The dns-managed-zone module creates a private Cloud DNS managed zone attached to the client VPC.
• For volumes created by netapp-volume, enabling dns_config is enough. The module creates and maintains the Cloud DNS A record automatically—you do not list endpoint IPs, write gcloud dns commands, or keep a separate record set in the blueprint.
• Client modules never see the IP list. They receive a single hostname from network_storage.server_ip and mount that name.

There are two ways to use this, depending on who provisions the storage:

1. Volumes created by Cluster Toolkit — Point dns_config at the private Cloud DNS zone. The volume module discovers every endpoint IP from the NetApp Volumes API, writes one A record containing all of them, and keeps that record aligned with the volume as Terraform applies. Downstream modules that use: the volume mount the hostname.
2. Pre-existing volumes and FlexCache — Those endpoints are not owned by netapp-volume, so you supply the IP list once on the Cloud DNS zone (recordsets) and set the hostname as server_ip on pre-existing-network-storage. Cloud DNS still answers the name; Cluster Toolkit does not invent IPs that it did not provision.

The first path is the integrated path: volume, Cloud DNS A record, and client mounts share one Terraform graph. Cluster Toolkit derives the mount hostname from the completed Cloud DNS record resource, so the A record exists before VM or scheduler modules attempt to mount it.

Slurm is a prominent beneficiary because its controller and compute nodes can all receive storage through module use: relationships. The mechanism is not Slurm-specific, however. Any Cluster Toolkit module that consumes the standard network_storage output can use the same hostname.

Create a Flex Unified large-capacity volume

The following excerpt is based on examples/netapp-volumes-slurm.yaml. Although that blueprint demonstrates Slurm, the network, DNS, pool, and volume modules are equally applicable to other VM-based environments.

First, create a private Cloud DNS zone attached to the client VPC:

  - id: gcnv_dns_zone
    source: modules/network/dns-managed-zone
    settings:
      zone_name: $(vars.deployment_name)-gcnv-dns
      # This is an example. Choose any suitable private DNS suffix.
      dns_name: gcnv.internal.
      visibility: private
      network_id: $(network.network_id)
      description: Private DNS for NetApp volume NFS endpoints

Cloud DNS zone names are fully qualified, so dns_name ends with a dot. The zone must be private: these endpoint addresses are reachable through private networking and should not be published in public DNS.

Next, create the Flex Unified scale-out pool and large-capacity volume:

  - id: netapp_pool
    source: modules/file-system/netapp-storage-pool
    use: [network, private_service_access]
    settings:
      pool_name: $(vars.deployment_name)-netapp-pool
      service_level: FLEX
      region: $(vars.region)
      zone: $(vars.zone)
      scale_type: SCALE_TYPE_SCALEOUT
      capacity_gib: 12288
      total_throughput_mibps: 256
      total_iops: 4096

  - id: homefs
    source: modules/file-system/netapp-volume
    use: [netapp_pool]
    settings:
      volume_name: homefs
      capacity_gib: 4800
      large_capacity_config:
        constituent_count: 48
      local_mount: /home
      protocols: [NFSV3]
      dns_config:
        managed_zone_name: $(gcnv_dns_zone.zone_name)

dns_config is the switch that turns on automatic Cloud DNS management. You name the private managed zone; Cluster Toolkit does the rest. Instead of returning the first endpoint IP as network_storage.server_ip, the volume module:

1. Reads all NFS endpoint IPs from the provisioned NetApp volume.
2. Creates a Google Cloud DNS A record in that private zone with every address (round-robin).
3. Returns the Cloud DNS hostname as the mount server.
4. Ties client mount outputs to that Cloud DNS record so Terraform does not mount before the A record exists.

If the volume’s endpoint set changes on a later apply, Terraform updates the same Cloud DNS A record. Clients keep mounting one hostname; they do not need a new IP list.

Any VM, image-building, or scheduler module that uses homefs receives the hostname automatically:

  - id: compute_nodes
    source: modules/compute/vm-instance
    use: [network, homefs]
    settings:
      # VM settings omitted

No list of addresses is needed on the client. Cluster Toolkit owns the Cloud DNS A record for this volume, so the client mount definition stays the same if the endpoint set changes and Terraform updates the record.

Choose a DNS-safe record name

If dns_config.record_name is omitted, Cluster Toolkit uses volume_name as the DNS label. That works for a volume such as homefs, but Flex Unified volume names can contain underscores, while DNS hostnames cannot.

Set an explicit DNS-safe name when the volume name includes an underscore:

  - id: scratchfs
    source: modules/file-system/netapp-volume
    use: [netapp_pool]
    settings:
      volume_name: eda_scratch
      capacity_gib: 4800
      large_capacity_config:
        constituent_count: 48
      local_mount: /scratch
      dns_config:
        managed_zone_name: $(gcnv_dns_zone.zone_name)
        record_name: eda-scratch
        ttl: 60

The module validates the effective record name during planning, so an underscore is reported before the Cloud DNS API is called. Every record name in a zone must also be unique.

Connect pre-existing volumes and FlexCache

Storage often has a longer lifecycle than compute. It may be provisioned by a storage team, managed in a separate Terraform deployment, or created as a FlexCache for data held in another environment. Cluster Toolkit still uses Google Cloud DNS for the hostname, but it cannot discover endpoint IPs from a volume it did not create. Place those addresses on the private Cloud DNS zone yourself and mount the hostname with pre-existing-network-storage.

vars:
  # Hostname suffix used by clients; omit the trailing dot here.
  storage_dns_name: storage.example.internal
  homefs_server_ips:
  - 10.10.0.10
  - 10.10.0.11
  - 10.10.0.12

deployment_groups:
- group: storage
  modules:
  - id: storage_dns_zone
    source: modules/network/dns-managed-zone
    settings:
      zone_name: $(vars.deployment_name)-storage-dns
      # Cloud DNS requires a trailing dot on the zone name.
      dns_name: $(vars.storage_dns_name).
      visibility: private
      network_id: $(network.network_id)
      recordsets:
      - name: homefs
        type: A
        ttl: 60
        rrdatas: $(vars.homefs_server_ips)

  - id: homefs
    source: modules/file-system/pre-existing-network-storage
    settings:
      # NFS clients receive a hostname without a trailing dot.
      server_ip: homefs.$(vars.storage_dns_name)
      remote_mount: /homefs
      local_mount: /home
      fs_type: nfs
      mount_options: "hard,rsize=262144,wsize=262144,vers=3,tcp,mountproto=tcp"

The community/examples/eda/eda-hybrid-cloud.yaml blueprint applies this pattern to tools, libraries, home directories, and scratch storage. Those file systems can be pre-existing NetApp Volumes or FlexCache exports.

The trailing-dot detail

DNS configuration and NFS mounts use slightly different forms of the same name:

• Cloud DNS zone: storage.example.internal. — fully qualified, with a trailing dot.
• NFS mount server: homefs.storage.example.internal — no trailing dot.

Some NFS clients and mount helpers can fail or behave unexpectedly when the server name ends in a dot. Cluster Toolkit removes the trailing dot from generated mount hostnames. When you configure pre-existing-network-storage directly, use the no-dot form yourself.

The record label passed through dns_config.record_name is relative to the selected managed zone. For example, both record_name: homefs and record_name: homefs. are normalized and combined with the managed zone suffix; users should normally use the simpler homefs form.

Operational considerations

Private-zone visibility

Attach the private zone to every VPC that contains clients, or use your organization’s DNS forwarding and peering design to make it resolvable from those networks. A correct A record is not enough if the client cannot query the private zone.

DNS caching

The module defaults to a short TTL suitable for distributing new mounts. DNS determines which endpoint a client selects when it resolves the name; it does not continuously rebalance an established NFS mount. Resolver caches, local caches, and the timing of mount operations therefore affect how clients spread across endpoints.

Endpoint health

Round-robin DNS publishes addresses; it does not probe endpoint health or remove an unhealthy address dynamically. Use the endpoint set supplied by NetApp Volumes and manage manually defined records as part of the storage lifecycle.

Protocol support

The Cluster Toolkit netapp-volume module supports NFSv3 and NFSv4.1. It does not provision SMB or iSCSI clients. pre-existing-network-storage remains the integration point for storage provisioned outside the module’s supported product surface.

Get started

1. Upgrade to Cluster Toolkit v1.104.0 or later.
2. Confirm that your volume type exposes multiple NFS endpoint addresses. Flex Unified large-capacity volumes are the primary use case; large-capacity Premium and Extreme volumes can use the same design.
3. Choose a private Cloud DNS suffix that follows your organization’s naming and network-resolution standards.
4. For a volume created in the blueprint, add dns-managed-zone and set dns_config on netapp-volume. Cluster Toolkit then creates and updates the Cloud DNS A record automatically.
5. For existing storage or FlexCache, publish endpoint IPs on the same Cloud DNS zone with recordsets and pass the no-dot hostname to pre-existing-network-storage.
6. Start with examples/netapp-volumes-slurm.yaml for an end-to-end Flex Unified example or community/examples/eda/eda-hybrid-cloud.yaml for pre-existing storage.

The client interface stays simple—a single NFS hostname—because Cluster Toolkit uses Google Cloud DNS and, for toolkit-provisioned volumes, manages the A records automatically. It works with Slurm, but it is a general storage integration pattern for any Cluster Toolkit environment that mounts NFS at scale.

For the two ways to attach NetApp Volumes to a cluster, and when to provision in the blueprint versus mount existing storage, see NetApp Volumes and Google Cluster Toolkit: Two Ways to Integrate Shared Storage. That overview will be updated for v1.104.0.