Google Cloud NetApp Volumes Flex Unified is now a first-class citizen in the Google Cluster Toolkit. Starting with release v1.102.0, the netapp-storage-pool and netapp-volume modules provision Flex Unified pools and volumes directly from a blueprint, including zonal and regional pools, large capacity (scale-out) pools, custom performance, and auto-tiering.
If you build EDA, HPC, or AI/ML clusters with gcluster, you can now describe your shared file systems in the same blueprint that creates your VPC, Private Service Access peering, Slurm cluster, and compute partitions — and you can size performance and capacity independently instead of buying throughput by the terabyte.
Why Flex Unified matters for technical computing
Standard, Premium, and Extreme service levels tie throughput to provisioned capacity: 16, 64, or 128 KiBps per provisioned KiB of volume capacity. That model is predictable, but it forces you to overprovision capacity when you only need more throughput.
Flex Unified breaks that link:
Capability | What it gives you |
|---|
Independent scaling | Set pool capacity, throughput, and IOPS separately. Start small and grow the dimension you actually run out of. |
Zonal or regional pools | Choose a single zone for lowest latency and cost, or a regional pool with an active and a standby zone for zone-failure resilience. |
Large capacity pools | Scale-out pools host large capacity volumes for multi-hundred-terabyte scratch, tool, and library file systems that need throughput well beyond a regular pool. |
Auto-tiering | Keep a hot tier for active data and let cold blocks move transparently to a cheaper tier. EDA tool trees, tape-out archives, and old regression results stop paying hot-tier prices. |
Sub-millisecond read latency | The characteristic that decides whether your job slots wait on metadata or run. |
Broad regional footprint | Flex Unified is available in more Google Cloud regions than Standard, Premium, or Extreme, and that list continues to grow. Check supported regions before you pick a region for your cluster. |
Flex Unified is the service level we want EDA teams to standardize on. This Cluster Toolkit release is an important milestone in that direction: the fastest NetApp Volumes service level for scale-out file workloads is now as easy to deploy as any other toolkit file system. Watch the Flex Unified space over the course of this year — performance characteristics keep improving.
What the modules do now
netapp-storage-pool
Set service_level: FLEX and the module configures Flex Unified for you. It sets mode: DEFAULT and type: UNIFIED automatically, so you never write those in a blueprint.
Pool layout | Blueprint settings |
|---|
Zonal | Set region and zone. Omit replica_zone. |
Regional | Set region, zone (active zone), and replica_zone (standby zone). Both zones must be in region and must differ. |
Large capacity (scale-out) | Set scale_type: SCALE_TYPE_SCALEOUT with region and zone. Scale-out pools are zonal only. |
Custom performance, auto-tiering, and plan-time minimums:
- Set
total_throughput_mibps in 1 MiB/s increments and, optionally, total_iops. Omit total_iops and Google Cloud derives it from throughput. Every pool includes 64 MiB/s and 1,024 IOPS by default; regular Flex pools scale to 5 GiBps and 160,000 IOPS, and large capacity pools go higher. See volume performance sizing. - Set
allow_auto_tiering: true, hot_tier_size_gib, and optionally enable_hot_tier_auto_resize so the hot tier grows when it fills. - Minimum capacity is 1024 GiB for Flex Unified and 6144 GiB for large capacity pools. The module enforces this at plan time, so you find out during
gcluster create instead of during an API error.
netapp-volume
Volumes inherit what the pool already knows. With use: [netapp_pool], Cluster Toolkit wires netapp_storage_pool_id, service_level, type, allow_auto_tiering, and scale_type from the pool automatically. Do not set region, zone, or location on the volume; the module derives location from the pool ID (the zone for a zonal pool, the region for a regional pool).
Topic | Behavior |
|---|
Flex large capacity volumes | Set large_capacity_config with constituent_count (48 is the typical value for current scale-out pools) and at least 4800 GiB. Do not set large_capacity; that flag belongs to Standard, Premium, and Extreme volumes, which now require at least 15 TiB and configure their multiple endpoints internally. |
Auto-tiering per volume | Set tiering_policy with tier_action: ENABLED and cooling_threshold_days. During a data migration, set hot_tier_bypass_mode_enabled: true so writes land on the cold tier instead of flooding the hot tier, then disable it when the migration finishes. |
Deletion policy | Set deletion_policy: PREVENT to block Terraform from deleting a volume, or ABANDON to drop it from state and keep it in Google Cloud. This is the answer to the biggest operational objection against provisioning storage inside a cluster blueprint. |
Volume names | Flex pools accept lowercase letters, numbers, and underscores; they do not accept hyphens. Standard, Premium, and Extreme volumes are the reverse. The module validates names at plan time. |
Both modules now document these behaviors in their READMEs, and NetApp Volumes is described alongside the other file systems in docs/network_storage.md.
Example: zonal Flex Unified pool with auto-tiering
- id: flex_pool
source: modules/file-system/netapp-storage-pool
use: [network, private_service_access]
settings:
pool_name: $(vars.deployment_name)-flex
service_level: FLEX
region: us-east1
zone: us-east1-b
capacity_gib: 4096
total_throughput_mibps: 256
total_iops: 4096
allow_auto_tiering: true
hot_tier_size_gib: 1024
enable_hot_tier_auto_resize: true
- id: home_volume
source: modules/file-system/netapp-volume
use: [flex_pool]
settings:
volume_name: eda_home
capacity_gib: 2048
local_mount: /home
protocols: ["NFSV3"]
deletion_policy: "PREVENT"
tiering_policy:
tier_action: "ENABLED"
cooling_threshold_days: 31
Example: large capacity scratch on a scale-out pool
- id: flex_pool_large
source: modules/file-system/netapp-storage-pool
use: [network, private_service_access]
settings:
pool_name: $(vars.deployment_name)-flex-large
service_level: FLEX
region: us-east1
zone: us-east1-b
scale_type: SCALE_TYPE_SCALEOUT
capacity_gib: 12288
total_throughput_mibps: 1024
total_iops: 16384
- id: scratch_volume
source: modules/file-system/netapp-volume
use: [flex_pool_large]
settings:
volume_name: eda_scratch
capacity_gib: 12288
local_mount: /scratch
protocols: ["NFSV3"]
large_capacity_config:
constituent_count: 48
What this changes for cluster operators
Outcome | How |
|---|
One blueprint, one lifecycle you control | VPC, Private Service Access, pool, volumes, Slurm, and partitions in a single deployment, with deletion_policy: PREVENT protecting the data you cannot afford to lose. Keep using separate deployment groups (base, software_installation, cluster) so you can tear down compute without touching storage. |
Right-sized performance for each file system | Give the tools volume modest throughput, give scratch the scale-out pool, and stop paying for capacity you only bought to buy IOPS. |
Cheaper long tails | Auto-tiering handles the reality of EDA data: a small working set, a very large cold set. Enable it on the pool once and set a cooling threshold per volume. |
Fail-fast blueprints | Plan-time validation of service-level minimums, zone membership, scale-type rules, and volume names means fewer failed applies after a 10-minute deploy. |
Hybrid and FlexCache
NetApp Volumes also offers FlexCache, which caches an on-premises ONTAP origin volume in Google Cloud so cloud compute reads at low latency without copying the whole dataset or paying repeated WAN costs. It is the standard way to burst an existing on-premises EDA flow into Google Cloud.
FlexCache volumes still need manual steps on the ONTAP origin, so the toolkit modules do not create them. Create the cache volume, then mount it in your blueprint with pre-existing-network-storage. The eda-hybrid-cloud blueprint follows exactly this pattern. The same applies to ONTAP-mode pools and volumes: the toolkit modules manage Default mode only.
Get started
- Upgrade to Cluster Toolkit v1.102.0 or later.
- Confirm your region supports Flex Unified and that your project has Flex Unified quota. Quota is tracked separately per service level.
- Connect NetApp Volumes to your VPC with Private Service Access, or let the
private-service-access module do it in the blueprint. - Start from
examples/netapp-volumes.yaml for a minimal end-to-end deployment, or from community/examples/eda for Slurm-based EDA reference architectures. - Read the module READMEs for
netapp-storage-pool and netapp-volume; both now document Flex Unified rules, minimums, and validation in detail.
For background on the service, see the NetApp Volumes overview and service levels. For the two ways to attach NetApp Volumes to a cluster and when to choose each, see NetApp Volumes and Google Cluster Toolkit: Two Ways to Integrate Shared Storage.
