In Part 1 of this series, we introduced how NetApp Harvest provides a unified observability layer for Google Cloud NetApp Volumes (GCNV) Flex Unified storage pools with ONTAP-mode enabled. In Part 2, we connected Harvest to Google Cloud Monitoring (Google Managed Prometheus) using the OpenTelemetry Collector to unlock 10-second real-time volume telemetry and PromQL alerting.
In this third installment, we will show you how to eliminate the monitoring VM entirely and run NetApp Harvest and the OpenTelemetry Collector as a single, self-healing serverless container on Google Cloud Run.
Why Cloud Run Instead of a VM?
In a traditional on-premises or IaaS environment, Harvest runs on a VM inside your private network so it can reach the ONTAP cluster management LIF. However, GCNV Flex Unified (ONTAP-mode) works differently: it exposes its ONTAP REST APIs securely through Google's control-plane proxy endpoint (netapp.googleapis.com) and authenticates using standard Google Cloud IAM OAuth 2.0 Bearer tokens.
Because no private network peering or static ONTAP passwords are required to poll the API, Google Cloud Run is a natural fit:
- Zero VMs to Manage: No guest OS patching, SSH keys, disk sizing, or Docker daemon maintenance.
- No Persistent Disk Required (No Historical Data Wiped on Restarts): A common question with Cloud Run is whether container restarts wipe out historical performance data. Because Harvest acts purely as a stateless poller (holding only the latest poll snapshot in memory on :2000/metrics) and OpenTelemetry streams every 10-second scrape directly into Google Cloud Monitoring, all previously ingested metrics are safely persisted in Google Managed Prometheus (GMP) for 24 months—stored at full granularity for 1 week, downsampled to 1-minute points for the next 5 weeks, and downsampled to 10-minute points for the remainder of the 24-month retention period. If Cloud Run replaces an instance, the collector restarts automatically within seconds; your historical time series remain intact in GMP, and only the few samples during the brief container restart and initialization window are missed..
- Keyless IAM Authentication: Instead of running the gcloud CLI on a VM or managing exported Service Account keys, the container fetches short-lived OAuth 2.0 tokens directly from the built-in Cloud Run Instance Metadata Server using its attached IAM Service Account.
- No Local Build Toolchain: Using a multi-stage Dockerfile and Google Cloud Build, we copy the pre-compiled binaries straight from the official NetApp Harvest (ghcr.io/netapp/harvest) and OpenTelemetry (otel/opentelemetry-collector-contrib) images—no local Go compiler or Docker installation needed.
Understanding the Serverless Architecture
Inside your Google Cloud project, a single always-on Cloud Run instance (min-instances=1, max-instances=1, CPU always allocated) runs both the NetApp Harvest Poller and the OpenTelemetry Collector Sidecar:
- Metadata Token Refresh (Every 5m): Harvest's credentials_script queries metadata.google.internal to mint a fresh OAuth 2.0 token for its dedicated IAM Service Account (roles/netapp.viewer).
- Harvest Poller (127.0.0.1:2000): Polls the GCNV ONTAP REST API (KeyPerf every 10s, Rest and Ems every 3m, as configured in Part 2) and exposes Prometheus metrics on local loopback.
- OpenTelemetry Collector (:8080 Health Probe): Scrapes 127.0.0.1:2000/metrics every 10 seconds, applies the label-cleaning and collision processors we covered in Part 2, pushes the batch to Google Managed Prometheus (GMP) (roles/monitoring.metricWriter), and serves an HTTP health check on port 8080 for Cloud Run liveness probes.
What Changes from the VM Setup in Part 1 & Part 2?
Before jumping into the commands, here is a quick summary of what changes when moving our setup from a VM to Cloud Run:
Component | VM Setup (Part 1 & Part 2) | Serverless Cloud Run Setup (Part 3) |
|---|
Compute & Build | Linux VM with Go, git clone, make build, and Docker Engine | Serverless Cloud Run container built in the cloud via Cloud Build |
Token Script (gcnv-token.sh) | Calls gcloud auth print-access-token | Queries the Cloud Run Metadata Server (metadata.google.internal) via curl + jq |
Prometheus Bind Address | 0.0.0.0:2000 | 127.0.0.1:2000 (private loopback inside the Cloud Run sandbox) |
TLS Verification | use_insecure_tls: true | use_insecure_tls: false (verifies netapp.googleapis.com TLS certificates) |
OTel Health Check | Not required on Docker VM | Adds health_check extension on 0.0.0.0:8080 to satisfy Cloud Run container probes |
Deploying Harvest on Cloud Run in 4 Steps
You can run the entire deployment directly from Google Cloud Shell—no VM, local Docker daemon, or Go installation required.
Prerequisites:
- A GCNV Flex Unified storage pool with ONTAP-mode enabled.
- Permissions in your GCP project to enable APIs, create a Service Account, and deploy to Cloud Run.
Step 1: Configure Least-Privilege IAM & Enable APIs
Set your project and pool environment variables, enable the required Google Cloud APIs, and create a dedicated, least-privilege Service Account (gcnv-harvest-sa) with read-only access to NetApp Volumes (roles/netapp.viewer) and metric write access to Cloud Monitoring (roles/monitoring.metricWriter):
# 1. Set your environment variables
export PROJECT_ID="your-gcp-project-id"
export REGION="us-central1"
export GCNV_LOCATION="us-central1"
export STORAGE_POOL_NAME="your-flex-unified-pool"
export SERVICE_ACCOUNT_NAME="gcnv-harvest-sa"
export ARTIFACT_REPO="gcnv-monitoring"
export SERVICE_NAME="gcnv-harvest-collector"
gcloud config set project "${PROJECT_ID}"
# 2. Enable required Google Cloud APIs
gcloud services enable \
run.googleapis.com \
artifactregistry.googleapis.com \
cloudbuild.googleapis.com \
monitoring.googleapis.com \
netapp.googleapis.com
# 3. Ensure Cloud Build can stage source archives in your project
export PROJECT_NUMBER=$(gcloud projects describe "${PROJECT_ID}" --format="value(projectNumber)")
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
--member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
--role="roles/cloudbuild.builds.builder"
# 4. Create a dedicated least-privilege Service Account for Cloud Run
gcloud iam service-accounts create "${SERVICE_ACCOUNT_NAME}" \
--display-name="GCNV Harvest Cloud Monitoring Collector"
export SA_EMAIL="${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
# 5. Grant read-only GCNV access and Cloud Monitoring metric writer access
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
--member="serviceAccount:${SA_EMAIL}" \
--role="roles/netapp.viewer"
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
--member="serviceAccount:${SA_EMAIL}" \
--role="roles/monitoring.metricWriter"
Step 2: Prepare the Cloud Run Configuration Files
Create a clean working directory for the container files:
mkdir -p gcnv-harvest-cloudrun/customconf/keyperf
cd gcnv-harvest-cloudrun
1. Metadata Server Token Script (gcnv-token.sh)
In Part 1, we used gcloud auth print-access-token. To keep our Cloud Run container lightweight and fast, we query the built-in Cloud Run Instance Metadata Server directly:
cat << 'EOF' > gcnv-token.sh
#!/bin/sh
TOKEN=$(curl -s -f -H "Metadata-Flavor: Google" \
"http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
| jq -r .access_token)
if [ -n "$TOKEN" ] && [ "$TOKEN" != "null" ]; then
echo "authToken: $TOKEN"
else
echo "Error: Failed to retrieve access token from GCP Metadata Server" >&2
exit 1
fi
EOF
chmod +x gcnv-token.sh
2. Harvest Configuration (harvest.yml.template & customconf/keyperf/default.yaml)
We reuse the 10-second KeyPerf override from Part 2 and template harvest.yml with environment variables so you can deploy or update pools via Cloud Run environment variables without rebuilding the container image:
cat << 'EOF' > customconf/keyperf/default.yaml
collector: KeyPerf
# Order here matters!
schedule:
- counter: 24h
- data: 10s
objects:
Aggregate: aggr.yaml
CIFSvserver: cifs_vserver.yaml
Cluster: cluster.yaml
EthernetSwitchPort: ethernet_switch_port.yaml
FlexCache: flexcache.yaml
LIF: lif.yaml
Lun: lun.yaml
Namespace: namespace.yaml
NFSv3: nfsv3.yaml
NFSv41: nfsv4_1.yaml
NFSv4: nfsv4.yaml
SystemNode: system_node.yaml
Volume: volume.yaml
EOF
cat << 'EOF' > harvest.yml.template
Exporters:
prometheus1:
exporter: Prometheus
addr: 127.0.0.1
port: 2000
Defaults:
use_insecure_tls: false
exporters:
- prometheus1
Pollers:
gcnv-pool-1:
datacenter: ${GCNV_LOCATION}
addr: netapp.googleapis.com/v1/projects/${GCP_PROJECT_ID}/locations/${GCNV_LOCATION}/storagePools/${STORAGE_POOL_NAME}/ontap
gcnv_ontap_mode: true
credentials_script:
path: /opt/harvest/gcnv-token.sh
schedule: 5m
timeout: 10s
collectors:
- Rest
- KeyPerf
- Ems
exporters:
- prometheus1
conf_path: /opt/harvest/customconf:/opt/harvest/conf
EOF
3. OpenTelemetry Collector Config (otel-config.yaml.template)
This uses the exact GMP pipeline we designed in Part 2 (memory_limiter, filter/drop_noisy, transform/collision, resourcedetection, and honor_timestamps: false), and adds the health_check extension on port 8080 so Cloud Run knows the collector pipeline is healthy:
cat << 'EOF' > otel-config.yaml.template
extensions:
health_check:
endpoint: "0.0.0.0:8080"
receivers:
prometheus:
config:
scrape_configs:
- job_name: harvest
scrape_interval: 10s
scrape_timeout: 10s
honor_timestamps: false
static_configs:
- targets: ['127.0.0.1:2000']
processors:
memory_limiter:
check_interval: 10s
limit_percentage: 80
spike_limit_percentage: 15
resourcedetection:
detectors: [gcp]
filter/drop_noisy:
metrics:
exclude:
match_type: regexp
metric_names:
- ".*_labels$"
- "^metadata_.*"
transform/collision:
metric_statements:
- context: resource
statements:
- set(attributes["service.instance.id"], "${STORAGE_POOL_NAME}:2000")
- context: datapoint
statements:
- set(attributes["exported_cluster"], attributes["cluster"])
- delete_key(attributes, "cluster")
batch:
send_batch_size: 500
timeout: 10s
exporters:
googlemanagedprometheus:
project: ${GCP_PROJECT_ID}
metric:
add_metric_suffixes: false
service:
extensions: [health_check]
telemetry:
logs:
level: info
pipelines:
metrics:
receivers: [prometheus]
processors: [memory_limiter, filter/drop_noisy, transform/collision, resourcedetection, batch]
exporters: [googlemanagedprometheus]
EOF
Step 3: Create the Entrypoint & Multi-Stage Dockerfile
Next, create entrypoint.sh to populate the runtime environment variables into /tmp and supervise both processes (exiting immediately if either process fails so Cloud Run automatically restarts the container), along with a multi-stage Dockerfile running as an unprivileged non-root user:
cat << 'EOF' > entrypoint.sh
#!/bin/sh
set -e
echo "==> Initializing GCNV Harvest Collector for Project: ${GCP_PROJECT_ID}, Pool: ${STORAGE_POOL_NAME} (${GCNV_LOCATION})"
envsubst '${GCP_PROJECT_ID} ${GCNV_LOCATION} ${STORAGE_POOL_NAME}' \
< /opt/harvest/harvest.yml.template > /tmp/harvest.yml
envsubst '${GCP_PROJECT_ID} ${GCNV_LOCATION} ${STORAGE_POOL_NAME}' \
< /opt/otel/otel-config.yaml.template > /tmp/otel-config.yaml
echo "==> Starting NetApp Harvest Poller on 127.0.0.1:2000..."
/opt/harvest/bin/poller --poller gcnv-pool-1 --promPort 2000 --config /tmp/harvest.yml &
HARVEST_PID=$!
echo "==> Starting OpenTelemetry Collector (health check on :8080)..."
/otelcol-contrib --config=/tmp/otel-config.yaml &
OTEL_PID=$!
trap "kill -TERM $HARVEST_PID $OTEL_PID 2>/dev/null || true" INT TERM
while kill -0 $HARVEST_PID 2>/dev/null && kill -0 $OTEL_PID 2>/dev/null; do
sleep 2
done
echo "==> A process exited unexpectedly. Triggering Cloud Run container restart..."
kill -TERM $HARVEST_PID $OTEL_PID 2>/dev/null || true
exit 1
EOF
chmod +x entrypoint.sh
cat << 'EOF' > Dockerfile
# Stage 1: Official NetApp Harvest image
FROM ghcr.io/netapp/harvest:latest AS harvest-source
# Stage 2: Official OpenTelemetry Collector Contrib image
FROM otel/opentelemetry-collector-contrib:0.105.0 AS otel-source
# Stage 3: Minimal Alpine runtime image
FROM alpine:3.20
RUN apk add --no-cache curl jq gettext ca-certificates && \
addgroup -S harvest && adduser -S -G harvest -u 10001 harvest
WORKDIR /opt/harvest
COPY --from=harvest-source /opt/harvest/bin ./bin
COPY --from=harvest-source /opt/harvest/conf ./conf
COPY --from=otel-source /otelcol-contrib /otelcol-contrib
COPY gcnv-token.sh /opt/harvest/gcnv-token.sh
COPY harvest.yml.template /opt/harvest/harvest.yml.template
COPY customconf /opt/harvest/customconf
COPY otel-config.yaml.template /opt/otel/otel-config.yaml.template
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /opt/harvest/gcnv-token.sh /entrypoint.sh && \
chown -R harvest:harvest /opt/harvest /opt/otel /tmp
USER 10001
EXPOSE 8080
ENTRYPOINT ["/entrypoint.sh"]
EOF
Step 4: Build & Deploy to Cloud Run
Finally, build the image in the cloud with Cloud Build and deploy the service to Cloud Run:
# 1. Create an Artifact Registry repository
gcloud artifacts repositories create "${ARTIFACT_REPO}" \
--repository-format=docker \
--location="${REGION}" \
--description="NetApp Harvest Collector for GCNV" 2>/dev/null || true
export IMAGE_URI="${REGION}-docker.pkg.dev/${PROJECT_ID}/${ARTIFACT_REPO}/${SERVICE_NAME}:v1"
# 2. Build and push the container image via Cloud Build
gcloud builds submit --tag "${IMAGE_URI}" .
# 3. Deploy to Cloud Run as an always-on singleton collector
gcloud run deploy "${SERVICE_NAME}" \
--image="${IMAGE_URI}" \
--region="${REGION}" \
--service-account="${SA_EMAIL}" \
--cpu=1 \
--memory=1Gi \
--no-cpu-throttling \
--min-instances=1 \
--max-instances=1 \
--port=8080 \
--ingress=internal \
--no-allow-unauthenticated \
--set-env-vars="GCP_PROJECT_ID=${PROJECT_ID},GCNV_LOCATION=${GCNV_LOCATION},STORAGE_POOL_NAME=${STORAGE_POOL_NAME}"
Note: Why these four Cloud Run flags matter:
- --no-cpu-throttling allocates CPU continuously so the 10-second background polling loop never pauses between HTTP health checks.
- --min-instances=1 keeps one warm container running 24/7.
- --max-instances=1 prevents horizontal autoscaling so multiple instances don't continuously poll the same pool.
(Note: During a new revision deployment, Cloud Run briefly starts the new revision before terminating the old one. Because our OpenTelemetry config in Step 2 explicitly pins service.instance.id to ${STORAGE_POOL_NAME}:2000 instead of the random container ID, both revisions write to the exact same GMP time series rather than creating duplicate series streams. If both revisions push a sample within GMP's 5-second minimum sampling window during the brief cutover, GMP simply ignores the single overlapping point and continues cleanly once the old revision stops. You can also pass --scaling=1 or --max=1 to enforce the single-instance cap at the service level.). - --ingress=internal and --no-allow-unauthenticated lock down the service endpoint so it is never exposed to the public internet.
Verify that Harvest and OpenTelemetry are running cleanly by streaming the container logs from Cloud Logging:
gcloud run services logs read "${SERVICE_NAME}" --region="${REGION}" --limit=50
Within 30 seconds, open Google Cloud Monitoring → Metrics Explorer, select Prometheus Target (prometheus_target), and query your 10-second volume metrics and PromQL dashboards exactly as shown in Part 2!
What You Achieved Today!
Congratulations! You have taken the real-time GCNV observability pipeline from Part 1 and Part 2 and made it 100% serverless on Google Cloud Run:
- Zero VMs or Persistent Disks: Fully managed container execution with 24 months of automatic metric retention in Google Cloud Monitoring.
- Enterprise-Grade Security: Keyless OAuth 2.0 token rotation via the Cloud Run Metadata Server, strict TLS verification, non-root execution, and internal-only ingress.
- Choosing Between a VM (Part 1 & 2) and Cloud Run (Part 3): Now that we have two deployment models, the right choice depends on who manages GCNV in your organization:
- Centralized Storage Administrator Team → Compute Engine VM: If a central storage team manages hundreds or thousands of GCNV storage pools across the enterprise, a GCE VM is the best fit. You can easily scale the VM vertically as your footprint grows, benefit from lower compute costs at scale (especially with GCE Committed Use Discounts), and run sideways helpers like the NetApp Harvest MCP Server on the same host.
- GCP Project or Application Team → Cloud Run: In Google Cloud, application and platform teams typically isolate workloads by GCP project, managing only one or two GCNV storage pools per project. Because their goal is workload and IAM isolation rather than massive single-instance scale, a right-sized Cloud Run collector (0.5–1 vCPU) has a trivial cost difference compared to a small VM while eliminating all OS patching and VM maintenance. And if they need helpers like the Harvest MCP Server, they can deploy it as a separate on-demand Cloud Run service that scales to zero and costs $0 when idle.
Catch Up on the Full Series & Get Involved!
Credits
A huge thank you to Pablo García Arévalo, Storage Specialist Customer Engineer at Google Cloud, who authored this installment. Pablo took the pipeline from Parts 1 and 2, designed and validated the Cloud Run deployment end to end.