Measuring Oracle Data Guard Fast-Start Failover RTO and RPO on Google Cloud NetApp Volumes
“Zero data loss” and “automatic failover” are common claims for Oracle Data Guard deployments. Those claims only hold weight when validated under real failure conditions, with evidence that would satisfy an Oracle DBA reviewing the numbers — not just an application log claiming success.
This article shares the results of a controlled evaluation of Oracle Database 19c Data Guard with Fast-Start Failover (FSFO) — Oracle’s capability to fail over to a standby database automatically, without manual intervention — running on Google Cloud NetApp Volumes (GCNV) iSCSI storage, across four failure scenarios.
Executive Summary
- What we tested: Oracle 19c Data Guard with FSFO in Zero Data Loss Mode, on two Google Compute Engine VMs backed by GCNV iSCSI storage, across a planned switchover, an instance crash, a VM-level outage, and a network partition.
- Observed RPO: 0 in all four scenarios. Observed RTO: approximately 45–58 seconds, app-visible.
- Why trust these numbers: two of the four scenarios were verified using Oracle’s own internal commit-ordering record — the System Change Number, or SCN, a monotonically increasing value stamped on every committed transaction — rather than relying on application-reported timestamps alone.
Test Environment
Component | Configuration |
|---|
Database version | Oracle Database 19c (19.3.0), Oracle Restart, single-instance (no RAC) |
Replication | Oracle Data Guard Broker with FSFO, Zero Data Loss Mode, 30-second failover threshold |
Protection mode | MaxAvailability with synchronous (SYNC) redo transport |
Compute | Two Google Compute Engine VMs (primary and standby) |
Storage | Google Cloud NetApp Volumes (GCNV), iSCSI-attached |
Volume manager | Oracle ASM (+DATA, +RECO, +FRA disk groups) |
Observer placement | Dedicated Compute Engine VM, independent of primary and standby |
App connectivity | Role-based Oracle service, registered only with the listener on the current primary |
Clock synchronization | All three hosts synced to Google’s internal NTP service; max offset 15 microseconds |
- Broker — manages the primary/standby relationship and its configuration.
- Observer — a separate process that continuously monitors the primary and triggers FSFO automatically if it becomes unreachable.
Architecture: primary and standby databases with synchronous redo transport, a Data Guard Observer, and shared Google Cloud NetApp Volumes storage
Methodology
An evaluation like this is only as credible as its measurement approach.
Environment validation
- Client timestamps are subject to clock skew and network latency, and say nothing about the database’s own transaction order.
- SCN avoids both problems — a value the application can’t influence or misreport.
- That’s why two of the four scenarios below use SCN-based verification rather than timestamps alone; after this point, we simply say “SCN-verified.”
Before injecting any failure, we confirm the environment is healthy:
- Broker configuration: status success, Observer connected, no warnings.
- Standby transport and apply lag: confirm at zero — the actual precondition for RPO = 0.
- Clock synchronization: confirm sub-20-microsecond offsets across primary, standby, and Observer, against Google’s internal NTP service.
Workload generation
- A continuous heartbeat writes a timestamped, sequenced row roughly once per second through the role-based Oracle service, so the client always lands on whichever host holds the primary role.
- For the switchover and instance-crash scenarios, the heartbeat table uses row-level dependency tracking to expose each row’s individual commit SCN.
- Each write captures its local transaction ID before committing, then reads back that row’s commit SCN immediately after, in the same round trip.
- Every failed write logs a distinct error code, distinguishing “no listener reachable” from “listener reachable, but service not yet registered.”
RPO verification
Immediately before each failure, we capture the last known-good row and its commit SCN as a checkpoint. After failover, RPO is confirmed by checking that:
- every acknowledged row is present on the new primary with matching content
- each row’s commit SCN is monotonically consistent with the pre-failure checkpoint (proof that commit order is preserved)
- no in-doubt or orphaned transactions exist
RTO measurement
- RTO is broken into phases, each anchored to an independently verifiable source: the Observer’s detection log, the Broker’s failover-start/succeeded entries, the standby’s recovery-and-open sequence in the alert log, and the heartbeat client’s outage window.
- An independent 1-second poller checks database open mode and listener registration on both hosts, so phase boundaries don’t depend solely on log-flush timing.
- Where two anchors cover the same instant, they are cross-checked against each other.
SCN-based RPO verification workflow: heartbeat transaction, commit, SCN readback, failure injection, automatic failover, and verification
Failure Scenarios
Four failure scenarios tested: planned switchover, instance crash, VM-level failure, and network partition with FSFO split-brain protection
How each failure was injected:
- Planned switchover — administrator-initiated role switch via the Broker
- Instance crash — abrupt Oracle instance termination; host and network stay up
- VM-level failure — primary compute instance stopped outright
- Network partition — primary’s connectivity to both standby and Observer cut, while it keeps running
The switchover and instance crash use the full SCN-verified methodology; the VM-level failure and network partition use row-and-timestamp verification. Both support an RPO = 0 conclusion, but we distinguish the two rigor levels rather than present all four as equally verified.
During the network partition test, direct inspection of the isolated primary (via a management path unaffected by the simulated outage) confirmed it had autonomously demoted itself from an open, writable state to a non-writable mounted state before connectivity was restored — the split-brain protection FSFO is designed to provide.
Results
Results summary: observed RTO by scenario as a bar chart, and observed RPO of zero across all four scenarios
RTO
Scenario | Detection | Threshold wait | Failover execution | App-visible RTO |
|---|
Planned switchover | n/a (planned) | n/a | ~12.8s (role transition) | ~45.5s |
Instance crash | ~1.0s | ~29.9s | ~22.6s | ~57s (56.9–57.7s, two anchors) |
VM-level failure | 5.8s | 30.0s | 13.2s | ~55.6s |
Network partition | ~23s* | ~15s* | 15.7s | ~57.9s |
* A silent packet-drop partition takes longer to detect than an active connection refusal, since TCP has to time out.
RPO
Scenario | Pre-failure checkpoint | Post-failover checkpoint | Commit SCN monotonic? | In-doubt transactions | RPO |
|---|
Planned switchover | seq_id=109, SCN 2732147 | seq_id=121, SCN 2833858 | Yes | 0 | 0 |
Instance crash | seq_id=158, SCN 2835025 | seq_id=161, SCN 2836545 | Yes | 0 | 0 |
VM-level failure | Last acknowledged row, verified present | — | Not captured (row/timestamp only) | Not checked | 0 |
Network partition | Last acknowledged row, verified present | — | Not captured (row/timestamp only) | Not checked | 0 |
Every acknowledged heartbeat row was confirmed present on the new primary in all four scenarios. For the two SCN-verified scenarios, commit SCNs increased monotonically across the failover and zero in-doubt transactions were found — meeting the verification criteria described in the Methodology section above.
Understanding the Observed RTO
A single headline RTO number hides where the time actually goes. Breaking down the ~45–58 second app-visible RTO by phase makes clear which parts of the stack are actually driving the delay:
- Detection (~1–23s): a crash or VM stop is detected almost immediately, since the connection is actively refused. A network partition takes longer (~23s) because a silent packet drop requires a TCP timeout.
- FastStartFailoverThreshold (~15–30s): a deliberate, configurable safety timer — 30 seconds in this environment — that FSFO waits out before acting, to avoid overreacting to a transient blip. This accounts for roughly half the total RTO in the crash and VM-stop scenarios.
- Failover execution (~13–23s): the standby completing redo apply and opening in the primary role.
- Service registration and client reconnect: the role-based service must register on the new primary’s listener, and the application must detect the failure and reconnect. This is the largest source of variance between scenarios.
GCNV sustained synchronous redo write latency low enough that storage never became the bottleneck on either the commit or apply path. But storage was only one input into the RTO. In this evaluation, Oracle’s configurable failover logic — principally the 30-second safety threshold — contributed more to the observed application-visible RTO than storage latency did.
RTO phase timeline from failure occurrence to application available, highlighting the FastStartFailoverThreshold wait and the observed 45-58 second application-visible RTO
Key Takeaways
- Independent timestamp anchors increase confidence. In the instance-crash scenario, the Observer’s detection log and the client’s last successful write agreed within 0.8 seconds, despite being captured by unrelated mechanisms.
- SCN verification is stronger evidence than application timestamps for RPO claims, since it comes from Oracle’s own commit-ordering record rather than anything the client reports.
- Remote command execution adds timing noise. Issuing a failure over a freshly established remote session adds a few seconds of round-trip uncertainty — use the Observer’s detection log or the client’s last successful write as ground truth instead.
- Oracle’s failover threshold, not storage, is usually the largest single contributor to app-visible RTO. Size expectations accordingly rather than assuming faster storage alone will shrink RTO.
- Methodology matters as much as the reported numbers. An RTO or RPO figure is only as credible as the measurement approach behind it.
Conclusion
Oracle Data Guard with Fast-Start Failover in Zero Data Loss Mode maintained an observed RPO of zero across all four failure scenarios tested, running on Google Cloud NetApp Volumes storage — with the switchover and instance-crash results additionally backed by Oracle’s own commit-SCN evidence. Observed RTO ranged from roughly 45 to 58 seconds.
These figures are specific to this test environment and workload. Validate RTO and RPO against your own infrastructure, workload characteristics, and application profile before using these numbers as a planning assumption.
