Skip to main content
This page documents the method WalletWall uses to move a production surface onto an owned BigQuery snapshot, as implemented in the migration tooling and runbooks. It is not a log of every rollout; the dated examples at the end are included only where they explain why a check exists.
The readiness gate is read-only. It observes, judges and reports; it never changes a route’s source, an IAM grant, a schedule or a table. Switching a surface is a deliberate deployment-configuration change made after reading the gate’s verdict, one route at a time.

The promotion path

1

Build and reconcile the owned data

The slice’s tables are backfilled and reconciled against the public source window by window, and the route’s owned read is compared with captured legacy responses. A slice does not proceed until reconciliation passes and the API-semantic comparison records which differences are expected.
2

Run in shadow

The route is switched to shadow. Readers still get the legacy answer. After each response the owned snapshot is read and compared, at most once per five minutes per instance, and the outcome is recorded. Shadow is a health signal for the owned path under real traffic, not proof of exactness.
3

Observe the scheduler

The scheduled refresh must complete healthily for an observation window (48 hours, so a six-hourly pipeline shows eight refreshes), and the serving read must stay inside its byte ceiling under real data growth.
4

Pass the readiness gate

An operator runs the cutover gate for the pipeline. Every gate below must pass on its own evidence; an observation that cannot be made is reported as unobservable, which is a failure, not a pass.
5

Switch, then watch the canary

The route’s selector is set to bigquery and the deployment is redeployed. The first 48 hours are the primary canary: the same gate is re-run on the live deployment, and the previous legacy source stays warm as the fallback.
6

Retire the fallback

A fallback is resilience only while its source is alive. Once a surface has proven itself on owned data, its selector and legacy path are removed, and an owned read that cannot be served answers 503 rather than an empty success. Stable Seer’s large flows and volume trend reached this step in November 2026.

What shadow compares

incomparable is a legitimate outcome, not a failure. When the legacy source covers a different population (the wallet-activity Dune feed lists labelled exchange wallets, the owned snapshot ranks all wallets) or has stalled, every comparison is incomparable by construction, and the slice’s agreement evidence comes from the reconciliation and API-semantic acts instead. Only allow-listed numbers and closed-vocabulary outcomes are ever logged; no address or hash leaves the comparison.

Agreement checks

  • Source reconciliation. Each loaded window is compared with the public source: row counts per hour or per day, shrink floors, identity checks. The gate requires the most recent reconciliation to be a PASS.
  • API-semantic comparison. The owned read is compared with captured production responses from the legacy path, and the verdict names the expected differences (for example, an ERC-1155 expansion that the owned data includes and the legacy feed did not, or tie-order differences among equal amounts).
  • Registry validity. Surfaces that classify stablecoins check the bundled stablecoin registry before they are judged.
  • Chain coverage. The multi-chain signer snapshot states which chains it covers; a relation that can no longer be read affordably is quarantined and named rather than silently dropped.

The readiness gate

The gate judges a pipeline from BigQuery’s own job history, the pipeline’s status read, and a credential-free probe of the public route. Every row below must pass. The gate reads job history rather than the routes’ own telemetry because the instance that reports telemetry is never the instance that served the route; that lesson was learned when both telemetry-based checks failed on 2026-10-25 while the job history showed the runtime’s reads plainly.

Production probes

The gate’s route probe is a plain GET with no credential, no redirects and a bounded body. It judges the response by the owned marker (provider: "bigquery", or the source header on the list-shaped route) and by the legacy response shape. Before a switch it must see the legacy shape; during the canary it must see the owned marker with a watermark inside tolerance. Surfaces that answer for one wallet have no probe, and the gate says so instead of inventing one.

Why a failed status read never blesses a deployment

  • A scheduled run whose refresh completed but whose status read failed keeps the refresh receipt, records the status read as failed and the health check as not run, and exits red. “Completed” and “verified healthy” are different states and the run reports which one it reached.
  • A gate observation the gate cannot make (an unavailable job census, a missing deployment boundary, an incomplete census page) is reported as unobservable and fails the gate.
  • Shadow telemetry is reported alongside the gate as non-authoritative; it moves no verdict.
  • The runtime’s result-completeness rule means a truncated read fails closed to the legacy path, so a gate cannot pass on a result the runtime would not actually serve.

Rollback

Rollback is one setting: set the route’s selector back to legacy (or remove it) and redeploy. Nothing else moves — no table is dropped, no grant changes, no job is stopped by hand. A refresh in flight finishes under its lease and budget; its data is simply not served. The scheduler may keep running, so a rolled-back surface can be re-promoted without rebuilding its data. A route whose fallback has been retired has no selector. Rolling it back means promoting the previous deployment; there is no other source to switch to.

Dated examples that shaped the method

The large-flows read scanned nine days of partitions and crossed 64 MiB after an on-chain burst. The refresh had completed; only the read was refused, and the legacy path answered. The fix was to bound the read to three days with a constant the planner can prune on and to tighten the staleness tolerance to two days — never to raise the ceiling.
The volume-trend read returned 671 rows; the runtime asked for a 500-row page and served the first page as a success, so five stablecoins were missing from the series until the route was rolled back to shadow the same day. The runtime client now requires every response to hold its whole result (the reported row total must equal the rows returned and no page token may be present) and the gate replays the operator’s full result through the same code.
One chain’s public transaction relation lost partition pruning overnight, which priced every read of it as a scan of its full history. The relation was quarantined, the hourly guard was split per chain so one relation’s pricing refuses only its own statement, and the snapshot states its covered chains.
The daily signer-snapshot rebuild exceeded its run budget as the snapshot grew. The planner refused before sending anything, the previous snapshot remained the serving state, and the budget and publication ranges were revised. A refusal that publishes nothing is the intended failure mode.

Where to go next

Data Serving Architecture

The request-time contracts, labels and fallback behaviour the gate protects.

Data Foundation Refresh Model

The lease, watermark and budget machinery behind every refresh the gate judges.