> ## Documentation Index
> Fetch the complete documentation index at: https://docs.walletwall.org/llms.txt
> Use this file to discover all available pages before exploring further.

# BigQuery Serving & Canary

> How a WalletWall surface is promoted from its legacy scheduled source to an owned BigQuery snapshot: shadow verification, agreement checks, the read-only readiness gate, production probes, and rollback.

This page documents the **method** WalletWall uses to move a production surface onto an owned BigQuery
snapshot, as implemented in the migration tooling and runbooks. It is not a log of every rollout; the
dated examples at the end are included only where they explain why a check exists.

> [!NOTE]
>
> The readiness gate is read-only. It observes, judges and reports; it never changes a route's source,
> an IAM grant, a schedule or a table. Switching a surface is a deliberate deployment-configuration
> change made after reading the gate's verdict, one route at a time.

## The promotion path

```mermaid theme={null}
stateDiagram-v2
    accTitle: Promotion path of one surface
    accDescr: A surface starts on its legacy scheduled source. It moves to shadow, where the legacy source still answers and the owned snapshot is read and compared off the response path. After an observation window and a passing readiness gate it is switched to bigquery, where the owned snapshot answers and the legacy path remains the fallback. From any state it can be rolled back to legacy by changing one setting.
    [*] --> Legacy
    Legacy --> Shadow: owned data complete and reconciled
    Shadow --> BigQuery: observation window and readiness gate pass
    BigQuery --> Legacy: rollback (one setting)
    Shadow --> Legacy: rollback (one setting)

    Legacy: legacy — the scheduled source answers, owned path untouched
    Shadow: shadow — legacy answers, owned read compared after the response
    BigQuery: bigquery — owned snapshot answers, legacy is the fallback

    class Legacy input
    class Shadow decision
    class BigQuery output

    classDef input fill:#FAF8F3,stroke:#C9A47A,color:#1E1A14,stroke-width:1.5px;
    classDef decision fill:#C9A47A,stroke:#8B6F47,color:#1E1A14,stroke-width:1.5px;
    classDef output fill:#E6DED2,stroke:#9A9186,color:#1E1A14,stroke-width:1.5px;
```

**1. Build and reconcile the owned data**

The slice's tables are backfilled and reconciled against the public source window by window, and
the route's owned read is compared with captured legacy responses. A slice does not proceed until
reconciliation passes and the API-semantic comparison records which differences are expected.

**2. Run in shadow**

The route is switched to `shadow`. Readers still get the legacy answer. After each response the
owned snapshot is read and compared, at most once per five minutes per instance, and the outcome
is recorded. Shadow is a health signal for the owned path under real traffic, not proof of
exactness.

**3. Observe the scheduler**

The scheduled refresh must complete healthily for an observation window (48 hours, so a six-hourly
pipeline shows eight refreshes), and the serving read must stay inside its byte ceiling under real
data growth.

**4. Pass the readiness gate**

An operator runs the cutover gate for the pipeline. Every gate below must pass on its own
evidence; an observation that cannot be made is reported as unobservable, which is a failure, not a
pass.

**5. Switch, then watch the canary**

The route's selector is set to `bigquery` and the deployment is redeployed. The first 48 hours are
the primary canary: the same gate is re-run on the live deployment, and the previous legacy source
stays warm as the fallback.

**6. Retire the fallback**

A fallback is resilience only while its source is alive. Once a surface has proven itself on owned
data, its selector and legacy path are removed, and an owned read that cannot be served answers
`503` rather than an empty success. Stable Seer's large flows and volume trend reached this step
in November 2026.

## What shadow compares

| Surface | Comparison | Outcomes |
| - | - | - |
| Wallet activity | Address membership of the legacy top-20 against the owned top-30, median and 90th-percentile deltas of transaction counts, window offset | `match`, `mismatch`, `incomparable`, `target_error` |
| Stable Seer large flows | Flows matched by transaction hash, sender, recipient, symbol and amount inside the overlapping time range | same |
| Quantum signature facts | Per-wallet row comparison: the legacy count may exceed the owned count (a known transfer-row overcount) but never fall below it; last-signature dates within a day | same |

`incomparable` is a legitimate outcome, not a failure. When the legacy source covers a different
population (the wallet-activity Dune feed lists labelled exchange wallets, the owned snapshot ranks
all wallets) or has stalled, every comparison is `incomparable` by construction, and the slice's
agreement evidence comes from the reconciliation and API-semantic acts instead. Only allow-listed
numbers and closed-vocabulary outcomes are ever logged; no address or hash leaves the comparison.

## Agreement checks

* **Source reconciliation.** Each loaded window is compared with the public source: row counts per hour
  or per day, shrink floors, identity checks. The gate requires the most recent reconciliation to be a
  `PASS`.
* **API-semantic comparison.** The owned read is compared with captured production responses from the
  legacy path, and the verdict names the expected differences (for example, an ERC-1155 expansion that
  the owned data includes and the legacy feed did not, or tie-order differences among equal amounts).
* **Registry validity.** Surfaces that classify stablecoins check the bundled stablecoin registry
  before they are judged.
* **Chain coverage.** The multi-chain signer snapshot states which chains it covers; a relation that
  can no longer be read affordably is quarantined and named rather than silently dropped.

## The readiness gate

The gate judges a pipeline from BigQuery's own job history, the pipeline's status read, and a
credential-free probe of the public route. Every row below must pass.

| Gate | What it proves |
| - | - |
| Runtime authentication healthy | Since the current deployment became ready, the runtime identity has executed at least one serving read whose statement is byte-for-byte the route's, against exactly the pipeline's serving tables, and none failed |
| Owned read healthy | The serving read's dry-run estimate fits the 64 MiB ceiling and returns at least one row |
| Watermark fresh | The watermark is no older than one cadence plus an hour (7 hours for six-hourly pipelines, 26 hours for daily ones; 74 hours for the daily holder cohort, whose balances trail the day by one refresh) |
| No reconciliation failure | The latest source reconciliation passed |
| Scheduler operating | Enough refreshes completed in the last 48 hours for the cadence, and the newest is recent |
| No unsettled mutation | No refresh statement is still running or pending, and none failed, in the census window |
| Fallback verified | The public route still answers from the legacy path while the runtime reads the owned target beside it (routes that answer one wallet have no public probe and report that explicitly) |
| Runtime read complete | For the volume-trend surface, the full operator result replayed through the runtime's one-page read is identical to what the runtime would serve |
| Slice completeness | The snapshot, newest day, newest flow or cohort ends at the watermark |

The gate reads job history rather than the routes' own telemetry because the instance that reports
telemetry is never the instance that served the route; that lesson was learned when both telemetry-based
checks failed on 2026-10-25 while the job history showed the runtime's reads plainly.

## Production probes

The gate's route probe is a plain `GET` with no credential, no redirects and a bounded body. It judges
the response by the owned marker (`provider: "bigquery"`, or the source header on the list-shaped
route) and by the legacy response shape. Before a switch it must see the legacy shape; during the
canary it must see the owned marker with a watermark inside tolerance. Surfaces that answer for one
wallet have no probe, and the gate says so instead of inventing one.

## Why a failed status read never blesses a deployment

* A scheduled run whose refresh completed but whose status read failed keeps the refresh receipt,
  records the status read as failed and the health check as not run, and exits red. "Completed" and
  "verified healthy" are different states and the run reports which one it reached.
* A gate observation the gate cannot make (an unavailable job census, a missing deployment boundary,
  an incomplete census page) is reported as *unobservable* and fails the gate.
* Shadow telemetry is reported alongside the gate as non-authoritative; it moves no verdict.
* The runtime's result-completeness rule means a truncated read fails closed to the legacy path, so a
  gate cannot pass on a result the runtime would not actually serve.

## Rollback

Rollback is one setting: set the route's selector back to `legacy` (or remove it) and redeploy. Nothing
else moves — no table is dropped, no grant changes, no job is stopped by hand. A refresh in flight
finishes under its lease and budget; its data is simply not served. The scheduler may keep running, so
a rolled-back surface can be re-promoted without rebuilding its data.

A route whose fallback has been retired has no selector. Rolling it back means promoting the previous
deployment; there is no other source to switch to.

## Dated examples that shaped the method

> [accordion] 2026-10-17 — a serving read outgrew its ceiling
>
> The large-flows read scanned nine days of partitions and crossed 64 MiB after an on-chain burst.
> The refresh had completed; only the read was refused, and the legacy path answered. The fix was to
> bound the read to three days with a constant the planner can prune on and to tighten the staleness
> tolerance to two days — never to raise the ceiling.

> [accordion] 2026-10-24 — a whole result or nothing
>
> The volume-trend read returned 671 rows; the runtime asked for a 500-row page and served the first
> page as a success, so five stablecoins were missing from the series until the route was rolled back
> to shadow the same day. The runtime client now requires every
> response to hold its whole result (the reported row total must equal the rows returned and no page
> token may be present) and the gate replays the operator's full result through the same code.

> [accordion] 2026-10-20 — a source relation quarantined
>
> One chain's public transaction relation lost partition pruning overnight, which priced every read of
> it as a scan of its full history. The relation was quarantined, the hourly guard was split per
> chain so one relation's pricing refuses only its own statement, and the snapshot states its
> covered chains.

> [accordion] 2026-10-30 and 31 — refusals before overspend
>
> The daily signer-snapshot rebuild exceeded its run budget as the snapshot grew. The planner refused
> before sending anything, the previous snapshot remained the serving state, and the budget and
> publication ranges were revised. A refusal that publishes nothing is the intended failure mode.

## Where to go next

> [card] [Data Serving Architecture](https://docs.walletwall.org/architecture/data-serving-architecture.md)
>
> The request-time contracts, labels and fallback behaviour the gate protects.

> [card] [Data Foundation Refresh Model](https://docs.walletwall.org/architecture/data-foundation-refresh.md)
>
> The lease, watermark and budget machinery behind every refresh the gate judges.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.