Skip to content
GRPNR.

Initial recommendation — Dual-run parity harness

2026-07-18 · Author: Fable (pre-research expert judgment). Status: challenged and superseded in part — see the post-research addendum at the end and executive-recommendation.md, which supersedes this document where they disagree.

Context inherited, not re-derived: the harness is already decided BUILD-flagship (plans/001-factory-apps-validation/00-SYNTHESIS.md, verdict #4 + architect addendum #4). This document is the product-shape opinion the research must attack.

1. What the product fundamentally is

An evidence machine for turning legacy OFF. Not a test framework, not a diff platform, not QA tooling in the generic sense. The program’s binding constraint is verification (analysis/legacy/SYNTHESIS.md §3: “the hard part is not code generation — it’s verification”), and the program’s currency is receipts (Zaruba’s audit law: every number carries source + window + confidence; falsification recipes ship with every audit). The harness’s product is therefore a machine-checked parity verdict with a replayable counterexample, per capability, wired to the Definition-of-DONE rows that pay the program’s bonuses.

Everything else — recorders, legacy lab, adapters, diff engines — is plumbing behind that one deliverable.

2. Who it serves first

The Orders flagship M2 gate. Zaruba’s schedule is explicit (reasons-and-decisions/03-orders-flagship.md): dual-run live by week 6, reads authoritative in M2, write ramp gated on ledger reconciliation ≥99.999%, money chain OFF inside M4. The 8-consecutive-week ledger proof is why dual-run starts in month 1.

  • Primary users: fleet agents (they consume verdicts mechanically via work-ledger rows) and the strike unit’s builders (they consume evidence bundles when a divergence blocks a unit).
  • Economic buyer: the program itself — gates pay bonuses only if legacy actually turns off; the harness is what makes “off” provable.
  • Secondary users: QSA/SOX sign-off (DONE row 9 requires auditor-grade evidence), Tomas as scope-contract owner.

3. The single most important customer problem

“Prove the new system behaves like the old one, when the old one is a 2013–2017 EOL estate nobody can safely run, and do it fast enough that the proof is not the schedule.”

Today’s gap, measured: parity machinery exists for exactly one domain slice (merchant-accounting’s reconciliation + cross-check oracles, ~300 LOC total); there is no characterization kernel, no legacy-runtime lab, no MBUS event comparison, and ASSESSMENT.md lists “no parity/dual-run harness” as a top-5 gap. Meanwhile 16/33 legacy repos are MBUS-coupled and the estate’s stacks (Ruby 1.9.3–2.6, Rails 3.2, JRuby 1.7) cannot be casually booted to observe behavior.

4. Simplest useful version

Four pieces, nothing more:

  1. Characterization kernel — record → normalize → replay → diff. Deterministic, content-addressed fixtures.
  2. Legacy-runtime lab — pinned EOL Docker images booting ONE endpoint chain (an Orders read path or one small long-tail Ruby service), enough to record ground truth.
  3. Orders adapter — generalizes Zaruba’s existing 3-oracle reconciliation pattern (feed vs ledger vs consumption anti-join) behind the kernel’s verdict interface instead of inventing a new comparison model.
  4. Verdict rows in work-ledger — PARITY | DIVERGED(counterexample) | UNVERIFIABLE(reason) per unit. The Encore surface is rows only; kernel + lab live outside Encore as a workspace package + CLI/CI runner (architect correction #4 — Cloud Run cannot run pinned EOL images or tap MBUS).

5. Primary user journey

Agent claims unit from work-ledger → runs harness verify <unit> → harness replays recorded traffic against legacy lab AND the new service → normalized diff → verdict:

  • parity ≥ threshold: unit promoted to PR lane, verdict row + evidence link attached.
  • parity < threshold: unit blocked, smallest counterexample attached, question minted into triage.

The human journey is the exception path: a builder opens the evidence bundle, sees the counterexample, decides “real regression / acceptable difference / add normalization rule,” and the decision is recorded (normalization rules are themselves auditable scope decisions).

CLI-first, CI-native, machine-readable. One command surface (harness record | verify | explain), JSON verdicts for agents, one human-readable evidence bundle per divergence. No interactive UI in v1 — the fleet is the main user and it reads structs, not screens. Human surface = the evidence bundle plus one Grafana panel (parity coverage / divergence rate) fed from ledger rows.

7. UX and UI direction

The diff report is the product’s face; design effort goes there, not into chrome.

  • Ive lens (focus, reduction): one verdict per screen-height. Verdict first, then the smallest failing counterexample, then everything else behind links. Complexity (normalization, replay mechanics) handled by the system, never exported to the reader.
  • Norman lens (affordances, feedback, error prevention): one obvious command; deterministic, idempotent output; the ledger makes the safe path the only path — a unit cannot reach the PR lane without a verdict row, so nobody has to remember to verify. Errors are prevented structurally, not by discipline.
  • Rams lens (honest, less-but-better): the report states its own limits — a normalization ledger section listing exactly what was ignored (timestamps, UUIDs, ordering) and why, and a replay recipe so any reader can falsify the verdict. Honesty and falsifiability are Zaruba’s audit law applied to our own tool; a parity report that hides its exclusions is a lie with good typography.

8. What should be removed from the initial concept

  • Standalone MBUS bridge — already dropped in 001; shadow-taps live inside adapters only.
  • Generic multi-domain platform ambition — per-domain adapters are the product; the shared kernel stays minimal and grows only per flagship gate.
  • Live production traffic shadowing in v1 — recorded-fixture replay first (assumption A4 fallback is already the plan of record); live shadow only where the ledger proof demands it, and that machinery may belong to Zaruba’s plane, not ours.
  • LLM-judge semantic equivalence as verdict authority — deterministic diff decides; an LLM may triage divergences (cluster, explain), never pass them.
  • Any Encore-hosted runtime for kernel/lab — settled by architect correction #4.

9. What should not be in the MVP

Embedding-based divergence clustering; auto-generated adapters; DB-state diffing beyond the oracle pattern already in the repo; multi-region replay; performance parity (the p99 ≤ +10% DONE row is production telemetry owned by the program, not a harness function — the harness proves functional parity); Teradata re-point verification (M2–M4 program work); UI beyond the evidence bundle.

10. Biggest usability risk

Diff noise → false DIVERGED → trust collapse → bypass. If timestamps, generated IDs, ordering, and float formatting drown the signal, agents burn tokens on phantom divergences, humans stop reading bundles, and the fleet routes around the harness — at which point the program is back to vibes at 99.999% stakes. Noise normalization is therefore kernel v0 scope, not polish. Second risk: legacy-lab boot pain making recording so slow that fixtures go stale and nobody refreshes them.

11. Fastest prototype that tests the concept

Days, not weeks: pin one EOL image, boot one endpoint chain in the lab, record ~100 requests through a recording proxy, replay against a deliberately-wrong stub, and produce the verdict + counterexample bundle. This tests the two riskiest assumptions at once — (a) EOL runtimes are bootable/recordable at acceptable cost, (b) the noise-normalization model produces zero false DIVERGED on a self-replay (record → replay against the same legacy instance must yield PARITY; that self-test is the harness’s own falsification recipe).

12. Platform

Workspace package (TypeScript) + CLI + CI runner, outside Encore; Docker for the lab; Encore surface = verdict rows in work-ledger; one Grafana panel. Not web, not mobile, not desktop, not conversational. API-first only in the sense that verdicts are structs.

13. Assumptions the research must attack

  1. Build-from-scratch kernel vs foundation — Scientist (GitHub, Ruby — same language as the legacy estate), Diffy (Twitter), GoReplay/Speedscale/Keploy, VCR/WireMock-class record-replay: does any of them cover >40% of the kernel honestly?
  2. Recorded replay vs live shadow — does the 99.999% ledger proof require live dual-run machinery, and if so, whose lane is it (harness vs Zaruba’s oracles)?
  3. Event parity ownership — MBUS shadow-taps in our adapters vs leaning entirely on Zaruba’s per-domain oracles.
  4. EOL lab feasibility — can Rails 3.2 / JRuby 1.7 / Ruby 1.9.3 stacks actually be pinned and booted (TLS-to-rubygems rot, native-gem builds, old JDKs), and at what setup cost per repo?
  5. The noise problem’s real size — published evidence from Diffy-class deployments on false-positive rates and what killed adoption.

Post-research addendum (2026-07-19)

The research ran (6 discovery agents, 13 documents, 3 adversarial audits). Verdict on the five assumptions above and the document as a whole:

  • §13.1 (foundation): confirmed BUILD on OSS components — no tool covers the kernel; adopt mitmproxy + lab base images, pattern-only from Diffy/ApprovalTests/Scientist. (open-source-analysis.md, build-vs-buy.md §4.)
  • §13.2 (recorded vs live): the load-bearing correction. Recorded replay cannot honestly clear the 99.999% money gate — every financial-grade precedent ran live (D6 Q4). Money gate = Zaruba’s oracle plane (Option B); the harness does functional/long-tail/DARK parity + the verdict/evidence wrapper + the two missing oracles. §1’s product identity stands; §8’s “recorded replay first” default was half-wrong, and the wrong half was load-bearing (research.md §1.1 A1).
  • §13.3 (event parity): shadow-taps stay harness-internal per-domain; confirmed zero OSS prior art — the top-risk build line.
  • §13.4 (EOL lab): feasible-but-unproven; “days not weeks” holds for the easy pilot, not for the 6-runtime Orders chain. Pilot ladder + rip-cord installed (validation-plan.md E2).
  • §13.5 (noise): nobody ever published a false-positive rate; the self-replay test in §11 is the only evidence source available anywhere, now the kernel’s ship-gate.
  • Mission reframe: from “prove parity” to “make the legacy-off decision safe and defensible” (Frame B), with two verdict tiers for the two customers (fleet agent vs program gate). Full reasoning: research.md §1.2, §3.5; final call: executive-recommendation.md.