Skip to content
GRPNR.

Made to Stick — Dual-run parity harness

SUCCESs analysis (template §10). Internal narrative for Tomas, C-level, and future agents reading the repo — not a pitch deck. Deliberately short.

Internal-product remap, stated once: buyer = program gates (Tomas’s Definition-of-DONE rows + Robert’s investment call); procurement = lane approval from Tomas; churn = a builder or agent routing around the harness (bypass); pricing = engineering weeks + agent tokens, not dollars-per-seat. Sections below are reinterpreted through this lens without re-flagging it each time.

One-sentence description

A CLI/CI tool that replays recorded legacy traffic against the new Encore services and returns a machine-checked verdict — PARITY, DIVERGED with a counterexample, or UNVERIFIABLE — so a capability can only reach the PR lane with evidence, never with vibes.

Internal value proposition

The program’s binding constraint is verification, not code generation (analysis/legacy/SYNTHESIS.md §3, confirmed fact). The harness is the only thing standing between “an agent says it’s done” and “the program can prove legacy is off.” The ≥99.999% ledger-reconciliation gate at week 6 is computed by Zaruba’s oracle plane in Core (Option B, architecture.md §2.3) — the harness does not produce that number. Without the harness, that computation still runs, but it has no verdict/evidence mechanism turning it into a PR-lane-gating artifact, and the rest of the surface it doesn’t cover has no mechanism at all — only a person reading two logs.

30-second explanation

Record real requests hitting the legacy stack in a pinned EOL Docker lab. Normalize away the noise that doesn’t matter (timestamps, generated IDs, ordering — logged in a normalization ledger so nobody has to trust that by faith). Replay the same requests against the new Encore service. Diff. Emit a verdict, and if it diverges, emit the smallest failing example plus the exact recipe to reproduce it. A unit cannot enter the PR lane without a verdict row in work-ledger — the safe path is structurally the only path, not a policy someone has to remember.

S — Simple

Commander’s intent, one sentence: No capability goes live until the harness can show, with a replayable counterexample, that it behaves like the system it replaces — or explicitly say it cannot yet tell.

Everything in the initial-recommendation’s four pieces (kernel, legacy-runtime lab, Orders adapter, work-ledger verdict rows) serves that one sentence. If a feature doesn’t produce a verdict or make a verdict trustworthy, it’s out of v1 (initial-recommendation §8–9, binding).

U — Unexpected

The counter-intuitive claim engineers won’t expect: the harness’s job is to catch itself lying. Most verification tooling is trusted by assumption; this one ships its own falsification recipe with every report (initial-recommendation §7, Rams lens) and its own self-test is a fixed point, not a vibe: record → replay against the same legacy instance must yield PARITY. If self-replay doesn’t come back PARITY, the harness is broken, not the migration — and that failure mode is cheap to produce on day one, before a single line of Encore code is judged.

This breaks the usual expectation that a test harness is above suspicion. Here, the harness is guilty until it proves itself innocent on its own recordings, every run.

C — Concrete

The demo, buildable in days (initial-recommendation §11), not weeks:

  1. Pin one EOL image, boot one endpoint chain in the legacy-runtime lab.
  2. Record ~100 real requests through a recording proxy (mitmproxy/GoReplay-class, outside-in capture — chosen because it needs no Gemfile changes on Ruby 1.9.3-era code; DESIGN, from D3).
  3. Replay those 100 requests against the real legacy instance → expect verdict PARITY across all 100. This is the self-replay falsification recipe, not a demo flourish — it’s the harness proving it isn’t lying to itself before it’s trusted to judge anything else.
  4. Replay the same 100 requests against a deliberately-wrong stub (one field renamed, one status code flipped, one field dropped).
  5. Watch DIVERGED appear with the smallest counterexample — not “87/100 differ,” but the one request that isolates the bug, plus the normalization ledger showing what was ignored and why it wasn’t the cause.

Before/after story, told the way it will actually happen at week 6: before the harness, a divergence in Orders write-ramp shows up as a support ticket three days later, after 1% of writes already went to the wrong ledger row. After the harness, the same divergence shows up as a DIVERGED verdict on unit N before the unit is promoted, with the exact request/response pair attached — the builder fixes it before the write ramp ever sees it. The dollar difference between those two stories is the ≥99.999% reconciliation gate itself: one bad unit at 1% ramp is the entire error budget for that day.

C — Credible

Credibility here isn’t a testimonial — it’s a mechanism, matched deliberately to Tomas Zaruba’s own audit law (source + window + confidence on every number; falsification recipes ship with every audit, per 03-orders-flagship.md and the context brief, confirmed fact):

  • Falsification recipes. Every evidence bundle includes the exact replay command a skeptic needs to reproduce or overturn the verdict. A verdict nobody can independently re-run is an opinion, not evidence.
  • Normalization ledger. Every report states what it ignored (timestamps, UUIDs, ordering) and why — the same move Zaruba’s own oracles make when they classify posted/received/unpublished in crossCheckRange rather than silently dropping mismatches (D-context, confirmed fact from the existing ~300 LOC merchant-accounting oracle).
  • Auditor-grade evidence. DONE row 9 requires QSA/SOX sign-off. An evidence bundle that already carries source, window, and confidence tags is the artifact an auditor can sign, not a screenshot of a passing test.
  • The credibility ceiling, stated honestly. External evidence on shadow-diffing tools shows the pattern works but the false-positive rate of the best-known implementation (Diffy) was never published anywhere (D1/D6 Q2, confirmed gap — not filled with invented numbers). We are not claiming precedent proves this harness will hit zero noise; we are claiming the mechanism (falsifiable, self-testing, ledger-honest) is sound regardless, and the self-replay test is how we find out early if it isn’t.

E — Emotional

What it protects: the gates that pay the program’s bonuses. DONE row 8 (ledger reconciliation ≥99.999%) is not a nice-to-have metric — it’s the number the entire M2→M4 schedule is conditioned on, and it is a number nobody can eyeball at that precision. Zaruba’s oracle plane in Core computes that number live (Option B, architecture.md §2.3); the harness is the mechanism that turns it — and every other functional-parity claim in the program — into an auditable verdict instead of review-meeting consensus.

The fear it removes: silent behavioral drift at 99.999% stakes — a fleet of agents rewriting a 1.05M-LOC estate, one of them quietly changes rounding behavior in a money path, and nothing catches it until a merchant’s payout is wrong three weeks later. Every rigorous published dual-run case (Zalando, Uber’s ledger migration, Slack’s Vitess cutover, GitHub’s Scientist) converged on the same lesson: don’t chase 100% parity by hand, define a noise floor with a stated source and gate on it mechanically (D2, confirmed fact, sourced in the case-study notes). The harness is that mechanism for this program; without it, the fear has no answer except “the humans were careful.”

S — Stories

The cautionary tale: two rotting half-stranglers. VIS3 and identity-service are half-finished strangler migrations already sitting in the estate (analysis/legacy/SYNTHESIS.md, confirmed fact) — proof, inside this program’s own codebase, of what happens when a migration loses its verification mechanism partway through and stalls in an unverifiable middle state nobody can finish or roll back cleanly. They are not hypothetical risk; they are the two visible fossils of “we started the cutover and couldn’t prove it was safe to finish.” The harness exists so Orders doesn’t become the third.

The deadline story: the week-6 gate. reasons-and-decisions/03-orders-flagship.md states it as a public commitment, not an internal target: dual-run must be live by week 6 (M2 checkpoint), reads authoritative in M2, and “if dual-run is not live, M4 slips, publicly” (confirmed fact, direct citation). That sentence is the whole business case in one line — there is no partial credit for a harness that arrives in week 8. The fastest-prototype plan in §C above is sized to that constraint: days, not weeks, to prove the two riskiest assumptions (EOL lab bootability, noise-normalization correctness) before the schedule commits real engineering weeks to the full four-piece build.

README headline + subheadline (for the harness repo)

Headline: Prove it, don’t promise it. Subheadline: Replays real legacy traffic against the new services and returns a verdict — PARITY, DIVERGED with a counterexample, or UNVERIFIABLE — so nothing reaches the PR lane on trust alone.

Strongest narrative

The self-replay falsification test (Unexpected + Concrete + Credible braided together): the harness proves it isn’t lying about the old system before anyone lets it judge the new one. That’s the one sentence worth repeating in every review meeting, because it answers the only question a skeptical auditor or a nervous C-level exec actually has — “how do we know the tool grading the migration isn’t itself broken?”

Flagged as abstract or unbelievable

  • The 99.999% self-replay-only claim is optimistic. Every real high-stakes precedent found (Uber, Stripe) achieved that precision with live continuous shadow traffic, not recorded-fixture replay alone (D6 Q4, confirmed fact). Recorded replay is the harness’s v1 mechanism (initial-recommendation §8, binding decision); whether it alone can clear the 99.999% ledger gate, or whether Zaruba’s live oracles have to carry that last mile, is explicitly still open (assumption #2, initial-recommendation §13) — the “before/after” story above is illustrative of the mechanism, not a guarantee this exact number is reachable from fixtures alone.
  • The false-positive/noise-floor number for our own harness does not exist yet. Diffy’s real-world false-positive rate was never published (D1/D6 Q2, confirmed gap). Our “noise normalization is kernel v0 scope” claim is a design commitment, not evidence it will work — the self-replay test is the earliest possible signal, but it is a necessary check, not a sufficient one.
  • The per-repo lab setup cost (2–5 engineer-days first repo, falling under a day for reuse) is MODELED by analogy, not measured on this program’s actual stacks (D3, explicitly flagged as inference, not sourced). Treat any schedule built on it as provisional until the first two adapters land.

Open questions

  1. Does the ≥99.999% ledger-reconciliation gate require live shadow-traffic machinery in addition to recorded-fixture replay, and if so, is that Robert’s harness lane or Zaruba’s oracle lane? (initial-recommendation assumption #2, unresolved)
  2. What is this harness’s own false-DIVERGED rate once it runs against a real Orders endpoint chain — the one number no published precedent supplies (D1/D6 Q2 confirmed gap)?
  3. Does the 2–5 engineer-day per-repo lab setup cost (MODELED, D3) hold on the actual Orders-chain stack matrix (6 distinct pinned runtimes, no reuse per D4), or does the second adapter cost >2× the first — the harness’s own top pre-mortem risk (context brief, known top risk)?
  4. Is there any surviving published evidence that shadow-traffic/dual-run parity checking specifically catches AI-agent-introduced regressions (as opposed to human-introduced ones)? Confirmed absent as of this research (D6) — worth one more explicit check before the program leans on this harness as its primary AI-regression backstop.