Research — Dual-run parity harness (the intellectual core)

2026-07-18 · Author: Fable. Role of this doc: brainstorm, problem discovery, advisory board, steelman. It attacks initial-recommendation.md; where they disagree, the later executive recommendation wins. Inputs read in full: context brief, initial recommendation, discovery notes D1 (OSS landscape), D2 (case studies), D3 (EOL lab), D4 (greenfield recon), D5 (legacy recon), D6 (verification science).

Internal-product reinterpretation (binding, per context brief): buyer = the program gates + Robert’s investment call; procurement = lane approval from Tomas; churn = the fleet routing around the harness (bypass); pricing = engineering-weeks + agent tokens. Every generic-SaaS beat below is scaled to that. Where I scale one down, I say so in one line rather than pad.

Evidence tags: MEASURED / MODELED / DESIGN, plus the epistemic split confirmed-fact / supported-interpretation / hypothesis / assumption / unknown. External claims carry URL + date pulled from the discovery notes.

1. Brainstorm (pm:brainstorm)

1.1 The assumptions in the initial recommendation, challenged

#	Assumption in initial-recommendation.md	Verdict after research	Why
A1	“Recorded-fixture replay first; live shadow only where the ledger proof demands it” (§8) is a safe default.	Half-wrong, and the wrong half is load-bearing.	D6 Q4: every credible primary-sourced financial-grade parity gate found (Uber experiment engine >99.999%, uber.com 2024-10-03; Uber LedgerStore 99.9999% upper bound; Stripe 99.9999% explainability, stripe.dev 2024-02-16) used live/continuous dual-run or dual-write, never recorded-fixture replay as the sole gate. No counterexample exists. Supported-interpretation: recorded replay de-risks earlier stages and non-ledger functional parity; it cannot honestly clear the 99.999% write-ramp row.
A2	The harness “generalizes Zaruba’s 3-oracle reconciliation” as the Orders adapter (§4.3).	Keep, but note only 1 of 3 oracles exists.	D4(b): order-store cross-check is built (`crossCheckBillingFeed`), provider-settlement is a typed stub, contract-check does not exist in any form. The reusable shape is real (`CrossCheckGap[]` is a ready-made counterexample payload, D4:52) but “generalize the 3-oracle pattern” is generalizing one working instance, not three.
A3	Noise normalization is a solvable kernel-v0 problem (§10).	Keep — but nobody has published that it is solvable at acceptable precision.	D6 Q2 + D1 §13: Diffy’s relative-disagreement noise floor is well-documented but its false-positive rate was never published by Twitter or anyone. The harness’s own self-replay test (record → replay against same legacy instance must yield PARITY, §11) is now the only evidence source available. Confirmed-fact: no external precision number to lean on.
A4	The EOL legacy lab is buildable in “days, not weeks” for the prototype (§11).	Keep for the prototype; reject as a general claim.	D3 + D5: prototype on one easy stack (forex-ng, Java 11, 2 endpoints, no DB) is genuinely days. The Orders chain needs 6 distinct pinned runtime images with no reuse (D5:54), and voucher-inventory-service (Ruby 1.9.3-on-JRuby-1.7, hand-patched Rails 3.2) is the hardest boot in the estate. MODELED 2–5 engineer-days per first repo of a stack combo, high end for the JRuby/Java-7 tier (D3 §8). Days-not-weeks is true per easy repo, false for the flagship chain.
A5	LLM triage-not-gate is the right call (§8).	Keep, fully corroborated.	D6 Q6: 2024–2026 LLM-judge-for-equivalence literature is benchmark-stage; the one survey found concludes “triage and assistance, not definitive gates” (arxiv 2510.24367). No paper trusts an LLM judge as sole equivalence gate.

The single most consequential correction: A1. The initial recommendation’s product identity (“an evidence machine for turning legacy OFF”) is right, but its default mechanism (recorded replay) is provably not what clears the row that turns the money off. That tension drives everything below.

1.2 Reframing the underlying problem (4 genuinely different frames)

Frame A — “Prove the new system behaves like the old one” (the stated frame). Parity as the product. This is the initial recommendation’s frame. It is a proxy frame: nobody is paid for parity; they are paid for legacy-off. Risk of this frame: it invites chasing 100% parity, which every rigorous source explicitly rejects (Zalando: “fixing those last few percentages has a cost higher than the value it brings,” engineering.zalando.com 2021-11; Uber set a 99.9999% upper bound deliberately). Keep as the mechanism, demote as the mission.

Frame B — “Make the legacy-off decision safe and defensible” (the economic frame). The buyer is a go/no-go decision, not a diff. The product is a decision backed by a replayable counterexample, sized to the money at stake. This frame correctly makes performance parity (p99), COLD/DARK coverage, and the noise budget first-class, because a defensible off-decision must account for what the comparison didn’t cover. TSB is the negative control: nine successful dress rehearsals + board sign-off + 1,600-person pilot still produced catastrophic failure (D2 §9) — because the decision rested on human judgment, not a structural machine-checked gate. This is the strongest frame. Adopt it as the mission.

Frame C — “Make agent output trustworthy at merge time” (the fleet frame). The primary user is a coding agent claiming a work-ledger unit; the product is a verdict row that gates promotion to the PR lane, so a unit cannot merge unverified. This is the Norman-affordance frame (safe path = only path). It is real and correct, but it is a delivery mechanism for Frame B, not a separate mission — the agent consumes the same verdict the off-decision consumes. Keep as the interaction model, not the mission.

Frame D — “Manufacture the evidence bundle Tomas and the QSA/SOX auditor sign” (the receipts frame). DONE row 9 requires auditor-grade evidence; Zaruba’s audit law is source + window + confidence on every number. The product is an evidence bundle with a normalization ledger (what was ignored and why) and a replay recipe (so any reader can falsify the verdict). This frame is what makes the harness auditable rather than merely useful. It is a facet of Frame B (a defensible decision must be an auditable one). Keep as a first-class output requirement.

Resolution: Frame B is the mission; C and D are how B is delivered and audited; A is the mechanism inside B. The initial recommendation conflates A with the mission; that is the reframe.

1.3 Fundamentally different solution shapes — kill or keep

Shape	What it is	Keep/Kill	Reason (decisive)
S1 — Live continuous shadow/dual-run	Diffy/Scientist/Uber model: fan out live traffic to legacy + candidate, diff continuously.	KEEP — but scoped to the money gate, and it is largely Zaruba’s plane, not the harness’s.	D6 Q3/Q4: this is the only pattern with evidence of clearing a five/six-nines financial gate. But D4(c): Zaruba’s R4 already declares rows 24/25 (order money → ledger) “YES — this IS the reconciliation harness” — the live oracle reconciliation over the billing-event feed. The live gate is oracle reconciliation that mostly exists in Core already; the harness contributes the generic verdict/evidence wrapper, not a new live-shadow rig. Live triplication of writes is unsafe by default even in Diffy’s own design (`POST/PUT/DELETE` ignored unless `-allowHttpSideEffects=true`, D1 §2a).
S2 — Recorded-fixture replay against an EOL legacy lab	Record legacy HTTP+bus → normalize → replay against candidate → diff → verdict. The initial recommendation’s core.	KEEP — but re-scoped to functional parity of read paths + the long-tail/DARK surface, NOT the money gate.	D6 Q4: recorded replay has zero evidence of clearing a financial-grade cutover gate. D5: it is the only way to characterize COLD/DARK use cases that live traffic never exercises (Uber’s own disclosed limitation: shadow validates only the hot path, D2 §6c). Recorded replay’s honest job is “does the candidate reproduce legacy behavior on inputs we captured” — a real, necessary, non-money job.
S3 — Test-impact-selected suites	TDAD-style graph impact analysis picks which existing tests an agent’s change must pass; no lab, no replay.	KEEP as the explicit fallback, not the primary.	D6 Q7: TDAD’s 6.08%→1.82% (~70%) is a single uncorroborated study, small sample, off-domain benchmark (SWE-bench Verified, 125 instances). Real but unreplicated. This is the context brief’s stated fallback if “harness became the schedule” (pre-mortem #2). It cannot prove legacy-off; it can keep agent regressions down cheaply while the harness is being built. Keep in the back pocket, cite as single-study.
S4 — Oracle-only per domain (no lab, no recorder)	Skip recording legacy entirely; build an independent oracle per domain (like the 3-oracle reconciliation) that checks the new system against an independent source of truth.	KEEP for money/ledger; KILL as a universal answer.	D4(b): works beautifully for reconciliation (feed vs ledger vs settlement — “independent, no Teradata”). But it requires an independent source of truth per domain; most read-path and long-tail behavior has no second oracle — the legacy system IS the only spec (Feathers’ characterization-test problem, D6 Q1). Where an independent oracle exists, prefer it over the lab (cheaper, no EOL boot). Where it doesn’t, you’re back to S2.
S5 — Process-only: rely on Zaruba’s use-case shadow-checkability gates + human sign-off	No harness. Zaruba’s R4 assigns every use case YES/COMPARE-ONLY/NO; a panel signs scope; builders judge divergences by hand.	KILL as sufficient; KEEP as the substrate the harness plugs into.	This is the steelman-against “it’s a feature of Zaruba’s plane.” D2 §9 (TSB) is the decisive kill: rehearsals + human sign-off is exactly what failed catastrophically at 5M-customer scale. Zaruba’s R4 defines what must be proven per use case but D4(c)/(d): his repo contains no mechanics for how (the decision log has zero hits for dual-run/shadow/parity). Process defines the gate; it does not execute it. The harness is the execution.

Synthesis of shapes: the product is not one shape — it is S4-where-an-oracle-exists (money gate, live, mostly Zaruba’s plane) + S2-elsewhere (functional/long-tail/DARK parity, recorded, the harness’s real net-new build) + S3-as-fallback, all wearing one verdict/evidence-bundle surface (Frames C+D). The initial recommendation’s error is presenting S2 as the thing that clears the money gate. It doesn’t; S4/S1 do.

2. Problem discovery (pm:discovery, template §4)

Core problem. The program cannot turn legacy OFF without a machine-checked, replayable, auditable proof that the Encore.ts rebuild behaves like a 2013–2017 EOL estate nobody can safely run — produced fast enough that the proof is not itself the schedule. (Confirmed-fact: verification is the binding constraint, analysis/legacy/SYNTHESIS.md §3 “the hard part is not code generation — it’s verification.”)

Who experiences it.

Fleet coding agents (primary, machine consumers): claim a work-ledger unit, need a mechanical PARITY/DIVERGED/UNVERIFIABLE verdict to promote to the PR lane. Without it they merge on vibes.
The 4-builder strike unit (evidence consumers): open the evidence bundle when a divergence blocks a unit; decide real-regression / acceptable-difference / add-normalization-rule.
The program gate itself (economic buyer): DONE rows (reconciliation ≥99.999%, error ≤ baseline +0.1pp, zero-RPS 30 days) pay bonuses only if legacy provably turns off.
QSA/SOX auditor + Tomas (sign-off consumers): DONE row 9 needs auditor-grade evidence.

Circumstances / frequency. Every unit the fleet claims on the Orders flagship and every subsequent flagship. Orders alone: 127 HTTP endpoints, 25+ MBUS topics + a Kafka signal, dual MySQL with a dedicated outbox DB, DARK Resque batch jobs (D5:7-13). MODELED 50–60+ distinct fixture classes for Orders alone (D5:64). Dual-run must be live by week 6 or M4 slips publicly (context brief; reasons-and-decisions/03-orders-flagship.md).

Cost of the problem (program economics). Confirmed-fact: program all-in $6.7M, M4 ≈ $0.9M, harness is Robert-lane, weeks not months, staged per flagship gate (context brief). The gates pay only if legacy turns off; a wrong off-decision on the money chain is a TSB-class event (£48.65m fine + ~£366m total cost at TSB, D2 §9 medium-confidence secondary sourcing). Supported-interpretation: the harness’s entire economic justification is that it converts a $0.9M milestone from “unclaimable without proof” to “claimable with proof,” and insures against a catastrophic false-off. It is cheap insurance on an expensive, irreversible decision.

How it is solved today. Two things, both partial:

Zaruba’s oracles + process. Merchant-accounting reconciliation.service.ts + crossCheckRange (~300 LOC, one domain slice, D4:17-23); R4 shadow-checkability table (YES/COMPARE-ONLY/NO per use case, D4:64); panel scope sign-off. This is real and it is the money-gate substrate.
Vibes for everything else. No characterization kernel, no legacy-runtime lab, no MBUS event comparison, no generic verdict, no evidence bundle, no work-ledger gating. ASSESSMENT.md lists “no parity/dual-run harness” as a top-5 gap (context brief). D4(a): zero verdict/question-minting surface exists in the greenfield repo.

Why today’s solution is insufficient.

Zaruba’s oracle covers exactly one domain (money/ledger) and even there only 1 of 3 oracles is built (D4:44-48). It says nothing about the ~40 non-ledger R4 use cases, the read paths, or the DARK batch surface.
Process (R4 + panel + human judgment) is precisely the configuration that failed at TSB. A hard structural gate (“a unit cannot reach the PR lane without a verdict row”) is the evidence-backed response (D2 §11.3).
Vibes do not survive 64 concurrent agents. No published evidence exists that shadow-traffic catches AI-agent regressions specifically (D6 confirm/overturn: CONFIRMED gap) — so the harness must generate that evidence for itself; it cannot cite anyone else’s.

Evidence the problem exists. Confirmed: the top-5 gap listing; the 300-LOC single-slice oracle; 16/33 legacy repos MBUS-coupled (context brief); 6 distinct EOL runtimes needed for one chain (D5:54); Zaruba’s decision log has zero entries designing the mechanism (D4:77). Supported-interpretation: the week-6 “or M4 slips publicly” clause is Tomas pricing the problem himself.

2.1 Five-class evidence separation

Confirmed facts.

No OSS/commercial tool covers the whole kernel (record+normalize+replay+diff+verdict, cross-language, incl. message bus) — D1 §0.
No OSS tool diffs JMS/STOMP/Kafka events between two systems — D1 §11.
opendiffy/diffy is CC BY-NC-ND 4.0, legally unusable by a commercial entity; twitter-archive/diffy is Apache-2.0 but archived 2020 by its own inventor as “outdated tech stack” — D1 §2a.
Every primary-sourced financial-grade parity gate used live dual-run, not recorded replay — D6 Q4.
Orders chain needs 6 distinct pinned EOL runtime images, no reuse — D5:54.
The greenfield repo has no verdict enum, no work-ledger, no evidence bundle, no question-minting — D4(a).
Zaruba’s dual-run is defined over THREE shadow-checkability classes (YES / COMPARE-ONLY / NO), not two — D4(c).

Strongly supported interpretations.

The 99.999% money gate is served by live oracle reconciliation (S4/S1), not by recorded replay (S2). (From D6 Q4 + D4(c) rows 24/25.)
The harness’s net-new value is functional/long-tail/DARK parity + the generic verdict/evidence-bundle wrapper, not the money gate itself.
The lab’s cost is dominated by the JRuby-1.7 / Ruby-1.9.3 / Java-7 tier and does not amortize cleanly across stack combos (linear, not sub-linear) — D3 §8, D5:54.

Reasonable hypotheses.

Diffy’s relative-disagreement noise floor (legacy-vs-legacy self-replay to establish a noise floor, then compare candidate divergence to it) will outperform hand-written normalization rules — D1 §2a, corroborated by initial-recommendation §11’s independent proposal.
A current apache/activemq-classic image can serve 2013–2017-era JMS/STOMP clients via wire back-compat — D3 §1 (flagged unverified there).
Test-impact-selected suites (S3) can hold agent regressions near the TDAD ~1.82% while the harness is built — single-study, D6 Q7.

Unsupported assumptions (to stop making).

“Recorded replay clears the 99.999% gate.” No evidence; contradicted by every real deployment.
“The 3-oracle pattern generalizes because it exists.” Only 1 of 3 oracles is built (D4:44-48).
“The legacy estate runs on K8s/AWS so mesh mirroring is available.” Unverified either way (D1 §14.3).

Unknowns requiring validation (spikes).

Are ruby:1.9.3 / jruby:1.7 / openjdk:7 tags actually pullable on a current Docker Engine (manifest-v1 deprecation, docker-library/ruby#452)? — D3 open questions.
Does eBPF capture (Keploy) work against JRuby 1.7 / Ruby 1.9.3 syscall/threading patterns? — D1 §14.4.
What is the harness’s own false-DIVERGED rate on self-replay? Nobody published theirs (D6 Q2) — this is the harness’s own falsification recipe and only evidence source.
Exact Resque/Quartz batch job inventory for Orders (DARK) — D5 open questions.

2.2 Is software the right solution, or does process (Zaruba’s audit use-cases) already cover it?

Decisive answer: software is required, but a smaller piece of software than the initial recommendation implies, and process defines its boundary.

Process (R4 shadow-checkability + panel + human sign-off) is necessary and already exists — it decides which use cases are YES/COMPARE-ONLY/NO. It is the substrate, not the answer. It does not execute a single comparison and it is the exact configuration that failed at TSB when relied on alone (D2 §9).
The money gate’s software largely also already exists in Zaruba’s plane (the oracle reconciliation over the billing-event feed, D4(b)/(c)). The harness should not rebuild it; it should wrap it in the generic verdict/evidence-bundle surface and own the parts Zaruba’s plane leaves empty.
The genuinely net-new software is: (i) the generic verdict interface + evidence bundle + work-ledger gating (Frames C/D — nothing like it exists, D4:34-36); (ii) the recorded-replay kernel + EOL lab for functional/long-tail/DARK parity where no oracle exists (S2); (iii) MBUS shadow-taps where a domain’s parity needs event comparison (no OSS exists, custom build, D1 §11).

So: build software, but build the wrapper + the no-oracle-coverage engine — not a second copy of the money gate, and not a general multi-domain platform. Process owns the scoping decision; the harness owns execution and evidence.

3. Expert advisory board (template §3)

All five are principle-based analytical simulations, not fabricated quotations or endorsements. Each reasons from the named public principles of the individual only.

Strongest argument for: the harness has exactly one job — make legacy-off provable. That is a knife-edge focus most infra tools never achieve. Ship the one command (harness verify <unit>) that produces the one artifact (the verdict + counterexample) and nothing else.
Strongest criticism: the initial recommendation is already smuggling in a platform (“generalize further per flagship gate,” “characterization kernel,” “adapters”). That is 1,000 things wearing a trench coat. Jobs kills the word “kernel” — it signals a framework nobody asked for.
Features he’d challenge: the multi-domain adapter abstraction; anything named “generic”; the second and third oracles that don’t exist yet; the Grafana panel (chrome).
Questions he’d demand: “What is the one unit of one flagship you can turn off next month, and does this produce the proof for that — or are you building a machine to build machines?”
Recommended change: collapse scope to the single Orders read-path proof end-to-end before any word like “adapter” or “kernel” appears in code. One widget, whole and working.
Likely MVP principle: One command, one artifact, one flagship. If it doesn’t turn one thing off, it ships nothing.
Evidence he’d require before backing investment: a live demo of one real Orders read-path unit going from claim → verdict → off-decision, not a slide about generalization.

3.2 Jony Ive (reduction; honesty of materials; complexity handled by the system, not exported)

Strongest argument for: the diff report is the product’s face and it can be beautiful in the Rams sense — a verdict that states its own limits (normalization ledger) and hands the reader a way to falsify it (replay recipe). That is honest material.
Strongest criticism: noise is the enemy of reduction, and the harness has no evidence it can suppress noise to a readable floor (D6 Q2: Diffy never published its false-positive rate). A report drowning in phantom divergences is dishonest typography — “a parity report that hides its exclusions is a lie with good typography” (initial-recommendation §7, kept verbatim as the design law).
Features he’d challenge: any verdict that shows more than one thing per screen-height; multi-field diff dumps; exposing replay mechanics to the reader.
Questions he’d demand: “When the machine is unsure, does it say UNVERIFIABLE plainly, or does it fake a verdict? Is the smallest counterexample truly the smallest?”
Recommended change: make UNVERIFIABLE a first-class, honestly-surfaced state (not an error), and make counterexample minimization a kernel requirement, not a polish pass.
Likely MVP principle: The reader sees the verdict, the single smallest counterexample, and the honest list of what was ignored — nothing else, ever.
Evidence he’d require: the self-replay test (record → replay same legacy instance → must yield PARITY) passing with zero false DIVERGED on a real recorded corpus. Until that holds, the material is not honest.

3.3 Marty Cagan (empowered teams; outcome over output; the four risks)

Cagan’s four product risks, scored for this internal product:

Value risk (will the buyer use it?): LOW-MEDIUM. The gate must consume verdicts (structural). But churn = fleet bypass is real if noise is high — agents route around a tool that cries wolf.
Usability risk: LOW for agents (they read structs), MEDIUM for builders (the evidence bundle is the whole UX and it’s unproven).
Feasibility risk: HIGH. The EOL lab (D3), the no-OSS MBUS shadow-tap (D1 §11), and the unpublished-noise-floor problem (D6 Q2) stack.
Viability risk (does it fit the business/program?): MEDIUM — “harness became the schedule” (pre-mortem #2) is the viability killer; under-resourcing is the opposite killer (Uber Invoicer: 10% progress in six months treating migration as side work, D2 §6d).
Strongest argument for: the outcome (legacy provably off) is unambiguous and measurable (DONE rows). That is a rare, clean outcome to build toward.
Strongest criticism: the plan is output-shaped (“build a kernel, a lab, adapters”) not outcome-shaped (“turn Orders reads off in M2”). Cagan reframes the roadmap around the flagship gate, not the tool’s feature list.
Features he’d challenge: anything not on the critical path to the M2 read-authoritative checkpoint.
Questions he’d demand: “What’s the riskiest assumption, and what’s the cheapest test of it this week?” (Answer: EOL-lab bootability + noise floor; test = the days-not-weeks prototype on forex-ng then giftcard_service, D5(e).)
Recommended change: run the four risks explicitly; attack feasibility first with a throwaway prototype before committing the flagship schedule to the harness.
Likely MVP principle: Discovery before delivery — prove the lab boots and the noise floor holds on one easy stack before Orders depends on it.
Evidence he’d require: a de-risking spike result on each HIGH-risk assumption before the week-6 dependency is accepted.

3.4 Teresa Torres (continuous discovery; opportunity-solution tree)

Outcome (root): Legacy provably OFF for the Orders flagship by M4 without the proof becoming the schedule.

OUTCOME: Orders legacy OFF by M4, provably, without the harness becoming the schedule
│
├─ OPPORTUNITY 1: "I can't trust the new system matches the old on money"
│   ├─ Solution 1a: Live oracle reconciliation over billing-event feed (S4/S1)  ← mostly exists (D4b/c), WRAP not rebuild
│   ├─ Solution 1b: Recorded-replay of money paths  ← KILL for the gate (no evidence clears 99.999%, D6 Q4)
│   └─ Solution 1c: Provider-settlement + contract oracles  ← build the 2 missing oracles (D4:44-48)
│
├─ OPPORTUNITY 2: "I can't observe legacy behavior at all — it's EOL and unbootable"
│   ├─ Solution 2a: Pinned EOL Docker lab + outside-in proxy capture (mitmproxy/GoReplay OSS)  ← core net-new, feasibility-HIGH
│   ├─ Solution 2b: eBPF capture (Keploy)  ← spike vs JRuby 1.7 first (D1 §14.4), don't commit
│   └─ Solution 2c: In-process VCR  ← KILL (requires legacy-code change, stuck on 2015 release line, D3 §5)
│
├─ OPPORTUNITY 3: "Divergences drown in noise → I stop trusting verdicts → I bypass"
│   ├─ Solution 3a: Self-replay noise floor (legacy-vs-legacy) à la Diffy  ← prototype, only evidence source (D6 Q2)
│   ├─ Solution 3b: Hand-written normalization ledger  ← keep as the auditable, human-curated complement
│   └─ Solution 3c: LLM triage/clustering of divergences  ← triage only, never gate (D6 Q6)
│
├─ OPPORTUNITY 4: "The invisible surface (batch/cron/MBUS) kills the cutover and I can't see it"
│   ├─ Solution 4a: MBUS shadow-taps in adapters  ← custom build, no OSS (D1 §11), scope per-domain only
│   ├─ Solution 4b: Outbox-DB durability capture (Orders orders_msg_prod)  ← must reproduce two coupled writes (D5:13)
│   └─ Solution 4c: Batch/cron capture via DB-log tailing  ← DARK, needs job inventory first (D5 open Qs)
│
└─ OPPORTUNITY 5: "Auditors/Tomas won't sign without receipts"
    ├─ Solution 5a: Evidence bundle = verdict + normalization ledger + replay recipe  ← first-class output (Frame D)
    └─ Solution 5b: Verdict rows in work-ledger gate PR-lane promotion  ← structural gate (Frame C, anti-TSB)

Strongest argument for: the tree exposes that Opportunity 1 (money) is mostly already solved in Zaruba’s plane, so the harness’s discovery energy belongs in Opportunities 2–5 — a genuine reprioritization the linear plan hides.
Strongest criticism: the initial recommendation jumps to solutions (kernel/lab/adapters) without mapping opportunities; it risks over-building 2a and under-attending 4 (the invisible surface that actually kills rewrites).
Questions she’d demand: “Which opportunity, if unsolved, sinks the flagship? (Answer: 4 — batch/MBUS invisibility.) Are you weighting discovery there or on the shiny kernel?”
Likely MVP principle: Smallest solution per opportunity, validated with a real recorded corpus, before widening.
Evidence she’d require: an assumption test per branch — especially 2a (lab boots), 3a (noise floor holds), 4a (a real MBUS event diffs cleanly).

3.5 Clayton Christensen (jobs-to-be-done; disruption; two separate “customers”)

The harness has two distinct customers hiring it for two different jobs. Conflating them is the platform-creep trap.

Customer 1 — the fleet coding agent.

Job: “When I claim a work-ledger unit, help me know whether my change is safe to merge so I don’t burn tokens or ship a regression.” Functional job: a mechanical verdict. Emotional job (for the human watching the fleet): confidence the 64-agent swarm isn’t silently corrupting Orders. Social job: a unit that merged carries a verdict row others can trust.
What it’s currently hired instead: vibes / nothing (no verdict surface exists, D4a).
Bar for firing the alternative: the verdict must be cheaper and faster than re-reading the diff by hand, and low-noise enough not to cry wolf. If it’s noisy, the agent re-hires vibes (bypass = churn).
Disruptive angle: the agent doesn’t need six-nines rigor per unit — it needs a good-enough, fast, non-noisy signal. Over-serving it with money-gate rigor is the classic incumbent over-shoot; keep the per-unit verdict lightweight (S2/S3), reserve six-nines for the money gate.

Customer 2 — the program gate (Tomas + QSA/SOX).

Job: “When I decide to turn legacy OFF, give me irreversible-decision-grade proof I can sign and defend to an auditor and a regulator.” Functional job: an auditable evidence bundle at ≥99.999%. Emotional job: not being the next TSB board. Social job: receipts that survive a QSA’s scrutiny.
What it’s currently hired instead: Zaruba’s oracle (money only) + rehearsals + human sign-off (the TSB-failure configuration for everything else).
Bar for firing the alternative: structural gate that a human cannot override with confidence alone (Norman affordance), backed by a live reconciliation number with a stated noise budget.
Disruptive angle: none — this customer wants more rigor, not less. Under-serving it is fatal. This is where live oracle reconciliation (S4/S1) belongs, not recorded replay.
Strongest argument for: separating the two jobs resolves the A1 tension cleanly — Customer 1 gets fast/cheap S2+S3 verdicts; Customer 2 gets rigorous S4/S1 evidence; one verdict/evidence-bundle vocabulary spans both.
Strongest criticism: the initial recommendation designs one artifact for both and thereby either over-serves the agent (too slow) or under-serves the gate (recorded replay can’t clear 99.999%). JTBD says build two service levels behind one surface.
Recommended change: two verdict tiers — a lightweight per-unit functional verdict (Customer 1) and a heavyweight money-gate reconciliation verdict (Customer 2) — explicitly labeled, never confused.
Likely MVP principle: Serve the agent’s job cheaply and the gate’s job rigorously; never make the agent pay six-nines cost or let the gate accept a token verdict.
Evidence he’d require: proof that the cheap tier actually reduces agent regression cost (S3/TDAD-style measurement on the real fleet) AND that the rigorous tier hits the reconciliation number on live data.

4. Steelman (template §13)

4.1 Strongest case FOR building the harness

Verification is the program’s binding constraint and its currency is receipts; the $0.9M M4 milestone is unclaimable and a false-off is TSB-catastrophic without machine-checked, replayable, auditable proof. Nothing off-the-shelf covers the kernel (D1 §0), no OSS diffs the message bus (D1 §11), and Zaruba’s plane covers only one domain’s money gate with 1 of 3 oracles built (D4:44-48). Every large engineering org that did this at scale (Uber, Slack, Shopify, Zalando, GitHub, LinkedIn — D2) built bespoke in-house dual-run tooling because no shared product fit; a purpose-built internal kernel is the empirically normal answer, not an exotic one. The structural gate (“no verdict row, no PR lane”) is the direct, evidence-backed antidote to the exact human-judgment failure that sank TSB (D2 §11.3). Cost is Robert-lane weeks against a $6.7M program; the ratio is cheap insurance on an irreversible decision.

4.2 Strongest case AGAINST (four named arguments, each taken at full strength)

(a) “It is a feature of Zaruba’s plane, not a product.” Zaruba’s R4 already defines dual-run over three shadow-checkability classes and declares the ledger rows “YES — this IS the reconciliation harness” (D4:64,69). The money gate — the only part with a hard number — is oracle reconciliation over the billing-event feed that already exists in Core (reconciliation.service.ts, D4:17). The billing feed’s own doc comment says it is “the seed of the >= 99.999% reconciliation check” (D4:13). So the load-bearing verification is a Core feature; the “harness” is a wrapper Robert wants to call a product. Full strength: the money gate does not need a separate harness at all.

(b) “Recorded replay cannot honestly support a 99.999% write-ramp gate.” D6 Q4 is unambiguous: no primary-sourced financial-grade cutover gate was ever cleared by recorded-fixture replay; every real one (Uber ×2, Stripe) used live/continuous dual-run or dual-write. Uber explicitly disclosed that shadow validation covers only the hot path, not historical data (D2 §6c). The initial recommendation’s “recorded replay first, live shadow only where demanded” defaults the harness’s core mechanism to the one mode with zero evidence of clearing the row that matters most. Full strength: the harness’s headline mechanism is disqualified from the headline gate.

(c) “The legacy-runtime lab is a tar pit.” D3 is a catalogue of unresolved rot: manifest-v1 deprecation threatens whether the oldest images even pull (docker-library/ruby#452, live/unresolved); RubyGems’ TLS 1.2 floor blocks fresh gem installs; native-extension builds need exact base-OS/OpenSSL pairings and archived apt sources; and no published writeup combines this project’s full difficulty stack end-to-end. Six distinct runtimes for one chain, no reuse, JRuby-1.7/Java-7 the highest-uncertainty tier (D5:54, D3 §8). The lab could consume the weeks the whole harness was budgeted for and still not boot voucher-inventory-service. Full strength: the lab is unbounded-downside effort feeding pre-mortem #2 (“harness became the schedule”).

(d) “The Orders adapter is the whole value; generalizing is waste.” Orders alone is 50–60+ fixture classes, 6 runtimes, DARK batch, dual-bus (D5). Getting just Orders off is the entire M4 prize. Every hour on a “generic kernel,” “adapter abstraction,” or second/third domain is speculative YAGNI against a one-flagship payoff. The 001 pre-mortem already flags “harness doesn’t generalize past Orders” with detection “2nd adapter costs >2× the first” — i.e. the org already suspects generalization won’t pay. Full strength: build one bespoke Orders proof, ship it, delete the platform ambition.

4.3 Independent advisory opinion (does not average the two)

I do not split the difference. Here is the call.

The harness is worth building — but it is a smaller, differently-shaped thing than the initial recommendation describes, and two of the four arguments-against are correct and must reshape it.

Concede (b) and (a)-partially: the money gate is live oracle reconciliation, and it is mostly Zaruba’s plane. Stop presenting recorded replay as what clears 99.999%. The harness does not build a second money gate. It contributes exactly two things to the money gate: (i) the two missing oracles (provider-settlement, contract-check — D4:44-48), because they are cheap independent-source oracles, not lab work; and (ii) the generic verdict + evidence-bundle wrapper (Frames C/D) so the reconciliation output becomes an auditable, PR-lane-gating artifact instead of a boolean logged by a cron with no alerting (D4:23). That wrapper is genuinely net-new and genuinely valuable, and it is not a Zaruba feature — his repo has no verdict enum, no bundle, no work-ledger gating (D4a). So (a) is half right: the money computation is Core’s; the money evidence surface is the harness’s.
Reject (a) as a whole and reject (d): the net-new value is the no-oracle surface, and it is not waste. The ~40 non-ledger R4 use cases, the read paths, and the DARK batch/MBUS surface have no independent oracle — the legacy system is the only spec (Feathers characterization problem, D6 Q1). That is exactly where recorded replay + the EOL lab is the honest tool (functional parity, not money parity), and where no part of Zaruba’s plane helps. This is also where the invisible surface that kills rewrites lives (batch/cron/MBUS, D5:62). Generalizing the wrapper across these use cases is not platform-creep — it is the point of Frame C (the fleet gates on it). Generalizing the lab and adapters across future flagships is speculative and should be capped: build the wrapper generic, build the lab/adapters strictly per-flagship-gate, exactly as the 001 synthesis already bounds it.
Take (c) seriously and defuse it with sequencing, not denial. The lab is a real tar-pit risk. The defense is the Cagan/Torres move: attack feasibility first, cheaply, before the flagship schedule depends on it. Run the D5(e) pilot ladder — forex-ng (self-test, zero complexity) → orders-ext (MBUS shadow-tap on a modern non-EOL stack) → giftcard_service (EOL Rails 3.2 boot rehearsal) — as throwaway spikes that de-risk the exact assumptions (pull-ability, TLS, native gems, noise floor, MBUS diff) before voucher-inventory-service’s JRuby boot is on the critical path. Cap total lab investment per flagship; the explicit fallback if the cap blows is S3 (impact-selected tests, TDAD ~70% single-study). The lab is only a tar pit if it is entered without a rip-cord; install the rip-cord.

Net shape of the product I’d back:

One verdict/evidence-bundle surface (generic, net-new, the real product) spanning two tiers: a cheap per-unit functional verdict for the fleet (S2/S3), and a rigorous money-gate reconciliation verdict for the program (S4/S1, wrapping Core’s existing oracle + the two missing oracles).
A recorded-replay kernel + EOL lab, scoped to functional/long-tail/DARK parity where no oracle exists, entered via the cheap-pilot ladder with a hard investment cap and S3 as the documented fallback.
MBUS shadow-taps per-domain only, custom-built (no OSS), scoped where a domain’s parity needs event comparison.
Explicitly NOT a second money gate, NOT a general multi-domain platform, NOT recorded-replay-as-the-99.999%-mechanism, NOT an LLM equivalence gate.

This is not the average of “build it” and “don’t.” It concedes the two correct criticisms (recorded replay ≠ money gate; money computation is mostly Zaruba’s), rejects the two wrong ones (it’s all a Zaruba feature; generalization is all waste), and re-centers the product on the surface only the harness can own: the auditable verdict that gates a 64-agent fleet and manufactures the receipts a QSA signs.

5. Open questions

Live vs recorded for the money gate — whose lane, resolved how? Evidence says the 99.999% row needs live reconciliation (D6 Q4); Zaruba’s R4 says that reconciliation “IS” the gate and it lives in Core (D4c). Does the harness own it, wrap it, or just consume it? Needs an explicit Robert↔Tomas lane decision, not an assumption. (Same as initial-rec open Q2, now sharper.)
Does any Encore-hosted harness piece survive the cross-plane rule? The billing-event feed is expose:false, reachable only via Core’s generated internal client, in-cluster (D4d:2). An external CLI/CI runner cannot call it. Resolve: a Core-side adapter service exposing an authenticated read, or run that hop inside the Encore boundary — either contradicts “kernel + lab live outside Encore.” Real seam (D4d).
Can the oldest images be pulled at all in 2026? ruby:1.9.3 / jruby:1.7 / openjdk:7 vs manifest-v1 deprecation (docker-library/ruby#452, live/unresolved). Rescue-pull-and-re-push workaround exists but must precede any unattended docker pull. Spike before the flagship depends on it (D3).
What is the harness’s own false-DIVERGED rate on self-replay? Nobody published theirs (D6 Q2); the self-replay test is the only evidence source. Must pass zero-false-DIVERGED on a real recorded corpus before agents trust it, or bypass/churn follows.
Is apache/activemq-classic wire-back-compatible with 2013–2017 JMS/STOMP clients? Unverified assumption (D3 §1); the whole MBUS shadow-tap plan rests on it.
What is the DARK batch surface for Orders? Resque/Quartz job classes are uninventoried (D5 open Qs); the 50–60+ fixture-class estimate stays MODELED until an app/workers/*.rb + schedule-file grep lands. The batch class cannot be captured by an HTTP proxy at all — it needs DB-log tailing or in-lab job instrumentation.
What is the stated noise budget for the 99.999% gate? Uber and Slack both accept a bounded non-zero floor with a stated source (S3 durability ~10 corruptions/trillion; replica-lag ≤1h) — D2 §11.1. The harness’s gate must define its own budget with a source (what corruption/staleness rate is structurally expected in the legacy estate?), not treat any nonzero diff as failure.
Where is the investment cap, numerically? “Weeks not months” and “cap per flagship” are DESIGN, not a number. The pre-mortem detection signal (“2nd adapter costs >2× the first”) needs a budget line and a named S3 fallback trigger, or “harness became the schedule” has no brake.