North Star — Dual-run parity harness

Product strategy for the parity kernel + legacy-runtime lab + work-ledger verdict surface (“the quality-control bench”). Internal product; template §11 reinterpreted per the context brief (buyer = program gates, procurement = lane approval from Tomas, churn = harness bypass, pricing = engineering time + tokens). Inherits binding facts from 00-context-brief.md and the challenged assumptions in initial-recommendation.md; does not re-litigate build/buy (settled BUILD) or the outside-Encore platform decision (settled, architect correction #4).

1. Vision

Every capability the strike unit keeps carries a verdict, not a vibe. Five years from now, “we turned legacy off” is a sentence backed by an evidence bundle any auditor can open and any engineer can falsify — not a changelog and a shrug.

2. Mission

Give the program a machine-checked parity verdict with a replayable counterexample, per signed capability, fast enough that producing the proof is never the reason a flagship gate slips.

3. Positioning

The program’s evidence machine, not generic QA. A generic QA platform answers “does this pass tests.” This answers a narrower, harder, program-critical question: “does this new capability behave like the legacy capability the panel signed off to keep.” It exists because Zaruba’s audit doctrine requires receipts (source + window + confidence) for every kept capability, and today exactly one domain slice (merchant-accounting’s ~300-LOC reconciliation oracle) produces that receipt. Positioning against near-neighbors:

Near-neighbor	Why it’s not this	Source
Generic test framework / CI gate	Verifies the new system against a spec humans wrote; has no legacy oracle, no EOL-runtime lab, no cross-language event diffing	D1
GitHub Scientist	Same-language live dual-execution only; legacy estate is EOL Ruby/Java, can’t run new TypeScript candidate code in the same process as legacy Rails	D1, D2 §1
Diffy / Hoverfly diff mode	Same-language, always-live, HTTP-only; assumes both sides are running services you can proxy to today — the legacy side here is a pinned-EOL stack that has to be booted in a lab, not proxied	D1
datafold/data-diff	Confirmed dead (official sunset 2024-05-17); also DB-row diffing, not behavior diffing	D1
MBUS/event diffing (any OSS)	No OSS tool diffs JMS/STOMP/Kafka events between two systems — confirmed gap	D1

Confirmed fact (D1): no tool covers the kernel end-to-end. Build is the only option that fits a pinned-EOL, cross-language, MBUS-coupled estate. This is not a novel claim being asserted here — it is the D1 discovery finding, restated as positioning.

4. ICP and beachhead

ICP: the Orders-flagship M2 gate (dual-run live by week 6, reads authoritative in M2, write ramp gated on ledger reconciliation ≥99.999% through M3), then subsequent flagships as the program schedules them.
Beachhead, sequenced: one long-tail pilot service before Orders, not instead of it. D5 ranks five candidates (forex-ng, orders-ext, giftcard_service, consumer-data, getaways-accounting-service) that each rehearse one piece of the Orders adapter — one endpoint chain, one EOL stack, one MBUS topic set — at a fraction of Orders’ 127-endpoint / 6-stack / 50-60-fixture-class surface (D5 §c/§d, MODELED). The pilot’s job is to retire the two riskiest assumptions from initial-recommendation.md §11 (EOL lab boot cost, noise-normalization self-replay) before the flagship clock is running. Days, not weeks, per the initial recommendation’s own prototype scoping. forex-ng is the committed first pilot — ranked cheapest (2 endpoints, no DB, no MBUS) and a modern, non-EOL Java 11 stack in D3’s lower-risk tier (validation-plan E2, architecture §8, prd §18).
Why not Orders first: Orders alone needs 6 distinct pinned legacy runtime images with no reuse and MBUS shadow-taps across 25+ topics (D5 §c). Discovering that the lab is infeasible on Orders’ hardest stack (Rails 3.2 / JRuby 1.7 / Ruby 1.9.3 — D3’s highest-risk tier, TLS-to-rubygems rot + native-gem builds + manifest-v1 pull risk, none confirmed infeasible but none confirmed solved either) during the week-6 gate is a program-level failure. Discovering it on a long-tail pilot is a Tuesday.

5. Core outcome

Legacy-off decisions backed by machine-checked evidence. Not “tests pass” — “this specific capability, signed by the panel, has a verdict + evidence bundle a QSA/SOX auditor and a skeptical engineer both accept.” The outcome is a decision artifact, not a test result.

6. Product principles

Verdict first, chrome never. No UI product in v1 (non-goal, §9). The evidence bundle is the product’s face (initial-recommendation §7); design effort goes there.
Structural gates beat human judgment under schedule pressure. TSB’s nine “successful” rehearsals + board sign-off still produced catastrophic failure at full-scale go-live (D2 §9) — the clearest negative-control evidence available. A unit cannot reach the PR lane without a verdict row; there is no “probably fine” path.
Noise normalization is the entire game, not polish. Every real tool that shipped (Scientist’s ignore blocks, Zalando’s header filtering + per-endpoint thresholds, Diffy’s A/A self-comparison) converged on this (D2 §11.5). Kernel v0 scope, not a v2 nice-to-have.
Nobody chases 100% parity — and says so with a number. Zalando: “fixing those last few percentages has a cost higher than the value it brings.” Uber’s ledger migration set an explicit upper bound (99.9999%) below 100% because ~10 corruptions/trillion records are structurally expected background noise (D2 §11.1). The harness states its noise floor and its source; it never treats nonzero divergence as automatic failure.
Side-effecting paths are never actually executed twice. Recorded-fixture replay by default; live shadow only where the ledger proof specifically demands it (§13 open question). Every source that discussed writes (Newman’s Spy, Uber’s Invoicer, Zalando’s non-idempotent caution) converged on never letting the candidate path fire the real side effect (D2 §11.4).
The tool proves its own claims or it doesn’t ship them. No published false-positive rate exists for Diffy’s noise-cancellation anywhere (D5/D6 Q2) — the harness cannot borrow anyone else’s track record. Self-replay-to-PARITY is therefore not a nice demo, it is the harness’s only falsification recipe for its own noise model (D6 §“Open questions” item 1).
Adapters are the product; the shared kernel stays minimal. Per-domain adapters generalize Zaruba’s existing 3-oracle pattern behind one verdict interface (initial-recommendation §4); the kernel grows only per flagship gate, never speculatively.

7. Moat

Accumulated fixtures + normalization rules + adapters per domain — evidence compounding. Concretely:

Every recorded fixture (masked, content-addressed) is reusable across every future re-verification of that capability — it does not decay unless the legacy behavior it captured was wrong to begin with.
Every normalization-ledger entry (a rule saying “ignore this field, here’s why”) is an auditable scope decision that the next adapter inherits for free where domains share shape (timestamps, UUIDs, ordering are near-universal; domain-specific rules compound per adapter family).
Every adapter is a solved instance of “how do we compare domain X” — the second adapter in a domain family (e.g. a second Orders-chain repo after orders-ext) costs a fraction of the first, per the same logic D3 uses for legacy-lab setup cost (2-5 engineer-days first repo, MODELED, well under a day for follow-on repos in a solved stack combination).
This is a moat against re-deriving the same evidence twice, not a moat against competitors — internal product, no market. The competitive framing in §3 exists to justify BUILD, not to describe a market position.

8. Strategic risks

Risk	Evidence	Mitigation already in scope
Harness doesn’t generalize past Orders (2nd adapter costs >2× the 1st)	Named risk, 001 synthesis	Beachhead pilot retires lab/noise risk before Orders; adapter cost tracked explicitly per D3’s MODELED follow-on-repo economics
“Harness became the schedule”	Named pre-mortem, 001 synthesis	Investment capped per flagship gate; fallback = impact-selected tests (TDAD’s ~70% regression-reduction figure — single uncorroborated study, small sample, off-domain benchmark, D6 Q7, cited as fallback economics only, not a proven number)
Under-resourcing the verification effort (the opposite failure)	Uber Invoice-gen: 10% progress in 6 months run as “extra work alongside the day job,” fixed only by dedicating a team (D2 §6d, §11.8)	State both directions of the risk explicitly (this document does); harness effort is a named Robert-lane workstream, not ambient side work
99.999% ledger-reconciliation gate may structurally require LIVE dual-run, not recorded-fixture replay	Every credible primary-sourced financial-grade parity example found (Uber, Stripe) uses live/continuous mechanisms for the final gate; no source anywhere shows recorded-replay-only clearing a financial cutover (D6 Q3/Q4, high-confidence inference from consistent absence of counterexamples)	Open question, escalated below (§13) rather than assumed either way — recorded replay de-risks earlier stages; final gate ownership (harness vs. Zaruba’s oracle plane) is unresolved by design, not by oversight
Diff noise → false DIVERGED → trust collapse → bypass	Named usability risk, initial-recommendation §10; corroborated structurally by every case study that shipped normalization tooling (D2 §11.5)	Kernel v0 = noise normalization; self-replay-to-PARITY falsification recipe is the harness’s only evidence source since no one else published theirs (D6 Q2)
Shadow/live traffic only validates the hot path; COLD/DARK capabilities need separate verification	Uber’s own disclosed limitation on the ledger migration — shadow validation “couldn’t guarantee historical data accuracy” (D2 §6c, §11.7)	Recorded-fixture replay (not live-traffic-dependent) is the v1 default specifically because it can target COLD/DARK capabilities live traffic would never exercise; batch/cron (DARK, structurally uncapturable by an HTTP proxy — D4) remains a named gap, not solved
EOL lab infeasible on the hardest stacks (Rails 3.2/JRuby 1.7/Ruby 1.9.3)	D3: highest-risk tier, unresolved Docker manifest-v1 deprecation issue, TLS-1.2 rubygems floor, no source demonstrating an end-to-end solve at this project’s full difficulty stack	Beachhead pilot targets an easier stack combination first (D3’s risk tiers are uneven; ActiveMQ and MySQL 5.x are lower-risk); Orders’ hardest-stack repos are sequenced, not first

9. Expansion

Old-estate 1→N split lanes, generalized past Orders, arrive only after a flagship gate proves the adapter economics (§8, row 1’s detection threshold). Sequence: pilot → Orders → next hub by in-degree (voucher-inventory 8, deal-catalog-jtier 7, users-service 6, billing-record 5 — context brief) → long tail. No parallel multi-domain build-out before Orders clears M4; the kernel’s minimalism principle (§6.7) is the guardrail against premature generalization.

10. Non-goals (explicit)

No live prod traffic ownership. Recorded-fixture replay is the harness’s default and v1 scope; if the ledger gate needs live traffic (§8 open risk), that machinery is Zaruba’s oracle plane to own and operate, not the harness’s.
No performance parity in v1. The p99 ≤ +10% DONE row is production telemetry the program owns; the harness proves functional parity only (initial-recommendation §9).
No UI product. CLI-first, CI-native, JSON verdicts for agents; one Grafana panel for humans. No interactive UI (initial-recommendation §6-7).
No generic QA platform. Adapters generalize only per flagship gate; the harness never becomes “test everything,” only “prove this signed capability matches.”

11. North Star metric

Candidates evaluated

#	Candidate	What it rewards	Failure mode
a	Units promoted to PR lane with PARITY evidence per week	Throughput	Explicitly named gameable in scope: raw verdict counts reward trivial fixtures — an agent can farm easy, low-value units for volume while HOT capabilities with real behavioral risk sit unverified
b	% of signed HOT/WARM capabilities with parity coverage	Coverage of the denominator that matters	Denominator is fixed by the audit panel (HOT/WARM classification), not by the harness team or the agents running it — resistant to the gaming in (a) by construction
c	Divergences caught pre-merge that would have shipped (true-positive catches)	Value delivered	Denominator is counterfactual and unmeasurable without human adjudication of “would it really have shipped broken” — no clean machine-checked measurement method exists; valuable as a quality sub-metric (§12), not a north star that must be computed weekly without dispute
d	Time from unit-ready to verdict	Velocity/friction	Says nothing about whether coverage or trust exist — a harness that verifies fast and shallow scores perfectly here while missing the program’s actual goal; belongs in the metric tree as time-to-value, not at the top

Winner: (b) — % of signed HOT/WARM capabilities with parity coverage

Decisive, not “it depends.” (b) wins because its denominator is the one thing in this program nobody can inflate or shrink for the metric’s benefit: the panel signs HOT/WARM scope independently of the harness (audit methodology, context brief). A capability either has a verdict-backed evidence bundle or it doesn’t. This is exactly what QSA/SOX sign-off (DONE row 9) and Tomas’s gate approval need to see, and it directly operationalizes the mission statement (§2): “every capability the strike unit keeps carries a verdict.” (a) fails the prompt’s own anti-gaming test. (c) is real but unmeasurable cleanly at reporting cadence — demoted to a quality sub-metric fed by periodic sampling (§12). (d) is necessary but not sufficient — a fast, low-coverage harness is a failed harness by this program’s own definition of done.

North Star: % of signed HOT/WARM Orders-chain capabilities with a current parity-coverage verdict (PARITY or DIVERGED-with-open-question; UNVERIFIABLE does not count as coverage).

Denominator source: the panel’s signed capability list per flagship (audit methodology doctrine). Numerator: capabilities carrying a non-stale evidence bundle (staleness threshold — see anti-metrics, §12). Reported per flagship, rolled up program-wide once ≥2 flagships have adapters.

12. Metric tree

Activation — first PARITY verdict produced end-to-end (record → normalize → replay → diff → verdict row in work-ledger) on the beachhead pilot. Binary, one-time, gates “the kernel exists” before Orders work starts.

Time-to-value — median hours from unit claimed in work-ledger to first verdict returned. Feeds engineering scheduling; not the north star (see §11 rejection of candidate d) but tracked so verification never silently becomes the schedule (named risk, §8).

Engagement — verify runs per agent per day. A proxy for “is the harness the path of least resistance for the fleet,” not for coverage; a healthy number here with flat coverage growth is itself a signal something is wrong (agents re-running instead of covering new capabilities).

Quality — false-DIVERGED rate.

Measurement method: self-replay sampling. Periodically re-run a recorded fixture against the same legacy instance that produced it; any verdict other than PARITY on a self-replay is a false DIVERGED by definition (initial-recommendation §11’s own falsification recipe; D6 Q2 confirms no external false-positive baseline exists to lean on instead, so this internal method is the harness’s only evidence source for its own precision).
Target: near-zero on self-replay; any nonzero rate triggers a normalization-ledger review before it’s allowed to accumulate.
Also feeds candidate (c) above as a bounded proxy: true-positive-catch estimates are only trustworthy once false-DIVERGED is known to be low.

Trust — bypass attempts (units that reached the PR lane without a verdict row — should be structurally zero per principle §6.2, so any nonzero count is a process breach, not a metric to optimize) and normalization-rule audit acceptance rate (% of normalization-ledger entries a human reviewer/auditor accepts on inspection vs. flags as scope-hiding — this is Zaruba’s audit-law honesty principle, §7 of the initial recommendation, made measurable).

Cost — verify cost/unit (tokens + lab minutes) and lab minutes/fixture (legacy-runtime lab boot + record time). Tracked against D3’s MODELED 2-5 engineer-day first-repo lab setup cost so the program can see whether follow-on repos are actually landing under a day as modeled, or whether that inference-by-analogy was wrong.

Reliability — kernel determinism: % of fixtures that reproduce the same verdict on repeat replay with no upstream change (should be ~100%; any drift means the normalization model or the recorder itself is nondeterministic, a kernel bug, not a legacy-system fact).

Guardrails and anti-metrics

Fixture staleness. A fixture that hasn’t been re-recorded since the legacy or candidate side last changed is coverage in name only. Track age-since-last-refresh per fixture; a capability whose fixture is stale beyond a threshold drops out of the north-star numerator even if its last verdict was PARITY.
Normalization-rule count growth without review. Every rule added to the normalization ledger without a logged human rationale is a silent widening of what the harness ignores — exactly the “a parity report that hides its exclusions is a lie with good typography” failure the initial recommendation names (§7). Track rule-count growth rate against audit-acceptance rate (trust metric above); growth outpacing review is a red flag, not neutral progress.
Coverage-without-depth gaming. Because (a) was explicitly rejected as gameable, watch for the north-star metric’s own gaming vector: an agent marking a capability “covered” via a trivially permissive fixture (one happy-path request) rather than a meaningfully risky one. Mitigation: coverage counts require the adapter’s fixture set to include the capability’s Given/When/Then use case scope (audit methodology’s own binding requirement), not merely “a fixture exists.”

Open questions

Does the 99.999% ledger-reconciliation gate require live dual-run machinery, and if so is that the harness’s lane or Zaruba’s? Every credible primary-sourced financial-grade example found (Uber, Stripe) uses live/continuous mechanisms for the final gate; no source shows recorded-fixture-replay-only clearing a comparable bar (D6 Q3/Q4). This is unresolved by design in this document — it needs a program-level decision, not a product-strategy assumption.
What is the legacy estate’s own structurally-expected noise floor, analogous to Uber’s “~10 corruptions/trillion records from S3’s 11-nines durability” reasoning (D2 §11.1)? Without a sourced floor, the ≥99.999% gate risks either chasing statistical noise or, worse, being set arbitrarily. This needs input from Zaruba’s plane on what the legacy MySQL/MBUS stack’s own known failure modes are.
Coverage-target trajectory for the north star — no interim percentage target by week 6 / M2 is set here; that belongs in the roadmap deliverable, not this document, but it should be pinned before execution starts so “% coverage” has a pass bar, not just a direction.
False-DIVERGED self-replay sampling cadence and threshold — this document specifies the method (self-replay) but not frequency or the numeric bar that triggers a normalization-ledger review; needs calibration once the pilot produces real fixture volume.