Product architecture — Dual-run parity harness (“quality-control bench”)

2026-07-18 · Author: Fable · Plan 002, template §12 (product architecture — deepest technical doc). Binding inputs: 00-context-brief.md, initial-recommendation.md, plans/001-.../00-SYNTHESIS.md verdict #4 + architect addendum #4, discovery notes d1–d6.

Reinterpretation note (internal product): “buyer” = program gates (Zaruba’s DoD rows + Robert’s investment call); “procurement” = lane approval from Tomas; “churn” = the fleet routing around the harness (bypass); “pricing” = engineering-days + LLM tokens. Every SaaS-template concept below is read that way; where a template section does not survive that translation it is scaled down in one line, not padded.

Confidence tags used throughout: confirmed fact / supported interpretation / hypothesis / assumption / unknown; numbers carry MEASURED / MODELED / DESIGN. External claims carry URL + date pulled from d1–d6.

1. Principles (design invariants)

Determinism is the product, not a feature. A verdict must be a pure function of (fixture set, adapter, normalization rules). Same inputs → same verdict, byte-for-byte, on any machine, forever. Any nondeterminism in the harness itself (map ordering, wall-clock, RNG, floating-point locale) is a defect of the same severity as a wrong verdict. Rationale: the harness’s own falsification recipe (self-replay must yield PARITY, initial-rec §11) only works if the pipeline is deterministic; and QSA/SOX evidence (DoD row 9) that cannot be re-derived identically is not evidence.
Replayability over liveness. Default mode is record-once → replay-on-demand against a lab booted when needed. Live dual-run is reserved for exactly the gate that provably requires it (money-chain ledger reconciliation — see §2 boundary). Justification: every financial-grade parity case found uses live traffic for the final gate (Uber, Stripe — d6 Q4), but recorded replay is what de-risks the hundred functional endpoints cheaply and reproducibly before that gate.
Graceful UNVERIFIABLE over false verdicts. Three verdict states, not two. When the harness cannot honestly compare (lab won’t boot, fixture is a DARK batch job an HTTP proxy can’t capture, downstream stub diverged), it emits UNVERIFIABLE(reason) — never a guessed PARITY or a noise-driven DIVERGED. A false PARITY ships a regression at 99.999% stakes; a false DIVERGED burns fleet tokens and collapses trust into bypass (initial-rec §10, the top usability risk). UNVERIFIABLE is the honest third answer that keeps both failure modes off the table.
Adapters are the product; the kernel stays small. Do not build a generic multi-domain parity platform. Build the Orders adapter, extract into the kernel only what a second adapter demonstrably reuses, and generalize further only when a flagship gate pays for it. Premature generalization is the named 001 risk #1 (2nd adapter >2× the first = failure signal). The kernel is plumbing; the per-domain adapter carries the judgment.
The report states its own limits. Every verdict ships a normalization ledger (what the diff ignored + why) and a replay recipe (how any reader falsifies it). A parity report that hides its exclusions is, in Zaruba’s audit terms, a lie with good typography.
The safe path is the only path. A unit cannot reach the PR lane without a verdict row in work-ledger. Verification is structural, not disciplinary — nobody has to remember to verify (Norman-lens; TSB’s negative control, d2 §9: rehearsals + human sign-off is not a substitute for a machine gate).

2. Context & boundaries

2.1 System context

The harness lives outside Encore as a TypeScript workspace package + CLI + CI runner (binding, architect correction #4 — Cloud Run cannot run pinned EOL images or tap MBUS). It touches four external planes: the legacy-runtime lab (which it owns), the new Encore.ts services under test, the work-ledger/triage surface inside Encore (its only write into the Encore world), and Grafana/OTel for its own observability.

C4Context
    title System context — dual-run parity harness
    Person(agent, "Fleet coding agent", "Primary user; consumes verdicts as work-ledger rows")
    Person(builder, "Strike-unit builder", "Consumes evidence bundles on DIVERGED")
    Person(auditor, "QSA / SOX / Tomas", "Consumes evidence for DoD sign-off")
    System_Boundary(harness, "Parity harness (outside Encore)") {
      System(kernel, "Parity kernel + adapters", "record → normalize → replay → diff → verdict")
      System(lab, "Legacy-runtime lab", "Pinned EOL Docker stacks; isolated, no egress")
    }
    System_Ext(encore, "Encore.ts services", "Candidate (new) system under test")
    System_Ext(ledger, "work-ledger + triage (in Encore)", "Verdict rows; question minting")
    System_Ext(obs, "Grafana / OTel", "Harness self-observability")
    Rel(agent, kernel, "harness verify <unit>")
    Rel(kernel, lab, "boot / record / replay")
    Rel(kernel, encore, "replay candidate requests")
    Rel(kernel, ledger, "write verdict rows / mint questions")
    Rel(builder, ledger, "read evidence bundle on DIVERGED")
    Rel(auditor, ledger, "read verdicts for sign-off")
    Rel(kernel, obs, "emit harness health metrics")

2.2 Trust boundaries

The legacy-runtime lab is a hostile-adjacent sandbox. Every image in it is intentionally EOL (Ruby 1.9.3–2.7, Rails 3.2/4.1, JRuby 1.7, Java 8/11 — MEASURED, d5 §c), i.e. carrying years of unpatched CVEs by design. It runs with: no egress (default-deny network policy; the lab reaches only its own compose-internal services and the recorder), isolated internal-only Docker network, non-root USER, read-only root filesystem + tmpfs scratch (d3 §6, standard hardening). The lab holds legacy credentials; those never enter Encore secrets (binding). The boundary between lab and everything else is treated like a boundary to untrusted code, because it effectively is.
Fixtures cross a PII/PCI boundary at capture. Every Orders-chain fixture class touches PII or PCI somewhere (card tokens/SEPA, GDPR-erasure payloads, device/fraud signals — MEASURED, d5 §b). Masking happens before a fixture is frozen and leaves the lab (§4). Unmasked capture never persists.
The Encore surface is verdict rows only. The kernel writes nothing into Core-owned tables directly; a work-ledger service inside Encore owns the rows, respecting the cross-plane proposal pattern and keyset-pagination convention (d4 §d).

2.3 Boundary vs Zaruba’s plane — who owns the LIVE dual-run at write-ramp

This is the load-bearing boundary decision. Zaruba’s vocabulary (d4 §c, 05a...appendix.md R4) defines dual-run over three shadow-checkability classes: YES (dual-run proves equivalence from production traffic), COMPARE-ONLY (writes execute once in legacy, new system computes-but-does-not-execute, outputs diffed), NO (verified another way). His R4 rows 24/25 (order money → ledger, voucher lifecycle → ledger) say verbatim “YES — this IS the reconciliation harness” — meaning the existing merchant-accounting reconciliation (reconciliation.service.ts, feeding on the keyset _billingEventsFeed, “the seed of the ≥99.999% reconciliation check”, d4 §a) already counts as the dual-run for the money rows.

Two options for who owns the live dual-run during the 1→100% write ramp:

	Option A — harness owns live shadow	Option B — Zaruba’s oracle plane owns the money-chain live gate (RECOMMENDED)
Mechanism	Harness gains a live-shadow mode: continuously mirror production write-path traffic to candidate, diff in-flight	Merchant-accounting reconciliation + cross-check oracles run continuously on the live billing-event feed; harness owns recorded-replay functional parity + MBUS shadow-taps only
Fit to evidence	Live shadow is what financial-grade cutovers use (d6 Q4) — but no OSS foundation exists (d1), and it duplicates machinery Zaruba already has	Reuses the one existing verdict-shaped code in the estate (d4 §b); recorded replay de-risks the broad endpoint surface where it is cheapest
Cost	New always-on live-mirror infra + the doubling-traffic tax (Zalando, GitHub “you take a performance hit” — d2 §6)	Near-zero new live infra; harness stays recorded-replay-first (principle 2)
Risk	Harness becomes the schedule (001 pre-mortem #2); live-shadow of a pinned EOL estate is exactly the mismatch d1 flags for Diffy	Two lanes must agree on the verdict contract; cold/DARK data still needs a backfill pass (Uber’s disclosed limitation, d2 §7)

Recommendation: Option B. The live money-chain ≥99.999% ledger-reconciliation gate belongs to Zaruba’s oracle plane (it already is the reconciliation harness for those rows, and every credible financial-grade precedent runs live continuous, not recorded replay — d6 Q4, confirmed pattern / interpretation on our specific estate). The parity harness owns recorded-replay functional characterization of the ~127 Orders endpoints + the MBUS event surface, and the normalization/verdict/evidence contract both lanes share. The harness explicitly does not clear the 99.999% gate by recorded replay — no evidence supports that reading (d6 Q4, confirmed absence of counterexamples). If a live-shadow mode is ever needed for a non-money domain, it is added to the kernel then, not speculatively now (principle 4).

3. Components

Each component: responsibility · inputs · outputs · failure behavior · build-vs-buy. “Buy” here means adopt-OSS; there is no commercial procurement.

3.1 Recorder (outside-in proxy)

Responsibility: capture the three traffic classes (sync HTTP, async MBUS+outbox, batch/cron) from the legacy lab without modifying legacy code.
Inputs: legacy traffic through a network-layer proxy (HTTP) + MBUS shadow-taps (§3.8); optional seed corpus of real requests.
Outputs: raw capture stream → handed to masker → fixture store.
Failure behavior: if capture drops or the proxy can’t attach, the recording is marked incomplete; no partial fixture is frozen. Batch/cron traffic is structurally uncapturable by an HTTP proxy (d5 §d) — the recorder emits UNVERIFIABLE(reason="batch-class, needs DB-txn-log tail or job instrumentation") rather than silently missing it.
Build-vs-buy: ADOPT mitmproxy as an external process (MIT, 44k stars, pushed 2026-07-18 — d1 §3, the most mature/active tool in the whole survey; supports client- and server-side replay natively). Rejected: GoReplay (LGPL core fine but masking + binary-protocol replay are Pro-gated — d1 §3, d3 §5); Keploy (eBPF capture unproven against JRuby 1.7 / Ruby 1.9.3 syscall patterns — d1, unknown); VCR (requires legacy Gemfile changes, stuck on 2015-era line for oldest stacks — d3 §5). mitmproxy has no built-in masking → masking is a custom addon downstream (§3.2), same for any tool since no free tool masks HTTP fixtures (d3 §7).

3.2 Fixture store (content-addressed, masked, versioned)

Responsibility: the source of truth for recorded ground truth. Content-addressed (hash of masked content = id), immutable once frozen, versioned per stack + normalization-rule-set.
Inputs: masked capture from recorder.
Outputs: deterministic fixture sets keyed by (domain, endpoint/topic class, stack version).
Failure behavior: a fixture that fails its own mask-verification gate (PII regex/entropy scan tripped) is quarantined, never stored.
Build-vs-buy: BUILD. No adoptable tool (VCR cassettes and WireMock mappings are format-adjacent but neither is content-addressed, masked, nor cross-language — d1 §4/§5). Small, boring, ours. Data model in §4.

3.3 Legacy-runtime lab

Responsibility: boot the pinned EOL stack for a domain, expose it to the recorder/replayer, tear down.
Inputs: an image-matrix profile per stack combination; masked DB seed.
Outputs: a running, health-checked legacy instance on the isolated network.
Failure behavior: if an image won’t pull (Docker manifest-v1/schema-1 deprecation, docker-library/ruby#452, live/unresolved as of 2026-07-18 — d3 §1) or bundle install can’t reach rubygems (TLS 1.2 floor since Jan 2018 — d3 §2), the lab fails the boot health check and the run is UNVERIFIABLE, never a false verdict against a half-booted app.
Build-vs-buy: BUILD (docker-compose profiles). Image matrix (MEASURED, d5 §c): Orders chain needs 6 distinct pinned runtime/framework images, no reuse — Ruby 2.4.6/Rails 3.2 (orders), Ruby 1.9.3-on-JRuby-1.7/Rails 3.2 hand-patched (voucher-inventory, the hardest boot in the estate), Ruby 2.6.10/Rails 4.1 (accounting), Ruby 2.7.5/Sinatra (users), Java 8/Spring+Hibernate 3.6 (billing-record), Java 8-11/Dropwizard (orders_mbus_client). Risk tiering (d3 §8): ActiveMQ low (a current apache/activemq-classic image exists — 5.x is not truly EOL), MySQL 5.x moderate (cytopia/mysql-5.6 third-party fills the gap), the Ruby/JRuby/Java-7 tier highest (manifest-pull + TLS + native-gem/archived-apt stack, no source demonstrates the full combination solved end-to-end in 2025–26). Mitigations baked in from day one: rescue-pull old images to an internal registry (auto-upgrades manifest), vendor gems / internal gem mirror (never touch live rubygems at boot). Per-repo first-boot cost ~2–5 engineer-days for a new stack combination, <1 day for follow-on repos reusing a solved combination (MODELED, d3 §8 — the follow-on figure is inference by analogy, not sourced).

3.4 Normalizer (rule engine + normalization ledger)

Responsibility: strip known-noise (timestamps, UUIDs, ordering, float formatting) from both sides before diffing, and record every rule it applied into the normalization ledger.
Inputs: legacy response + candidate response (or bus event pair); the active rule set.
Outputs: normalized pair + a normalization-ledger fragment (rule id, field path, rationale).
Failure behavior: a rule that would suppress a whole response body (over-broad) is rejected at rule-authoring time by the self-replay gate (§8); an unknown field that no rule covers passes through to the differ unmodified (fail toward DIVERGED, not toward hidden PARITY).
Build-vs-buy: BUILD. Two noise strategies, both used: (a) explicit hand-curated normalization ledger (Scientist ignore blocks, Zalando header-filtering — d2 §5; the load-bearing feature, not polish), and (b) A/A self-replay noise floor — replay the same recorded request twice against the lab (or two lab instances) to auto-learn inherently-noisy fields, borrowed conceptually from Diffy’s relative-disagreement-rate mechanism (d1 §2a, d6 Q2). We steal Diffy’s design, not its code (opendiffy is CC BY-NC-ND, not usable; twitter-archive is abandoned Apache-2.0 — d1 §2a). Note: Diffy’s real false-positive rate was never published anywhere (d6 Q2, unknown) — we cannot lean on its track record, we prove precision ourselves via the self-replay gate.

3.5 Differ (deterministic structural diff)

Responsibility: structural diff of normalized pairs — JSON bodies, raw bodies, headers (allowlisted), status codes, and bus-event envelopes/payloads.
Inputs: normalized legacy + candidate pair.
Outputs: an ordered, minimal set of field-level differences (the counterexample seed).
Failure behavior: must be total and deterministic — never throws on malformed input, emits a structured “unparseable, cannot compare” that becomes UNVERIFIABLE.
Build-vs-buy: BUILD. Nothing adoptable: Diffy blocked (license/archived/live-only architecture mismatch — d1 §2a), Hoverfly diff-mode is Go-native and “live-vs-recorded-snapshot” framed with no verdict/severity/bus model (d1 §2b, ~25% fit), no OSS diffs two systems’ JMS/STOMP/Kafka events at all (d1 §11, confirmed gap). The pure-function shape already exists in-estate to generalize: computeReconciliation (reconciliation.utils.ts:20-41) / crossCheckBillingFeed (reconciliation.utils.ts:79-104) are pure, deterministic, and return a gaps: CrossCheckGap[] (interfaces/merchant-accounting.interfaces.ts:140-146) that is the closest thing to a counterexample payload in the whole repo (d4 §b).

3.6 Verdict engine (PARITY / DIVERGED / UNVERIFIABLE + thresholds per traffic class)

Responsibility: turn a diff set + noise floor into one verdict, with per-traffic-class thresholds and an expected-no-op class.
Inputs: diff set, A/A noise floor, adapter-declared thresholds.
Outputs: PARITY | DIVERGED(smallest counterexample) | UNVERIFIABLE(reason).
Failure behavior: ambiguous → UNVERIFIABLE, never a coin-flip.
Build-vs-buy: BUILD. Critical subtlety from d4 §b (design source: _documentation/backend/ledger-design.md:36-46): a naive “not in ledger ⇒ DIVERGED” misclassifies every lifecycle/void event (REDEEMED, UNREDEEMED, FULFILLED, VIEWED, CANCEL, RESCIND_AUTHORIZATION) that is correctly received but never posted, per the ConsumptionStatus enum (interfaces/merchant-accounting.interfaces.ts:62-66). The verdict model therefore needs a three-way classification mirroring ConsumptionStatus (posted / received-expected-no-op / unpublished-gap), not binary PARITY/DIVERGED. Thresholds are per-adapter config with program-owned values (§9 ADR-5); the harness does not invent the 99.999% number, it reads it. No source chases 100% parity — every rigorous case defines a noise budget with a stated source (Zalando “fixing those last few percentages has a cost higher than the value it brings”; Uber ledger set a deliberate 99.9999% upper bound — d2 §11, MEASURED).

3.7 Per-domain adapter SPI

Responsibility: the interface a domain implements to plug into the kernel. This is the product.
An adapter implements: (1) which fixture classes exist for the domain + how to record them, (2) the independent oracle source(s) to compare against, (3) domain-specific normalization rules, (4) thresholds (from program config), (5) the counterexample shaping.
Shape (DESIGN, d4 §b): generalize the existing pattern — Oracle<TFeedEvent, TCheckSource> = (feedWindow, independentSource) => { posted | gap | unverifiable }[], with the Orders adapter wrapping the existing 3-oracle reconciliation almost verbatim: order-store cross-check (built: crossCheckBillingFeed, exposed as _reconciliationOrderCrossCheck / POST /merchant-accounting/internal/reconciliation/order-cross-check; catches “the exact gap class the 15 legacy [Quartz] jobs find” per ledger-design.md:36-46), provider-settlement (typed stub only — interface fixed at interfaces/merchant-accounting.interfaces.ts:171-188, endpoint _reconciliationProviderSettlement is TODO pending the payment adapter), contract-check (not built, not stubbed — no mention anywhere else in the repo). Do not assume all three oracles are reference implementations — only 1 of 3 exists in code (d4 open items).
Build-vs-buy: BUILD, and build only Orders first (principle 4). Extract kernel API from the second adapter, not by speculation.

3.8 MBUS shadow-tap

Responsibility: capture and diff async bus events — legacy JMS/STOMP (messagebus client) and Kafka — including the outbox durability contract, not just payload parity.
Inputs: tap points on the domain’s topics; the corresponding outbox-DB rows.
Outputs: bus-event fixtures (topic, envelope, body) + the outbox row asserting “durably queued exactly once”.
Failure behavior: a tap that can’t attach → that topic class is UNVERIFIABLE.
Build-vs-buy: BUILD, highest-uncertainty piece of the whole kernel (d1 §11 — no OSS foundation exists for cross-system bus diffing). Scope is real: Orders alone has 25+ jms.topic subscriptions + a separate Kafka txn.signals.high; orders_mbus_client consumes 8 more; Orders runs its own outbox DB (orders_msg_prod), users-service a bespoke sweep daemon (MEASURED, d5 §a/d). Harness-internal only — the standalone MBUS bridge app is DROPPED (binding, 001).

3.9 Evidence-bundle generator

Responsibility: produce the one human-readable artifact per divergence — verdict, smallest counterexample, normalization ledger, replay recipe.
Inputs: verdict + diff + normalization fragments + fixture ids.
Outputs: a content-addressed evidence bundle (auditor-grade, DoD row 9).
Build-vs-buy: BUILD. Study ApprovalTests’ data model for how approved baselines are versioned/auditable (d1 §9, ~15% fit — worth the pattern, not the engine).

3.10 work-ledger client

Responsibility: write verdict rows into the Encore-side work-ledger; the only harness write into Encore.
Inputs: verdicts from the kernel.
Outputs: append-only verdict rows; gates PR-lane promotion.
Nearest existing precedent, rejected as insufficient (d4 §a): workflow-management’s workflow_executions table is the closest thing in the repo to a “unit of work with a status,” but it is Temporal-run-shaped, not agent-task-shaped — work-ledger cannot be built as a thin view over it; it needs its own new schema.
Constraints (binding, d4 §d): new Encore service self-registers via registerInternalService from encore.service.ts; append-only rows use keyset pagination ({cursor, from, to, limit} → {items, nextCursor}, the _billingEventsFeed reference convention); writes into any Core-owned table go through a Core proposal endpoint (submit → validate → apply/reject → audit log), never a direct table write; layered controller→service→repository, ...defaultTableColumns, migrations via pnpm drizzle only.
Build-vs-buy: BUILD (Encore service). No verdict/work-ledger surface exists in-repo (d4 §a, confirmed).

3.11 Triage hook (question minting)

Responsibility: on DIVERGED, mint a deduped triage question for a human/agent group.
Build-vs-buy: BUILD thin, piggyback existing. Reuse Tasks TaskType.INFORMATION_REQUEST (tasks.interfaces.ts:10-14, alongside APPROVAL/SIGNATURE) + createFromWorkflow idempotent mint path (tasks.service.ts:43-84, stamped on workflowId/runId/signal, d4 §a), targeting assigneeUserId or assigneeGroupSlug (interfaces/tasks.interfaces.ts:52-56). Two frictions to resolve: (a) minting is not a standalone primitive — it structurally requires an owning Temporal workflow run, so the harness needs a thin wrapper workflow or a deliberate bypass that loses idempotency; (b) no assigneeGroupSlug registry exists — routing DIVERGED questions to a group is greenfield even in Zaruba’s repo. Completion is also not a harness read: a Task’s complete() (Admin-ACL-gated) or completeByToken() writes into Postgres and signals back into the owning workflow (_workflowSignalSend, tasks.service.ts:173-181) — the harness must treat “triage answered” as an external signal into work-ledger, not as a Task read. Recommendation: thin wrapper workflow, accept the coupling, define the first slug set.

3.12 CI runner

Responsibility: run harness verify in CI, gate promotion.
Build-vs-buy: BUILD — first backend CI surface in the repo. No backend-ci.yml exists; only frontend workflows (frontend-ci.yml, frontend-deploy.yml, deploy-frontend.yml, publish-distributed-clients.yml, d4 §e, confirmed). Mirror frontend-ci.yml shape (path-triggered, git diff-scoped, concurrency cancel, timeout-minutes: 20), invoking the existing pnpm validate gate (root package.json:20: check:ci && docs-sync --check && type-check && test) plus harness verify. Test runner is vitest across all workspaces, mirrored source tree convention; merchant-accounting and order already carry tests/utils/reconciliation.utils.test.ts — the reference test to model the generalized oracle’s own tests on (d4 §e).

3.13 LLM triage assistant (ADVISORY ONLY)

Responsibility: cluster and explain divergences to speed human triage. Never a verdict authority.
Hard rule: the deterministic differ (§3.5) is the only verdict authority. An LLM may cluster similar DIVERGEDs, draft a plain-language “likely cause,” and suggest a candidate normalization rule for human approval — it may never pass, fail, or mint a PARITY. Evidence: LLM-as-judge-for-equivalence literature (2024–26) is benchmark-stage and uniformly frames LLM judges as “triage and assistance, not definitive gates” (d6 Q6, arxiv 2510.24367, 2025-10-29, moderate confidence); no paper recommends trusting an LLM judge as sole equivalence gate. Cost controls in §5.
Build-vs-buy: BUILD thin wrapper; POSTPONE past MVP (initial-rec §9 — embedding clustering is explicitly not-in-MVP).

4. Data architecture

4.1 Fixture entity model

Fixture
  id                  content hash of masked payload (address = identity)
  domain              e.g. "orders"
  class               endpoint/topic equivalence class (e.g. "order.create.local")
  traffic_class       sync_http | mbus_event | batch (batch = capture-metadata only)
  stack_version       pinned image tag the fixture was recorded against
  request             method/path/headers(allowlisted)/body   (masked)
  response            status/headers(allowlisted)/body          (masked)  [sync_http]
  bus_envelope,body   topic + envelope + payload                (masked)  [mbus_event]
  outbox_assertion    "durably queued exactly once" ref          [mbus_event, publish side]
  mask_manifest       which rules masked which fields (audit)
  recorded_at, frozen_at
  retention_class     PII | PCI | non-sensitive  → drives expiry
NormalizationRule
  id, field_path, matcher, rationale, author, approved_by, self_replay_passed
  version, supersedes
Verdict  (also mirrored as a work-ledger row)
  fixture_set_id, adapter, rule_set_version, unit_id
  state               PARITY | DIVERGED | UNVERIFIABLE
  counterexample      minimal diff  [DIVERGED]
  reason              [UNVERIFIABLE]
  evidence_bundle_id, created_at

4.2 Lifecycle: record → mask → freeze → replay → expire

record (raw, in-lab, never persisted unmasked) → mask (PII/PCI scrubbed by rules; mask-verification gate) → freeze (content-address, immutable, versioned by stack + rule-set) → replay (deterministic, N times, same result) → expire (retention-class-driven deletion).

4.3 Retention / deletion (PII/PCI)

Assumption (needs security sign-off, binding): masking removes PANs/IBANs/emails/device-ids at capture; a masked fixture is the lowest-sensitivity artifact that still proves parity. No off-the-shelf tool masks HTTP fixtures (d3 §7, confirmed) → custom scrubber (masking pattern: anonymize before data leaves the lab, inline during capture, never after landing — d3 §7).
Retention class drives expiry: PCI fixtures shortest-lived and access-logged; GDPR-erasure fixtures (orders_mbus_client consumes gdpr.account.v1.erased by construction — d5 §b) must themselves honor erasure — a fixture derived from an erased account is deleted on erasure propagation. Open for security sign-off: exact retention windows per class (unknown — not set by any source).

4.4 Source of truth, versioning, audit trail

Source of truth for ground truth = the frozen fixture set (not the live legacy estate, which cannot be kept always-on — d1 §2a boot-pain). For the money-chain live gate, source of truth = the billing-event feed in Zaruba’s plane (§2.3, Option B).
Versioning: a verdict is only meaningful as (fixture_set_version, adapter, rule_set_version). Bumping any of the three re-opens verdicts — a stale fixture against a new stack is UNVERIFIABLE until re-recorded.
Normalization-rule audit trail: rules are themselves auditable scope decisions (append-only, supersedes chain, approved_by, self_replay_passed). Adding a rule that turns a DIVERGED into PARITY is a recorded decision with an author, not a silent config edit — this is Zaruba’s audit law applied to our own tool.

5. AI section

Where LLM is used: exactly one place — the advisory triage assistant (§3.13): clustering many DIVERGEDs, drafting plain-language cause explanations, proposing candidate normalization rules for human approval.
Hard rule (restated because it is the whole point): the deterministic differ is the only verdict authority. No LLM output ever becomes a verdict, gates a promotion, or mints a PARITY. This is principle-based and evidence-backed (d6 Q6), not a stylistic preference.
Cost controls: advisory only runs on DIVERGED (not on the PARITY majority path — the fleet’s default outcome should cost zero tokens); batch-cluster per triage session, not per divergence; cache explanations by counterexample hash (identical counterexample → cached explanation); hard per-run token cap; log every run in the project’s LLM run-evidence ledger (lane + billed cost re-summed from real usage — no silent spend, per global rule). POSTPONE entirely past MVP — the fleet reads structs, not prose, in v1.

flowchart LR
  D[DIVERGED verdicts] --> C{cluster by counterexample hash}
  C -->|cached| X[reuse explanation]
  C -->|new| L[LLM: explain + suggest rule]
  L --> H[human triage: approve rule / real regression / accept]
  H --> R[(NormalizationRule\nappend-only, approved_by)]
  L -.advisory only.-> V[[deterministic differ = sole verdict authority]]
  style V fill:#222,color:#eee

6. Integrations

Integration	Direction	Mechanism	Retry / idempotency
work-ledger	harness → Encore	verdict rows via work-ledger service (proposal endpoint if Core-owned)	rows keyed by `(unit, fixture_set_version, rule_set_version)`; re-verify is idempotent upsert, append-only history
Triage	harness → Encore Tasks	`INFORMATION_REQUEST` via `createFromWorkflow`, stamped	idempotent on workflow stamp; retry finds existing task, does not duplicate (d4 §a)
Zaruba oracles	harness ← read-only	generalize `crossCheckBillingFeed` shape; read the billing-event feed via keyset pagination	read-only; harness never writes into merchant-accounting; feed is append-only keyset
CI	CI → harness	new `backend-ci.yml`, path-triggered, diff-scoped, runs `pnpm validate` + `harness verify`	`concurrency` cancels superseded runs; verify is deterministic → safe to re-run
Grafana / OTel	harness → obs	emit parity coverage / divergence rate / self-replay health	metrics only; local-first

The billing-event feed itself (_billingEventsFeed.controller.ts:1-38, GET /order/internal/billing-events, expose:false auth:false) is consumed today only by merchant-accounting’s reconciliation service and its own subscription (billingEvent.subscription.ts, topic billing-event) — the harness would be its second consumer ever, not plugging into an established multi-consumer feed (d4 §a).

Integration seam to resolve (d4 §d, confirmed): the internal reads the harness wants (_billingEventsFeed, reconciliation endpoints) are all expose:false, reachable only via the Encore-generated internal client in-cluster. An external CLI runner cannot call them directly → needs either a purpose-built Core-side authenticated read adapter, or that one hop runs inside the Encore boundary. This is a real seam, not assumed away.

7. Security

7.1 Threat model

Threat	Vector	Guardrail
EOL image exploitation	Every lab image carries unpatched CVEs by design	Hostile-adjacent sandbox: no egress, internal-only network, non-root, read-only FS + tmpfs (d3 §6); EOL-alerting explicitly allowlists lab-tagged images while staying strict fleet-wide
Poisoned fixture	A malicious/corrupt fixture drives a false PARITY	Content-addressing (tampering changes the id); fixtures immutable once frozen; self-replay gate (§8) fails if a fixture no longer reproduces PARITY against its own stack
PII/PCI leakage	Unmasked capture persists or leaves the lab	Mask-before-freeze; mask-verification gate quarantines any fixture tripping PII scan; retention-class expiry; access-logged PCI class (§4.3)
Accidental replay against prod	Replayer points at a production endpoint instead of candidate/lab	Target allowlist (replayer refuses any host not on the explicit candidate/lab list); lab has no network route to prod (no egress); mutating verbs OFF by default (Diffy’s own default — POST/PUT/DELETE gated behind an explicit flag, d1 §2a)
Legacy credential exposure	Legacy creds leak into Encore secrets or CI logs	Legacy creds stay in the lab, never in Encore secrets (binding); lab env is compose-local, not committed; CI never echoes lab env

7.2 Secrets & isolation

Legacy creds: lab-local only. Encore secrets: candidate/work-ledger only. The two secret domains never mix (binding).
Isolation is the same boundary as untrusted-code isolation — the lab is treated as compromised-by-assumption and given nothing it doesn’t need (least privilege; disposable, loopback-only).

8. Deployment & ops

Local-first. Runs on WSL2 + docker-compose. No cloud dependency until the billing gate clears (binding). The kernel is a workspace package + CLI; the lab is compose profiles; the Encore surface is verdict rows.
CI runner: new backend-ci.yml (first backend CI in the repo), mirroring frontend-ci.yml (d4 §e).
Observability of the harness itself: the harness’s own health is a first-class signal — the self-replay gate is the harness’s falsification recipe (record → replay against the same legacy instance MUST yield PARITY; d6 open-Q1 notes this is now the only evidence the harness will have, since Diffy-class false-positive rates were never published). Metrics to Grafana/OTel (local-first): self-replay pass rate (must be 100%), parity coverage, divergence rate, UNVERIFIABLE rate (rising UNVERIFIABLE = lab-boot rot, the second usability risk — initial-rec §10), fixture staleness, lab boot success rate. No existing dashboard to extend: crossCheckRange today runs only via the daily cron merchant-accounting-daily-cross-check, logging a structured report with no alerting and no queryable history (d4 §e) — the harness panel is new build, not an addition to an existing surface.
Fastest prototype (initial-rec §11, DESIGN): days not weeks — pin one EOL image, boot one endpoint chain (forex-ng is the ranked cheapest pilot: 2 endpoints, no DB, no MBUS — d5 §e), record ~100 requests, replay against a deliberately-wrong stub, emit verdict + bundle. Tests the two riskiest assumptions at once: EOL bootability and zero-false-DIVERGED on self-replay.

flowchart TB
  subgraph WSL["WSL2 workstation / CI runner (local-first)"]
    CLI[harness CLI]
    subgraph Lab["Legacy-runtime lab (compose, no egress, non-root)"]
      L1[Ruby 2.4/Rails 3.2 · orders]
      L2[JRuby 1.7/Rails 3.2 · voucher-inv]
      MQ[ActiveMQ · JMS/STOMP]
      DB[(MySQL 5.x seed · masked)]
    end
    PX[mitmproxy recorder]
    FS[(fixture store · content-addressed)]
    CLI --> PX --> FS
    CLI --> Lab
  end
  subgraph Encore["Encore.ts (Cloud Run)"]
    CAND[candidate services]
    WL[(work-ledger · verdict rows)]
    TASKS[Tasks · triage]
  end
  CLI -->|replay| CAND
  CLI -->|verdict rows| WL
  CLI -->|mint question| TASKS
  CLI -->|self-replay health| GRAF[Grafana/OTel]

9. ADR table

#	Decision	Choice	Alternatives (rejected)	Trade-off	Reversibility
1	Recorder	ADOPT mitmproxy (external process)	GoReplay (masking/binary Pro-gated, d1/d3), Keploy (eBPF unproven vs JRuby 1.7), VCR (legacy code change)	No built-in masking → custom scrubber; new-language process boundary	High — recorder is behind an interface; swap substrate without touching kernel
2	Fixture format	BUILD content-addressed, masked, versioned JSON manifest	VCR cassettes, WireMock mappings (neither masked/addressed/cross-language)	Own the format; small maintenance	Medium — format migration is a re-record, not a rewrite
3	Diff engine	BUILD deterministic structural differ	Diffy (license/archived/live-only), Hoverfly diff-mode (Go, no verdict/bus model)	Full build, but no adoptable option exists (d1)	Low — the differ is the core; replacing it is replacing the product
4	Lab orchestration	BUILD docker-compose profiles	Kubernetes/Signadot (live-shadow/K8s model, wrong fit for pinned-EOL Docker — d1 §10), Testcontainers	Compose is local-first, matches “no cloud until billing gate”	High — profiles are declarative; re-target later
5	Verdict-threshold ownership	Program-owned values, adapter-declared config	Harness-invented thresholds	Harness reads the 99.999% number, never invents it; keeps parity honest	High — config, not code
6	Live-shadow ownership	Zaruba’s oracle plane owns money-chain live gate; harness = recorded-replay + MBUS (Option B)	Harness owns live shadow (Option A)	Reuses existing reconciliation; harness stays recorded-replay-first; two lanes must share verdict contract	Medium — a kernel live-shadow mode can be added later if a non-money domain needs it
7	Adapter SPI shape	`Oracle<TFeedEvent,TCheckSource>` generalizing `crossCheckBillingFeed`; Orders first	Generic multi-domain platform up front	Avoids premature generalization (001 risk #1); extract kernel from 2nd adapter	High — SPI extracted empirically, cheap to reshape before adapter #2
8	Evidence storage	BUILD content-addressed bundle + verdict rows in work-ledger (Encore/Postgres)	Bundle-in-git (ApprovalTests style), DB blob	Auditor-grade, replayable; two stores (bundle object + row)	Medium
9	Noise strategy	Normalization ledger (explicit) + A/A self-replay floor (Diffy pattern, not code)	Diffy-code adoption (blocked), pure hand-rules, pure statistical	Two mechanisms to maintain; but the ledger is the load-bearing feature (d2 §5)	High — rules are versioned config
10	LLM role	Advisory triage only; deterministic diff is sole verdict authority	LLM-as-judge gate	Zero-token PARITY path; matches all 2024–26 evidence (d6 Q6)	High — advisory layer is removable without touching verdicts

10. Mermaid diagrams

Diagrams 1 (system context) and 7 (deployment) are in §2.1 and §8; the AI-assist flow is in §5. The remaining four:

Containers

flowchart LR
  subgraph Kernel["Parity kernel (workspace pkg)"]
    REC[recorder] --> MASK[masker] --> STORE[(fixture store)]
    STORE --> NORM[normalizer + norm-ledger]
    REPLAY[replayer] --> NORM
    NORM --> DIFF[differ] --> VE[verdict engine]
    VE --> EB[evidence-bundle gen]
    SPI[[adapter SPI]] -.configures.-> NORM & DIFF & VE
    ORD[Orders adapter\nwraps 3-oracle] --> SPI
    TAP[MBUS shadow-tap] --> STORE
  end
  VE --> WLC[work-ledger client]
  VE --> TRI[triage hook]
  EB --> WLC

Core data flow — record → verify

sequenceDiagram
  participant Lab as Legacy lab
  participant Rec as Recorder
  participant Mask as Masker
  participant FS as Fixture store
  participant Cand as Candidate (Encore)
  participant Norm as Normalizer
  participant Diff as Differ
  participant VE as Verdict engine
  Lab->>Rec: legacy traffic (HTTP + bus)
  Rec->>Mask: raw capture
  Mask->>FS: freeze (content-addressed, masked)
  Note over FS: fixture = ground truth
  FS->>Cand: replay same requests
  Cand-->>Norm: candidate responses
  FS-->>Norm: legacy responses
  Norm->>Diff: normalized pair (+ norm-ledger)
  Diff->>VE: minimal diff set
  VE->>VE: PARITY / DIVERGED / UNVERIFIABLE

Primary request — agent verify

sequenceDiagram
  participant Agent as Fleet agent
  participant CLI as harness verify
  participant K as Kernel
  participant WL as work-ledger
  participant T as Triage
  Agent->>CLI: claim unit, run verify <unit>
  CLI->>K: replay fixtures vs candidate
  K-->>CLI: verdict
  alt PARITY
    CLI->>WL: verdict row → unit promoted to PR lane
  else DIVERGED
    CLI->>WL: verdict row (blocked) + evidence bundle link
    CLI->>T: mint deduped question
  else UNVERIFIABLE
    CLI->>WL: verdict row (reason) → not promotable, no false block
  end

Integration flow — ledger / triage

flowchart LR
  K[kernel verdict] --> WLC[work-ledger client]
  WLC -->|proposal endpoint if Core-owned| WL[(work-ledger rows\nkeyset, append-only)]
  WL --> PR{PR-lane gate}
  K -->|DIVERGED| TRI[triage hook]
  TRI -->|INFORMATION_REQUEST\nworkflow-stamped| TASKS[Tasks service]
  TASKS --> HUMAN[builder / group queue]
  HUMAN -->|decision| RULE[(normalization rule\napproved_by)]
  WL --> GRAF[Grafana panel]

11. MVP vs post-MVP · build/buy/postpone/avoid · risks · complexity

11.1 MVP architecture (Orders flagship, week-6 gate)

Four pieces (initial-rec §4): characterization kernel (record→normalize→replay→diff→verdict, deterministic, content-addressed) · legacy-runtime lab booting ONE endpoint chain · Orders adapter wrapping the 3-oracle pattern · verdict rows in work-ledger. Prove it on forex-ng (self-test) before Orders. Noise normalization is kernel v0 scope, not polish (the top usability risk, initial-rec §10).

11.2 Post-MVP

Second adapter (extract the real kernel API here — the generalization test); MBUS shadow-tap hardening across both bus technologies; advisory LLM triage; A/A self-replay noise floor automation; broader long-tail adapters per flagship gate.

11.3 Build / buy / postpone / avoid

Build: differ, verdict engine, fixture store, adapter SPI, MBUS shadow-tap, evidence bundle, work-ledger client, CI runner, masker (no free HTTP-fixture masker exists).
Buy/adopt (OSS): mitmproxy (recorder substrate); apache/activemq-classic (current image); cytopia/mysql-5.6 (third-party MySQL 5.x); design references only — Diffy’s noise-floor idea, ApprovalTests’ baseline-versioning, Scientist’s ignore/percentage-ramp vocabulary.
Postpone: LLM triage; A/A automation; live-shadow mode; adapters #2+.
Avoid: opendiffy/diffy (CC BY-NC-ND, legally blocked); GoReplay Pro dependency; Signadot/Speedscale (live-shadow/K8s model, wrong fit); DB-value diffing (data-diff dead + out of scope); Pact (different problem); any Encore-hosted kernel/lab (architect correction #4); building a generic platform before adapter #2 pays for it.

11.4 Top technical risks

Harness doesn’t generalize past Orders (001 risk #1). Detection: 2nd adapter costs >2× the first. Mitigation: extract kernel API from adapter #2, never speculatively; Orders’ own ~50–60+ fixture classes (MODELED, d5 §d) already justify the narrow first bite.
EOL lab won’t boot at acceptable cost (highest-uncertainty, d3 §8). No source shows the full Ruby 1.9.3/JRuby 1.7 + TLS + archived-apt + native-gem stack solved end-to-end in 2025–26 (evidence gap, not a confirmed failure). Mitigation: rescue-pull + internal registry, vendored gems, the “days not weeks” forex-ng spike first.
MBUS shadow-tap is net-new with no prior art (confirmed gap, d1 §11). Mitigation: rehearse on orders-ext (modern stack, shares a fraud/payment topic family with Orders — d5 §e) before combining bus-tap with an EOL boot.
Diff noise → false DIVERGED → bypass (top usability risk). Mitigation: normalization ledger as v0 scope + self-replay gate; UNVERIFIABLE as the honest third state.
Recorded replay cannot clear the 99.999% money gate (d6 Q4 — every financial-grade precedent runs live). Mitigation: Option B — that gate is Zaruba’s oracle plane, not the harness; harness does functional parity, not the money-chain final gate.
“Harness became the schedule” (001 pre-mortem #2) — and its opposite, under-resourcing (Uber Invoicer: 10% progress in six months as side-work — d2 §6d). Mitigation: cap investment per flagship gate; fallback = impact-selected tests (~70% regression reduction, but single-study, uncorroborated, off-domain — d6 Q7, cite as such).

11.5 Operational complexity estimate (MODELED)

Lab setup: ~2–5 eng-days first repo per stack combination, <1 day follow-on (d3 §8).
Orders chain: 6 distinct pinned images, no reuse (d5 §c); ~50–60+ fixture classes for Orders alone (d5 §d).
Ongoing: fixture staleness refresh + lab-boot maintenance is the recurring cost (the second usability risk); the self-replay gate must stay green as the harness’s own regression test.
The harness is Robert-lane, weeks-not-months, staged per flagship gate (economics anchor, context brief) — the complexity is real but bounded by scoping to Orders first and refusing to generalize early.

Open questions

Money-chain live gate ownership (partially resolved, needs Zaruba confirmation). Option B assigns the live ≥99.999% reconciliation to Zaruba’s oracle plane. His decision log has zero logged decisions on dual-run/shadow/parity (d4 §c) — this boundary is proposed here, not yet ratified by him. Needs explicit agreement on the shared verdict contract.
Internal-endpoint seam (d4 §d): an external CLI runner cannot call expose:false internal reads directly. Purpose-built Core-side authenticated read adapter, or run that hop inside Encore? Unresolved; affects the work-ledger-client and Zaruba-oracle-read integrations.
Retention windows per PII/PCI class (§4.3): unknown, needs security sign-off before any fixture is frozen.
EOL bootability at full combined difficulty (d3 open Qs): is jruby:1.7 / openjdk:7 actually pullable on a current Docker Engine? Does a current ActiveMQ serve 2013–17-era JMS/STOMP clients? Both plausible-but-unverified; the forex-ng→giftcard_service pilot ladder is designed to answer them cheaply.
Batch/cron traffic class is DARK (d5 open Qs): Resque/Quartz job inventories for orders/users-service/voucher-inventory are not itemized. An HTTP proxy cannot capture them; DB-txn-log tailing or job instrumentation is needed. Sizing this class needs a repo-level grep before the 50–60+ fixture estimate firms from MODELED to MEASURED.
Fixture-only replay precision (d6 open-Q1): Diffy’s real false-positive rate was never published anywhere; the self-replay gate is now the only evidence the harness’s noise model works at acceptable precision — it must be proven, not assumed.
assigneeGroupSlug registry (d4 §a): no slug enumeration exists; routing DIVERGED questions to a group is greenfield even in Zaruba’s repo. First slug set to be defined.