Build vs buy — dual-run parity harness
2026-07-18 · Author: Fable · Template §16. Binding inputs: 00-context-brief.md, initial-recommendation.md, and this plan’s competitive-analysis.md, open-source-analysis.md, architecture.md, research.md. This document does no new research — it decides build/buy/partner at the option and component level, and names the conditions under which the decision flips.
Internal reinterpretation (binding, context brief): “buy” = adopt an OSS/commercial dependency through security review; “procurement” = lane approval from Tomas + a security sign-off, not a purchase order; “vendor” = any dependency (OSS maintainer or commercial SaaS) whose survival/licensing we don’t control; “cost/pricing” = engineering-days + agent tokens, not dollars-per-seat. The 001 synthesis already recorded BUILD-flagship (verdict #4). My job here is to attack that verdict at full strength, then conclude with a component split, not to re-ratify it by assertion.
Confidence tags: confirmed fact / supported interpretation / hypothesis / assumption / unknown. Numbers carry MEASURED / MODELED / DESIGN.
1. The honest attack on BUILD (steelman, before the matrix)
Four arguments against building, each taken at full strength (from research.md §4.2). Two are correct and reshape the build; two are wrong. Naming which is the whole point of this document — an unattacked BUILD verdict is a vibe.
| # | Argument against build | Verdict | Consequence |
|---|---|---|---|
| a | “It’s a feature of Zaruba’s plane.” The money gate — the only row with a hard number — is oracle reconciliation over the billing-event feed, and that already exists in Core (reconciliation.service.ts; the feed’s own doc comment calls itself “the seed of the ≥99.999% reconciliation check”). |
Half-correct. | The money computation is Core’s — do NOT build a second money gate. But Core has no verdict enum, no evidence bundle, no work-ledger gating (research.md §2.1, confirmed fact). The money evidence surface is net-new and ours. → Partner for the gate, build the wrapper. |
| b | “Recorded replay cannot honestly clear a 99.999% write-ramp gate.” Every primary-sourced financial-grade cutover used live/continuous dual-run, never recorded-fixture replay (Uber experiment engine >99.999%, uber.com 2024-10-03; Stripe 99.9999%, stripe.dev 2024-02-16). No counterexample exists. | Correct. | The harness’s headline mechanism is disqualified from the headline gate. → Recorded replay owns functional/long-tail/DARK parity; the live money gate is Zaruba’s oracle plane (Option B, architecture.md §2.3, ADR-6). |
| c | “The EOL lab is a tar pit.” Manifest-v1 pull rot (docker-library/ruby#452, live/unresolved), RubyGems TLS-1.2 floor, native-gem builds, 6 distinct pinned runtimes for one chain with no reuse — no published writeup solves the full stack end-to-end. | Real risk, not a kill. | Defuse by sequencing, not denial: the forex-ng → orders-ext → giftcard_service pilot ladder de-risks bootability/TLS/native-gems/MBUS-diff before voucher-inventory’s JRuby-1.7 boot is on the critical path, with a hard investment cap and S3 (impact-selected tests) as the documented rip-cord. |
| d | “The Orders adapter is the whole value; generalizing is waste.” Getting just Orders off is the entire M4 prize; every hour on a ‘generic kernel’ is YAGNI. | Wrong about the wrapper, right about the lab. | Build the verdict/evidence wrapper generic (the fleet gates on it — Frame C); build the lab and adapters strictly per-flagship-gate. 001 already bounds it this way (kernel small, adapters per gate). |
Net: BUILD survives the attack, but as a smaller, differently-shaped thing — a generic verdict/evidence wrapper + a recorded-replay kernel scoped to the no-oracle surface, explicitly NOT a second money gate and NOT a general platform. That reshaping is what the option comparison below has to land.
2. The eight options
Every option scored against the parity kernel: record HTTP+bus → normalize → replay → diff → verdict, cross-language, against a pinned-EOL estate, producing an auditable PARITY/DIVERGED/UNVERIFIABLE verdict wired to work-ledger.
2.1 Build from scratch (everything, including the recorder)
Write a bespoke HTTP capture/replay proxy in addition to the differ/verdict/lab. Rejected. mitmproxy (MIT, 44k★, pushed 2026-07-18 — open-source-analysis.md §2.6) already solves TLS interception, scriptable capture, and both replay directions; rebuilding that burns Robert-lane weeks for zero differentiated value. The differentiation lives entirely on top of the recorder (normalize/diff/verdict/evidence), never in it. Building the recorder from scratch is the one place “build” would be strategically irrational.
2.2 Build on OSS foundation — RECOMMENDED SPINE
Exact composition (open-source-analysis.md §4, architecture.md §11.3):
- Adopt as substrate: mitmproxy (HTTP recorder, external process) ·
apache/activemq-classic(MBUS lab image — current, not truly EOL) ·cytopia/mysql-5.6(MySQL 5.x lab image, third-party) · VCR/WebMock + WireMock/Mountebank (legacy-lab outbound-dependency isolation only, so the lab boots deterministically). - Pattern-only, adopt zero code: Diffy’s relative-disagreement-rate noise floor · ApprovalTests’ versioned approved-baseline data model · Scientist’s control/candidate vocabulary + “always return control” discipline · Hoverfly diff-mode’s live-vs-recorded comparison shape.
- Build: differ, verdict engine, fixture store, masker, adapter SPI, MBUS shadow-tap, evidence bundle, work-ledger client, CI runner.
This is the spine because the adoptable OSS covers exactly the commodity leg (HTTP capture/replay, lab base images) and nothing more; the differentiated legs have no honest buy option.
2.3 Buy commercial (Speedscale, Signadot)
Both are live-shadow, Kubernetes/service-mesh-native; free tiers are local-dev only, everything cluster-scale is Enterprise-gated (competitive-analysis.md §1.10). Rejected on three independent grounds:
- Substrate mismatch — they assume a long-running K8s deployment with a routing mesh; the legacy estate is pinned-EOL Docker booted ad hoc. The model does not physically apply (unknown whether the estate even runs on K8s/AWS —
research.md§5 open Q). - Procurement reality for a 4-person confidential program (principle-based simulation): pricing unpublished; a real eval needs a security review of a closed-source control plane touching PII/PCI-adjacent traffic, a paid trial before knowing if the model fits, and a champion burning weeks on a vendor POC instead of Orders-flagship work. On a week-6 deadline, that lane does not close in time. And the program is confidential — a vendor POC means exposing the rebuild’s shape to an outside party, a confidentiality cost on top of the schedule cost.
- Audit trail — DoD row 9 needs QSA/SOX-defensible evidence; a fully-owned tool is far easier to reason about than a paid vendor’s closed control plane sitting in the audit chain.
Do not spike commercial shadow platforms before M2. Revisit only if live-shadow is later confirmed load-bearing and on the harness’s lane (unlikely — that’s Zaruba’s plane, §2.6).
2.4 Integrate multiple tools without a kernel
The closest achievable stitch: mitmproxy (capture/replay) + Hoverfly diff-mode (HTTP diff) + WireMock/VCR (lab isolation) + borrowed Diffy/ApprovalTests patterns. Rejected — the ceiling is too low. Blended coverage ≈ 35–40% of the kernel (competitive-analysis.md §3.1, MODELED), entirely on the HTTP leg, at fixture-vs-live granularity, 0% on MBUS and 0% on the verdict/evidence layer. No combination clears even the 60% bar the analysis tested. Worse, integration-without-a-kernel produces no single deterministic verdict authority — the exact property DoD row 9 and the fleet gate both require. You’d still build 60%+, and the 60% you’d build is the differentiated part.
2.5 Partner (Zaruba’s plane owns it all; we contribute adapters)
Zaruba’s R4 declares the ledger rows “YES — this IS the reconciliation harness.” Partial adopt, not whole. Correct for the money-chain live gate: it already largely exists in Core, and live oracle reconciliation is what the evidence supports (Option B). Wrong as the whole answer: his repo contains the scoping decision (YES/COMPARE-ONLY/NO per use case) but zero execution mechanics — the decision log has no dual-run/shadow/parity entries (research.md §2.1, confirmed fact). Process defines the gate; it does not execute it. The ~40 non-ledger use cases, read paths, and DARK batch/MBUS surface have no independent oracle — the legacy system is the only spec. Partnering for the money gate is right; partnering for everything leaves the harness’s actual net-new surface unbuilt.
2.6 White-label
Reinterpreted internally: rebrand someone else’s tool as ours. Nonsense here — there is no tool to rebrand; no product covers the kernel, and the differentiated legs don’t exist in any codebase to relabel. One line, dismissed.
2.7 Managed service + lightweight software (= manual QA panel + scripts)
Human testers exercise both systems, aided by scripts, and sign off. Rejected — direct negative-control evidence. TSB is the receipt: nine successful dress rehearsals + a 1,600-person pilot + board sign-off still produced a catastrophic full-scale failure (£48.65m fine, ~£366m total cost — Slaughter and May review via research.md §2, d2 §9, medium-confidence secondary sourcing). Rehearsal-plus-human-judgment is exactly the configuration that fails at financial stakes under schedule pressure. It also does not scale to 64 concurrent agents and produces no replayable counterexample. Keep human judgment for the DIVERGED triage step (which regressions are real), never as the gate.
2.8 No new software (Zaruba’s use-case gates + 100%-test-suite Bun pattern only)
The cheapest option: rely on R4 scoping + “100% of the existing test suite passes” as the gate (Anthropic’s Bun port, ~1M LOC / <2 weeks / ~64 agents / $165K API list — research/02-web-research.md). Rejected as sufficient. The Bun gate requires the old system to already have a trustworthy test suite; the legacy estate mostly doesn’t (that’s why characterization testing exists as a category). It produces pass/fail, not an auditor-grade counterexample, and covers neither MBUS parity nor the DARK surface. Kept only as a fallback: S3 impact-selected tests (TDAD 6.08%→1.82%, ~70% reduction — single uncorroborated off-domain study, research.md §1.3) is the documented rip-cord if the lab blows its investment cap — it holds agent regressions down cheaply but cannot prove legacy-off.
3. Assessment matrix
Options scored against the nine criteria. ✓ = meets, ~ = partial/needs work, ✗ = fails, — = n/a. “Defensibility” = does the option compound auditable evidence over time (Frame D) or produce throwaway signal.
| Option | Time to 1st verdict | Week-6 fit | Total cost (incl. maint) | EOL-lab compat | Bus-event parity | Auditor-grade evidence | Vendor/dep risk | Reversibility | Defensibility |
|---|---|---|---|---|---|---|---|---|---|
| 1 Build all (incl. recorder) | ✗ slow (rebuild commodity) | ✗ | ✗ highest | ✓ | ✓ | ✓ | ✓ low | ✓ | ✓ |
| 2 Build on OSS foundation | ~ days to pilot verdict | ✓ if lab boots | ~ Robert-lane weeks | ✓ designed for it | ✓ (custom taps) | ✓ | ✓ low (mitmproxy MIT; patterns not code) | ✓ high (recorder behind iface) | ✓ compounds |
| 3 Buy commercial | ✗ procurement | ✗ won’t close | ? unpublished + lock-in | ✗ K8s-native | ✗ | ✗ closed plane | ✗ high | ✗ re-architect | ~ |
| 4 Integrate, no kernel | ~ fast but partial | ~ | ~ | ~ HTTP only | ✗ 0% | ✗ no single authority | ~ | ~ | ✗ 40% ceiling |
| 5 Partner (Zaruba owns all) | ✓ for money gate | ✓ money only | ✓ near-zero new | ✗ no lab | ✗ | ~ money only | ✓ | — | ~ money only |
| 6 White-label | — nothing to label | — | — | — | — | — | — | — | — |
| 7 Manual QA + scripts | ✗ human-paced | ✗ | ✗ human time | ~ judgment | ~ judgment | ✗ (TSB) | ✓ | — | ✗ no receipts |
| 8 No new software (Bun/R4) | ✓ immediate | ✓ | ✓ cheapest | ✗ | ✗ | ✗ pass/fail only | ✓ | ✓ | ✗ weak signal |
No single row clears the matrix. Option 2 clears the most, but leaves the money gate (where Option 5 wins) and carries the lab’s EOL risk (where Option 8/S3 is the fallback). The answer is not one option — it is a component split across 2 + 5, with 8/S3 held as the rip-cord.
4. Recommendation — component-level split
Build on the OSS foundation (Option 2), partner for the money gate (Option 5), hold S3 (Option 8) as the capped fallback. Concretely, per component:
| Component | Decision | What | Why |
|---|---|---|---|
| HTTP recorder | ADOPT (buy/OSS) | mitmproxy, external process, custom masking addon | Mature/MIT/most-active tool surveyed; building it from scratch is the one irrational build. No free tool masks HTTP fixtures → masker is the only custom bit. |
| Lab base images | ADOPT (buy/OSS) | apache/activemq-classic, cytopia/mysql-5.6 |
ActiveMQ is not truly EOL (lowest lab risk); MySQL third-party image fills the stale-official gap. |
| Lab dependency isolation | ADOPT (buy/OSS), narrow | VCR/WebMock (Ruby outbound), WireMock/Mountebank (non-Ruby) | Stub what legacy calls out to; not the recorder, not the kernel. |
| Noise floor | PATTERN-ONLY from Diffy | Relative-disagreement-rate A/A self-replay; adopt zero code | opendiffy is CC BY-NC-ND (legally blocked); twitter-archive is abandoned. Diffy never published its false-positive rate → prove precision ourselves via the self-replay gate. |
| Baseline/approval data model | PATTERN-ONLY from ApprovalTests | Versioned, human-approved, auditable baselines → the normalization ledger | Best existing model for the human-decision audit trail; no capture/diff/bus engine to adopt. |
| Vocabulary + safety discipline | PATTERN-ONLY from Scientist | control/candidate framing, “always return control”, mutating-verbs-off-by-default (also Diffy’s default) | Same-process execution model is structurally wrong here; keep only the words + the write-safety default. |
| Differ | BUILD | Deterministic structural diff, HTTP + bus envelopes; generalize crossCheckBillingFeed’s pure CrossCheckGap[] shape |
No adoptable option; Hoverfly diff-mode (~25% fit, Go, no verdict/bus model) would mean building 75% around a borrowed 25% plus a new-language dep. |
| Verdict engine | BUILD | PARITY/DIVERGED/UNVERIFIABLE, three-way (posted / received-expected-no-op / unpublished-gap), program-owned thresholds | The differentiated deliverable; no buy option exists; thresholds read the 99.999% number, never invent it. |
| Fixture store + masker | BUILD | Content-addressed, masked, versioned | No free HTTP-fixture masker exists; VCR/WireMock formats aren’t content-addressed/masked/cross-language. |
| MBUS shadow-taps | BUILD, highest-uncertainty | JMS/STOMP/Kafka cross-system diff + outbox durability | Confirmed market gap — zero OSS/commercial does this. Budget as the top-risk line item, not a footnote. |
| Evidence bundle, work-ledger client, CI runner, triage hook | BUILD | Verdict + counterexample + normalization ledger + replay recipe; verdict rows gating PR-lane | Frames C/D — the surface only the harness can own; nothing in-repo or in-market provides it. |
| Money-chain live ≥99.999% gate | PARTNER (Zaruba’s oracle plane) | Live reconciliation over billing-event feed stays in Core; harness contributes the 2 missing oracles (provider-settlement, contract-check) + the generic verdict/evidence wrapper | Every financial-grade precedent runs live continuous (Uber/Stripe); the computation already exists in Core; recorded replay has zero evidence of clearing this row. |
| Regression fallback | HOLD S3 (Option 8) | Impact-selected tests (TDAD ~70%, single-study) | The rip-cord if the EOL lab blows its per-flagship investment cap; keeps agent regressions down while the lab is de-risked. Cannot prove legacy-off — fallback only. |
One-line summary: adopt mitmproxy + lab base images + VCR/WireMock; pattern-only from Diffy/ApprovalTests/Scientist/Hoverfly; build the differ/verdict/fixture-store/masker/MBUS-tap/evidence-bundle/work-ledger surface; partner with Zaruba’s oracle plane for the money gate; hold impact-selected tests as the capped fallback.
Economics check (why the ratio is cheap): program all-in $6.7M, M4 ≈ $0.9M (MEASURED, context brief). The harness is Robert-lane weeks-not-months, staged per flagship gate; lab setup MODELED at ~2–5 eng-days first-of-a-kind stack combo, <1 day follow-on (open-source-analysis.md §3). It converts a $0.9M milestone from “unclaimable without proof” to “claimable with proof” and insures against a TSB-class false-off. Cheap insurance on an irreversible decision (supported interpretation).
5. Conditions under which the recommendation flips
Each is a concrete, watchable trigger — not “it depends.”
-
EOL lab blows its investment cap. If the pilot ladder (
forex-ng → orders-ext → giftcard_service) cannot boot an EOL Rails-3.2/JRuby-1.7 stack within the capped budget (manifest-v1 pull fails unrecoverably, TLS/native-gem builds prove intractable) → drop the recorded-replay lab for that domain, flip to S3 impact-selected tests + oracle-only where an independent oracle exists. The harness shrinks to the wrapper + money-gate partnership; the lab-build is abandoned, not the harness. (Detection:architecture.md§11.4 risk 2; rip-cord already documented.) -
Internal MBUS-diffing precedent surfaces. If a bespoke message-bus comparison already exists in
repos/monorepo-development/or Zaruba’s oracle code (unchecked —open-source-analysis.mdopen Q2) → don’t build shadow-taps; wrap the existing mechanism. This removes the single highest-uncertainty build line. -
Tomas assigns the whole verification lane to Zaruba’s plane. If the Robert↔Tomas lane decision (
architecture.mdopen Q1) hands all execution — not just the money gate — to Core → the recommendation collapses toward Option 5: harness = the verdict/evidence wrapper + adapters only, no independent lab. (Unlikely given Core has zero execution mechanics today, but it’s a decision, not a fact.) -
A license/hosting shift makes a commercial or blocked tool viable. If an OSI-licensed Diffy successor emerges, or a shadow-diff vendor ships an air-gapped, source-available, PII-safe on-prem tier at trivial procurement cost → re-open the differ/recorder buy decision. Today opendiffy is CC BY-NC-ND and every commercial option is K8s-native + Enterprise-gated; that’s what makes build correct, and it’s a property that can change.
-
Recorded replay is shown sufficient for the money gate. No evidence supports this today (every precedent runs live). If a rigorous primary source demonstrates recorded-fixture replay clearing a five/six-nines financial-cutover gate → the harness could own the live gate too, and Option 5’s partnership narrows to a read-only feed integration. Treat as unknown, not expected.
-
Integration ceiling rises above ~60%. If a single tool appears that natively covers record+normalize+replay+diff+verdict across a cross-language boundary including a message bus (nothing does today — confirmed gap) → Option 4 becomes viable and BUILD narrows to adapters + evidence wrapper. This is the market gap closing; watch for it, don’t wait for it.
Open questions
- Money-gate lane, unratified. Option B (partner) assigns the live ≥99.999% reconciliation to Zaruba’s plane, but his decision log has zero logged dual-run/shadow/parity decisions — this boundary is proposed, not agreed. Needs an explicit Robert↔Tomas lane decision + a shared verdict contract before M3. (
architecture.mdopen Q1,research.md§5 Q1.) - Internal-endpoint seam. The billing-event feed and reconciliation reads are
expose:false, reachable only via Core’s in-cluster client. An external CLI/CI runner can’t call them → a Core-side authenticated read adapter, or that hop runs inside Encore — either partially contradicts “kernel + lab outside Encore.” Affects both the work-ledger-client and the Zaruba-oracle-read integrations. (architecture.mdopen Q2.) - EOL bootability at full combined difficulty — is
jruby:1.7/openjdk:7actually pullable on a current Docker Engine, and doesapache/activemq-classicserve 2013–17-era JMS/STOMP clients? Both gate the flip-condition-1 trigger; the pilot ladder is designed to answer them cheaply before the flagship depends on them. - Self-replay false-DIVERGED rate. Diffy’s real-world precision was never published anywhere; the self-replay gate is the only evidence the noise model works at acceptable precision. Must be proven zero-false-DIVERGED on a real corpus before the fleet trusts it, or bypass/churn follows. No external track record can shortcut this.
- The investment cap, numerically. “Weeks not months” and “cap per flagship” are DESIGN, not a number. Flip-condition 1 has no brake until the cap is a budget line with a named S3-fallback trigger. (
research.md§5 Q8.) - GoReplay-Pro paid masking vs custom mitmproxy scrubber — unresearched; no free OSS tool masks HTTP fixtures (as opposed to DB rows). A cheap build-vs-buy sub-decision, deferred until the masker is scoped.