Pre-mortem — Dual-run parity harness
2026-07-18 · Author: Fable (pm:pre-mortem, template §14). Inputs read in full: 00-context-brief.md, initial-recommendation.md, prd.md, research.md, and plans/001-factory-apps-validation/00-SYNTHESIS.md (pre-mortem + risk register). It is 18 months later. The harness failed badly. This document extends 001’s 5 entries — it does not repeat them.
Extends, doesn’t repeat. 001’s pre-mortem was written before an adapter existed; its 5 entries (lane politics, schedule capture, ledger/Temporal drift, triage-shipped-clever, blind concurrency scale-up) are portfolio-level. This document is harness-internal: the specific mechanics of how the parity kernel, legacy-runtime lab, and verdict surface fail once they’re real. Where a theme recurs (schedule capture, FM9; boundary politics, FM8), the entry below cites the 001 origin and adds the granular failure path the PRD’s own text already half-predicts.
Internal-product framing (binding, context brief): “buyer” = program gates; “procurement” = Tomas’s lane approval; “churn” = fleet bypass; “pricing” = engineer-days + agent tokens. No SaaS-template padding below.
Evidence discipline. Numbers carry MEASURED / MODELED / DESIGN. Claims are confirmed fact / supported interpretation / hypothesis / assumption / unknown. Section citations point at prd.md / research.md / 00-SYNTHESIS.md sections already in evidence — no new external sourcing is invented here; where a new metric is proposed it is marked new.
Failure modes
Each entry: description, root cause, likelihood, impact, earliest warning signal, leading indicator metric, prevention, mitigation, owner, pivot trigger, kill trigger.
FM1 — Adoption: agents bypass, builders ignore bundles
Two facets of one mechanism. Agents work around work-ledger claim-before-work entirely (direct push, manual PR) once harness verify latency or noise exceeds the “cheaper than re-reading the diff” bar (JTBD-1, research.md §3.5). Builders facing a DIVERGED bundle rubber-stamp “acceptable difference” without reading the counterexample, defeating the one human checkpoint (prd.md §7).
| Field | Value |
|---|---|
| Root cause | NFR-2’s verify-latency budget is DESIGN, not measured (prd.md §10); the noise floor is unproven (D6 Q2, no external false-positive baseline); the Ive-lens bundle design is unvalidated against real builder cognition (persona note, prd.md §5: “assumption, not measured”). |
| Likelihood | Medium — already the PRD’s own top usability risk (prd.md §17 row 1), but nothing forces early detection before Orders is load-bearing. |
| Impact | High — full regression to vibes at 99.999% stakes; the exact TSB configuration (human judgment substituting for a structural gate) reappears one layer inside the tool built to prevent it. |
| Earliest warning signal | Merges land with no matching work-ledger verdict-row reference; CI logs show repeated harness verify timeout/retry followed by an ungated merge path. |
| Leading indicator metric | Bypass-attempt count (target: structurally zero, prd.md §22 Trust metric); normalization-rule audit-acceptance rate approaching 100% with near-zero rejection (proxy for unread bundles). |
| Prevention | Enforce FR-C2 at the data/proposal-endpoint layer exactly as spec’d, fail closed; set a real NFR-2 number from pilot telemetry before Orders scale instead of leaving it DESIGN; require builders’ disposition notes to cite the specific counterexample line (cheap forcing function). |
| Mitigation | Weekly audit of merge history for gate gaps; any gap is a write-ledger integrity incident, not a backlog item — escalate to Tomas immediately. |
| Owner | Robert (gate integrity); strike-unit lead (builder discipline). |
| Pivot trigger | Measured verify latency exceeds one CI-step timeout on real Orders volume — shrink verdict scope (functional-only, defer long-tail classes) before adoption erodes further. |
| Kill trigger | A bypass path is found that the data-layer gate cannot close — FR-C2 is named in the PRD as the harness’s only truly load-bearing structural claim, the one thing that “cannot degrade gracefully” (prd.md §9.6). If it can be routed around, halt and re-architect the gate before any further adapter work. |
FM2 — Trust: false-DIVERGED storm, and the worst case — one false PARITY ships a money bug
Two shapes, one root cause: an unproven noise model (D6 Q2, confirmed gap — nobody has ever published a Diffy-class false-positive rate). (a) Noise floor too tight: legitimate legacy nondeterminism drowns the signal, every unit reads DIVERGED, trust collapses (feeds FM1). (b) The worst case: noise floor too loose, or a normalization rule wrongly masks a real divergence, or the three-way classifier (FR-K6) misclassifies a genuine gap as an expected no-op — a real money-chain regression reads PARITY, ships, and only reconciliation catches it, if anything does, after real money has moved. This is the exact “false-off” the harness exists to prevent (prd.md §2, TSB analogy) — caused by the harness itself.
| Field | Value |
|---|---|
| Root cause | (a) hand-curated rules incomplete on day one, self-replay unproven on a real corpus (open Q4). (b) a rule added to silence a false DIVERGED masks a true one — the mechanism FM7 names explicitly — or FR-K6’s posted/received-no-op/unpublished-gap classifier gets a real gap wrong. |
| Likelihood | (a) Medium-high pre-tuning, falls with self-replay maturity. (b) Low per unit, but the harness gates essentially every Orders-chain unit including the money chain, so expected-value impact is dominated by (b) despite low per-unit odds. |
| Impact | (a) Medium, recoverable. (b) Catastrophic — a TSB-class event manufactured by the tool built to prevent one; destroys the program’s ability to trust any future verdict, harness or otherwise. |
| Earliest warning signal | (a) DIVERGED rate spikes disproportionate to real code-change volume. (b) Zaruba’s live reconciliation (prd.md §13) disagrees with a unit the harness already passed PARITY — the two-lane design is itself the tripwire, since functional PARITY and live reconciliation are two independent checks on the same money path by construction. |
| Leading indicator metric | False-DIVERGED rate via self-replay sampling (prd.md §22, target near-zero). New: PARITY-then-reconciliation-disagreement count on money-chain units — must be exactly zero, tracked per unit, never sampled (sampling is acceptable for functional units, not for money). |
| Prevention | FR-K8 self-replay gate as a hard merge precondition (already spec’d, NFR-5); keep the §13 two-lane boundary intact so functional PARITY is never the sole gate on the money chain; FR-V4’s append-only, rationale-bearing ledger keeps every masking decision reviewable. |
| Mitigation | Any PARITY/reconciliation disagreement freezes the responsible normalization-rule version immediately, auto-reverts to the prior version pending review, and triggers a full audit of every unit verified under that version — regardless of blast radius, reported to Tomas + QSA. |
| Owner | Robert (kernel/noise model); Tomas (§13 boundary enforcement). |
| Pivot trigger | False-DIVERGED rate fails to approach zero after reasonable normalization effort on the pilot — prd.md §24 already names this; pivot to S3 before Orders depends on the noise model. |
| Kill trigger | One confirmed PARITY-then-reconciliation-disagreement on a real money-chain unit. Not a threshold — a single confirmed instance voids the harness’s authority as a gate pending full re-verification of every prior PARITY verdict issued under that adapter version. Stricter than any kill trigger currently in prd.md §24 — see “what changes today” below. |
FM3 — Generalization: the adapter cost curve
Extends 00-SYNTHESIS.md risk register row 1 and pre-mortem #2 with the mechanics visible now that an adapter exists: adapter #2 doesn’t just cost >2× adapter #1, it scales worse than linearly, because each domain’s independent-oracle coverage, MBUS topic shape, and EOL runtime combination is closer to bespoke than the “extract the kernel from adapter #2” principle (FR-A4) assumes — and D5’s own finding that lab cost does not amortize sub-linearly across stack combos already predicts this for the lab half.
| Field | Value |
|---|---|
| Root cause | The kernel API is deliberately undesigned until adapter #2 exists (principle 8, prd.md §4) — the first generalization attempt is discovery-by-failure, on the program’s clock. |
| Likelihood | Medium — the org already suspected it before any adapter #2 existed (“detection: 2nd adapter costs >2× the first” was written in 001, pre-flagship). |
| Impact | Medium-high — burns exactly the budget line that’s still unset (§25 Q5); risks quietly becoming FM9 under a different name. |
| Earliest warning signal | A kernel refactor for adapter #2 requires reworking adapter #1’s contract — a true minimal-kernel extraction should not need this; estimate-vs-actual eng-days for adapter #2 exceeds adapter #1’s actual cost before 50% complete. |
| Leading indicator metric | Adapter #2 eng-days ÷ adapter #1 eng-days, tracked from day 1 of Stage 2 — not just at completion. |
| Prevention | Name the 2× threshold as a hard stage-gate before Stage 2 starts, not just a “detection signal” (prd.md §20); timebox kernel-extraction work separately from adapter-specific work so the ratio isn’t conflated. |
| Mitigation | If trending over 2× at the halfway mark, freeze kernel generalization immediately — don’t wait for completion — and finish adapter #2 bespoke, exactly as adapter #1 was built. |
| Owner | Robert. |
| Pivot trigger | Cost trend crosses 1.5× at 50% complete — open the freeze conversation early. |
| Kill trigger | Adapter #2 crosses 2× (prd.md §20 Stage 2, already the named trigger) — freeze generalization permanently for this era; kernel stays adapter-#1-shaped; future domains build bespoke per the per-domain economics in §21. |
FM4 — The lab tar pit
Extends steelman argument (c) (research.md §4.2c) and prd.md §17 risk row 2 with the failure’s actual shape: EOL boot costs (manifest-v1 pull failures, RubyGems TLS floor, native-gem/OpenSSL pairing hunts) consume the pilot ladder’s time budget before Orders’ 6-runtime chain is attempted, and voucher-inventory-service’s JRuby-1.7/Ruby-1.9.3 boot — already the estate’s hardest — becomes open-ended debugging with no published precedent to bound it (D3, confirmed: “no writeup found combines this project’s full difficulty stack”).
| Field | Value |
|---|---|
| Root cause | The full difficulty stack (old-enough Ruby + TLS workaround + archived apt + JMS/STOMP + PII masking + hardening) has never been solved end-to-end by anyone found in research — MODELED 2-5 eng-days carries a point estimate, not a real confidence interval. |
| Likelihood | High for the hardest tier specifically (voucher-inventory); Low-Medium for the rest (D3’s own risk tiering: ActiveMQ low, MySQL 5.x moderate, Ruby/Rails/JRuby/Java-7 highest). |
| Impact | High if unbounded — the single highest-uncertainty line item in the harness (research.md §4.2c). |
| Earliest warning signal | giftcard_service (Rails 3.2, the MVP’s own rehearsal target, prd.md §18 M2) fails to boot within its capped spike time-box; rescue-pull doesn’t resolve cleanly on the first easy image. |
| Leading indicator metric | Lab-boot success rate + duration per stack tier (FR-L5) — track variance, not mean; the tar-pit risk lives in the tail. |
| Prevention | MVP M2 already requires “a dated pass-or-fail recipe… a written failure with the exact blocker is an acceptable MVP outcome, silent abandonment is not” (prd.md §18) — hold this discipline for every subsequent image, especially voucher-inventory, before it’s on Orders’ critical path. |
| Mitigation | Per-image hard time-box; on timeout, that domain’s units fall to S4 (oracle-only, if one exists) or S3 (impact-selected tests) rather than blocking on the boot — prd.md §23 already names this pivot. |
| Owner | Robert + fleet ops (lab lifecycle, prd.md §5 persona table). |
| Pivot trigger | Any single image exceeds its time-box by 2× → invoke §23’s pivot for that domain specifically, not the whole harness. |
| Kill trigger | Voucher-inventory (or any Orders-chain image) cannot boot at all after rescue-pull + vendored-gem mitigations are exhausted → that repo loses recorded-replay coverage; §13’s Option B (oracle-first) becomes mandatory for its units, and UNVERIFIABLE(lab_boot_failed) is the honest verdict rather than a forced boot. |
FM5 — Fixture staleness
Frozen fixtures silently stop representing legacy behavior — either the “lab-recipe rot vs. legacy change” distinction (prd.md §15) blurs in practice, or the weekly rebuild+self-replay check (DESIGN, “no measured basis yet”) is skipped once pilot pressure lifts.
| Field | Value |
|---|---|
| Root cause | For genuinely frozen legacy (orders ~2017), staleness is mostly lab-recipe rot, not legacy drift — but the design deliberately avoids auto-invalidating fixtures on a failed rebuild (“avoids a cascading false-DIVERGED storm,” §15), so a rotted lab can silently coexist with unrevalidated fixtures. |
| Likelihood | Medium — the weekly cadence is DESIGN with no measured basis; nobody has committed to actually running it. |
| Impact | Medium — a stale fixture set produces verdicts against a lab that no longer represents even the frozen baseline; PARITY becomes “matches an old recording,” not “matches legacy.” |
| Earliest warning signal | Lab rebuild job fails or is skipped >2 cycles; fixture age (days since last successful re-record) climbs with no alert. |
| Leading indicator metric | Fixture staleness — already a §22 guardrail (“a stale fixture drops out of the numerator even if its last verdict was PARITY”); the gap is enforcement cadence, not the metric’s existence. |
| Prevention | Wire the weekly rebuild+self-replay job into the same CI/Grafana surface as everything else (FR-I4); set an actual staleness threshold (currently undefined) before Orders scale. |
| Mitigation | On detected staleness, re-record affected fixture classes before trusting any verdict against them; treat a stale-fixture PARITY as UNVERIFIABLE until re-validated, never as a pass. |
| Owner | Fleet ops (lab, recorder, fixture cadence — explicit owner already named, prd.md §5). |
| Pivot trigger | Staleness metric shows >X% of active fixture classes untouched for >1 rebuild cycle (X unset — same numeric gap as FM9’s cap). |
| Kill trigger | None at the whole-harness level — this degrades correctly to per-fixture UNVERIFIABLE by design (NFR-3 spirit); escalate only if staleness detection itself is found silently broken, which is a monitoring failure, not a fixture failure. |
FM6 — PII/PCI incident from fixtures
A fixture carrying unmasked PII/PCI (card tokens, SEPA, address, device/fraud IDs, OAuth tokens — every Orders-chain fixture class touches one, D5 §b) leaves the lab boundary before masking sign-off, or a masking rule has a gap the scanner misses, and real customer data lands in a fixture store, Grafana panel, evidence bundle, or QSA hand-off.
| Field | Value |
|---|---|
| Root cause | No off-the-shelf tool masks recorded HTTP fixtures (D3 §7, confirmed — GoReplay’s masking is Pro-only); the harness builds and validates a custom scrubber under schedule pressure, and masking coverage must be proven per fixture class, not asserted once (journey.md §1.6, cited prd.md §14) — an obligation easy to shortcut racing the week-6 clock. |
| Likelihood | Low-Medium per fixture class if the scanner gate holds; but the class count is large (MODELED 50-60+ for Orders alone) and every one touches sensitive data, so aggregate probability across the full surface is not small. |
| Impact | Severe — a real leak inside an internal tooling project whose entire premise is regulator-defensible evidence; undermines DONE row 9 directly and can trigger a compliance incident independent of the migration’s success. |
| Earliest warning signal | The unmasked-fixture scanner (NFR-3, already a CI gate) flags a hit; a fixture class is recorded before its masking-rule sign-off lands — the policy exists (§12: “recording without sign-off = security incident, fixture quarantined + destroyed”), the risk is whether it holds under time pressure. |
| Leading indicator metric | Unmasked-fixture scanner hit rate (target zero); count of fixture classes recorded before masking sign-off (target zero, currently unmeasured). |
| Prevention | Keep NFR-3 a hard blocker with no “record now, mask later” exception, even under week-6 pressure; close the retention-window question (§19/§25 Q6, currently open) before MVP M6 ships, not after. |
| Mitigation | On a confirmed leak: quarantine + destroy the fixture (policy already stated); audit every fixture recorded under the same masking-rule version; notify security + Tomas immediately; run this as its own security postmortem, independent of harness delivery status. |
| Owner | Security reviewer (masking sign-off, explicit gate owner, prd.md §5). |
| Pivot trigger | None — there is no acceptable “pivot” on PII handling; any gap found is fixed before recording resumes for that class. |
| Kill trigger | A confirmed leak of real customer PII/PCI outside the lab boundary → halt all recording program-wide pending a full masking-pipeline audit. This kill trigger does not currently exist in prd.md §24 — see “what changes today” below. |
FM7 — Noise-rule sprawl (the honest-lie failure)
The normalization ledger accumulates rules added to silence individual false-DIVERGED incidents, each locally justified, that in aggregate quietly widen what the harness ignores until real divergences pass through unflagged. The PRD’s own design law: “a parity report that hides its exclusions is a lie with good typography” (prd.md §4 principle 6) — this is that lie, produced honestly, one reasonable rule at a time.
| Field | Value |
|---|---|
| Root cause | FR-V4 makes ledger writes append-only with rationale — good against silent edits, but nothing caps or reviews aggregate growth; the guardrail (“normalization-rule growth outpacing audit-acceptance,” prd.md §22) is named but carries no numeric threshold. |
| Likelihood | Medium-high over the harness’s life — the predictable failure mode of any allowlist-style suppression system left to grow indefinitely; a mechanism risk, not a Groupon-specific one. |
| Impact | High — converges with FM2(b) but arrives gradually and structurally rather than from one bad rule, harder to detect and harder to pin on a single incident. |
| Earliest warning signal | Ledger rule count grows faster than fixture-class count; a rule’s stated field/topic scope broadens on a later revision without a narrower rule being retired. |
| Leading indicator metric | New: ledger rule count ÷ fixture-class count ratio over time — the existing audit-acceptance metric measures rejection at write time, not aggregate drift after acceptance. |
| Prevention | Periodic full-ledger review (not just per-addition review) — every N rules or every flagship gate, re-derive whether each rule’s rationale still holds against a fresh self-replay sample; require each rule to cite the specific false-DIVERGED incident that motivated it, not just prose rationale. |
| Mitigation | On a review finding a rule masking real divergence: revert it, re-run every verdict issued under its lifetime, treat the finding with the same severity as FM2(b) (freeze + audit). |
| Owner | Robert (MVP scale) → strike-unit co-review (post-MVP, prd.md §12). |
| Pivot trigger | Full-ledger review interval slips past 2 flagship gates without happening — schedule it explicitly rather than deferring to “post-MVP.” |
| Kill trigger | A full-ledger review finds >1 rule masking real divergence in the same review — treat as systemic, same severity as FM2’s single-instance kill; halt new-rule additions until the review process itself is redesigned. |
FM8 — Boundary conflict with Zaruba’s plane
Extends 00-SYNTHESIS.md pre-mortem #1 (lane politics) to the specific §13 money-gate boundary, which the PRD itself flags “proposed, not ratified” (prd.md §13, §25 Q1): the harness and Zaruba’s oracle plane build overlapping reconciliation/verdict machinery independently, because no explicit Robert↔Tomas lane decision exists — two verdict surfaces, two “sources of truth” for the same money question, discovered only when they disagree (also FM2(b)’s trigger condition).
| Field | Value |
|---|---|
| Root cause | Zaruba’s decision log has zero dual-run/shadow/parity entries (research.md §5 Q1, confirmed) — the boundary is Robert’s proposal, not a negotiated agreement; the internal-endpoint seam (§25 Q2: billing-event feed is expose:false, in-cluster only) is unresolved and could be closed unilaterally by either side, in a way that contradicts “kernel + lab outside Encore.” |
| Likelihood | Medium — a coordination-failure pattern, not a technical one; 00-SYNTHESIS.md pre-mortem #1 already shows this class materializing under less complexity. |
| Impact | Medium-high — duplicate machinery wastes both teams’ effort and turns FM2’s intended cross-check safety net into an unplanned ambiguity instead. |
| Earliest warning signal | Zaruba’s team starts building a verdict/evidence concept independently (decision-log entry, PR, thread) without referencing the harness’s contract; the internal-endpoint seam gets resolved unilaterally. |
| Leading indicator metric | New: days since §13’s boundary was flagged “proposed, not ratified” (accruing since 2026-07-18) without a closed Robert↔Tomas decision. |
| Prevention | Close §25 Q1 (money-gate lane) and Q2 (internal-endpoint seam) before M3, as the PRD already states — as a precondition, not a nice-to-have; a written, shared verdict contract is the actual artifact, not a verbal understanding. |
| Mitigation | If duplicate machinery is found, the harness’s wrapper always defers to Zaruba’s oracle plane’s money-gate computation as source of truth (§13 Option B is already the recommendation) — reconcile by adoption, not competition. |
| Owner | Robert (proposes), Tomas (ratifies) — joint, per §12 roles table. |
| Pivot trigger | No ratification by M2 (dual-run live checkpoint) — escalate explicitly rather than letting ambiguity ride into write-ramp. |
| Kill trigger | Tomas assigns the whole verification lane to Zaruba’s plane (§23, already a named pivot) — collapse the harness to verdict/evidence wrapper + adapters only, no independent lab. Scope-kill for the lab specifically, not a full-harness kill. |
FM9 — Schedule capture (“harness became the schedule”)
Extends 00-SYNTHESIS.md pre-mortem #2 with the concrete mechanics now visible: the investment cap is still DESIGN, not a number (prd.md §25 Q5, confirmed gap, restated three separate times across §17/§24/§25) — so the kill-criterion rip-cord in §24 has no brake to actually pull. The harness consumes the Orders schedule through concrete channels — FM3 (adapter overrun), FM4 (lab tar pit), FM7 (ledger review debt) — each individually mitigated but collectively eating the same uncapped budget.
| Field | Value |
|---|---|
| Root cause | §24 states the cap “requires a budget line with a named S3 trigger — currently DESIGN, not a number.” Without a number, “cap investment per flagship” is a stated intention, not an enforceable mechanism. |
| Likelihood | High if the number is never set — near-certain given the pattern, not merely probabilistic: 001’s pre-mortem #2 already predicted the qualitative shape; the PRD confirms the quantitative gap remains open 18 months of runway later. |
| Impact | High — M4 slips, publicly, exactly as the context brief’s own week-6 rule states, except caused by the harness itself, not by legacy complexity. |
| Earliest warning signal | Cumulative harness eng-days (kernel + lab + adapter + review overhead) crosses an informal, undocumented sense of “weeks” without any written trigger firing, because none exists. |
| Leading indicator metric | Cumulative harness eng-days vs. a stated budget — cannot be tracked until the budget number exists. The leading indicator for this failure mode is, today, “the leading indicator does not exist.” |
| Prevention | Set the number now — the one item in this document that is a same-day PRD edit, not a future practice (see “what changes today” below): a specific eng-week cap per flagship with the S3 fallback pre-authorized to trigger automatically at the cap, not on a discretionary call. |
| Mitigation | If the cap is blown before it’s set, retroactively apply the nearest named sub-trigger (adapter #2 >2×, self-replay non-zero after reasonable effort, lab-boot cap) as the closest available brake, and set the number immediately for future gates. |
| Owner | Robert. |
| Pivot trigger | Cumulative spend crosses 75% of the (currently unset) cap. |
| Kill trigger | Cap fully consumed pre-M4 with the flagship still ungated → invoke §24’s rip-cord (freeze generalization, ship Orders bespoke proof, hold the line at S3) — already written, just needs a number to fire on. |
FM10 — Dependency rot (mitmproxy/GoReplay breakage)
The recorder (mitmproxy, FR-K1) breaks compatibility with an EOL stack’s TLS/HTTP quirks after an upstream release, or drops support for ancient protocol variants the legacy estate speaks. A live dependency on actively-maintained OSS pointed at deliberately unmaintained targets is a mismatch that widens over time, not a one-time integration cost.
| Field | Value |
|---|---|
| Root cause | mitmproxy is chosen because it’s actively maintained (44k★, pushed 2026-07-18, D1 §3) — but active maintenance optimizes for current TLS/HTTP, not 2013-2017-era client behavior; the PRD currently pins no version discipline for the harness’s own tooling. |
| Likelihood | Low-Medium in the MVP window, rising over the harness’s multi-year operating life as the legacy estate stays frozen while the tool ecosystem moves on. |
| Impact | Medium — a recorder break stops new fixture capture (not existing verdicts), degrading gracefully into FM5 (staleness) rather than a silent wrong answer, provided FR-K8 self-replay is itself unaffected. |
| Earliest warning signal | A routine mitmproxy upgrade changes captured fixture content for a previously-stable class; a CI dependency-update PR touching the recorder is followed by a self-replay failure. |
| Leading indicator metric | Self-replay pass rate immediately after any recorder/tooling version bump (should stay 100%; any dip is the tripwire) — reuses the existing self-replay metric as the regression detector. |
| Prevention | Pin the recorder’s version explicitly, no auto-upgrade in the lab’s CI path; extend NFR-5’s self-replay-gate discipline (currently scoped to “kernel/normalization-ledger” changes) to tooling version bumps as well — the PRD doesn’t currently name this case. |
| Mitigation | Roll back to the last known-good recorder version on a detected break; content-addressing (FR-K3) means old fixtures stay valid and diffable regardless of current recorder version, limiting blast radius to new recording only. |
| Owner | Fleet ops (lab lifecycle). |
| Pivot trigger | Recorder incompatibility found with >1 stack in the estate — evaluate GoReplay (D1’s named alternative) as a second recording path rather than patching around one tool indefinitely. |
| Kill trigger | None at the whole-harness level — degrades to “cannot record new fixtures for affected classes,” falling back to existing frozen fixtures (stale but valid) or S3. |
FM11 — Operator bus-factor (Robert-only knowledge)
The MVP-scale roles table (prd.md §12) names Robert as sole approver for normalization-rule changes and the primary deeply-knowledgeable operator of kernel, lab, and adapter internals. If Robert is unavailable, no one else can diagnose a false DIVERGED, approve a rule, or judge whether a lab-boot failure is expected rot vs. a real problem — the harness stalls exactly when the fleet most needs a verdict.
| Field | Value |
|---|---|
| Root cause | Explicit, deliberate MVP simplification (§12: “Change a normalization rule: Robert (MVP scale); strike unit co-review post-MVP”) — named without a trigger for when it must end. |
| Likelihood | Medium — single-operator bottlenecks are a near-inevitable risk in any MVP-scoped internal tool; this program already runs Robert at fleet-orchestration scale (64 concurrent agents) with limited slack. |
| Impact | Medium-high — a stalled harness mid-write-ramp (M3) has the same practical effect as FM9 but triggered by availability, not scope creep; the same bottleneck is also a single point of failure for FM7 and FM2(b) (unreviewed solo rule changes). |
| Earliest warning signal | Normalization-rule changes queue unreviewed >48h during an active flagship window; a lab-boot failure sits undiagnosed because only Robert recognizes the error signature. |
| Leading indicator metric | Median time-to-review for a normalization-rule change; count of harness-critical decisions made by exactly one person with no rationale trail beyond the ledger entry itself. |
| Prevention | Pull “strike unit co-review post-MVP” forward to before Orders write-ramp (M3), not after — money-chain stakes at M3 are exactly when solo-operator risk is least acceptable; write a short runbook for the three most common lab-boot failure signatures (manifest-v1, TLS, native-gem) so diagnosis isn’t tacit. |
| Mitigation | If Robert is unavailable during a live gate, explicitly degrade to UNVERIFIABLE (never a guessed PARITY) for any unit requiring judgment — FR-K6’s “never a coin-flip” principle extends naturally to operator unavailability. |
| Owner | Robert (until co-review lands); Tomas (approves pulling co-review forward if asked). |
| Pivot trigger | Any single incident where a stalled review measurably delays a flagship-gate deadline — pull co-review forward immediately, don’t wait for “post-MVP” to arrive on its own schedule. |
| Kill trigger | None directly — an operational risk to mitigate, not a kill condition; but a sustained solo-bottleneck stall during M3 write-ramp should trip the same “harness became the schedule” kill trigger (§24) since the effect on the flagship is identical regardless of cause. |
FM12 — Evidence rejected by QSA/SOX
The evidence bundle, built to the Ive/Rams falsifiability law, doesn’t satisfy an actual auditor’s requirements — because no auditor has touched program artifacts yet (prd.md §5, §25 Q10: “assumption, not measured”) — and the mismatch surfaces at DONE row 9, the last gate, after the bundle format is already baked into every verdict issued.
| Field | Value |
|---|---|
| Root cause | The QSA/SOX persona and its evidence-bundle requirement are explicitly flagged as assumption, not measured engagement, in two places — the harness is designed against a simulated auditor requirement (falsifiability, replay recipe, source+window+confidence) that is principled but unvalidated against what a real QSA asks for (chain-of-custody, specific retention proof, QSA-tooling-specific format — none knowable without engagement). |
| Likelihood | Medium — the design principles likely overlap substantially with real audit needs, but “overlap” is not “satisfies,” and audits are particular about form as well as substance. |
| Impact | High if discovered late — DONE row 9 blocks at the worst possible time, after M4 money-chain work is otherwise complete; reformatting thousands of already-issued bundles is expensive. |
| Earliest warning signal | No QSA/SOX engagement has happened by the time the Orders adapter produces its first real bundles (MVP M4) — §25 Q10 already flags this needs confirmation “before finalizing the evidence-bundle format,” implying it hasn’t happened. |
| Leading indicator metric | New: days since program start without a QSA/SOX bundle-format review — a scheduling gap, analogous to FM8’s tracking of the unratified money-gate boundary. |
| Prevention | Schedule an early, low-stakes QSA/SOX review of one sample bundle (the forex-ng pilot bundle is real and cheap to show) rather than waiting for DONE row 9 to be first contact. |
| Mitigation | FR-V5’s reproducibility (any past verdict reproduces exactly from its cited hash) means bundles can be reformatted from underlying data without re-verification if rejected late — the expensive part, the verdicts themselves, survives a format change. |
| Owner | Tomas (auditor relationship); Robert (bundle-format implementation). |
| Pivot trigger | No QSA/SOX contact scheduled by M2 (dual-run live checkpoint) — raise explicitly rather than letting it ride to M4. |
| Kill trigger | None as a harness kill on its own — FR-V5 insures against catastrophe; becomes a real kill-risk only compounded with fixture retention already having expired for early units (§25 Q6, currently open) — Q6 must close before Q10’s risk is fully bounded. |
FM13 — Kill-criteria never enforced
prd.md §24 names three kill criteria and §23 names five pivot criteria — but naming a trigger is not the same as having someone whose job is to watch for it and pull it. Without an assigned, scheduled check, a kill criterion fires quietly in the data while everyone is heads-down building, and gets rationalized away (“one more sprint”) exactly as sunk-cost dynamics predict.
| Field | Value |
|---|---|
| Root cause | §23/§24 are well-designed triggers with no named cadence (“review every N weeks / at every checkpoint”) and no named person whose explicit job includes checking them; Robert as sole operator (FM11) is also the person with the least structural distance from the sunk-cost bias the criteria exist to counteract. |
| Likelihood | Medium-high — the generic failure mode of every pre-written kill switch in every project; the exact reason TSB’s board sign-off (a human judgment gate) failed with nine rehearsals of contrary evidence in hand (D2 §9, research.md) — the harness’s own kill criteria are subject to the identical human-judgment-under-pressure failure the harness exists to remove from the legacy-off decision, applied one level up to the harness-continuation decision. |
| Impact | High — the meta-risk that makes every other kill trigger in this document (FM2, FM3, FM4, FM9) theoretical rather than operative. |
| Earliest warning signal | A kill/pivot leading-indicator metric crosses its threshold (per the individual FMs above) and no decision gets made within one flagship-checkpoint cycle. |
| Leading indicator metric | New: for each named trigger in §23/§24, days-since-threshold-crossed without an explicit recorded go/no-go decision — a meta-metric over the other FMs’ metrics. |
| Prevention | Attach an explicit review cadence to §23/§24: check all pivot/kill triggers at every flagship checkpoint (M2, M3, M4 minimum), not on discretion; make the check itself an auditable, dated decision record — apply principle 2 (“structural gates beat human judgment under schedule pressure,” prd.md §4) to the harness’s own continuation decision, not just to unit promotion. |
| Mitigation | If a trigger is found crossed silently, treat the delay itself as a finding — apply the trigger’s action retroactively now, and add the missed-detection interval as a case study in the review process. |
| Owner | Tomas — the one role structurally positioned to override Robert’s sunk-cost incentive, since he already owns program-level go/no-go (§12: “Approve adapter scope beyond Orders: Tomas”). |
| Pivot trigger | This FM’s own trigger — the first missed kill/pivot check at a flagship checkpoint — should immediately institute the cadence, not wait for a second miss. |
| Kill trigger | None of its own — this FM is enforcement infrastructure, not a harness condition; but its unmitigated presence is what upgrades every other FM’s kill trigger from “theoretically stops the harness” to “actually stops the harness,” which is why it ranks among the 5 most dangerous below. |
The 5 most dangerous, ranked
- FM2(b) — one false PARITY ships a money bug. The single scenario that turns the harness from insurance into the incident it was built to prevent. Catastrophic, low-probability-per-unit, dominates expected value because it gates the money chain specifically.
- FM13 — kill-criteria never enforced. The meta-risk. Every other kill trigger in this document, including FM2’s, is inert without an assigned cadence and owner to pull it under pressure — the exact TSB failure mode, one level up.
- FM9 — schedule capture with no numeric cap. Not probabilistic — near-certain given the pattern already visible three times in the PRD’s own text (§17, §24, §25 Q5). The rip-cord exists; the number to pull it at does not.
- FM6 — PII/PCI incident from fixtures. Severe, non-recoverable-by-design impact (a real leak is a compliance event independent of the migration’s fate) across a large fixture surface (50-60+ classes, every one sensitive).
- FM7 — noise-rule sprawl (the honest-lie failure). The slow-burn precursor to FM2(b) — harder to detect than a single bad rule because it’s built from many individually-reasonable ones, and the PRD’s own aggregate-growth guardrail has no numeric threshold yet.
What changes in the PRD today, because of this document
- Add a fourth kill criterion to §24: one confirmed PARITY-then-reconciliation-disagreement on a money-chain unit. Currently §24 has three performance-shaped triggers; none covers a single false PARITY discovered via Zaruba’s independent reconciliation (FM2). This is stricter than any existing trigger and should read as: any confirmed instance freezes the responsible normalization-rule version and forces re-verification of every unit issued under it — not a threshold, a single-instance trip.
- Add a fifth kill criterion to §24: a confirmed PII/PCI leak outside the lab boundary halts all recording program-wide pending a full masking-pipeline audit (FM6). Currently no kill trigger in §24 covers a security incident at all — every named trigger is harness-performance-shaped.
- Set the investment cap numerically, now, not as a future practice. Proposed (DESIGN, this document’s proposal — ratify or replace, do not leave unset): 40 eng-days total harness investment through Orders M4, split as lab-boot ladder ≤15 eng-days (within the MODELED 12-30 eng-day range for 6 non-reusable runtime combos at 2-5 days each, biased toward the low end with FM4’s rip-cord as the release valve), kernel+adapter ≤15 eng-days, review/hardening ≤10 eng-days. This gives FM9’s leading indicator something to be measured against — today it cannot be tracked at all.
- Attach an explicit kill/pivot review cadence to §23/§24, owned by Tomas, checked at every flagship checkpoint (M2/M3/M4 minimum) as a dated, recorded go/no-go decision — not discretionary (FM13). This is the single edit that makes every other kill trigger in the PRD, including #1 and #2 above, actually enforceable rather than theoretical.
- Pull “strike unit co-review” for normalization-rule changes forward from post-MVP to before Orders M3 write-ramp (FM11), and require every rule to cite the specific false-DIVERGED incident that motivated it, not just prose rationale (FM7). Money-chain stakes at M3 are exactly when solo-operator, unreviewed rule-writing is least acceptable — this closes FM11 and materially slows FM7 with one process change.
Open questions
- What is the actual numeric investment cap, ratified by Robert and Tomas together? §25 Q5 (proposed here at 40 eng-days, DESIGN) needs a real decision, not a research-agent default.
- Who signs off on the kill-criteria review cadence, and what happens on a missed check? FM13’s prevention names Tomas as owner; the escalation path for a missed review is not yet designed.
- Does Zaruba’s oracle plane’s reconciliation output actually get compared against harness verdicts automatically, or does FM2(b)’s tripwire depend on a human noticing a disagreement? The PRD’s §13 boundary makes the two lanes independent by design but does not currently specify an automated cross-check — without one, FM2’s “earliest warning signal” is aspirational, not built.
- What is the real (not modeled) false-DIVERGED rate once self-replay runs on a genuine multi-week Orders corpus? Every noise-related failure mode in this document (FM1, FM2a, FM7) inherits this same unresolved number from
research.mdD6 Q2 — it is the single most load-bearing unknown in the entire product. - Has any part of this pre-mortem’s proposed 40-eng-day cap already been informally exceeded by the time this document is read, given the pilot ladder (forex-ng, giftcard_service, orders-ext) may already be underway? If so, FM9 is not a future risk — it is a present one, and the cap should be set against actuals-to-date, not zero.