Pre-mortem — work-ledger, 18 months on
Framing conceit (brief §14): it is 2028-01. work-ledger launched in mid-2026 and failed badly. This document is the retrospective written from that failure — root causes, the signals we ignored, and the triggers that should have made us pivot or kill sooner. Written before launch so those triggers get instrumented on day one, not discovered at the post-mortem.
Builds on plan 001’s five pre-mortem seeds (
00-SYNTHESIS.mdlines 119-125) — of which #1 (lane never existed), #3 (two sources of truth), #5 (blind scale-up) are work-ledger-specific — and goes deeper into the product-specific failure surface. Evidence tiers marked: confirmed fact / supported interpretation / hypothesis / assumption / unknown. Ground-truth IDs [E1]-[E10] inplans/001-factory-apps-validation/01-deep-dive.md.
0. How to read this
Every failure family carries the eleven fields the brief demands. To keep it dense, likelihood (L), impact (I), score, and owner live in the master table (§1); the per-family blocks (§3+) carry the remaining seven — root cause, earliest warning signal, leading indicator, prevention, mitigation, pivot trigger, kill trigger. Scores are L×I on a 1-5 scale each. The top three by decision-relevance get an extra scrutiny paragraph (§3.1-3.3).
Two definitions kept distinct, because the brief lists both:
- Earliest warning signal — the first cheap-to-observe symptom once the failure is already starting (coincident/lagging).
- Leading indicator — a forward metric you can watch on a dashboard that trends toward the failure before it lands.
1. Ranked failure register
| # | Failure family | L | I | Score | Owner | Extra scrutiny |
|---|---|---|---|---|---|---|
| F1 | Ledger theater — agents claim but state rots, dashboards lie, Robert stops looking | 4 | 4 | 16 | Robert | ✔ §3.1 |
| F2 | Claim contention / lease bugs at 64-concurrent — double work returns, trust gone | 3 | 5 | 15 | Robert | ✔ §3.2 |
| F3 | A1 politics / Zaruba absorbs it — lane never granted, revoked, or he builds unit-tracking into workflow-management and work-ledger becomes a shadow system |
3 | 4 | 12 | Tomas (grants) / Robert (builds) | ✔ §3.3 |
| F4 | Single-fleet irrelevance — Tomas’s fleet never claims through it; kill-gate panel uses his own evidence instead | 4 | 3 | 12 | Robert | |
| F5 | Cost attribution garbage-in — agents misreport, numbers diverge from Anthropic billing, finance-grade claims from vibes-grade data | 4 | 3 | 12 | Robert | |
| F6 | Schema ossification — early wrong unit granularity (too coarse / too fine) makes every later flagship fight the model | 3 | 4 | 12 | Robert | |
| F7 | Operational neglect — no backups, work_unit_events grows unbounded, reconciliation audit quietly disabled |
3 | 4 | 12 | Robert | |
| F8 | Two-truths collapse — a Temporal-disagreement incident kills trust in both stores | 2 | 5 | 10 | Robert | |
| F9 | Scope creep to PM tool — UI, priorities, comments creep in; becomes a worse Linear nobody maintains | 2 | 3 | 6 | Robert | |
| F10 | Program-level: the rebuild program pivots or dies — ledger outlives its purpose | 2 | 2 | 4 | Robert / Tomas |
Tiebreak note (F3 vs F4-F7, all score 12): F3 gets the third scrutiny slot on decision-relevance, not arithmetic — it is plan 001’s own #1 pre-mortem and the single live threat the prosecution case leans on hardest (platform absorption). The other 12s are treated in full but without the extra paragraph.
Reading of the register (supported interpretation): the two highest scores (F1, F2) are the product’s own reason to exist failing at its own job — nobody trusts the record, or the claim primitive lets double-work back in. Everything ranked 3-8 ultimately degrades into F1: a service nobody trusts is a service nobody looks at. F1 is the sink, not a peer.
2. How the failures cascade
flowchart TD
F3[F3 A1 politics /<br/>Zaruba absorbs] --> SHADOW{work-ledger is a<br/>shadow / duplicate}
F4[F4 single-fleet<br/>irrelevance] --> SHADOW
F6[F6 schema<br/>ossification] --> ROT
F5[F5 cost<br/>garbage-in] --> LIE
F7[F7 operational<br/>neglect] --> ROT
F2[F2 claim/lease<br/>bugs] --> DOUBLE{double-work<br/>returns}
F8[F8 two-truths<br/>collapse] --> LIE
DOUBLE --> LIE
SHADOW --> LIE
ROT[state rots vs reality] --> LIE{dashboards<br/>lie}
LIE --> F1[F1 LEDGER THEATER:<br/>Robert stops looking]
F1 --> DEAD[(dead weight —<br/>fold into monitor<br/>or kill, §25 PRD)]
F9[F9 scope creep] -.slow parallel path.-> DEAD
F10[F10 program<br/>pivots/dies] --> EXPORT[graceful shutdown<br/>= final disposition export<br/>§3.10]
The diagram is the argument for where to spend prevention budget: guard the two sinks (dashboards-lie and its terminal, ledger-theater) and most upstream families lose their teeth.
3. Failure families in detail
3.1 — F1 · Ledger theater ★ top-3
Description. Agents call claim because the CLAUDE.md paragraph tells them to, but the rest of the lifecycle rots: units sit in claimed forever because agents crash without releasing and nobody tuned the lease; verified/done transitions get skipped because the real work landed as a PR out-of-band; cost_cents is null on half the units; evidence_url is a placeholder. The Grafana panels render confident numbers over incomplete data. Robert glances at v_queue_depth, sees it climbing for reasons the panel can’t explain, stops trusting it, goes back to reading PRs directly. The ledger is now write-only theater — the exact “worse Linear that two people use” 01-initial-recommendation.md §“what it fundamentally is” warns against, arrived at by neglect rather than by adding screens.
| Field | Value |
|---|---|
| Root cause | The claim call is mandatory (it is how you get work), but every downstream write is optional and unenforced. Agent-usability of the full loop is untested — it’s the riskiest assumption in the whole plan (Cagan/Torres advisory; 12-prd.md R3, Q-untested #3), yet nothing forces transition/cost/dispose the way claim is forced. |
| Likelihood | Medium-High (4). With two human readers and no downstream enforcement, drift-to-disuse is the default, not the exception. |
| Impact | High (4). The service becomes dead weight; the audit record it exists to produce is full of holes exactly when Tomas needs it (feeds F4, F8). Not data-corrupting, so not a 5. |
| Earliest warning signal | Units in claimed/in_progress older than one lease-TTL with no terminal event; evidence_url null-rate on done units > 0. |
| Leading indicator | % of fleet work that flows through claims trending below ~80%, and verified-throughput (09-north-star.md NSM: units/week reaching a terminal disposition with cost AND evidence both populated) flat while raw claimed count rises. The gap between “claimed” and “verified with evidence” IS the theater metric. |
| Prevention | Error-strings-as-documentation so the loop is legible zero-shot (07-ux-ui.md §3, five verbatim strings); full-state returned on every call so an agent can’t lose track of where a unit is; the walking-skeleton under-1-minute usability self-test (07-ux-ui.md prototype spec) run against a real agent given only the OpenAPI schema, no prose — before the disposition/cost layer is built (Torres/Ive “zero-shot usability proof”). |
| Mitigation | Stale-lease auto-reclaim already appends a stale_reclaimed event (10-architecture.md §stale-takeover) so a dead agent’s unit doesn’t sit forever. Add a single “incompleteness” panel — units terminal-without-evidence, units claimed-past-2×-TTL — so the theater is visible as theater instead of hiding behind a healthy-looking queue-depth number. |
| Pivot trigger | Verified-throughput flat for 2 weeks while claim-count grows → the loop is too awkward; simplify the API surface (fewer required fields) before adding anything. |
| Kill trigger | <50% of fleet work through claims after 4 weeks → kill or fold into the fleet monitor’s schema (12-prd.md §25 kill/fold, Q1). This is the PRD’s headline kill trigger and F1 is what it detects. |
Extra scrutiny. F1 is ranked first because it is both the most likely and the terminal state every other failure decays into (see §2). It is also the hardest to detect honestly, because the failure mode is a dashboard that looks fine — queue-depth is a real number even when every other column is rotting. The defense that matters is therefore not another green panel but the one panel designed to go red: incompleteness. The plan already commits to reporting disposition-mix alongside the headline NSM precisely so a wall of DEFERRED (the cheapest disposition to assign) can’t masquerade as progress (09-north-star.md gaming-risk note); extend that instinct to the whole loop — every headline number pairs with a completeness denominator. The honest test is behavioural and cheap: does Robert choose to open the ledger instead of reading PRs, in week 3, unprompted? If a human with two readers won’t look at it voluntarily, no amount of schema rigor saves it. That is a one-week observation, and it should gate the disposition/cost build, not follow it (Cagan/Torres “smallest live test of the riskiest assumption”).
3.2 — F2 · Claim contention / lease bugs at 64-concurrent ★ top-3
Description. The one invariant the product exists to hold — one claimant per unit — breaks under real concurrency. Two agents hold the same unit and both do the work; or the stale-takeover path reclaims a unit whose original agent was merely slow (not dead), and now two attempts race to done. Double tokens burned, merge collision, and — worse than the cost — the first time Robert sees a duplicate, trust in the whole ledger is gone, because “no double-claims” is the one promise it made.
| Field | Value |
|---|---|
| Root cause | Not the SKIP LOCKED clause itself (battle-tested — Oban/graphile-worker/pg-boss; 10-architecture.md §claim). The bugs live in the custom parts around it: the stale-lease reclaim predicate (state='claimed' AND lease_expires_at < now()), the lease TTL (an uncalibrated knob, not a first-principles constant — 10-architecture.md line 232), and the fact that the atomic-claim concurrency test is NEW infrastructure — no Docker-PG vitest pattern exists anywhere in the repo today (repoRecon §5; contradicts the “house pattern” claimed in 00-SYNTHESIS.md line 27). A miscalibrated lease is not a cosmetic bug: arXiv:2606.19616 measured advisory-lease-only conditions producing the highest redundant-rediscovery rate of any tested design (priorArt digest §2, §4). |
| Likelihood | Medium (3). The proven clause pulls base rate down; the uncalibrated lease + custom stale path + never-before-written concurrency test pull it back up. |
| Impact | Catastrophic (5). Voids the product’s sole reason to exist. 09-north-star.md makes double-claims a hard-zero correctness invariant — one occurrence stops the concurrency ramp, not an SLO you amortize. |
| Earliest warning signal | Two claimed events for the same unit_id inside one lease window in work_unit_events; a merge collision on the PR lane traced back to two agents on one unit. |
| Leading indicator | Stale-reclaim rate rising as concurrency ramps (units reclaimed / units claimed) — the stall-list panel (07-ux-ui.md) is the designated feedback loop for tuning lease TTL. A reclaim rate climbing with the fleet size means the lease is too short and slow-but-alive agents are being pre-empted. |
| Prevention | The atomic-claim test written as real Docker-PG concurrency infrastructure (two concurrent claimers → exactly one 200 + one 409, verbatim string; 12-prd.md AC), acknowledged as new infra, not inherited. Lease TTL derived from measured attempt duration, not guessed. Heartbeat extends the lease so long-running-but-alive agents aren’t reclaimed (10-architecture.md §lease+heartbeat). |
| Mitigation | Every reclaim bumps attempt_no and appends stale_reclaimed, so a double-attempt leaves a durable trace (priorArt: race-to-close is otherwise invisible in git history) — detection survives even when prevention fails. A caught duplicate freezes the ramp at current concurrency until root-caused. |
| Pivot trigger | Stale-reclaim rate > a tuned threshold at any ramp step → hold the ramp, lengthen lease, before proceeding (09-north-star.md ramp gating 10→50→64). |
| Kill trigger | A confirmed double-claim that the schema could not detect (i.e. the append-only trace failed too) → the concurrency guarantee is unsound; stop the fleet ramp and re-architect the claim path before any further scale. |
Extra scrutiny. This is the highest-impact family and the one whose mitigation the plan has most thoroughly designed — yet the residual risk is real and specifically not where instinct points. The danger is not that SKIP LOCKED fails (it won’t at 10→64; the vacuum/MultiXact pressure point is a thousands-of-workers problem, irrelevant here — priorArt digest, 10-architecture.md scale-note). The danger is the lease, because it is the one number in the system that cannot be derived from first principles and must be tuned against a physical reality — agent attempt duration — that nobody has measured yet (this is the Ponytail “leave the calibration knob, the real clock drifts” principle applied literally). Too long, and a dead agent’s unit is stranded (feeds F1). Too short, and a live-but-slow agent is pre-empted and F2 fires. There is no safe default; there is only the stall-list feedback loop and the discipline to ramp concurrency on that metric rather than the calendar. The plan’s hard-zero framing is correct and should not be softened into a percentage: the first undetected double-claim is not a bug to log, it is a trust event, and trust in this system is asymmetric — built slowly across clean reconciliations, destroyed by one caught discrepancy (09-north-star.md moat framing).
3.3 — F3 · A1 politics / Zaruba absorbs it ★ top-3
Description. Three variants of the same organizational failure: (a) the CODEOWNERS-partitioned lane (assumption A1) is never granted — the Tomas conversation drags, Robert builds nothing for a quarter; (b) it’s granted then revoked when Tomas decides factory services don’t belong in his repo; (c) the most insidious — Tomas, who shipped the two-plane arch, tasks, full workflow-management, and a same-day auth rewrite across 81 commits solo in days (repoRecon §1, [E8]), simply adds unit-tracking columns to workflow-management himself, and work-ledger becomes a redundant shadow of his store. Nobody decided to kill it; it was out-competed by the platform owner building the trivial version faster than the lane negotiation resolved.
| Field | Value |
|---|---|
| Root cause | A1 is an assumption, untested — flagged as the top pre-mortem risk in plan 001 (00-SYNTHESIS.md pre-mortem #1, risk register A1 Medium) and unresolved as of 2026-07-18 (01-deep-dive.md assumption log A1). The durable per-unit table is a trivial addition to workflow-management for the person who owns it; the prosecution case (Count 6) rates platform-absorption live, not theoretical. |
| Likelihood | Medium (3). Plan 001’s own rating; A-lite fallback exists but variant (c) has no clean fallback. |
| Impact | High (4), asymmetric. Variant (a)/(b): a quarter lost, recoverable via A-lite (Robert-owned repo, API-only — portfolio unchanged, 00-SYNTHESIS.md A1 fallback). Variant (c): work-ledger is redundant — but this outcome is partly acceptable, because the durable table getting built anywhere satisfies the actual need (temporalCase verdict: “the table gets built regardless; standalone-vs-schema is an org-boundary call, not build/no-build”). |
| Earliest warning signal | No OWNERSHIP.md agreement within 2 weeks of the research/14 Tomas conversation (00-SYNTHESIS.md risk A1 detection). For variant (c): a commit to workflow-management adding a state/disposition/cost column. |
| Leading indicator | research/14 answer unsent or unanswered; Tomas’s commit velocity into adjacent surfaces (workflow-management, tasks) — the faster he moves there, the higher the absorption odds. |
| Prevention | Lead the A1 ask with the minimum — an OWNERSHIP.md + directory-convention boundary, not a standing review commitment; CODEOWNERS later on his startup-mode schedule (architect addendum). Make work-ledger a zero-PR-against-core service so it asks him for nothing ongoing (08-made-to-stick.md “asks Tomas for nothing”). Design the schema so it can move: it’s placement-agnostic by construction (12-prd.md §17). |
| Mitigation | Trigger A-lite immediately on the detection signal — do not wait out the negotiation (00-SYNTHESIS.md pre-mortem #1 response). For variant (c): fold gracefully — if his columns cover claim+disposition+cost, adopt them and delete work-ledger; the fold-into-monitor/platform path is a first-class kill trigger, not a defeat (12-prd.md §25, Q1). |
| Pivot trigger | A1 unresolved 2 weeks after the answer draft is sent → pivot to A-lite placement (no re-scope, just a different repo). |
| Kill trigger | Tomas ships durable unit-tracking (state + disposition + cost, readable past 72h) in workflow-management → fold: work-ledger’s reason to be a separate service is gone; migrate any rows and retire it. |
Extra scrutiny. The uncomfortable truth this pre-mortem must state plainly: F3 variant (c) is not purely a failure — it is a partially-good outcome wearing a failure’s clothes. Every pro-build digest concedes the durable table gets built regardless (temporalCase, build-vs-buy verdict); the prosecution’s sharpest charge is that naming it a separate service is redundant packaging. So if Tomas absorbs it correctly — durable, dispositioned, cost-attributed, readable at M12 — the program still gets what it needs and loses only Robert’s ownership of the surface. The genuine failure inside F3 is narrower and worth isolating: a quarter burned waiting for a lane that A-lite made unnecessary from day one, or an absorbed version that is worse than work-ledger’s spec (state that purges, no evidence links, no completeness discipline) shipped because it was faster. The prevention is therefore not to defend the service boundary politically but to make the boundary cheap to abandon: keep the schema portable, the asks minimal, and A-lite pre-warmed, so that whichever way A1 resolves — grant, fallback, or absorption — the table survives and the program’s audit record survives with it. Owning the surface is negotiable; the durable, evidence-bearing table is not.
3.4 — F4 · Single-fleet irrelevance
Description. Tomas’s fleet runs Core domain-migration workflows on a different execution model in a different plane and never claims through work-ledger; at the kill gate Tomas reads his own evidence (his workflow store, his ADR log), not v_disposition_summary. The “cross-fleet system of record” framing was oversized: it is a single-fleet Robert tool that happens to have a stakeholder-facing view nobody uses.
| Field | Value |
|---|---|
| Root cause | Open question #3 (01-initial-recommendation.md) is untested — “does anyone besides Robert’s fleet ever adopt it?” — and two execution models across two planes (temporalCase B3/B5) mean Tomas’s units aren’t even shaped to claim. The system-of-record framing rests on an assumption, not evidence. |
| Likelihood | Medium-High (4). Nothing in the plan obliges Tomas’s fleet to adopt; the default is that it doesn’t. |
| Impact | Medium (3). The claim + cost + disposition value for Robert’s fleet stands alone and still justifies the build (dispositions alone were the prior justification). What’s lost is the “cross-fleet” scope, not the tool. |
| Earliest warning signal | Zero units with fleet='tomas' after the first kill gate; Tomas’s M4 gate decision cites a source other than v_disposition_summary. |
| Leading indicator | Tomas’s explicit yes/no (Q3) on whether he’ll read the disposition table vs. the ADR log he already reads — get it in writing before the disposition schema is final (12-prd.md R4, Q3; Cagan/Torres “this is a phone call, not a research doc”). |
| Prevention | Don’t build cross-fleet affordances speculatively — the fleet column is free-form text, not an enum (10-architecture.md line 101, “YAGNI”), so single-fleet reality costs nothing. Route component-tier dispositions (~dozens, 26 repos) through the existing ADR/decision-log Tomas already reads; reserve the table for unit-tier cardinality (~1000) ADRs can’t carry (temporalCase; Christensen tier-split). |
| Mitigation | Reframe honestly as a single-fleet tool if Q3 comes back no — drop the “system of record” language, keep the claim+cost+disposition core. The value was never contingent on two fleets. |
| Pivot trigger | Tomas answers Q3 “ADR log, not the table” → collapse cross-fleet framing; make the disposition half ADR-backed and keep only claim + cost/queue (12-prd.md §25 fold row). |
| Kill trigger | Combined with F1 (Robert’s fleet also < 50% adoption) → nothing writes through it → kill. Single-fleet alone is a scope correction, not a kill. |
3.5 — F5 · Cost attribution garbage-in
Description. Agents self-report cost_cents through the external-claim path, and the numbers are wrong: token estimates not billed cost, missing retries, double-counted attempts. v_cost_per_unit and the cost/merged-unit panel show finance-grade precision built from vibes-grade inputs. The first time anyone reconciles a monthly figure against Anthropic billing (or ccusage) the divergence is large enough that no cost number from the ledger is trusted again — poisoning the NSM, whose definition requires cost populated to count a unit as verified (09-north-star.md).
| Field | Value |
|---|---|
| Root cause | The ledger treats agent-reported cost as authoritative, but the billed truth lives in OTel (OTEL_RESOURCE_ATTRIBUTES="worktree=…,agent=…", research/12 §2) and ultimately Anthropic’s billing. Two independent cost paths with no reconciliation is a dual-source problem (the same anti-pattern F8 names, applied to money). |
| Likelihood | Medium-High (4). Self-reported cost diverging from billed cost without a reconcile is near-certain; the global rule “re-sum billed cost from real usage, never the estimate” (CLAUDE.md §LLM Run Evidence) exists precisely because this always drifts. |
| Impact | Medium (3). Cost claims become untrustworthy and the NSM’s cost gate rots; but claim-safety and disposition integrity are unaffected, so not a 5. |
| Earliest warning signal | Ledger monthly cost total vs. ccusage/billing total diverging > a few %; cost_cents null or implausibly round on real units. |
| Leading indicator | Reconciliation delta (ledger cost vs OTel-derived billed cost per agent/worktree) tracked weekly — the same audit muscle as the state reconciliation (10-architecture.md §weekly reconciliation), pointed at money. |
| Prevention | Make OTel the source of truth for cost and have the ledger’s cost be a joined/projected value keyed on worktree/agent, not an agent-authored free number — cost recorded per attempt but reconciled against billed OTel, not accepted blind. Where the fleet monitor already computes cost/merged-unit from OTel (research/12 §2), the ledger should read that, not re-derive it (Christensen/Cagan “one place cost is recorded”). |
| Mitigation | Label cost confidence the way the repo already labels facts (MEASURED vs MODELED, 00-SYNTHESIS.md risk register) — a reconciled cost and an agent-estimate are not the same datum and shouldn’t render identically. |
| Pivot trigger | Reconciliation delta > ~10% → stop treating agent-reported cost as authoritative; switch the cost column to OTel-joined. |
| Kill trigger | (Scope, not service) — drop ledger-owned cost entirely and read cost from the monitor if no reconciliation path holds; the cost column is the most foldable part. |
3.6 — F6 · Schema ossification
Description. The unit granularity chosen in week one turns out wrong — units are too coarse (one row per repo, so attempts/cost/blocking-questions can’t attach where the work actually happens) or too fine (one row per file-edit, so the table is 100k rows of noise and v_disposition_summary is unreadable). By flagship 2, every new domain fights the model; the 7-state enum and 4-way disposition either don’t fit or force lossy mapping, and migrating a live SSOT with an audit trail is expensive and risky.
| Field | Value |
|---|---|
| Root cause | Granularity and the disposition taxonomy were fixed a priori from the legacy sample [E6] and the decommission-ledger merge, before real fleet units existed to validate them. Specifically DEFERRED vs DROPPED may be a distinction without an operational difference at the gate (Jobs/Ive challenge; 12-prd.md Q4), and 7 execution states may be more than the loop needs (PRD open question, state-granularity 7-vs-5). |
| Likelihood | Medium (3). Some model revision is near-certain; a painful one depends on how early real usage corrects it. |
| Impact | High (4). Migrating a live append-only SSOT is the expensive kind of change; every flagship built on the wrong grain compounds the cost. |
| Earliest warning signal | Agents systematically registering units at a different grain than designed (e.g. splitting one unit into sub-units via external_ref hacks); disposition assignments clustering on one value because the others don’t fit. |
| Leading indicator | Disposition mix skew (e.g. DEFERRED >> everything, the cheapest-to-assign state — already a flagged gaming risk, 09-north-star.md); attempt_count distribution bimodal in a way the grain didn’t anticipate. |
| Prevention | Treat the enum and grain as provisional for the first 2 weeks of live use, not a locked spec (Torres “hypothesis set to test, not a finished design”); decide DEFERRED-vs-DROPPED collapse and 7-vs-5 states from real data, explicitly deferred as open questions (12-prd.md Q4) rather than settled. The append-only work_unit_events log makes grain-history recoverable, softening a later re-grain. |
| Mitigation | Enum widening is a cheap Drizzle migration (generated, pnpm drizzle); enum narrowing or re-grain is the costly direction — so start narrow and widen (fewer states, collapse DEFERRED into DROPPED unless data demands the split) rather than shipping speculative granularity. |
| Pivot trigger | Two weeks of data show the grain is wrong → re-grain now, while the table is small, not at flagship 2 when it’s 1000+ rows. |
| Kill trigger | n/a — schema is fixable; this family never kills the product, it taxes it. The kill path is only via F1 if the tax makes the tool not worth using. |
3.7 — F7 · Operational neglect
Description. It’s an internal tool with a solo operator, so the boring parts get skipped: no Postgres backup of the SSOT, so a bad migration or disk loss erases the M12 audit record irrecoverably; the work_unit_events append-only table (and its projection-dedup rows) grows unbounded and the claim hot-path slows; the weekly reconciliation cron throws once, nobody notices, and it’s quietly disabled — so the one detector for F8 and F5 is off and theater (F1) creeps back invisibly.
| Field | Value |
|---|---|
| Root cause | Internal tooling for two humans attracts zero ops attention until it hurts; the reconciliation audit is the plan’s central credibility mechanism (08-made-to-stick.md) but a cron that fails silently is worse than no cron — it implies a check that isn’t running. |
| Likelihood | Medium (3). Default trajectory for a solo-operated internal service. |
| Impact | High (4). SSOT data loss on an audit record meant to survive to M12+ is close to catastrophic (the whole point is durability past Temporal’s 72h purge, temporalCase B1); a silently-disabled audit re-opens F1/F5/F8 undetected. |
| Earliest warning signal | No backup exists / restore never drilled; reconciliation cron last-success timestamp stale; work_unit_events row count growing with no retention/partition plan. |
| Leading indicator | Claim p99 latency creeping up (the PRD NFR: claim p99 < 200ms) — the append-only table bloating shows up here first; cron success-heartbeat gap. |
| Prevention | A restore drill as an explicit NFR (12-prd.md NFR set names it) — a backup you haven’t restored isn’t a backup. Keyset pagination on work_unit_events (auditlog precedent, 10-architecture.md) keeps reads bounded as it grows. Make the reconciliation cron’s failure an alert, not just its mismatches — a dead check must page. |
| Mitigation | The append-only log means point-in-time recovery is possible if backups exist; the dedup unique index (source_event_id) makes re-projection idempotent, so a restore-then-replay is safe. |
| Pivot trigger | Claim p99 > 200ms sustained → partition/archive work_unit_events before it degrades the hot path. |
| Kill trigger | n/a — neglect is preventable operations, not a product-viability verdict. Its danger is enabling the families that do kill (F1/F8). |
3.8 — F8 · Two-truths collapse
Description. Ledger state and Temporal execution state disagree during an incident; because neither can be presumed right, trust in both collapses — plan 001 pre-mortem #3. The classic form: work_units.state says done while Temporal says the workflow failed.
| Field | Value |
|---|---|
| Root cause | Any bidirectional coupling between the two stores; the dual-write anti-pattern (priorArt digest — the well-documented failure the one-way projection design exists to avoid). |
| Likelihood | Low (2) — and lower still in v1 for a specific, load-bearing reason (see scrutiny). Structurally mitigated: projection is one-way by construction, code-review-enforced (10-architecture.md §one-way invariant). |
| Impact | Catastrophic (5) when it fires — trust in both stores gone at the worst possible moment (an incident). |
| Earliest warning signal | First weekly-reconciliation mismatch between work_units.state and Core workflow_executions.status (10-architecture.md §weekly reconciliation — this query is the designated detector). |
| Leading indicator | Any code-review-approved write from work-ledger toward execution state (the invariant breach precedes the incident); reconciliation mismatch count > 0. |
| Prevention | One-way projection by construction: work-ledger reads workflow_executions via the generated client only, never writes execution state back (P2, the core/internal boundary in root CLAUDE.md); everything enumerable is enum + CHECK so state can’t be set to an illegal value out of band. |
| Mitigation | Recover by idempotent read-before-write, never an out-of-band manual edit (priorArt: Vitess recovery discipline); a mismatch is an alert, not an auto-fix — a human reconciles, the log records it. |
| Pivot trigger | Repeated mismatches on the same code path → that path is doing a hidden dual-write; excise it. |
| Kill trigger | n/a via this family alone — the fix is architectural, not abandonment. |
Note (supported interpretation, corrects a stale assumption). In v1 the classic two-truths incident cannot fire, because there are no real fleet-unit Temporal workflows — repoRecon found exactly one toy example-internal-workflow repo-wide, and all v1 units come through the external-claim path with a null workflow_id. So the reconciliation cron in 10-architecture.md (“for every unit with a non-null workflow_id…”) has nothing to compare against in v1 — it is inert until projection lands (post-MVP). The actual v1 truth-divergence is ledger-state vs. reality — did the agent really do what it reported verified/done — for which there is no automated oracle at all. That makes F8’s v1 manifestation a sub-case of F1 (theater), detectable only by the incompleteness panel and human spot-check, not by the reconciliation query everyone assumes is guarding it. Don’t let the (correct, but post-MVP) one-way-projection mitigation create false confidence that v1 is protected — in v1 it isn’t, because there’s nothing to reconcile.
3.9 — F9 · Scope creep to PM tool
Description. A priority field here, a comment thread there, a little status web page “just for Tomas,” and work-ledger drifts into a worse Linear that two people use and nobody maintains — the fate 01-initial-recommendation.md names as the thing to resist from day one.
| Field | Value |
|---|---|
| Root cause | The gravitational pull of “while we’re here, let’s add…”; every defunct competitor (Vibe Kanban/Bloop Apr 2026, Terragon Feb 2026) shipped exactly this shape (competitive digest). |
| Likelihood | Low (2). Strong structural guards: the anti-UI stance is committed (07-ux-ui.md), and 04-user-stories.md carries 7 explicitly-rejected stories (kanban, comments, priorities, notifications, assignee-routing, embedding-dedup, multi-tenant roles) each with a one-line reason. Internal 2-human tools have little organic pressure to grow screens. |
| Impact | Medium (3). Maintenance burden and blurred purpose, but not data-corrupting. |
| Earliest warning signal | A PR adding a non-enumerable free-text field beyond question/evidence_url; a request for a sortable web view before the 3rd-human trigger. |
| Leading indicator | Any feature request that isn’t claim/transition/block/dispose/cost/read — the rejected-stories list is the tripwire. |
| Prevention | The rejected-stories list as a standing “no” (04-user-stories.md); the one server-rendered page has a falsifiable build trigger — the 3rd human who needs interactive non-SQL access (today there are 2; 07-ux-ui.md, 12-prd.md §22 N1) — so “a page for Tomas” is answerable with “he’s reader #2, the trigger is #3.” |
| Mitigation | Free text confined to question and evidence_url by schema (01-initial-recommendation.md §usability risk) — creep has to fight the data model, not just taste. |
| Pivot trigger | Sustained pull for interactive reads from a real 3rd human → build the one read-only server-rendered table, nothing more (01-initial-recommendation.md platform verdict). |
| Kill trigger | If it becomes a maintained web app, it has already failed as a ledger — but that’s F1 (dead weight) by another route, not a separate kill. |
3.10 — F10 · Program-level: the rebuild program pivots or dies
Description. Over 18 months the rebuild program itself changes shape — Orders flagship stalls at a kill gate, the estate strategy pivots, Groupon re-orgs the effort — and work-ledger is left tracking work that no longer matters. This is the one family where the right response is not to prevent the failure but to exit gracefully.
| Field | Value |
|---|---|
| Root cause | work-ledger is downstream of a program run by a decision-bound owner at real kill gates (M4/M8/M12, [E1]); the program ending is a normal outcome the gates are designed to produce, not a failure of the ledger. |
| Likelihood | Low-Medium (2) over 18 months. |
| Impact | Low (2) — if graceful shutdown is a feature. Catastrophic only if the ledger dies the way the two stranglers did: in silence, taking its record with it [E6]. |
| Earliest warning signal | A kill gate returns “don’t proceed”; program direction memo; Robert’s fleet reassigned. |
| Leading indicator | Verified-throughput trending to zero for reasons outside the ledger (no new units registered because there’s no work, not because the tool broke). |
| Prevention | n/a — you don’t prevent a program from ending; you make the ledger’s death clean. |
| Mitigation → the feature | Graceful shutdown = final disposition export (see below). |
| Pivot trigger | Program pivots to a new domain → the ledger follows (it’s placement- and domain-agnostic; fleet/domain are free-form). No rebuild needed. |
| Kill trigger | Program ends → run the export, commit the artifact, retire the service. This is success, not failure. |
Graceful shutdown as a first-class feature (design spec). The inverse of [E6]’s rotting stranglers — a migration dying in silence with no record of what happened to each unit — is a migration that ends with a permanent, human-readable record of every disposition. Spec:
GET /export(orpnpm work-ledger:export) → one flat file (CSV + JSON) of everywork_unit:external_ref, finalstate,disposition(MIGRATED/DROPPED/DEFERRED/DEAD),cost_cents,evidence_url, terminal-event timestamp, and the fullwork_unit_eventstrail per unit.- The artifact is committed to the append-only decision-log repo (
reasons-and-decisions/, git-durable forever, the medium Tomas already reads) at program end — so the record outlives the service, the Postgres instance, and the billing account. - Shutdown procedure = export → verify row count against
v_disposition_summary→ commit artifact → drop the service. The ledger’s whole purpose (“no unit ever in an ambiguous state,”01-initial-recommendation.md) is most fulfilled at the moment it produces this file and stops.
This is cheap (one query, one commit), and it converts F10 from a failure family into a planned, dignified ending — the single place in this pre-mortem where the correct outcome is to let it die well.
4. Consolidated instrumentation — the one dashboard to build day one
The prevention across all ten families reduces to a small set of forward metrics. Instrument these before first fleet use, not after the post-mortem:
| Metric | Detects | Red when |
|---|---|---|
| % fleet work through claims | F1, F4 | < 80% trending down; < 50% = kill/fold trigger |
| Verified-throughput (units/wk terminal with cost + evidence) | F1, F5 | flat while claimed count rises |
| Incompleteness count (terminal-without-evidence; claimed past 2×TTL) | F1, F8-v1 | any sustained non-zero |
| Stale-reclaim rate (reclaimed / claimed) | F2 | rising with concurrency |
Double-claim count (two claimed per unit per lease) |
F2 | > 0 ever (hard-zero, freezes ramp) |
| Cost reconciliation delta (ledger vs OTel/billing) | F5 | > ~10% |
State reconciliation mismatches (ledger vs workflow_executions) |
F8 | > 0 (inert in v1 — nothing to compare) |
| Reconciliation-cron last-success age | F7 | stale (the check itself died) |
| Claim p99 latency | F7 | > 200ms sustained |
| Disposition mix skew | F6, F1 | one value (esp. DEFERRED) dominating |
Five of these (queue depth, stall list, cost/merged-unit, rework, disposition summary) are already the committed v1 Grafana panels (07-ux-ui.md). The additions this pre-mortem demands are incompleteness, double-claim count, cost-reconciliation delta, and cron-liveness — the panels designed to go red, because a pre-mortem’s job is to build the alarms, not the applause.
5. What we’ll be glad we did (if this doc worked)
Supported interpretation. If work-ledger doesn’t fail, the reasons will trace back to four decisions this pre-mortem argues for making now:
- The incompleteness panel existed, so F1 (theater) was visible as theater instead of hiding behind a healthy queue-depth number — and Robert kept looking because the tool told the truth about its own gaps.
- The lease was tuned from measured attempt duration on the stall-list feedback loop, not guessed, so F2’s double-claims never returned and the concurrency ramp stayed gated on data.
- A-lite was pre-warmed and the schema kept portable, so F3 resolved — grant, fallback, or absorption — without a lost quarter and without a worse table shipping in work-ledger’s place.
- Graceful shutdown was built as a feature, so when the program ended (F10) the ledger produced its permanent disposition record and died well — the exact opposite of the two stranglers rotting in silence that justified building it in the first place [E6].
The through-line: the failure this pre-mortem most fears is not a crash but quiet decay into a record nobody trusts (F1, the sink for §2). Every prevention above is a way of making decay loud — and a loud failure is a fixable one.