07 — Pre-Mortem: Fleet Monitor

It is 2027-01. The fleet monitor was built per 01-initial-recommendation.md (ADOPT + thin layer, local-first) and it failed — not necessarily by crashing, mostly by becoming irrelevant, untrusted, or a time sink. Internal-tool failure modes only; no market/pricing ceremony (2 operators, no customers).

Evidence key: VERIFIED = pulled live 2026-07-18 (R2/R5/R6, cited by packet). PRIOR-RESEARCH = sourced to an earlier doc via R7, not rechecked this pass. ASSUMPTION = no source, stated as such. Likelihood/Impact are DESIGN judgments (no failure has happened yet — nothing to measure), not measured rates.

Where this pre-mortem’s evidence conflicts with 01-initial-recommendation.md, it’s called out inline (modes 4, 14) rather than silently aligned.

Failure modes

1. Trust collapse

A false/stale needs-input status — or one missed — sent once, and the operator reverts to tmux polling permanently. The product’s entire value is the operator trusting the list enough not to check manually. Root cause: status derived from unstable/inferred signals rather than an authoritative one — Claude Code’s session transcript format is explicitly “not stable for external parsing,” and there is no documented cross-machine session-enumeration API (VERIFIED, R6 §5/§7); homegrown multi-host bridging compounds this. Likelihood: H Impact: fatal, no recovery path (Norman lens, 01-initial-recommendation.md: “false status is a mapping failure — worse than no status”). Signal: Robert tab-cycles tmux “just to check” more than once in a day despite the board showing 0 blocked. Prevention: derive status only from claude agents --json (documented CLI contract), never transcript-parse; show “unknown,” never a guessed status. Mitigation: on any detected mismatch, kill that signal source immediately and fall back to raw Agent View until repaired — don’t “fix it later” while showing wrong data. Kill/pivot trigger: 2 independent false-negative incidents in one week → board demoted to advisory-only, Agent View becomes primary again.

2. Silent telemetry gaps

Hooks/OTel break on a Claude Code version bump; exporters stop, the dashboard freezes on last-known-green, the fleet burns unattended. Root cause: base stack is a pinned fork of ColeMurray/claude-code-otel (469★, 13mo stale per PRIOR-RESEARCH R7(b)) — “pin, don’t track” means it does not auto-adapt when Anthropic renames/adds metric attributes; Claude Code’s own doc surface shows real version churn within this research window alone (VERIFIED, R6, v2.1.139→v2.1.212+). Likelihood: M-H Impact: high — false-green is worse than known-down because the operator stops checking (same end state as mode 1, different cause). Signal: any core metric (e.g. claude_code.session.count) flatlines while tmux panes show live agents. Prevention: a synthetic “collector alive” heartbeat, checked by a watchdog outside the OTel pipe (plain cron+curl); pin Claude Code’s own version alongside the OTel fork, bump both together, tested. Mitigation: heartbeat failure → red “telemetry unknown since HH:MM” banner, degrade to Agent View. Kill/pivot trigger: >2 silent gaps in a quarter with real time lost → drop custom OTel parsing, subsist on claude agents --json polling alone.

3. Anthropic obsolescence (good failure — define the absorb)

Agent View / Agent Teams ships fleet-wide needs-input aggregation before or instead of the custom layer maturing; custom code is dead weight. Root cause: the product’s own thesis is “thin layer over what Anthropic doesn’t cover” — Anthropic ships fast (Agent View: preview → JSON output → forkable, inside this one sweep, VERIFIED R6) and the covered gap is designed to shrink, not an accident. Likelihood: M — same-host coverage is squarely Anthropic’s target; cross-machine and Zaruba-watch are explicitly not covered by anything found today (VERIFIED, R2 “Bottom line for the ADOPT verdict”). Impact: LOW/positive — this is the win condition, treat it as a good outcome, not a threat. Signal: a claude agents release adds cross-host or cross-team needs-input aggregation. Prevention: build panels as independently-killable units from day one; keep Zaruba-watch and ledger-join panels structurally separate from the needs-input list — those are the parts least likely to ever ship first-party. Mitigation: delete the obsoleted panel that same week; no parallel redundant UI “just in case.” Kill/pivot trigger: any Anthropic release covers needs-input + inline reply across this-host AND multi-host → retire the custom board entirely, keep only Zaruba-watch + ledger metrics.

4. Work-ledger never materializes

Ledger (BUILD THIN, separate Encore service) is decided but, per every doc read, “not reported executed” (PRIOR-RESEARCH, R7(d)) — cost/merged-unit and rework rate (2 of the canonical 4 metrics, R7(c)) never land; monitor stuck at queue-depth + stall-detection only, forever. Root cause: ledger competes for the same single operator’s (Robert’s) time as the monitor and the actual rebuild; 01-initial-recommendation.md’s own open question 6 — does v1 degrade gracefully without it — is unanswered in any doc read (R7(e)#4). This is a live contradiction, not resolved by the doc under test. Likelihood: M Impact: M — monitor stays functional (2/4 metrics) but the headline pitch (“cost per merged unit… in one glance”) silently becomes false. Signal: 4+ weeks post-launch, ledger service shows zero claimed units in prod. Prevention: design v1 to run session-level-only by default — show “N/A — ledger pending” for the two dependent metrics, not a guess and not a blank. Mitigation: if ledger stalls, ship the 2-metric board as the permanent v1 and revise the one-sentence pitch honestly rather than keep promising four. Kill/pivot trigger: 8 weeks, zero ledger units claimed → formally cut cost/merged-unit + rework-rate from the board copy, not just leave them silently absent.

5. Metric-wall creep

The board grows panels one “just this one” at a time until it’s the NOC-style wall the Ive lens explicitly rejected — in reverse of the ~80% panel-removal target (01-initial-recommendation.md). Root cause: Grafana makes adding a panel free to the adder; the cognitive-load cost shows up later, on someone else’s glance, with no natural forcing function to remove anything. Likelihood: H — default outcome for any Grafana-based tool absent an explicit governor. Impact: M — doesn’t break function, erodes the “glance, don’t analyze” value prop; the empty-state goal (“0 blocked,” large) gets buried. Signal: board needs scrolling, or panel count doubles from launch within 3 months, or a panel goes unglanced 30 days. Prevention: hard panel ceiling (4 metrics + needs-input list, nothing else) enforced as a review question on any new panel: “what decision does this change?” Mitigation: quarterly cull — anything nobody can name a decision it drove gets deleted, no grandfathering. Kill/pivot trigger: board needs scrolling to see the needs-input list → immediate reset to baseline, treated as regression not feature.

6. Zaruba-watch overreach

Watching Tomas’s deployments/workflows — his explicit ask (2026-07-18) — creeps from “what he asked for” into de facto surveillance: broader queries than needed, assignee-level drill-down, a digest that reads as monitoring-him rather than status-for-him. Root cause: the watch token (g-api-key, VIEWER role, minted once — VERIFIED, R5 §“Watch-surface map” rows 1a/2) is technically capable of reading more than executions+tasks status; nothing structurally caps it once it exists. Likelihood: M — no incident yet, but the token is broad-by-default across the workflow-management API, not scoped per-definition. Impact: H — this is the only relationship risk in a 2-person program. Signal: Robert checks task-assignee/who’s-behind data out of curiosity rather than to route a stuck task; or Tomas asks “why do you know that.” Prevention: scope queries to execution status + task status/dueAt only (surfaces 1a+2 per R5’s own v1 recommendation); write the scope decision into the ADR Tomas already expects (MADR + owner-domain + Decided-by:, PRIOR-RESEARCH R7(f)) so it’s auditable by him. Mitigation: on any discomfort signal, narrow the token immediately and show him the exact query log — don’t argue the scope was fine. Kill/pivot trigger: any explicit discomfort from Tomas → drop to the single coarsest signal (FAILED/TERMINATED counts only) until he re-authorizes anything richer.

7. Maintenance tax

Pinned forks (claude-code-otel base, dashboard 25255, claude-squad/agent_farm if adopted) rot; the “glance a day” tool becomes recurring unplanned work. Root cause: every OSS dependency named this pass shows real churn in one sweep — Terragon dead, Vibe Kanban quiet, Crystal renamed, omnara pivoted, the OTel fork itself 13mo stale (VERIFIED, R2). Pin-don’t-track avoids upstream breakage but converts it into owed upgrade labor, solo. Likelihood: H — 5 of ~10 tracked tools changed status in one sweep (R2 “Bottom line”). Impact: M — stable short-term (pinned), compounds; a forced upgrade (CLI breaks the old hook/metric format) can become a multi-day project. Signal: >2 hours/month spent on monitor upkeep unrelated to a deliberate feature add. Prevention: fixed, calendared monthly maintenance slot; track pinned-version staleness as a line item, checked monthly, not discovered via breakage. Mitigation: log any month where upkeep time exceeds time saved — 3 such months in a row is data, not a feeling. Kill/pivot trigger: 3 consecutive months of upkeep > value → strip to the single highest-value surface (needs-input list) and drop the rest.

8. Single-host loss

Local-first docker-compose, one box; the machine dies (disk failure, WSL2 corruption — a known class of issue on this exact platform), Prometheus/Grafana history and any local ledger cache is gone. Root cause: deliberate MVP choice, not an oversight — cloud is billing-gated (PRIOR-RESEARCH, R7(a)); this tool is a view, the real system of record (ledger’s Postgres if built; Zaruba’s own workflow-management DB) lives elsewhere. Likelihood: L Impact: L if the ledger stays the record of truth (design intent) — H only if Robert has drifted into treating Grafana history as the record, which the doc explicitly says not to do. Signal: none as drift (point failure) — the precondition is “is there a backup” going unanswered for months. Prevention: weekly cron export of the compose data volume to a second location; explicitly no HA effort — not worth building for 2 operators. Mitigation: rebuild from the declarative compose file in minutes; accept Grafana history is gone; the one thing needing real backup (ledger Postgres, if built) gets standard pg_dump, nothing novel. Kill/pivot trigger: none — correct response is rebuild; escalates only if it recurs (signals a bad host, not a bad design).

9. Stall-detection cry-wolf

active_time.total plateau-vs-wall-clock heuristic fires amber on agents that are fine (reading a big file, a slow tool call, Robert mid-review) until amber is tuned out — and the real stall hides in the noise the day it matters. Root cause: “stall” is inferred from a proxy, not ground truth; the Norman lens already flags this needs “an honest confidence signal, not a binary lie” (01-initial-recommendation.md) — the risk was anticipated, but no concrete threshold/calibration is specified in any doc read. Likelihood: M-H — uncalibrated proxy detection reliably over-fires before tuning. Impact: H — causes mode 1 via the opposite mechanism (false positives, not negatives; same end state: ignored board). Signal: amber-resolves-with-zero-intervention ratio rising over time. Prevention: graded confidence (quiet 5/15/30min+), not binary amber, matching the lens already committed to; calibrate thresholds against real session logs before defaulting it on. Mitigation: if false-positive rate is high, widen the threshold immediately — don’t let operators self-train to ignore the color. Kill/pivot trigger: self-resolved-without-action rate >50% over 2 weeks → demote stall detection from a glance-color to a passive queryable stat until recalibrated.

10. Operator bypass

claude agents alone is good enough for the common case — same-host needs-input, inline reply, JSON output already shipped first-party (VERIFIED, R6 §3) — and the custom board goes unopened after week 2. Root cause: the board’s differentiators (cross-host, Zaruba-watch, ledger metrics) are exactly the parts most likely to be incomplete early (see modes 4, 14); Agent View already covers the common single-host case well. Likelihood: M Impact: this is the kill signal, defined below, not a background risk. Signal: board not opened for 5+ consecutive working days after week 2 (measurable via a “last viewed” timestamp on the local page). Prevention: don’t reimplement what Agent View already does — link/embed it for that, lead the custom board with the actually-differentiated panels (Zaruba-watch, ledger). Mitigation: if bypass is happening, ask why in the monthly maintenance slot before adding more panels — the answer is almost never “needs more panels.” Kill/pivot trigger: adoption-failure kill, explicit: board unopened 5 consecutive working days, twice, within the first 6 weeks post-launch → stop custom-board development, keep Zaruba-watch/ledger data as a CLI query or daily-digest message only, no persistent UI.

11. OTel cardinality/cost blowup at 64 sessions

Per-worktree OTEL_RESOURCE_ATTRIBUTES tagging × the metrics schema’s optional attributes (model/query_source/agent.name/skill.name/mcp_tool.name, VERIFIED R6 §1) × up to 64 concurrent sessions — classic Prometheus cardinality explosion. Root cause: the schema is rich by design and the plan explicitly wants per-agent attribution; richer attribution directly means more label combinations = more time-series. Likelihood: M — 64 is the proven ceiling (PRIOR-RESEARCH, R7(b)), not routine load; real risk depends on label hygiene (OTEL_METRICS_INCLUDE_SESSION_ID defaults true, VERIFIED R6 §1 — an easy accidental cardinality multiplier). Impact: M-H if hit — Prometheus OOMs or query latency degrades exactly when the fleet is busiest, worst possible timing. Signal: prometheus_tsdb series count / query p95 climbing as concurrency ramps 10→50→64. Prevention: turn off OTEL_METRICS_INCLUDE_SESSION_ID for aggregate panels; pre-aggregate at the collector before ingestion; load-test cardinality at 64 synthetic sessions before the real ramp. Mitigation: if hit, drop the highest-cardinality label first (session.id > agent.name > skill.name), accept coarser attribution over a dead Prometheus. Kill/pivot trigger: main-board query p95 exceeds a few seconds at real fleet load → cut cardinality immediately; if it recurs after one cut, drop to per-model attribution only.

12. Cloud migration pain

Billing gate clears (R5 notes the adjacent infra project’s billing is already “PENDING a quota increase” — actively in motion, VERIFIED) and moving off single-box docker-compose is a rewrite, not a redeploy, because local-first assumptions are baked into schema/interaction model, not just deploy config. Root cause: “Not in MVP” (01-initial-recommendation.md) explicitly excludes auth/multi-tenant/network binding for speed — correct short-term, but nobody has flagged which exclusions are structural (schema/code) vs merely configuration, so the migration cost is unknown until attempted. Likelihood: M — this program’s billing gates have a track record of eventually clearing; it’s a “when.” Impact: M — bounded (thin layer, not the whole rebuild program), but costs calendar time at the moment attention is probably wanted elsewhere. Signal: a concrete cloud target date gets set while the exclusion list is still unrevisited. Prevention: at MVP ship time, write down which exclusions are structural vs config, so the migration estimate exists in advance. Mitigation: if migration is a rewrite, treat it as a fresh v2 build under the same thin-layer discipline rather than patching local-only assumptions into a cloud shape. Kill/pivot trigger: migration estimate exceeds the original MVP build cost → don’t migrate; stay local-first permanently, solve remote access narrowly (SSH/Tailscale tunnel) instead.

13. Zaruba-watch API drift

Workflow-management’s REST surface or its VIEWER token (minted once, VERIFIED R5) changes on Tomas’s side — field rename, pagination change, token rotation — and the pane silently stops updating. Root cause: zero coordination mechanism exists today — a token polled against an API entirely Tomas’s to change, no contract test, no deprecation channel (VERIFIED, R5: “poll a REST endpoint,” nothing more). Likelihood: M — both repos are under active development (VERIFIED, R5: “up to date,” recent commits); an external VIEWER-role consumer is exactly what breaks silently on an internal refactor. Impact: M — degrades exactly when it matters, on the pane least glanced (same stale=green shape as mode 5/9). Signal: executions/tasks poll returns 0 rows or an unexpected schema for more than one cycle — needs its own heartbeat (mode 2’s pattern, second pipeline). Prevention: same heartbeat discipline as mode 2; ask Tomas for an informal heads-up channel before endpoint shape changes — costs nothing to ask in a 2-person program. Mitigation: on drift, fail loud (“Zaruba-watch stale since HH:MM”), fall back to nothing, ping Tomas directly. Kill/pivot trigger: 3+ silent-drift incidents in a quarter → downgrade to a manual daily check (Robert reads his dashboard himself) rather than maintain a brittle scraper against a moving target.

14. Zaruba-watch blind to the deploy that matters most

R5’s own v1 scope covers Temporal-orchestrated workflow runs and (once PAT-unblocked) frontend CI — backend Encore Cloud deploys, “presumably what Tomas cares about most,” are entirely outside GitHub Actions and were left unverified this pass (VERIFIED-as-gap, R5 item 4). The monitor can show “all green” while structurally unable to see the deploy pipeline that matters. Root cause: v1 scope was picked by what’s cheap/available today (existing REST endpoints), not by what Tomas most wants watched — this is a real gap in the doc under test, not resolved by 01-initial-recommendation.md’s open question 3. Likelihood: M-H — this is a named, unverified gap today, not hypothetical. Impact: H — undermines the premise of “watching his deployments,” the ask that expanded this project’s scope; confident-but-incomplete coverage is worse than admitted no-coverage (mode 1’s shape, applied to Tomas’s trust). Signal: Tomas mentions a backend deploy issue the monitor never surfaced. Prevention: before calling Zaruba-watch “done,” investigate Encore Cloud’s dashboard/API/webhook surface (R5’s own flagged open action); until resolved, label the watch surface honestly to Tomas: “workflow + frontend CI only, backend deploy not yet covered.” Mitigation: if the gap surfaces via a real incident, close it same-week if Encore Cloud exposes a webhook; don’t let it sit as known-broken. Kill/pivot trigger: if Encore Cloud is verified to expose no usable API/webhook → tell Tomas directly rather than fake coverage; honest fallback is “paste me the deploy link” until Encore ships one.

15. Bus factor / solo maintainer

Robert is sole builder and maintainer; Tomas is explicitly read-only by MVP scope (01-initial-recommendation.md). If Robert is unavailable for an extended stretch, nobody can fix a break or explain a panel. Root cause: deliberate 2-operator scoping for speed has the side effect of zero redundancy — same shape as mode 7, but the failure axis is availability, not effort. Likelihood: L-M over a multi-month program — ordinary life events, not a crisis. Impact: L — the monitor is a view over other systems of record; its downtime doesn’t lose data, it just goes dark until Robert returns. Signal: none as ongoing drift — precondition is “could this restart from docker-compose up with zero tribal knowledge” going untested. Prevention: keep the stack to compose-file + README, no undocumented manual step (e.g. a hand-run token mint) without writing down exactly how to redo it. Mitigation: acceptable answer if Robert’s out: “it stays down until he’s back” — do not build redundancy nobody asked for. Kill/pivot trigger: none required, accepted risk — escalate only if downtime during Robert’s absence compounds with mode 14 (Tomas loses real information he needed) — that combination is the only version worth planning for.

The 3 leading indicators to actually instrument

1. Dual heartbeat, not data volume. A synthetic “pipe alive” check on both telemetry paths (Claude Code OTel and the Zaruba-watch REST poll), independent of the pipes themselves — covers modes 2 and 13, the two silent-gap failure classes, with one mechanism.
2. Board-open frequency. A single “last viewed” timestamp on the local page — the strongest, cheapest available adoption signal, and it directly instruments mode 10’s kill trigger.
3. False-status rate, both directions. Count false negatives (missed needs-input) and false positives (stall cry-wolf) separately — covers modes 1 and 9, the two ways trust actually dies.

Kill criterion (whole product)

Board unopened 5 consecutive working days, twice, in the first 6 weeks (mode 10) OR two false-negative needs-input incidents in one week (mode 1) — whichever trips first — kill the custom board. Fall back to Agent View + a plain script that polls Zaruba-watch and posts a daily-digest message; no persistent UI is maintained past that point. Do not keep building panels to “fix” a trust or adoption failure — neither is a panel problem.