04 — PRD: Fleet Monitor

2026-07-18. Internal 2-operator tool. This PRD re-tests 01-initial-recommendation.md against the research pass (R1, R4, R5, R6, R7 packets, all dated 2026-07-18) rather than assuming it. Evidence labels: VERIFIED (source+date), PRIOR-RESEARCH (dated doc, not rechecked today), ASSUMPTION (my inference, flagged). Numbers carry MEASURED / MODELED / DESIGN passports.

Exec summary

Ship the smallest thing that gets Robert off tab-cycling and gives Tomas passive visibility into his own deployments — in that order, using what already exists before building anything. The research pass confirms the ADOPT-first direction but shrinks the build far below what 01 and research/12 scoped: two of Robert’s own rung-2 tools (fleetops, ghmonitor) already cover most of session-status and CI/deploy watching (VERIFIED-TODAY, R4). Claude Code’s native claude agents Agent View already ships peek-and-reply without attaching (VERIFIED-TODAY, R6). Tomas’s highest-value watch surface (workflow executions + tasks) is a two-endpoint REST poll against his existing API, zero new infra (VERIFIED-TODAY, R5). What’s left to actually build is thin: a ledger-joined metrics view (degrades gracefully pre-ledger), the Zaruba poller, a daily digest, and a wave report. No docker-compose OSS fork is required to get F1–F3 working; it’s still useful for aggregate cost/token panels but is no longer the critical path.

Corrections to `01-initial-recommendation.md` (evidence-driven, not silent)

Q1 (rung-2 check) resolves YES, and it changes the plan, not just answers a question. 01 scoped F1 as new build (hook ingest → SQLite → web page). fleetops already discovers every live session, derives status (working/idle/needs-answer/waiting/stalled), reads token/context-window from transcripts, and ships fleet snapshot as a JSON dump built for exactly this (VERIFIED-TODAY, R4). F1 is now REUSE, not BUILD.
The OSS anchor 01/research/12 implied (disler’s hook→bus→UI pattern) has a live license blocker (LICENSE 404s, license.spdx_id: null) and is separately moot: Claude Code’s own hooks docs already document HTTP-forward-to-collector as a first-party mechanism (VERIFIED-TODAY, R1 + R6). No OSS repo needs to be forked or cloned for the ingest pattern at all.
Q3 (Zaruba scope) resolves narrowly, and differently than 01 implied. The highest-leverage, zero-blocker surface is workflow_executions + tasks REST polling (VERIFIED-TODAY, R5) — not GitHub Actions (blocked on an unissued PAT) and not Temporal direct (nothing provisioned). A real gap 01 didn’t name: backend deploys run through Encore Cloud, invisible to .github/workflows/* entirely — unverified what Encore Cloud exposes, flagged not dismissed (R5).
01’s F4 framing (“join work-ledger rows”) undersells the dependency. The ledger is decided-not-built (PRIOR-RESEARCH, R7d), and its own PRD explicitly excludes Temporal projection from v1 — it needs Robert’s fleet to be separately wired via hooks to call claim/transition/block (PRIOR-RESEARCH, plans/002-work-ledger-product/01-initial-recommendation.md). That’s two unbuilt things stacked, not one. This is why the degradation path below is required, not optional.
Kanban stays removed — 01 already cut it from concept; research/12’s Vibe Kanban donor mention is superseded, treated here as a deliberate override, not an oversight (R7e#1).

Problem statement + evidence

Two distinct, evidenced problems, not one:

Robert’s triage blindness. At N>10 concurrent Claude Code sessions, Robert cannot tell which agents are blocked/awaiting-input/failed without tab-cycling; blocked agents idle silently (wasted wall-clock + tokens); the program has no cost/quality accounting per merged unit (PRIOR-RESEARCH, 01). Not hypothetical: a 34-agent fleet ran today, 2026-07-18 (~14:00–14:30, 25 repo profilers + 8 sub-profilers + 1 merge, ~2.1M subagent tokens) with no persisted observation layer — Fable orchestrated live, nothing recorded fleet state independently of the run itself (VERIFIED-TODAY, R4 item 4, docs/llm-runs.md).
Tomas’s blind spot into his own deployments. Tomas explicitly asked (2026-07-18, 16:52 GChat) that Robert’s tooling also watch his deployments/workflows (PRIOR-RESEARCH, 01) — a distinct ask the fleet-monitor scope didn’t originally contain, and one that R5’s live re-check shows is mostly not a build problem: the signal already exists as authenticated REST endpoints in his own app (VERIFIED-TODAY, R5).

Both problems share a root cause worth naming: the observation layer has never been anyone’s job. Robert’s fleet runs live-orchestrated with no standing collector; Tomas’s execution/task data is reconciled and exposed but nobody polls it. Fleet Monitor’s job is specifically to be that missing observer — not to replace the systems of record on either side (session state stays Claude Code’s; workflow state stays Tomas’s workflow_executions table).

Goals / non-goals

Goals:

G1 — median time-to-unblock <60s once flagged.
G2 — “0 blocked” is a state the operator trusts at a glance.
G3 — Tomas gets passive visibility into his workflow executions/tasks without logging into anything.
G4 — cost/merged-unit and rework-rate become measurable once the ledger + fleet-hook integration exist — not claimed before then.
G5 — reuse native Claude Code capability and Robert’s existing rung-2 tools before writing anything new.

Non-goals:

Multi-host/cross-machine fleet aggregation (single WSL2 host today — fleetops is WSL2-only, VERIFIED-TODAY R4).
Auth/multi-tenant, mobile, Tomas write-access, auto-remediation, triage dedup (separate app), cloud deploy (PRIOR-RESEARCH 01 “Not in MVP”, unchanged).
A growing analytics suite — panel count should shrink, not grow.
GitHub Actions watching in v1 (blocked on PAT; ghmonitor already covers it — add later, near-zero effort).
Real-time Encore Cloud backend-deploy status in v1 (surface unverified, see F5).

Product principles

Attention router, not analytics suite. One prioritized list; needs-input floats to top; aggregate health is underneath, not primary (PRIOR-RESEARCH, 01).
Trust SLO over feature count. A wrong needs-input flag, once, sends the operator back to tmux permanently (PRIOR-RESEARCH, 01 “Biggest usability risk”). False status is worse than no status — every metric below either has an honest confidence label or doesn’t ship.
Adopt-first, in a specific order (this is the pass’s main correction): native Claude Code capability (claude agents, OTel, hooks) → Robert’s existing rung-2 tools (fleetops, ghmonitor, tokenomics) → license-clean OSS component → new build, only for the delta none of the above cover.

Personas + JTBD

Robert (operator, primary). “When I have 10–64 agents running, I want to instantly see which ones need me, so I can unblock them without tab-cycling and without learning to distrust the list.” Also: “When a wave finishes, I want to know what it cost and whether the fleet is trending better or worse, so I can decide whether to scale concurrency.”

Tomas (consumer, secondary, read-only). “When I’m not actively watching, I want to know if my workflows/deployments are stuck or failing, so I don’t find out from a downstream complaint.” No dashboard login, no write access, GChat digest only (PRIOR-RESEARCH, 01; house channel per R7f).

Functional requirements

F1 — Unit/session board (needs-input list). REUSE, not build. Data source: fleet snapshot (JSON dump, VERIFIED-TODAY R4) + claude agents --json (native, requires v2.1.139+, VERIFIED-TODAY R6) for session status (working/needs-input/idle/done/failed). The thin layer’s only new work here is rendering both feeds as one sorted list (needs-input first) instead of two separate tools — a view, not a collector.

F2 — Inline unblock path. REUSE for local sessions: native Agent View already supports peek-and-reply (Space) without attaching, and Enter/attach for full context (VERIFIED-TODAY R6). fleetops additionally jumps to the wezterm pane on Enter (VERIFIED-TODAY R4). No new build for single-host. Gap, out of scope: no documented cross-machine session enumeration exists (VERIFIED-TODAY R6) — if the fleet ever spans >1 host, this breaks; not a v1 problem since the fleet is single-host today.

F3 — Telemetry ingest (OTel + hooks). BUILD (config, not code):

Enable CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_METRICS_EXPORTER=prometheus (or otlp), OTEL_RESOURCE_ATTRIBUTES="worktree=…,agent=…" per dispatch — not CLAUDE_CODE_RESOURCE_ATTRIBUTES, which does not exist (PRIOR-RESEARCH, research/12, fact-checked).
OTEL_LOG_USER_PROMPTS stays default-off, OTEL_LOG_TOOL_CONTENT/OTEL_LOG_RAW_API_BODIES never enabled (VERIFIED-TODAY R6) — see confidentiality NFR.
Metrics consumed: claude_code.cost.usage, .token.usage, .active_time.total, .pull_request.count, .commit.count (full metric list VERIFIED-TODAY R6).
Hooks used for stall/lifecycle signal where OTel’s 60s export interval is too coarse: SubagentStart/SubagentStop, Stop, PermissionRequest — forwarded via the documented HTTP-POST-to-collector mechanism (VERIFIED-TODAY R6). No OSS pattern-steal needed (correction #2 above) — this is a first-party recipe, not a borrowed architecture.

F4 — Factory metrics (rework rate, cost/merged-unit, queue depth, stall detection). BUILD, ledger-dependent for two of the four. Definitions:

Metric	Formula	Ledger dependency	Day-1 available
Stall detection	`claude_code.active_time.total` flat (Δ≈0) over window W while session wall-clock age keeps climbing past W	None — pure OTel/session data	Yes
Queue depth	`count(units) GROUP BY state` from ledger’s queue-depth SQL view (PRIOR-RESEARCH, work-ledger `01`)	Ledger read	No — day-1 proxy: `count(sessions) GROUP BY status` from `claude agents`/`fleet snapshot` (MODELED — session ≠ unit; a unit can span retries/multiple sessions)
Cost/merged-unit	Σ(ledger `cost` column) over units reaching `verified` disposition in period ÷ count(merged units)	Ledger read	No — day-1 proxy: Σ `claude_code.cost.usage` tagged by `OTEL_RESOURCE_ATTRIBUTES` worktree ÷ `claude_code.pull_request.count` in same window (MODELED — PR count is not unit count; will misattribute multi-session units)
Rework rate	`count(units verified→reopened post-merge)` ÷ `count(merged units)`, window W	Ledger transition log (needs unbroken unit identity across sessions)	No, and no proxy shipped. File-touch-reopen heuristics were considered and rejected — too noisy to be trustworthy (Norman: false status worse than none). Displays `—` until ledger + fleet-hook integration both exist.

F5 — Zaruba watch pane. BUILD, small, scoped per R5’s v1 recommendation:

Poll POST /workflow-management/executions (status, progress, timestamps) and GET /tasks (assignee, dueAt, linked workflowId), joined on workflowId/runId — turns “stuck in WAITING_FOR_APPROVAL” into “task X, owned by Y, overdue since Z” (VERIFIED-TODAY R5).
Auth: one g-api-key token, VIEWER role, minted via tokenCreate — no new infra, no OSS tool (VERIFIED-TODAY R5).
Alert on FAILED/TERMINATED counts and on rows stuck IN_PROGRESS past an expected duration (threshold: DESIGN, no baseline — start at 2× observed median, tune after week 1).
The interceptor names originally suspected as pollable (_ensureInProgress, _progressUpdate, workflowStatusUpdate) are not monitor-facing — they’re Temporal-worker-side, expose:false, in-cluster only. The monitor reads their effect (the workflow_executions row), not the calls themselves (correction confirmed VERIFIED-TODAY R5).
Explicitly not v1: GitHub Actions (blocked on a PAT Tomas already approved but hasn’t provisioned — ghmonitor covers it day one once it lands, VERIFIED-TODAY R4/R5); Temporal direct (cluster not provisioned, billing PENDING, VERIFIED-TODAY R5); Encore Cloud backend-deploy status (unverified what it exposes — real gap, flagged for a follow-up spike, not silently dropped, R5).

F6 — Daily digest to Tomas (GChat). BUILD, small. Content: FAILED/TERMINATED execution counts, overdue task counts from F5, each number passport-tagged (MEASURED, live REST poll). GChat is house standard for stakeholder-facing comms (PRIOR-RESEARCH, R7f). Terse, receipts-first — no marketing prose, matches Tomas’s own audit-law style.

F7 — Wave report. BUILD, small — a formatted read, not a new data source. A “wave” = one fleet run (e.g., today’s 34-agent profiling run, R4 item 4). Report = agents dispatched, units claimed/merged/blocked, cost, errors, joined against the ledger when it exists, else docs/llm-runs.md + OTel session totals (this repo’s existing manual-run evidence surface per global CLAUDE.md’s LLM Run Evidence rule — reuse, don’t duplicate).

Non-functional requirements

Requirement	Target	Passport
Needs-input staleness	≤15s (local JSON reads, sub-second in practice)	DESIGN
Zaruba pane staleness	≤20 min total = Tomas’s own 15-min reconciliation cron (VERIFIED-TODAY R5, “self-healing mirror of Temporal”) + ≤5 min monitor poll interval	DESIGN (poll interval) / MEASURED (cron cadence)
False-needs-input budget	<2%/week	DESIGN target — no baseline exists; establish + measure in the trial itself
Latency	Board render <1s; Zaruba poll latency unverified today (network+auth path not load-tested)	ASSUMPTION (board) / unverified (Zaruba)
Single-host	WSL2 localhost only — matches `fleetops`/`ghmonitor` scope and the billing gate (no cloud infra provisioned)	PRIOR-RESEARCH (00-SYNTHESIS.md) + VERIFIED-TODAY (R5, billing PENDING)
Zero-auth-localhost	No auth layer in MVP	PRIOR-RESEARCH, `01`
Confidentiality	`OTEL_LOG_USER_PROMPTS` stays OFF (default); no prompt/code/tool-content persisted, only structured metric/event/hook metadata; Zaruba execution/task data (Groupon business data) never leaves localhost — same reason Grafana Cloud is NOT-RECOMMENDED regardless of billing status	VERIFIED-TODAY (R1, R6)

Data requirements

Session/fleet state: ephemeral, re-derived each poll from fleet snapshot + claude agents --json — no independent store for F1.
Ledger-joined metrics: read-only queries against work-ledger’s Postgres views when it exists — this app owns no ledger data, never writes to it.
Zaruba poll cache: small local store (SQLite) of last-seen workflow_executions/tasks rows, purely for delta/alert detection — the REST list endpoints don’t push (VERIFIED-TODAY R5).
No long-term analytics store in MVP — explicitly removed from concept (PRIOR-RESEARCH, 01).
Claude session JSONL already has 30-day default retention (VERIFIED-TODAY R6) — don’t duplicate it, read it live instead.

Integrations

Integration	Role	Status
`claude agents --json`	F1/F2 session status + inline reply	Native, VERIFIED-TODAY R6
OTel export + hooks HTTP-forward	F3 metrics/events feed	Native, VERIFIED-TODAY R6
`fleetops` (`fleet snapshot`)	F1 session discovery	Reuse, VERIFIED-TODAY R4
`ghmonitor`	GH Actions/deploys	Reuse, post-MVP (PAT-blocked, R4/R5)
`workflow-management` REST (executions, definitions)	F5	New client, VERIFIED-TODAY R5
`tasks` REST	F5 join	New client, VERIFIED-TODAY R5
Work-ledger (read-only)	F4 true metrics	New client, when ledger ships (not yet, R7d)
GChat webhook	F6 digest	New, house standard (R7f)
`docs/llm-runs.md`	F7 wave-cost source	Reuse, existing evidence surface

Instrumentation of the monitor itself

The monitor logs its own poll health per source (last success timestamp, error count) — a monitor that silently stops polling is the exact “false status” failure Norman’s lens warns against, so surface it as a line in the board itself, not a side dashboard. If digest/report generation ever calls an LLM (e.g., to summarize a wave in prose), that run must be logged in docs/llm-runs.md per the global CLAUDE.md LLM Run Evidence rule — flagged now as a build constraint so it isn’t missed later.

Degradation path — no work-ledger (REQUIRED, answers `01` open question 6)

Yes, v1 ships and degrades gracefully. The ledger is decided-not-built, and even once built, v1 explicitly excludes Temporal projection — it needs a second, separately-scheduled integration (fleet hooks calling claim/transition/block) before it can feed F4 at all (PRIOR-RESEARCH, R7d + work-ledger 01). Fleet Monitor must not block on either. Per-metric fallback is specified in the F4 table above: stall detection ships day-1 with no dependency; queue depth and cost/merged-unit ship day-1 as clearly-labeled MODELED session-level proxies; rework rate has no safe proxy and shows — rather than a fabricated number until both ledger and hook integration exist. This is a deliberate reversal of 01’s implicit “MVP joins ledger rows” framing.

MVP scope

F1 board = fleet snapshot + claude agents --json rendered as one sorted list. Reuse, no new collector.
F2 inline unblock = native Agent View peek/reply + fleetops pane-jump. Reuse, no new build.
F3 telemetry = enable OTel env vars + OTEL_RESOURCE_ATTRIBUTES at dispatch; fork ColeMurray/claude-code-otel’s docker-compose skeleton (pin, don’t track — 13mo stale but small/correct) + import dashboard 25255 for aggregate cost/token panels. Config + frozen fork, not new code.
F4 metrics view = one local page: day-1 metrics (stall, proxy queue-depth, proxy cost/unit) now; ledger-true metrics wired in when the ledger ships. New build.
F5 Zaruba poller = workflow_executions + tasks REST poll, one VIEWER token. New build, small.
F6 daily digest = GChat webhook off F5 data. New build, small.
F7 wave report = formatted read of docs/llm-runs.md + F4 data. New build, small.

Explicitly excluded from MVP: GitHub Actions/ghmonitor integration (blocked on PAT); Temporal direct access; infra deploy status (nothing provisioned); Encore Cloud backend-deploy status (unverified surface); auth/multi-tenant; mobile; Tomas write-access; auto-remediation; triage dedup; cloud deploy; a fabricated rework-rate proxy; kanban board.

Post-MVP

GitHub Actions pane (ghmonitor reuse) once the PAT lands — near-zero effort.
Temporal direct once grpn2-infra’s cluster is provisioned.
Encore Cloud backend-deploy status once its API/webhook surface is investigated — dedicated follow-up spike, may be the deploy signal Tomas cares about most; do not let it stay permanently out of scope by default.
True rework rate once ledger + fleet-hook integration both exist.
Multi-host aggregation if the fleet ever spans >1 machine.
Self-hosted Grafana on monorepo-infra’s own stack once billing/cluster clear — not Grafana Cloud SaaS, ever, per the confidentiality NFR (Groupon data stays local regardless of billing status).

Acceptance criteria — 1-week live trial

Deployed on the next real wave (≥10 concurrent agents), running 7 consecutive days.
Board operational (F1) ≥95% of wall-clock time during the trial (self-reported via monitor-health instrumentation).
Zero missed needs-input events — every genuine stall/block was surfaced within the staleness SLO, checked retrospectively against session transcripts.
False-needs-input rate <2% observed; if higher, this is a trust-SLO failure (see kill criteria).
Operator-minutes/day spent locating blocked agents measured before/after — no pre-existing baseline exists, so the trial establishes baseline and result together (this is the “fastest concept test” 01 already specified).
Tomas receives a GChat digest (F6) every trial day with passport-tagged (MEASURED) execution/task numbers.
Cost/merged-unit and rework-rate correctly display as unavailable rather than a fabricated number for the full trial (ledger integration is not expected to land within the week) — verified by inspection.

Success metrics

North Star: median time-to-unblock (needs-input flagged → operator action taken), measurable day-1 from session/hook timestamps, no ledger dependency.

Alternative considered	Why not North Star
Cost/merged-unit	Lagging, aggregate; not measurable day-1 (needs ledger + hook integration per degradation path); doesn’t drive daily operator behavior — it’s a guardrail, not a steering metric
Merged-units/day	Throughput, easily gamed by small trivial units; the stated problem is silent stalls, not insufficient throughput — wrong metric for the product’s actual job
Operator-minutes/unit	Conflates real decision/review work with triage friction; time-to-unblock isolates specifically the attention-routing job this product does, without contaminating it with work the product isn’t trying to speed up

Guardrails: false-needs-input rate (if this rises to game the North Star, the product has failed its own trust principle); total fleet cost (should not silently balloon); queue-depth trend (should not indicate quietly-abandoned work).

Anti-metrics: dashboard panel count (should shrink, not grow — Ive’s ~80% removal projection for a typical Grafana LLM dashboard, PRIOR-RESEARCH 01); metrics displayed that never changed a decision, reviewed qualitatively at the week-1 retro (Norman’s alarm-fatigue risk).

Pivot criteria

If F5 (Zaruba pane, near-zero build) delivers more measured value in week 1 than F1–F4 (more moving parts, still building trust) — re-sequence: ship F5 as the de facto v1, keep relying on native claude agents + fleetops directly for fleet triage a while longer. If the false-needs-input rate is elevated but root-causes to one fixable signal (e.g., a stall threshold miscalibration) — tune it, don’t scrap the board.

Kill criteria

Trust SLO violated (one bad needs-input event not fixed within one iteration sends the operator back to tmux permanently, 01’s own named risk) — kill the custom board, fall back permanently to native claude agents + fleetops. Robert’s concurrent-session count drops below ~10 and stays there — the triage-at-scale problem this product exists for no longer exists; kill, don’t maintain. Anthropic ships native cross-session needs-input aggregation in Agent View that fully covers F1–F3 — kill the custom board, keep only F4/F5/F6/F7 (not Anthropic’s problem to solve).