06 — Fleet Monitor Architecture

2026-07-18. Grounded only in packets R1 (OSS landscape refresh), R4 (local rung-2 tooling scan), R5 (Zaruba watch surface), R6 (Claude Code OTel/hooks/Agent-View docs), R7 (prior-research constraints). Tests 01-initial-recommendation.md against this evidence; contradictions are called out inline, not smoothed over.

Headline correction to 01: the “(b) Build” half of 01’s simplest-useful-version is smaller than 01 assumed. R4 (VERIFIED 2026-07-18) found five purpose-built personal TUIs already on this machine — fleetops does the local session-liveness/status board 01 scoped as new build, ghmonitor does GitHub Actions/deployment watching Tomas asked for. R5 (VERIFIED 2026-07-18) found Tomas’s highest-value watch surfaces (workflow_executions, tasks) need no new infra at all — a REST poll against his existing API. The actual build is the merge layer: one bespoke queue view joining fleetops + hooks + (future) ledger + groupon2 poll data. Details below.

1. Principles

Adopt-first. Every layer checks “does this already exist on this machine or in OSS” before writing code. R1+R4 show most of the surface does.
Local-first. docker-compose, localhost binding, no cloud until the billing gate clears (01; R7a — cloud is blocked on billing regardless of preference).
Boring components. Prometheus, Loki, Grafana, SQLite, cron. Nothing novel.
Replaceable, pinned. Every adopted fork/dashboard/TUI sits behind a thin seam; pin exact versions, never track upstream (§7).
No premature services. No new Encore service, no new database beyond one SQLite file, until the work-ledger itself (a separately-decided, not-yet-built app — R7d) exists to justify a real join.
Telemetry never contains prompt/code content. Hard gate, non-negotiable, enumerated in §6 — applies to Claude Code OTel/hooks AND to anything pulled from groupon2 (task/workflow payloads may carry merchant data).
Monitor is a projection, never a second state machine. Applies to the ledger join and to the groupon2 WatchItem cache alike — the monitor never writes back to either source of truth.

2. System context

flowchart LR
  subgraph FLEET["Robert's Claude Code fleet, <=64 sessions"]
    CC["Claude Code sessions"]
  end

  CC -- "OTel metrics" --> OTELC["OTel Collector (adopt)"]
  CC -- "hook events (stripped)" --> ING["ingest service (build-thin)"]
  CC -. "session json + /proc" .-> FO["fleetops TUI (reuse, R4)"]

  OTELC --> PROM[("Prometheus")]
  OTELC --> LOKI[("Loki, reinstated")]
  PROM --> GRAF["Grafana (adopt: 25255 + panels)"]
  LOKI --> GRAF

  FO -. "fleet snapshot poll" .-> ING
  ING --> DB[("SQLite: HookEvent, Question, WatchItem")]

  LEDGER[("work-ledger Postgres — future, built elsewhere")] -. "read-only views" .-> GRAF
  LEDGER -. "read-only views" .-> BOARD

  DB --> BOARD["Board UI: attention-router queue (build-thin)"]
  GRAF -. "linked, not embedded" .-> BOARD

  WP["groupon2 watch poller (build-thin)"] -- "REST poll: executions, tasks (g-api-key)" --> WMDB[("Zaruba workflow-management API")]
  WP -- "REST via GH PAT, when provisioned" --> GHM["ghmonitor (reuse, R4)"]
  GHM --> DB
  WP --> DB

  BOARD --> DIGEST["daily digest script (cron)"]
  DIGEST -- "webhook" --> GCHAT[("Google Chat")]

Note: no arrow to Temporal directly, and no arrow to Encore Cloud. Both are deliberate — Temporal is already reconciled into workflow_executions every 15 min by Tomas’s own cron (R5, surface 1d: “you may not need direct Temporal access at all”), and Encore Cloud’s own deploy signal is unverified, not wired — see §10 risk 3.

3. Component table

Component	Responsibility	In	Out	Failure behavior	Build vs buy
OTel Collector	Receive CC telemetry, fan out	OTLP from CC sessions	Prometheus scrape, Loki push	Export is fire-and-forget from CC’s side (R6); collector outage loses metrics window, never blocks agents	Adopt — config forked from ColeMurray/claude-code-otel skeleton (R1)
Prometheus	Store/scrape metrics	Collector scrape	Grafana datasource, queue-depth fallback query	Gap in history only	Adopt
Loki	Store log-shaped events (tool_result/api_error text, content-free)	Collector push	Grafana datasource, log search	Gap in log history only	Adopt — reinstated, see ADR-9
Grafana	Aggregate health/cost dashboards, underneath the board	Prom + Loki + ledger PG (read-only)	Human view, linked from board	Dashboards unavailable; board UI unaffected (decoupled by design)	Adopt/import — dashboard 25255 (R1)
hooks → ingest service	Receive stripped hook JSON, write HookEvent/Question	Local HTTP POST from CC hooks	SQLite rows	Hook POST failure is non-blocking per CC hook semantics (R6) — agent keeps running; ingest downtime = stale board only	Build-thin
fleetops (reused)	Local session liveness/status (working/idle/needs-answer/stalled?/shell)	`~/.claude/sessions/*.json` + `/proc`	`fleet snapshot` JSON	Local-only; if unavailable, ingest falls back to hook-derived state alone (degraded, not blocked)	Reuse, zero build (R4)
SQLite store	HookEvent, Question, WatchItem cache	ingest + poller writes	Board UI reads	Single-file; mitigated by cron backup (§7)	Build-thin (schema only)
Board UI	The one prioritized queue — 01’s primary surface	SQLite + ledger PG views (if present) + fleetops snapshot	Robert’s daily-driver page	Ledger unreachable → cost/rework show “unavailable,” never fabricated; SQLite unreachable → stale-timestamp banner	Build-thin, bespoke — not a fork (ADR-3)
groupon2 watch poller	Poll workflow-management REST	`/executions`, `/definitions`, `/tasks` (`g-api-key`, VIEWER role)	WatchItem rows	Poll failure → stale WatchItem timestamp shown, never silently dropped	Build-thin — a poll loop, not a service (R5)
ghmonitor (reused)	GitHub Actions run/deployment status	GH Actions API via PAT	Status shown directly, optionally polled into WatchItem	PAT missing = feature absent (current state, not an error)	Reuse, zero build (R4) — blocked on PAT (R5)
digest script	One daily GChat post	Board’s SQLite/queries	GChat webhook	Missed digest = no post that day, retried next cycle	Script, not a service
work-ledger (external)	Unit × state × cost × disposition SSOT	Internal-plane workflow stream (own build)	Read-only Postgres views	Absent = degradation mode (§4)	Not this app’s build — separate plan (R7d)

4. Data model

Entities

Entity	Owner / source of truth	Key fields
Unit	work-ledger (when it exists) — monitor never owns this	unit_id, state, disposition, claimed_by, cost, evidence_url
Session	fleetops / CC session files (local, per-host)	session_id, host, cwd/project, status, pid, last_seen
HookEvent	monitor’s own SQLite (ephemeral bookkeeping, not business data)	session_id, event_name (SessionStart/Stop/PreToolUse/PostToolUse/…), ts, tool_name, decision — never tool params/content
Question	ledger’s `block()` call when present; else a HookEvent-derived flag	question_id, unit_id or session_id, short summary (not full transcript), link
MetricSample	computed by ingest/board at read time	metric name, value, window, passport: MEASURED / MODELED / DARK / DESIGN (R7f)
WatchItem	Zaruba’s workflow-management DB — monitor’s copy is a poll cache only	source (execution/task/gh-run), external_id, status, assignee, last_polled, link

The rule, stated once, applied twice: the monitor is a projection of external systems of record — work-ledger for fleet-unit state, Zaruba’s workflow-management API for groupon2 state. It writes to neither. If the two ever disagree with what the monitor shows, the monitor is wrong by construction, not the source.

Retention: Prometheus/Loki keep short windows (days–weeks) — 01 explicitly removes “historical analytics suite” from concept, so long retention isn’t a requirement; tune later if the digest needs longer look-back (DESIGN, not packet-sourced). HookEvent in SQLite: mirror Claude Code’s own 30-day session retention default (cleanupPeriodDays, R6) — DESIGN choice, no reason to keep longer than the source. WatchItem: rolling cache, last-N-days; Zaruba’s DB is the durable record, not this cache. Unit/disposition history: entirely the ledger’s retention concern, out of scope here.

No-ledger degradation mode (answers 01’s open question 6, closes R7d’s flagged gap): decisively, yes, v1 degrades gracefully. Queue depth and stall detection run off session/OTel data alone — no ledger required (R7c/d). Cost/merged-unit and rework rate are structurally impossible without the ledger’s attempt/transition history and the board shows them as “unavailable — ledger required,” never an approximated number. This is Norman’s “honest confidence signal, not a binary lie” (01) applied literally to metric availability, not just to stall confidence.

5. Metric definitions (code-ready)

# 1. Rework rate — REQUIRES ledger (transition history)
rework_rate(window) =
  count(u in units where u.reached_merged_before(now)
        and exists(transition t: t.unit_id = u.id
                   and t.ts > u.first_merged_at
                   and t.state in ('in_progress','blocked')))
  / count(u in units where u.merged_at in window)

# 2. Cost per merged unit — REQUIRES ledger join to OTel cost
cost_per_merged_unit(window) =
  sum(otel.claude_code.cost.usage
      where session.unit_id in {u.id for u in units if u.disposition == 'MERGED'
                                                       and u.merged_at in window})
  / count(units where disposition == 'MERGED' and merged_at in window)
  # passport: MEASURED if OTel cost export present for every attempt session;
  # MODELED (ccusage-style token*price fallback, R1) for sessions with no OTel export

# 3. Queue depth — session/OTel-only fallback exists
queue_depth(state) =
  # with ledger:
  count(units where units.state == state)
  # without ledger (MODELED, weaker — session count is a proxy for unit count):
  count(fleetops_sessions where status == state)

# 4. Stall detection — session/OTel-only, always available
def is_stalled(session, now, STALL_THRESHOLD_S=600):  # DESIGN default, tune later
    wall_elapsed = now - session.last_active_time_sample_ts
    active_delta = otel.claude_code.active_time.total(now) \
                  - otel.claude_code.active_time.total(now - STALL_THRESHOLD_S)
    open_tool_span = exists(HookEvent: PreToolUse fired, no matching PostToolUse yet)

    if open_tool_span:
        return NOT_STALLED   # long single tool call (build/bash) is expected, not a stall
    candidate = wall_elapsed > STALL_THRESHOLD_S and active_delta ~= 0
    if not candidate:
        return NOT_STALLED

    corroborated = fleetops.status(session.id) in ('stalled?', 'waiting')
    return HIGH_CONFIDENCE_STALL if corroborated else LOW_CONFIDENCE_STALL
    # LOW_CONFIDENCE renders amber, not red — never a single-signal red alert (01 biggest usability risk)

False-positive handling is structural, not a tunable: an open tool span always wins over the plateau heuristic, and a single signal never reaches “needs human” (red) — it takes fleetops corroboration to escalate. This directly targets 01’s named #1 usability risk (“a needs-input list that is stale or wrong ONCE sends the operator permanently back to tmux”).

6. Security / confidentiality

localhost binding. docker-compose stack (ingest, Grafana, board UI) binds 127.0.0.1 only — matches 01’s “Not in MVP: auth/multi-tenant.”
Never collected, by hard gate: prompt text, assistant response text, tool params/command content, full tool I/O, raw API bodies. Concretely: OTEL_LOG_USER_PROMPTS, OTEL_LOG_ASSISTANT_RESPONSES, OTEL_LOG_TOOL_DETAILS, OTEL_LOG_TOOL_CONTENT, OTEL_LOG_RAW_API_BODIES all left off/unset (R6 — off is the documented default for the privacy-sensitive ones; this pins it, doesn’t rely on default drift). Metrics attributes (tool_name, decision, model, cost/token counts) are coarse metadata, not content, and stay on — they’re what the four metrics are built from.
Hook payload stripping. The ingest service forwards only session_id, hook_event_name, tool_name, decision, timestamps from the hook stdin JSON (R6 §2); transcript_path, cwd beyond a coarse project label, and any tool-specific fields are dropped at the ingest boundary, not filtered downstream.
groupon2 data handling. WatchItem stores status/metadata only (state enum, ids, timestamps, assignee) — never workflow input/output payloads or task body text, which can carry merchant/customer data. The poller requests list/filter endpoints (status fields), not full record bodies.
Secrets. GitHub PAT and the g-api-key token (VIEWER role, minted via tokenCreate, R5) are referenced only — stored per ~/s/groupon/ conventions (CLAUDE.md), injected as env vars at container/process start, never committed to compose files or this repo.

7. Ops

docker-compose layout: otel-collector, prometheus, loki, grafana, ingest (hook receiver + board UI, one small service), one named volume for the SQLite file. groupon2-watch-poller and the digest script run as host cron, not containers — they’re thin polling loops with no state of their own beyond what they write to the shared SQLite file. fleetops and ghmonitor run outside compose entirely — they’re already-working personal binaries invoked as subprocesses/cron by the poller; dockerizing someone else’s working TUI is unrequested scaffolding.

Single-host reality. The whole stack lives on one machine (today: Robert’s WSL2 box). That matches 01’s platform verdict (“not desktop, not mobile, not SaaS”) and R5/R7’s local-first framing. No HA, no failover — not a gap, a scope decision (ADR-10).

Backup. Nightly sqlite3 <db> .backup <dated-file> via cron. No replication tooling (litestream etc.) — data volume is small (event/cache rows, not the ledger’s business data) and this isn’t the system of record for anything; skip until a real reason appears.

Upgrade posture for pinned forks. ColeMurray/claude-code-otel’s docker-compose skeleton and dashboard-25255’s JSON are vendored at import time — exact commit SHA / JSON snapshot checked in, never git pull upstream (R1: “pin, don’t track” — ColeMurray is 13 months stale, dashboard 25255 has no visible last-updated date). Re-test against R6’s documented OTel schema when Anthropic changes metric/event names, or when panels visibly break — not on a calendar.

8. ADR table

#	Decision	Alternatives	Recommendation	Reversibility
1	Storage for ledger-derived metrics	Replicate ledger rows into monitor’s own store vs read live via read-only Postgres view	Read live view — no replication, no second state machine (§4 principle)	High — swapping to a cache later is additive
2	Board UI platform	Web vs TUI	Web (matches 01’s platform verdict); fleetops (TUI) stays the per-host session-liveness source, not the primary UI	Medium — UI framework choice, data model unaffected
3	Fork-vs-build board	Fork hoangsonww/Claude-Code-Agent-Monitor or tombelieber/claude-view (R1) vs build bespoke merge view	Build-thin, bespoke — both candidates solve a narrower single-machine session-dashboard problem; this product’s join (fleetops + hooks + ledger + groupon2) is what no OSS tool anticipates	High — small enough to rewrite if a better donor appears
4	Session signal: poll vs stream	Hooks (push) vs OTel metrics (60s export, R6) vs polling `claude agents --json`/fleetops (pull)	Both, complementary, not either/or — hooks give session-lifecycle/question state OTel doesn’t carry; OTel gives cost/token truth hooks don’t carry	High — additive, no lock-in
5	Grafana embedded vs linked	iframe embed in board UI vs separate tab/link	Linked — 01: Grafana is “underneath,” not primary; an iframe is speculative polish (ladder rung 1)	High — trivial to add an iframe later if ever justified
6	Ingest DB engine	SQLite vs Postgres	SQLite for HookEvent/WatchItem cache; Postgres stays the ledger’s own store, not duplicated here	Medium — schema is small, migration is cheap if volume ever justifies Postgres
7	Hooks vs OTel-only	Drop hooks, rely on OTel alone	Keep both — OTel’s metric set (R6) has no “needs-input”/session-lifecycle signal; hooks are the only source for that	High
8	Multi-host now vs later	Provision for Tomas’s own board instance now vs single-host today	Later — single WSL2 host covers both operators today (01); revisit only if Robert’s fleet moves off one machine or Tomas wants his own instance	High — docker-compose → remote collector is additive
9	Log backend	Keep 00-SYNTHESIS’s silent drop of Loki vs reinstate	Reinstate — tool-decision/error-event log search needs a log store; Prometheus alone can’t hold event text (content-free) usefully. Diverges from `00-SYNTHESIS.md`’s architecture diagram, which dropped Loki without a stated reason (R7e#2)	High — one more compose service, additive
10	groupon2 watch: build vs reuse	Build a new watcher service vs reuse existing surfaces	Reuse — poll workflow-management REST directly (R5, zero new infra) + ghmonitor as-is for GH Actions (R4, zero build) once PAT lands	High — both are thin poll loops, swappable

9. MVP vs post-MVP

MVP (v1, ship first — every item below has zero infra blockers as of 2026-07-18):

fleetops reused as-is for local session liveness — nothing to build (R4).
hooks → ingest → SQLite for HookEvent/Question (needs-input signal).
OTel Collector → Prometheus + Loki → Grafana, ColeMurray skeleton forked + dashboard 25255 imported, localhost only.
groupon2 watch poller: workflow_executions + tasks REST poll — one token, two endpoints, zero new infra (R5 v1 recommendation, verbatim).
Board UI: queue depth + stall detection live day one (session/OTel-only); cost/merged-unit + rework rate show “unavailable — ledger required” until work-ledger ships.
One daily GChat digest script.
ghmonitor pointed at groupon2 repos as soon as the PAT lands — not MVP-blocking, but zero-build once unblocked (R4/R5).

Post-MVP, explicit triggers:

Cost/merged-unit + rework rate → unblocked when work-ledger ships (separate plan, R7d — not reported executed as of this pass).
GitHub Actions ingestion via ghmonitor → unblocked when Tomas’s already-approved PAT is actually provisioned (R5, currently blocked).
Encore Cloud backend-deploy signal → needs a dedicated follow-up spike first (R5 flags this unverified, and it’s likely the deploy signal Tomas cares about most — see risk 3).
Temporal-direct / infra-deploy watch (R5 surfaces 1e, 3b, 5) → nothing is running (0% infra provisioned); revisit when grpn2-infra billing clears.
Multi-host aggregation → only if Tomas runs his own board instance (ADR-10).
Cloud-hosted observability (Grafana Cloud, Datadog) → blocked by the billing gate regardless of preference (R7a, 01).
Triage-app integration → separate factory app (01, R7); fleet-monitor does not absorb it.
Auth/multi-tenant, mobile, Tomas write-access, auto-remediation → explicitly out per 01, unchanged by this pass.

Components to avoid entirely: any kanban board (01’s explicit removal wins over research/12’s Vibe Kanban donor mention — R7e#1, resolved in favor of the later, doc-under-test); disler/claude-code-hooks-multi-agent-observability (no license, legal risk, R1); Grafana dashboard 25052 (Azure-only ingestion, GCP org mismatch, R1); Grafana Cloud’s packaged Claude Code integration (cloud-only, R1); a direct Temporal client/cluster poller (nothing running to poll, R5); a from-scratch GitHub Actions watcher (ghmonitor already is one, R4); replicating ledger rows into a second Postgres instance (violates §4’s projection-only principle).

10. Technical risks, ranked

Ledger dependency is load-bearing and unbuilt. Two of four canonical metrics are inert until work-ledger ships, and R7d confirms this exact question was unanswered in every prior doc. Mitigation: hard “unavailable” state (§4), never an approximated stand-in.
Stall-detection false positives break trust in one shot. 01 names this as the biggest usability risk; a long single-tool-call session or a legitimate WAITING_FOR_APPROVAL hold could plateau active_time without being stalled. Mitigation: open-tool-span exclusion + two-signal corroboration before red (§5) — but this is a heuristic, not a proof, and needs a real trial before trusted unattended.
Encore Cloud backend-deploy blind spot. R5 flags this unverified and calls it “likely the deploy Tomas cares most about” — the GH-Actions-centric watch design is structurally blind to it (Encore Cloud deploys outside .github/workflows/* entirely). Telling Tomas “deployments are watched” without this spike overpromises.
Fork/dashboard staleness with no maintainer signal. ColeMurray/claude-code-otel is 13 months stale; dashboard 25255 has no visible last-updated date (R1). Pin-don’t-track means schema drift from Anthropic’s side won’t be caught automatically. Mitigation: smoke-test on import, re-check when panels visibly break.
groupon2 API surface is young and actively developed. No contract test is named for workflow-management’s REST shape (R5); Zaruba’s endpoints could change without notice. Mitigation: WatchItem ingestion must tolerate missing/renamed fields defensively — this is exactly the drift a projection layer should expect.
Soft coupling to five personal tools outside this plan’s control. fleetops/ghmonitor/bridge/ground-control/tokenomics are actively maintained (R4) but are side projects with their own release cadence; a JSON-shape change in any one degrades the merge layer silently unless treated as a versioned contract. Mitigation: smoke-test fleet snapshot-style outputs on each reuse point, not just at build time.
Credential handling gates R5’s “zero blockers” claim. The PAT and g-api-key token are live credentials crossing from Zaruba’s system into Robert’s tooling; process risk (wrong storage location, accidental commit) is the actual blocker, not technical difficulty. Mitigation: ~/s/groupon/ convention only, enforced before first poll runs, not after.

Open items this doc does not resolve (flag, don’t guess): Encore Cloud’s deploy API/webhook surface (risk 3, needs its own spike); the exact GChat digest content/format (out of scope for architecture, product-copy decision); whether claude agents --json (R6, local-only, v2.1.139+) should also feed the board alongside fleetops, or whether that’s redundant with fleetops’s own read of the same session files — not decided here, low cost either way, defer to implementation.