Observability, Triage, and Safety Controls

Telemetry spine

OpenTelemetry — standardizes traces/metrics/logs; Collector receives/processes/exports to multiple backends. Every task emits a trace with spans: inventory, plan, edit, build, test, repair, create_check, open_pr, await_stakeholder, merge_queue. Gives latency attribution, stuck-state detection, cost-to-outcome analysis.

Primary metrics (operational, not vanity)

queue depth, queue age, active vs waiting agents
path-lock contention
LLM: request rate, 429/5xx rate, token volume, prompt-cache hit rate, cost
sandbox startup latency
build/test pass rate, repair-loop count
PR creation rate, PR merge rate, post-merge rollback rate
stakeholder question volume, answer latency, SLA breach rate

Prometheus recording rules precompute expensive aggregates; Grafana Alerting spans metrics + logs.

Stack choice

Stack	Strength	Trade-off
OTel + Prometheus + Grafana	Best default for platform engineering; standards-based, low lock-in	More assembly + ops effort
Datadog	Fastest unified APM/logs/traces/correlation + trace-volume controls	Vendor cost + lock-in
Honeycomb	Trace-first debugging, high-cardinality queries	Usually paired with extra metrics tooling

Human-in-the-loop triage

Design as structured interrupt, not free-form chat.

Question taxonomy: semantic ambiguity · policy/approval · missing test oracle · owner confirmation · dependency/version choice · security/privacy constraint · runtime-operational dependency.

Routing: CODEOWNERS first → historical reviewers/component owners → Jira issue + Slack thread when ownership unclear.

Stakeholder UI shows only the minimum decision payload: problem statement, impacted files/components, proposed default, blast radius, deadline/SLA, one-click actions (Approve / Option A/B / Escalate / Need meeting / Block permanently). Target: answerable in under a minute. Full reasoning transcripts kill response rates.

SLAs: same business day for merge-blocking questions on protected branches; next business day normal; auto-escalation on breach to owning team channel + issue tracker. Questions are resumable workflow events (Temporal signals + durable state), never lost chat messages.

End-to-end task workflow

flowchart TD
    A[Task selected] --> B[Analysis agent builds impact graph]
    B --> C{Question needed?}
    C -- Yes --> D[Triage agent classifies + routes] --> E[Stakeholder answers UI/Slack/Jira] --> F[Workflow resumes] --> G
    C -- No --> G[Code agent edits in sandbox]
    G --> H[Test agent: affected build/test/lint]
    H --> I{Passes?}
    I -- No --> J[Repair loop within retry budget] --> H
    I -- Yes --> K[CI/check agent posts annotations]
    K --> L[PR agent opens draft PR]
    L --> M[Reviewer bot + CODEOWNER review]
    M --> N{Required checks + approvals?}
    N -- Yes --> O[Merge queue]
    O --> P[Post-merge canary + metrics watch]
    P --> Q{Healthy?}
    Q -- Yes --> R[Close task]
    Q -- No --> S[Rollback / revert / pause wave]

Safety layers

Minimum: affected-project build/test · lint/type-check · dependency review (fails PRs with vulnerable packages) · CodeQL static analysis · required checks · code-owner review · protected branches · merge queue (validates against latest busy branch head) · push protection (secrets).

Rollback — three standardized levels

PR rollback — close/revert before merge.
Post-merge revert — automated revert PR + merge-queue revalidation.
Wave rollback — disable migration policy/agent rule, pause affected task queues.

Runtime behavior changes → combine with service canaries / feature flags. Pure structural refactors → canary is the merge queue + post-merge error budget.