Monitoring, Rate Limits, and Comparative Metrics
Centralized Monitoring & Budget Dashboard
Sits on an enterprise AI gateway (LiteLLM proxy); captures every LLM call, tool invocation, human-in-the-loop interaction.
- Tracks TPM/RPM across models; projects remaining API headroom to prevent rate limits stalling the pipeline.
- Virtual billing keys attribute spend to microservice / refactoring task / business unit; budgets monitored per unit.
- Budget exhaustion policy: microservice budget spent → restrict agent to lower-cost local models or pause its queue until next billing cycle.
- Redis circuit breaker: opens after 5 consecutive failures, fast-fails Redis calls (0ms), falls back to bounded PostgreSQL queries — prevents gateway thread exhaustion during cache degradation. 100% API availability during cache outage.
- Monitors Temporal workflow histories; alerts when an agent loop approaches the ~500-iteration history limit so engineers intervene before termination.
- HTTP 429 → exponential backoff retry policy.
Limit table
| Parameter | Default threshold | Enforcement level | Rationale |
|---|---|---|---|
| Temporal workflow history | 500–600 loop iterations | Orchestration | History limits degrade performance |
| Model rate limits | 100,000 TPM / 100 RPM | Agent (LiteLLM) | Cap per-agent throughput, protect API headroom |
| Session rate limits | 50,000 TPM / 50 RPM | Session (LiteLLM) | Stop runaway session monopolizing shared quota |
| Iteration limits | Max 25 iterations | Workflow (LiteLLM) | Prevent infinite retry loops |
| Redis circuit breaker | 5 consecutive failures | Gateway (LiteLLM) | Fast-fail to PostgreSQL during cache outage |
| Temporal payload limit | 2 MB per call | System | Prevent gRPC serialization failures; larger contexts to object storage |
Gateway flow: Agent call → LiteLLM proxy → Redis circuit breaker (closed: Redis cache for key/budget/rate counters; open: bounded PostgreSQL fallback) → token-aware token bucket (RPM+TPM check) → dispatch to provider, or queue/429.
Comparative metrics: unguided vs guided vs human
| Metric | Raw autonomous (unguided) | Guided (this blueprint) | Human benchmark |
|---|---|---|---|
| Primary focus | Minor renames, shallow adjustments, bulk annotations | High-impact structural changes, method extraction, dependency uncoupling | Balanced structural + logical design |
| Structural impact | >91% of unguided commits = annotation modifications | 2.8×–5× greater codebase-health improvement | Strategic design-level transformation |
| Code smells introduced | Assertion roulette 15%, long statements 8%, unnecessary abstractions 8% | Localized smells caught in iterative validation loops | Minimal localized; occasional design coupling |
| Token efficiency | High churn, trial-and-error edits | ~50% lower token spend (targeted execution paths) | N/A |
| Defect density | +46% corrective maintenance, +45% bug-fixing post-merge | Substantial reduction | Baseline |
Supporting evidence
- Unguided agents are conservative — default to safe shallow changes because legacy code too complex to modify safely unguided. When they attempt deep refactoring: Cursor Agent mean smells 87.1 → 107.0 per commit; unguided Claude Code 179.1 → 214.6 per commit (highest absolute increase).
- Unhealthy code costs tokens: agents burn up to 50% more tokens on same tasks in degraded codebases; verbose languages (Java) ~120% more output tokens in iterative refactoring on degraded code. Code health metrics are an economic lever, not just quality.
- Agent PR merge-rate pattern: high merge rates for well-defined standardized migrations (docs, build configs, CI upgrades); frequent rejection for complex bug fixes / perf optimizations (compiler failures, test regressions, too-many-files changes).