Models, Tooling, and Cost Model
Model strategy
Hybrid routing: frontier model only for ambiguous planning + hard repair loops; cheaper models for classification, summarization, Q&A drafting, boilerplate transformations. Tool use matters more than raw chat capability — the useful loop is “model plans → tools inspect repo/build/test → model revises.”
Cloud models = fastest start (minimal serving ops, tool calling, increasing data-residency/private-networking support). OpenAI: project-level data controls, regional processing/storage, Private Link. Vertex AI: data residency + regional processing. Strong isolation constraints → keep code in-region, route minimal task context to model, or self-host for most sensitive repos.
Self-hosted attractive at stable high volume / strict data locality / repeated-workload economies: vLLM (OpenAI-compatible API, low lock-in), KServe (K8s-native autoscaling, canary rollouts), Triton (dynamic batching, queue policies). Trade-off: less marginal inference cost, more platform engineering + eval responsibility.
Fine-tuning: not the first lever. First build: task decomposition, stable tool schemas, retrieval over repo context, approved exemplars, eval harness. Fine-tune only after a large corpus of approved migrations/review comments/stakeholder answers exists. Prompt engineering + retrieval + tool use + caching move the needle more safely early.
Platform comparison
| Option | Strengths | Considerations |
|---|---|---|
| OpenAI API | Tool calling, prompt caching, Batch API (−50%, separate higher-rate pool), data controls, regional options, ecosystem | Cost rises fast with oversized prompts / top-tier models on routine work |
| Anthropic Claude API | Tool use, prompt caching, message batches first-class; Sonnet/Haiku tiers documented | Validate operational features in vendor docs during procurement |
| Gemini API / Vertex AI | Function calling, context caching, Batch API (−50%), data residency on Vertex, aggressive flash-tier pricing | Some features preview/beta; verify model/service maturity for production |
| Self-hosted open models | vLLM OpenAI-compat, KServe canaries, Triton batching | Platform burden, harder eval, GPU ops |
Internal apps/services to build
| Service | Purpose | Core integrations |
|---|---|---|
| Agent Manager | Task intake, scheduling, quotas, locks, retries, escalation | Workflow engine, inventory DB, LLM gateway, policy engine |
| Triage UI | Stakeholder questions, one-click answers, SLA tracking | CODEOWNERS/PRs, Slack, Jira |
| Sandbox Runner | Reproducible isolated edit/build/test jobs | K8s Jobs, object storage/cache, secrets/identity |
| PR Reviewer Bot | Checks, annotations, review comments, requested actions | GitHub App auth, Checks API, review comments, webhooks |
| Monitor Dashboard | Fleet ops, cost, lock contention, PR survivability, SLA | OTel Collector, backends, alerting |
| Inventory Graph Service | Canonical component metadata + blast-radius graph | Git metadata, manifests, lockfiles, dependency submissions, traces |
Required integrations
VCS/PR APIs · CI checks/status · issue-tracker comments+webhooks · chat messaging · object storage/artifacts · secrets manager · workload identity · telemetry export. On GitHub critical set: App auth, Checks, review comments, protected branches, merge queue, webhooks.
Monorepo tooling (TS + Go)
| Approach | Recommended use | Why |
|---|---|---|
| pnpm + TS project references + go.work + Nx | Best starting point for incremental migration | Native TS/Go ergonomics, good CI “affected” execution, low migration overhead |
| Bazel + rules_js/rules_ts + rules_go | When cross-language graph fidelity + remote caching become strategic | Reproducibility, remote caching, multi-language scale |
| Turborepo + native Go tooling | JS/TS-heavy repo, Go stays a smaller island | Excellent JS/TS scheduling/caching, lighter than Bazel |
Avoid big-bang monorepo platform rewrite during a language migration. Adopt Bazel only if cross-language build graph fidelity, cache reuse, or reproducibility become major blockers. For an agent fleet what matters: fast reliable affected builds + stable hermetic sandboxes, not build-system purity.
Cost model (base case)
Assumptions: 1,000 logical agents · 200 active max · 4,000 completed tasks/day · 20 min avg sandbox time · 0.6M input + 0.06M output tokens/task · ~40% cacheable prompt prefix → ~72B input + 7.2B output tokens/month.
- Runner fleet: ~40,000 sandbox hours/month → low-five-figure or lower monthly compute (CPU-bound), storage second-order.
- OpenAI list pricing: ~$89k/mo (cached-prefix-heavy cheap lane) · ~$223k/mo (premium lane) · ~$130k/mo (mixed routing).
- Gemini flash-tier: materially lower for same volume (verify production maturity).
- Batch APIs (−50%): offline inventory, repo summarization, nightly evals.
Critical optimizations: sparse checkout, affected-project context retrieval, prompt caching, batch offline work, strict model routing.