Skip to content
GRPNR.

Web Research — Agentic Mass Refactoring, Verified Evidence (2026-07-18)

Deep-research harness: 5 search angles, 21 sources fetched, 103 claims extracted, top 25 adversarially verified (3 refuters each, 2/3 kills) → 21 confirmed, 4 refuted. Full raw output: raw-deep-research-output.json.

Confirmed findings

1. Production-proven case studies (~1M-LOC scale)

Case Numbers Source
Google (FSE 2025, arXiv 2504.09691) 39 migrations, 12 months, 3 developers: 595 changes / 93,574 edits; 74% of changes LLM-generated; est. 50% time saving. Caveat: all int32→int64 ID migrations; estimate self-reported. high confidence, 3-0
Airbnb Enzyme→RTL (Mar 2025) 3,500 test files in 6 weeks vs 1.5-year manual estimate. First bulk run: 75% of files in 4 hours. 4 days of pipeline refinement → 97%. Last 3% manual (~1 week), seeded by LLM output. high, 3-0
Anthropic Bun port Zig→Rust (Jul 2026) ~1M lines in <2 weeks, ~64 parallel agents; gate = 100% of existing test suite in CI before merge; 19 post-merge regressions (all fixed). Cost: 5.9B uncached input + 690M output tokens ≈ $165K API list price. Largest verified single agentic migration. high, 3-0
Amazon Q Java 8→17 10 min avg transformation/app (longest <1h) — but transformation-time only, not end-to-end. The famous “1,000 apps in 2 days by 5 people” claim was REFUTED 0-3 — do not cite. medium

2. The winning pipeline architecture (Airbnb, most reusable finding)

Per-file/unit state machine (refactor → fix tests → fix lint/tsc → complete) with:

  • machine-checkable validation gates between states,
  • brute-force retry loops feeding validation errors + latest file version back into dynamic prompts — most files pass ≤10 attempts, long tail retried 50–100×; retries were the primary success driver, not prompt perfection,
  • large curated context: 40k–100k-token prompts pulling up to 50 related files; choosing the right files mattered more than prompt wording.

Maps 1:1 onto Temporal: state machine = workflow, retry budget = activity retry policy, gates = activity boundaries.

3. Anthropic’s verification architecture (2-1, medium)

  • Specialized review subagents per failure-mode category (8 in the Bun case).
  • Adversarial reviewer pairs in separate contexts; third agent as tie-breaker.
  • Single build daemon — only process allowed to rebuild; compiler/test failures automatically become next work-queue items (“the queue writes itself”).
  • Qualifier: composited from two internal projects (Bun + a 165K-LOC Python→TS port); the blog itself says “don’t follow this guide blindly.”

4. Agent failure modes (measured)

  • ~6.08% test-level regression base rate for a vanilla agent (562 pass-to-pass failures on 100 SWE-bench instances); graph-based impact analysis (code→test dependency mapping, run high-impact tests pre-submit) cut it to 1.82% — a ~70% reduction (arXiv 2603.17973 / TDAD; small open-weight model, single-author preprint — mechanism is model-agnostic, numbers may not transfer).
  • Agents do not improve code quality by default: smell reduction ≈ 0 (Cohen’s d −0.027); a Jan 2026 follow-up shows smell increases for Cursor and Claude Code output.
  • Agents skew mechanical: 35.8% low-level edits vs 24.4% human; 53.9% of agent refactorings are tangled into commits with unrelated intent → review-burden inflation. (AIDev corpus, Java-heavy, 89% Codex commits — medium confidence.)
  • Consequence: delegate routine transformation to agents, keep design-level restructuring human-planned, gate quality explicitly in CI, add tangling detection/commit-splitting.

5. Merge queue (medium)

  • GitHub’s move from batch deploy trains to a merge queue cut average wait-to-ship 33% (self-reported). Queue = speculative grouping + required checks on groups + auto-eviction of failures; main never advances past a failed required check.
  • The guarantee is only as strong as required-check coverage and test determinism — flaky false-passes still land breakage; flaky false-failures stall the queue at volume. Flake elimination is a prerequisite for scaling merge volume.
  • GitHub’s own throughput numbers (“500 engineers / 2,500 PRs a month”) REFUTED 0-3.
  • Community evidence (Aviator, unverified-tier): scoping merge-queue CI to affected targets only is the key throughput lever in monorepos (hours → minutes).

6. Human-in-the-loop triage

  • The proven primitive pair: require_approval (block specific tool calls until human responds) + human_as_tool (open-ended agent question to a human) — from HumanLayer, which is itself vendor-deprecated (pivot to CodeLayer). Both “adopt HumanLayer” claims refuted 0-3.
  • Verdict: build thin — two primitives + ownership-based routing + chat delivery. No maintained off-the-shelf alternative survived verification. No measured SLA/escalation-queue data exists anywhere — genuinely open design space.

7. Fleet observability

  • Claude Code natively exposes 8 Prometheus-compatible OTel metrics (sessions, token usage, cost USD per model, active time, LOC, commits, PRs, edit-tool decisions) via CLAUDE_CODE_ENABLE_TELEMETRY=1; structured OTel logs per event. Community Grafana dashboard ID 25255 visualizes them already (siblings 24993, 25052). High confidence, verified against official docs current July 2026.
  • Not covered — needs thin custom layer: task success rate, review-rejection rate, rework rate, per-migration-unit progress.
  • OTel GenAI semantic conventions (gen_ai.*) = vendor-neutral instrumentation standard for anything custom.

Hard limits of the evidence

  • Nothing verified documents >~64 concurrent agents. The 1000-agent target is extrapolation; staged ramp with regression-rate + rework-rate telemetry as the scaling gate.
  • Axis 2 (Temporal fleet orchestration, sandboxing at scale, model routing) produced essentially no surviving published case studies — the Temporal recommendation rests on the stack already being self-hosted + the state-machine mapping. Temporal vendor claims (unverified tier): one worker process can manage hundreds-to-thousands of workflows; sandbox providers (Docker/Modal/Daytona/E2B) attach as activities.
  • Worktree community lore (unverified tier): teams cap at 8–10 concurrent worktrees per host before management overhead wins — plain worktrees don’t scale to fleets without orchestration.
  • Nearly every headline metric is first-party self-report — verified as accurately quoted, not audited. Vendor fleet-scale numbers specifically failed verification twice (Amazon, GitHub) while per-unit numbers held.
  • Cost: $165K/1M LOC is notional API list price excl. cache; linear 5M-LOC extrapolation ($0.8–1M) is order-of-magnitude only.

Refuted (do not cite)

  1. Amazon “1,000 production apps upgraded in 2 days by 5 people” (0-3).
  2. GitHub merge-queue monorepo throughput “500 engineers / 2,500 PRs/month” (0-3).
  3. HumanLayer as adoptable off-the-shelf triage product (0-3).
  4. HumanLayer Slack/Email routing as available feature (0-3, deprecated).

Open questions carried into the plan

  1. Does any deployment orchestrate ~1000 concurrent coding agents? (No published evidence.)
  2. Do the 6.08%→1.82% regression numbers transfer to frontier models on TS/Go?
  3. Quantitative effectiveness of mutation testing / shadow traffic / contract tests against agent regressions — no surviving evidence.
  4. Routing/SLA design that prevents stakeholder overload at hundreds of questions/day — no measured data anywhere.