Skip to content
GRPNR.

R3 — Human-in-the-loop question triage for AI agent fleets: 2025-2026 landscape

Research date: 2026-07-18. Purpose: does anything already solve “50 agents hit the same ambiguity, one human answers once, all resume, answer persists” for an internal 2-human + 64-agent setup? Full raw notes below; distilled report returned separately (max 900 words).

1. HumanLayer — STATUS: pivoted away from its original HITL-router product; original repo deprecated

  • Source: https://github.com/humanlayer/humanlayer (fetched 2026-07-18)
    • Verbatim maintainer note (Dex Horthy, founder) pinned in the repo: “public issues repo for humanlayer - the code here is pretty much all deprecated - you can try the rebuild of humanlayer at https://humanlayer.com - thanks for all your support - dex”
    • Repo stats at fetch time: 11.1k stars, 927 forks, 2,098 commits, 40 open issues, 0 open PRs. Latest tagged release: codelayer-0.20.0, dated 2025-12-23.
    • Repo is now effectively an issue tracker for a legacy product, not the live codebase.
  • Source: https://finance.biggo.com/news/15099f5634f5ab9a (BigGo Finance, 2026, covering Dex Horthy interview)
    • HumanLayer originally founded 2020 as “Metalytics”; pivoted to an AI-agent platform in late 2024.
    • Horthy ran a “dark factory” experiment July–November 2025: fully automated agent pipeline (agents write/review/deploy code, zero human reads a line). Within 3 months the codebase degraded so badly a single bug took weeks to debug — direct negative evidence against the “agents self-resolve without humans” approach this whole research thread is checking alternatives to.
    • Current pivot (2026): HumanLayer is now building a “collaborative AI IDE” (real-time Google-Docs/Slack-style shared editing with agents, “sync engine with durable streams” for human steering while the agent works). Horthy publicly proposed “killing the pull request” on 2026-07-14.
    • Net: the original “contact_human tool-call + approval decorator + Slack/email routing” product (what people cite as “HumanLayer” in 2024-25 blog posts) is not the thing being actively built in mid-2026. It still nominally exists (approval decorators, routing to Slack/Email/Discord, escalation/timeout, “learning” auto-approval from prior decisions — per https://humanlayer.systems/index-en and https://www.everydev.ai/tools/humanlayer, both undated marketing pages, evidence weight low) but the founder’s own words say the code is deprecated and effort has moved to the IDE.
    • Fit verdict: Not a fit as a standing dependency. Original product is founder-confirmed deprecated; the team’s current focus (collaborative IDE) is a different product shape (interactive pairing, not a fan-in question-triage queue). No evidence anywhere of duplicate-question merging across agent instances — HumanLayer’s model was always 1 agent-run : 1 pending approval.

2. LangChain — LangGraph interrupt() + Agent Inbox

  • Source: https://www.langchain.com/blog/making-it-easier-to-build-human-in-the-loop-agents-with-interrupt (LangChain blog) — interrupt() pauses graph execution, persists state via LangGraph’s checkpointer, human decision (approve/edit/reject/respond) resumes it.
  • Source: https://docs.langchain.com/oss/python/langchain/human-in-the-loop — HITL middleware pauses on risky tool calls (file write, SQL) pending review.
  • Source: https://github.com/langchain-ai/agent-inbox/blob/main/README.md and direct repo fetch 2026-07-18: “📥 An inbox UX for interacting with human-in-the-loop agents.” 1,000 stars, 12 open issues, 9 open PRs, 365 commits, TypeScript 98.5%. No specific last-commit date surfaced by the fetch (page didn’t render commit timestamps), so activity level as of July 2026 is not independently confirmed beyond “not archived, has recent open PRs.”
  • Secondary source (blog, lower confidence): https://prompts.brightcoding.dev/blog/stop-building-bad-ai-agents-agent-inbox-fixes-hil describes it as something LangChain “quietly dropped” — i.e., a low-profile companion tool, not a flagship product with marketing push.
  • Architecture: each interrupt() call surfaces one pending item in the inbox; the inbox is a UI for triaging many different pending interrupts from many different graph runs/threads. That is genuinely N:1 (many threads → one inbox), which is half of the target problem. But nothing in the docs or README describes content-based deduplication — two threads independently hitting the identical ambiguous question show up as two separate inbox items, not one merged item; and there’s no described mechanism for writing the human’s answer back into a shared knowledge store consumed by future/other runs (each interrupt’s answer resumes only its own thread).
  • Fit verdict: Closest primitive match for the “one inbox for many agents” half of the problem, actively maintained as of 2026, but it is a many-questions-to-one-queue UI, not a many-agents-one-question-merge engine — the dedup and write-back-to-memory pieces would have to be built on top.

3. gotoHuman

  • Source: https://www.gotohuman.com/ (product site, 2026) — “Supervise AI agents in one flexible platform.” Framework/model-agnostic; Agent Inbox concept where reviewers see items assigned to them or the whole team; customizable review templates (text/image/markdown/JSON); GDPR-compliant, EU-hosted, SOC 3 + ISO 27001 certified core services (self-reported, not independently verified here).
  • Source: https://n8n.io/integrations/gotohuman/ and n8n community post — gotoHuman node is “verified” on n8n cloud, positioned as “the only AI-first human-in-the-loop solution available to all n8n users” (vendor claim).
  • Source: LinkedIn post (2026) — named customer PayFacto (payments company, ~$10B annual transaction volume) using it for sales/marketing automation review, not engineering/agent-fleet ops.
  • Could not find: GitHub repo activity (github.com/gotohuman has repos but no star/commit data surfaced), funding amount, pricing, or any mention of cross-agent question deduplication. Product model is per-item review (one AI output → one reviewer decision), same 1:1 shape as HumanLayer’s original approach.
  • Fit verdict: Proprietary SaaS, alive, but built for content-review / approval-per-output workflows (marketing copy, sales actions), not for merging identical clarifying questions across a coding-agent fleet. No dedup or write-back-to-memory feature found.

4. Claude Agent SDK / Claude Code — AskUserQuestion, hooks

  • Source: https://code.claude.com/docs/en/agent-sdk/overview and https://platform.claude.com/docs/en/agent-sdk/hooks — AskUserQuestion is a built-in tool; hook events include PreToolUse, PostToolUse, Stop, SubagentStart/Stop, Notification, PermissionRequest, UserPromptSubmit. PermissionRequest hook can be used to push external notifications (Slack/email/push) when Claude is waiting on approval.
  • Source: https://code.claude.com/docs/en/agents (Claude Code subagents docs) — hard constraint, directly on point for a 64-agent fleet: “A subagent cannot ask you a clarifying question (AskUserQuestion is unavailable to subagents),” and “a background subagent auto-denies any tool call that would otherwise prompt for permission.” This means in the current product, only the top-level/orchestrating agent session can ever surface a question to a human at all — subagents silently auto-deny instead of escalating.
  • Source: https://github.com/anthropics/claude-code/issues/59908 (fetched 2026-07-18) — opened 2026-05-17, still open, labeled area:hooks, duplicate (implying an earlier, presumably still-open, related report — likely issue #13830 found in search, “Feature Request: Add notification hook support for AskUserQuestion events”). Verbatim problem statement from the issue: “When Claude Code awaits user input via AskUserQuestion tool, no hook event fires. Users relying on hooks for async-workflow notifications (terminal bell, system notification, etc.) silently miss these ‘Claude needs your input’ moments.” Confirmed inconsistency table in the issue: Stop hook works, Notification hook works for permission prompts, but no hook fires for AskUserQuestion as of Claude Code v2.1.143 (the reporter’s version).
  • Fit verdict: Directly relevant negative evidence for the target architecture. As of July 2026, Claude Code’s own subagent model has no path for a subagent to ask a human anything — by design, it auto-denies. And even the top-level agent’s AskUserQuestion doesn’t reliably notify external channels (open bug, 2 months old, unresolved). A 64-agent fleet built on Claude Code subagents cannot rely on the SDK to surface human-facing questions from worker agents at all; any HITL layer for such a fleet must be built as custom tooling (e.g., a custom MCP tool that proxies “ask human” calls out of subagents into an external queue), not assembled from AskUserQuestion + hooks out of the box.

5. CrewAI / AutoGen (now Microsoft Agent Framework)

6. Slack/Teams approval bots

7. Multi-agent coding orchestrators (adjacent tooling for a 64-agent fleet)

  • Source: search summary citing https://github.com/BloopAI/vibe-kanban and coverage articles (2026) — Vibe Kanban: kanban board for running many parallel Claude Code/Codex/other coding agents. Company behind it (Bloop) shut down 2026-04-10; project continues as Apache-2.0, community maintained, but the paid cloud product and remote services were sunset — negative evidence of commercial fragility in exactly this adjacent space (managing many concurrent coding agents).
  • Conductor (Mac app, isolated git-worktree-per-agent, diff viewer, PR flow) and Crystal (renamed “Nimbalyst”) are the other commonly cited tools in this category per 2026 blog roundups (Nimbalyst, Munder Difflin blog, agentsroom.dev — all secondary/low-confidence sources, not fetched directly). None of the search summaries surfaced a unified question-inbox or notification-merge feature in any of them; they solve workspace isolation and diff review, not human-question triage.
  • Fit verdict: No fit found. This category solves “run N agents without them stepping on each other’s files,” not “route their questions to a human without duplication.”

8. Academic / research: merging duplicate questions, answer write-back

  • Source: https://arxiv.org/pdf/2512.13154 — “MAC: A Multi-Agent Framework for Interactive User Clarification in Multi-turn Conversations” (arXiv, submitted ~Dec 2025). Fetched directly 2026-07-18. Architecture: a supervisor agent handles domain-agnostic ambiguity, expert agents handle domain-specific underspecification before API calls; explicitly limits to one clarification per turn to control latency. This is single-user, single-conversation multi-turn clarification (one user, one agent pipeline) — not the fan-in case of many independent agent instances hitting the same ambiguity simultaneously. Fetch could not confirm any deduplication-across-agents or memory-write-back mechanism in the visible sections (methods/results pages weren’t part of the fetched excerpt) — treat as inconclusive on those two specific sub-questions rather than a firm “absent.”
  • General agent-memory research (source: https://thenuancedperspective.substack.com/p/designing-agentic-memory-in-2026, and a cited Dec-2025 107-page survey “Memory in the Age of AI Agents” by Hu et al., not independently fetched) discusses write-back patterns generally — e.g., “Codex… stor[es] memory in explicit manifests where every entry is version-controlled and human-reviewed” — but none of the surfaced sources tie this specifically to HITL-answer-persistence for deduplicating future agent questions.
  • Fit verdict: No paper or product found that directly targets “N concurrent agents ask the same/similar question → merge to 1 → 1 human answer → all N resume → answer persists to shared memory/skill/doc.” Closest adjacent research (MAC) addresses a different problem shape (single-conversation ambiguity, not fleet-wide dedup).

Bottom line

Searched: HumanLayer (repo direct fetch), LangChain agent-inbox (repo direct fetch) + LangGraph interrupt docs, gotoHuman (product + n8n listing), Claude Code/Agent SDK docs + a live open GitHub issue (direct fetch), CrewAI/AutoGen/Microsoft Agent Framework docs, Slack approval-bot patterns, Vibe Kanban/Conductor/Crystal coding-agent orchestrators, and one arXiv paper (direct fetch) on multi-agent clarification. Every tool and paper found implements a 1 agent-instance : 1 pending question/approval shape (fan-in to a queue at best, per LangGraph Agent Inbox). None implement content-based merging of duplicate/similar questions across concurrently running agent instances, and none implement automatic write-back of the human’s answer into a shared memory/skill/doc store that other agents consult before asking. Claude Code’s subagents specifically cannot ask a human anything today (auto-deny by design) — a hard blocker for a Claude-Code-subagent-based 64-agent fleet unless a custom “ask human via external queue” MCP tool is built. This looks like a genuine, currently unfilled gap as of 2026-07-18, not a “go install X” situation.