14 - Validation Plan: Atlas (the answer loop)

Passport convention (binding): every number carries MEASURED, MODELED, DARK, or DESIGN. No em-dashes. Plain English, receipts not adjectives.

Purpose: the PRD (12-prd.md) already specifies what Atlas does. This document specifies how we find out, cheaply, whether any of its load-bearing assumptions are false before we spend a full build on them. Every one of the four core sizing numbers named in the PRD (duplicate rate, interrupts/day, re-ask rate, answers-worth-persisting) is DARK today. This plan is the instrument that turns them MEASURED, in the cheapest order that disproves the concept fastest if it deserves to be disproved.

Stance: do not assume the concept is good. Every assumption below is written as a falsifiable claim with a number that kills it, not a feature to justify.

1. How to read this document

Section 2 is the full assumption inventory: 12 assumptions across the six risk categories, each with the 13 fields requested (category, importance, current evidence, uncertainty, validation method, sample, evidence required, success threshold, failure threshold, cost, effort, sequence, decision triggered).
Section 3 details the four cheapest disproving tests, mapped onto the assumptions they resolve.
Section 4 gives the actual execution order (not the same as the order the four tests are usually listed in) and a diagram.
Section 5 is the decision map: which numbers trigger build, pivot, or kill.
Section 6 names counters the PRD does not yet instrument, surfaced by writing this plan.

2. Assumption inventory

Index

ID	Category	Assumption (one line)	Importance	Uncertainty	Sequence	If it fails
VA-1	Desirability	The fleet generates enough duplicate questions for dedup to matter	Critical	High (DARK)	1 (wave 1)	Kill dedup, keep bare triage
VA-2	Desirability	Tomas answers in the tasks-UI queue, not GChat out-of-band	Critical	Medium	1 (wave 1)	Pivot to queue latency/UX, not scope
VA-3	Desirability	Enough answers are worth persisting to justify write-back at all	High	High (DARK)	2 (wave 1-2)	Kill write-back, keep triage+resume only
VA-4	Usability	Agents search the corpus before filing a question	Critical	High (DARK)	2 (wave 1)	Pivot: make search a forced pre-step, not optional
VA-5	Usability	The five-field capture/answer forms get filled with real content, not rubber-stamped	High	Medium	2 (wave 1)	Pivot: tighten field prompts, add garbage detection
VA-6	Feasibility	Temporal signal-resume works across N attached workflows off one completed task	Critical (build-blocking)	High (DARK)	0 (pre-wave spike)	Fall back to N-completion-calls (FR-14 fallback)
VA-7	Feasibility	Deterministic exact-key dedup catches a meaningful share of real duplicates	Medium	High (DARK)	1 (wave 1)	Accept exact-key is decorative; value is Robert’s manual merge, prioritize Stage-1 fuzzy sooner
VA-8	Viability	Loop’s LLM/token cost is less than the interrupts + re-asks it prevents	High	Very high (DARK, uninstrumented)	2 (wave 1, needs new counters)	Kill or narrow write-back drafting; keep deterministic parts only
VA-9	Viability	Write-back PR review does not become the new bottleneck	High	Medium	2 (wave 1-2)	Pivot to sampled/batched review (P-1)
VA-10	Trust	Agents that consume a past answer produce measurably better/correct output, not just fewer interrupts	Critical	Very high (DARK)	3 (wave 2, needs corpus depth)	If a consumed answer produced silently wrong work: treat as SAFETY KILL trigger
VA-11	Trust	One bad written-back answer does not collapse Tomas’s trust in the whole corpus	Critical	Medium	Ongoing, checked wave 2+	If out-of-band answer rate spikes after a known-bad write-back: pivot per P-4, audit gate
VA-12	Data	Question text is safe to store/render given PII/secrets/confidentiality risk	Critical (go/no-go gate)	Medium	0 (pre-wave red-team)	Hard block go-live until regex/gate fixed (SAFETY KILL)

Ordering note: “Sequence” above is logical dependency order, not the four-test numbering given in the brief. Section 4 reconciles the two.

VA-1. Duplicate rate is high enough to justify dedup

Category: Desirability.
Importance: Critical. This is the PRD’s own R6, “the single riskiest” (12-prd.md §23). If false, the flagship benefit does not exist.
Current evidence: DARK. No fleet-scale duplicate-question data exists in public literature (checked twice per PRD §2). VIS (~2013) and orders (~2017) freezes are named as high-volume “is this alive” generators, but that is MODELED, not measured.
Uncertainty: High.
Validation method: Concierge/wizard-of-oz week (Test 1, §3.1). Robert reads every incoming agent-questions task by eye and hand-merges duplicates for one real fleet wave.
Sample: Wave 1, one real fleet wave, one week, throttled cohort (PRD §28).
Evidence required: Raw (un-deduped) question count vs deduped queue count Tomas actually saw, side by side, at week end.
Success threshold: Duplicate/attach rate >= 15% (DESIGN, matches PRD §32 kill floor).
Failure threshold: Duplicate/attach rate < 15% AND re-ask rate not falling AND answers-worth-persisting < 5/week, all three together (PRD §32 KILL/COLLAPSE condition).
Cost: Zero build. Robert’s reading time only, existing substrate (INFORMATION_REQUEST + agent-questions queue is already built).
Effort: Low. One person, one week, no code.
Sequence: Runs first among live-wave tests, in parallel with VA-6 and VA-12 (pre-wave spikes).
Decision triggered: >=15%: proceed to build the dedup layer for real. <15% alone: watch, do not kill yet (only the three-way AND kills). All three low: collapse to plan-001 bare triage (PRD §32).

VA-2. Tomas answers in the tasks-UI queue, not GChat out-of-band

Category: Desirability.
Importance: Critical. Named directly by Tomas (“vic agentu mi nepomuze,” he is decision-bound) and flagged as a champion-dependency single point of failure (PRD §23 R5).
Current evidence: ASSUMPTION. “Inside Zaruba’s tasks UI” is explicitly unverified; no frontend completion route was confirmed in the substrate read (PRD §12, §07 point 4).
Uncertainty: Medium. The backend contract (POST /tasks/:id/complete) is real regardless of frontend; what is unverified is whether a human-friendly surface exists or whether Tomas will use curl/CLI/dictation.
Validation method: Same wave-1 concierge week (Test 1). Track every answer’s channel: in-UI, curl/CLI, or GChat relay outside the queue entirely.
Sample: Wave 1.
Evidence required: Count of answers given in-queue vs out-of-band (GChat) during the wave; verify whether a frontend completion surface exists at all before the wave starts.
Success threshold: Out-of-band answer rate stays low and flat across the week (no DESIGN number exists yet; treat any out-of-band answer as a logged incident, not a rate to tolerate).
Failure threshold: Out-of-band rate rising over the week, or Tomas defaulting to GChat because the queue is slower (P-4).
Cost: Zero build.
Effort: Low, observational.
Sequence: Concurrent with VA-1.
Decision triggered: Rising out-of-band rate triggers P-4: fix queue latency/UX before adding any scope. This is also the leading indicator for VA-11 (trust collapse).

VA-3. Enough answers are worth persisting to justify write-back

Category: Desirability.
Importance: High. Sizes the expansion beyond bare triage (01-initial-recommendation.md’s original fastest-test framing).
Current evidence: DARK.
Uncertainty: High.
Validation method: Count write-back-worthy answers during the concierge week and the following week (Robert judges “worth persisting” by eye, same wizard-of-oz posture as dedup).
Sample: Wave 1-2.
Evidence required: Weekly count of answers Robert judges durable/reusable vs one-off/contextual.
Success threshold: >= 5/week (DESIGN, PRD §32 kill floor).
Failure threshold: < 5/week, combined with the other two legs of the §32 KILL/COLLAPSE test.
Cost: Zero build (this is a counting exercise against wave-1 output, not new software).
Effort: Low.
Sequence: 2, depends on wave-1 answers existing.
Decision triggered: >=5/week: build the write-back worker for real. <5/week alone: watch. All three legs low: collapse per §32.

VA-4. Agents search the corpus before filing a question

Category: Usability.
Importance: Critical. Search-before-ask is the entire mechanism that makes G1 (“spend Tomas’s decision-hours once”) possible; without compliance, the corpus is written but never read, and Atlas degrades to write-only logging.
Current evidence: DARK. No corpus exists yet to search (cold start, PRD §8 variant B); compliance has never been observed on this fleet.
Uncertainty: High. Agents optimizing for task completion have no intrinsic reason to search first unless the skill/proxy makes it the path of least resistance; NFR-1’s own reasoning (“if search is slow, agents skip it”) applies equally to “if search is optional, agents skip it.”
Validation method: Instrument search-before-ask compliance (PRD §17 counter, already specified) from day one of wave 1; do not wait for a dedicated test.
Sample: Wave 1, and again wave 2 once the corpus has depth (cold start means wave-1 compliance is uninformative about search value, only about search habit).
Evidence required: Fraction of ask_question calls preceded by a search_answers call, per agent session.
Success threshold: >= 70% (DESIGN, no prior basis; set as a working target, not a proven number).
Failure threshold: < 40% (DESIGN). Below this, search is decorative and the loop is write-only.
Cost: Zero extra build; the counter is already in FR/§17 scope.
Effort: Low, instrumentation only.
Sequence: 2, measured continuously from wave 1 onward.
Decision triggered: Below 40%: stop treating “search-first” as a design principle that holds by itself; make the skill/MCP proxy call search_answers automatically before ask_question is even reachable (remove the choice, do not ask agents to comply).

VA-5. The five-field forms get filled with real content, not rubber-stamped

Category: Usability.
Importance: High. The PRD names this risk explicitly: “agents do not abandon forms, they rubber-stamp them” (07 §7.1, cited PRD §12). If true of the capture form, dedup keys are garbage-in; if true of Tomas’s answer form, scope/expiry/confidence are decorative and NFR-7 (100% passport completeness by construction) becomes 100% complete nonsense.
Current evidence: ASSUMPTION, sourced from a UX document’s prior observation of agent form behavior generally, not measured on this fleet or this form.
Uncertainty: Medium.
Validation method: Manual audit: Robert reads a sample of filled forms from wave 1 and judges whether each field carries real information or boilerplate (e.g. searched[] containing an actual grep command vs the literal string “searched the codebase”).
Sample: Wave 1, n = 20 (DESIGN) forms, capture and answer forms both.
Evidence required: Per-field judgment (real content / boilerplate) across the sample.
Success threshold: >= 80% (DESIGN) of fields carry real, specific content.
Failure threshold: < 50% (DESIGN).
Cost: Zero build, Robert’s reading time.
Effort: Low.
Sequence: 2, after wave 1 forms exist.
Decision triggered: Below 50%: tighten field prompts with concrete examples (already attempted in FR spec, “paste the exact grep/search calls,” PRD §12) and consider blocking submission on pattern-matched boilerplate (e.g. reject searched[] values under N characters or matching a stopword list). This is a cheap fix if caught early; expensive if the dedup layer is already keyed off garbage.

VA-6. Temporal signal-resume works across N attached workflows off one completed task

Category: Feasibility.
Importance: Critical, build-blocking. Named by the PRD as “the highest technical risk” and the single item to “prototype FIRST, before any docs/glossary work” (PRD §23 R1, §24 OQ-1).
Current evidence: Unproven at N>1 on this substrate (03-journey.md fan-out gap; 04-user-stories.md CP-5 flagged ASSUMPTION).
Uncertainty: High.
Validation method: Resume smoke test (Test 4, §3.4): isolated harness, no live fleet wave needed.
Sample: Not a fleet wave. A synthetic test: N workflows (start with N=2, then N=10, DESIGN) deliberately blocked on one canonical task via _taskCreateFromWorkflow/TaskWorkflowStamp, one completion, one fan-out.
Evidence required: resumed == attached count, exact, plus timing per resume.
Success threshold: 100% resume rate at N=10 (zero tolerance per NFR-6, “a dropped resume is a hard bug, not a soft degradation”).
Failure threshold: Any workflow left stranded, or a duplicate resume, at any N tested.
Cost: Half a day of engineering time, no production wiring, no wave dependency.
Effort: Low-medium (one spike).
Sequence: 0, before anything else. This is the cheapest test in the whole plan relative to its blast radius and should not wait for a live wave.
Decision triggered: Pass: proceed with the attachment-signal design (FR-14). Fail: switch immediately to the documented fallback, N completion calls per attached asker (04 CP-5 fallback), before building anything else on top of fan-out.

Category: Feasibility.
Importance: Medium. Distinct from VA-1 (whether duplicates exist) - this asks whether the deterministic tier (FR-6/7/8, no fuzzy, no LLM) catches any of them, or whether real paraphrased questions never share a normalized key and all dedup value comes from Robert’s manual fuzzy judgment in wave 1.
Current evidence: DARK beyond the unit-level self-check (FR-6 requires a runnable assert that two paraphrases of one entity-question produce the same key; that proves the function works, not that real questions collide on it).
Uncertainty: High. Legacy-code questions about the same entity are plausibly phrased in many different ways (“is VIS still alive,” “who owns the VIS service,” “can I touch VIS code”) that a lowercase/stopword/entity-token normalizer may or may not collapse to one key.
Validation method: During the wave-1 wizard-of-oz week, run the normalizer on every incoming question and separately record Robert’s manual true-duplicate judgments; compare which true duplicates the exact-key tier actually caught.
Sample: Wave 1, all questions filed.
Evidence required: Of Robert-judged true-duplicate pairs, the fraction that also share an exact normalized key.
Success threshold: >= 30% (DESIGN) of true duplicates caught by exact-key alone.
Failure threshold: < 10% (DESIGN).
Cost: Zero extra build, the normalizer already exists per FR-6; this is a comparison of two logs already being kept.
Effort: Low.
Sequence: 1, alongside VA-1.
Decision triggered: Below 10%: exact-key dedup is decorative on this question population; the deterministic tier was low-value to build, and Stage-1 fuzzy automation (PRD §26) should be re-prioritized sooner than its current volume gate implies, because the alternative today is Robert doing 100% of the merging by hand indefinitely.

VA-8. Loop’s LLM/token cost is less than the interrupts and re-asks it prevents

Category: Viability.
Importance: High. This is the actual economic case for building Atlas at all, and it is currently absent from the PRD’s instrumentation (§17 has no cost counter). Every LLM call in the loop (question-schema drafting assistance if any, write-back drafting, per C5 the one place an LLM appears with provenance) costs real tokens; the PRD never compares that spend to the value of a prevented interrupt.
Current evidence: DARK, and uninstrumented. Not merely unmeasured, there is currently no counter that would measure it even if wave 1 ran today.
Uncertainty: Very high.
Validation method: Add a per-stage token/cost counter (ask_question validation, write-back draft generation) before wave 1 starts; log every LLM call the loop makes with cost, per the user’s own LLM-run-evidence discipline (no silent spend). Compare weekly loop spend against a MODELED cost-per-interrupt (Tomas’s architect time, converted to a rough per-interrupt cost, MODELED, e.g. 15 minutes of decision-time per raw interrupt, a placeholder pending a real number).
Sample: Wave 1 onward, continuous.
Evidence required: Dollars/tokens spent by the loop per week vs (interrupts prevented x modeled cost-per-interrupt).
Success threshold: Loop cost per prevented re-ask < modeled cost of one interrupt (ratio < 1, DESIGN).
Failure threshold: Ratio > 1 sustained across 2+ weeks.
Cost: Small, a logging addition, no new infra.
Effort: Low-medium (new counters, not new architecture).
Sequence: 2, but the counters must exist before wave 1, not be retrofitted after.
Decision triggered: This finding stands regardless of outcome: the PRD ships without this counter today. It should be added as a §17 counter before wave 1, or Atlas will run its first real wave financially blind, which the program’s own credo (no silent spend on LLM runs) does not permit for anything else it operates.

VA-9. Write-back PR review does not become the new bottleneck

Category: Viability.
Importance: High. Named directly in the PRD as R4/RA-4 and OR-3: “the one place Atlas could recreate the bottleneck it exists to kill.”
Current evidence: DARK, bounded only by the untested assumption that answers-worth-persisting stays low.
Uncertainty: Medium.
Validation method: Track wall-clock time Robert spends reviewing write-back PRs during wave 1-2, against the answers-worth-persisting count (VA-3).
Sample: Wave 1-2.
Evidence required: Review-hours/week vs PRs/week.
Success threshold: < 20 write-backs/week (DESIGN, matches PRD P-1) and review time not crowding out Robert’s operator duties.
Failure threshold: > 20/week, or review time measurably competing with other duties.
Cost: Zero build, time-tracking only.
Effort: Low.
Sequence: 2.
Decision triggered: Above threshold: pivot to sampled/batched review (P-1), never to full review of every answer at scale.

VA-10. Agents that consume a past answer produce measurably better/correct output

Category: Trust.
Importance: Critical, and the least-addressed assumption in the PRD as written. The PRD’s counters (re-ask rate, activation event) measure whether an agent stops asking after finding an answer. None measure whether the agent’s resulting work was actually correct. A silent failure mode exists that is arguably worse than the status quo: an agent finds a stale, wrong-scope, or misapplied answer, does not ask (because the corpus “worked”), and produces silently wrong work with zero interrupt and zero visibility. The PRD’s own cold-start language (“hit = no interrupt, the product worked invisibly,” 01-initial-recommendation.md) assumes a hit is a good outcome; it is only a good outcome if the served answer was actually right for that use.
Current evidence: DARK. No mechanism in the PRD samples downstream code/decision correctness attributable to a corpus hit.
Uncertainty: Very high.
Validation method: Write-back shadow test (Test 2, §3.2), extended with a fidelity audit: for a sample of corpus hits, Robert manually checks whether the agent’s resulting work matches the served answer’s rationale and scope, not just whether it stopped asking.
Sample: Wave 2 (corpus needs depth; judging this at wave 1 cold-start is meaningless per PRD’s own variant-B guidance).
Evidence required: n = 10 (DESIGN) audited corpus-hit outcomes, pass/fail on fidelity.
Success threshold: >= 90% (DESIGN) of audited hits show correct application of the served answer.
Failure threshold: Any confirmed case of an agent silently misapplying a stale or wrong-scope answer to produce incorrect work that shipped without a human catching it.
Cost: Zero build, added review time on top of the shadow test.
Effort: Medium (requires reading agent output, not just logs).
Sequence: 3, after corpus has depth (wave 2+).
Decision triggered: A confirmed silent-misapplication case is treated as the PRD’s own SAFETY KILL trigger (“a wrong written-back answer trusted and acted on downstream that the human-gate did not catch,” §32), not a soft finding. This is a genuine gap versus the PRD’s stated counters and should be added to §17 as its own line, not folded silently into activation/re-ask numbers.

VA-11. One bad written-back answer does not collapse Tomas’s trust in the corpus

Category: Trust.
Importance: Critical. The PRD states this risk in its own words: a wrong doc “collapses trust in all docs, not gracefully” (§23 R2, citing R8 Sherlock-adjacent precedent).
Current evidence: Medium, by analogy (R8, general trust-collapse literature cited in research), not observed on this program.
Uncertainty: Medium.
Validation method: Watch the out-of-band answer rate (VA-2’s counter, PRD’s own “champion canary,” §17) in the period immediately following any known write-back correction (a superseded answer, FR-22/23).
Sample: Wave 2+, whenever the first supersede event happens.
Evidence required: Out-of-band rate in the 1-2 weeks before vs after a correction event.
Success threshold: No measurable rise in out-of-band rate after a correction is published (trust survives a correction handled transparently).
Failure threshold: A measurable rise following a correction.
Cost: Zero build, reuses an existing counter.
Effort: Low.
Sequence: Ongoing from wave 2 onward, triggered opportunistically by the first real correction.
Decision triggered: Rise detected: treat as P-4 (champion drift), audit whether corrections are being surfaced clearly enough (supersede chain, EXPIRED warnings) before adding any further scope.

VA-12. Question text is safe to store/render given PII/secrets/confidentiality risk

Category: Data.
Importance: Critical, go/no-go gate (SEC-1/SEC-3, PRD §18, §27 AC-8).
Current evidence: Medium. A regex denylist design exists (DR-2, gitleaks-style patterns) and a unit-level test is specified (a seeded secret pattern flips classification). Not tested against adversarial or legacy-specific formats (Czech-locale PII, non-standard legacy credential formats from 1.05M LOC of EOL Ruby/Java).
Uncertainty: Medium.
Validation method: Red-team pass before go-live: Robert crafts 10-20 (DESIGN) adversarial quoted-data samples deliberately designed to evade the denylist (unusual key formats, Czech PII patterns, data shaped like code but containing real-looking secrets) and runs them through the ingest classifier.
Sample: Pre-wave, synthetic, zero real questions needed.
Evidence required: Catch rate on the adversarial set; also verify GET /tasks?assigneeGroupSlug= visibility scoping (SEC-1) does not leak full question text to any VIEWER-role employee outside the queue.
Success threshold: 100% catch on the adversarial set (zero tolerance, matches the PRD’s own framing for stale-served and false-merge).
Failure threshold: Any miss.
Cost: Zero build, red-team is manual test-case authoring against existing code.
Effort: Low-medium.
Sequence: 0, before any real question is filed, gating go-live per PRD §27 AC-8.
Decision triggered: Any miss: hard block go-live (PRD’s own SAFETY KILL framing) until the regex/gate is fixed and re-tested. Do not soften this to “acceptable miss rate.”

3. The four cheapest disproving tests

These are the tests explicitly prioritized for this plan: cheap, fast, and each capable of killing or redirecting a major piece of the concept before a full build.

3.1 Concierge/wizard-of-oz week

What: Run the zero-PR v1 triage live on one real fleet wave. Robert reads every incoming agent-questions task and merges duplicates by hand (no automated fuzzy matching). Count questions filed, duplicate/attach rate, interrupts/day, time-to-unblock.
Resolves: VA-1, VA-2, VA-7 directly; feeds VA-3, VA-4, VA-5 as a byproduct of the same wave.
Cost/effort: Zero build. Uses substrate already in place (INFORMATION_REQUEST, group queue, Temporal resume). One week of Robert’s reading/judging time.
Numbers produced: Raw question count, deduped queue Tomas actually saw, dup rate, interrupts/day before/after, median time-to-unblock (stamped vs poller).
Decision: Feeds VA-1/VA-3’s build/kill gate (PRD §32) directly.

3.2 Write-back shadow test

What: Persist wave-1 answers to a scratch corpus (not yet wired into production write-back). Measure re-ask rate in the next wave with the corpus available for search (ON) vs not available (OFF).
Resolves: VA-3 (is the corpus worth having), VA-10 (does consuming it actually help, via the fidelity-audit extension).
Practical caveat, stated plainly: a true randomized ON/OFF split on one live fleet is operationally hard, there is one fleet, not two. The honest design is a cohort split within wave 2 (some agent instances get corpus access, some do not, same task types) rather than a time split (which confounds “corpus ON” with “corpus has more depth by week 2 regardless”). Robert must choose the split before wave 2 starts; a time-split result is directional evidence at best, not a clean ON/OFF comparison.
Cost/effort: One day to stand up a scratch corpus and wire search_answers to it for the ON cohort. No production write-back worker needed yet.
Numbers produced: Re-ask rate, ON cohort vs OFF cohort; fidelity-audit pass rate on a sample of ON-cohort hits (VA-10).
Decision: Re-ask rate materially lower ON than OFF, with fidelity holding: build the real write-back worker. Re-ask rate flat or fidelity failing: write-back is not paying for itself yet, or is actively unsafe; do not wire it into production.

3.3 Digest concierge

What: Hand-write a 5-line digest (no software) and send it to Tomas for 3 days. Observe: does he read it, does he reply, does he ever reference it unprompted.
Resolves: Part of VA-2 (is a digest even wanted) and directly tests PRD’s own P-5 (“if Tomas skims and ignores the digest line, delete it at zero cost”).
Cost/effort: Zero build. Robert’s time to compose 5 lines/day for 3 days.
Numbers produced: Read/ack rate over 3 days (binary per day: opened/replied/referenced, or not).
Decision: Ignored across all 3 days: delete the digest concept entirely before building any digest rendering code, per P-5, “do not staff, design, or improve it.” Read or referenced at least once: proceed to spec the generated digest (FR-25) for real.

3.4 Resume smoke test

What: Block N workflows (start N=2, then N=10) on one canonical task via a synthetic harness. Complete the task once. Verify all N resume, none double-resume, none strand.
Resolves: VA-6, the single highest technical risk in the whole PRD (R1), explicitly flagged “prototype this FIRST.”
Cost/effort: Half a day, isolated harness, zero live-wave dependency, zero production wiring.
Numbers produced: resumed == attached (exact count), any duplicate resume, any stranded workflow, resume latency per workflow.
Decision: 100% resume at N=10: build FR-14 as designed (attachment-signal fan-out). Any failure at any N: switch immediately to the documented fallback (N completion calls, 04 CP-5) before writing another line of Atlas code on top of fan-out; every other feature (dedup, write-back, digest) is worthless if the resume mechanism itself is not solid, since a silent stranded agent defeats the product more completely than any dedup miss.

4. Execution sequence

The four tests are not equally early. Resume smoke test and the SEC-1/VA-12 red-team pass are pure spikes with zero live-wave dependency and the highest cost-of-being-wrong (a broken fan-out or a PII leak poisons everything built after it); they run first, before any real question is filed. The concierge week and digest concierge can start the same week since neither depends on the other. The shadow test needs wave-1 output to exist first.

flowchart TD
    A["Day 0-1: Resume smoke test (VA-6)\nsynthetic harness, N=2 then N=10"] -->|"pass"| C
    A -->|"fail"| A2["Switch to N-completion-calls fallback\nbefore building anything else"]
    A2 --> C
    B["Day 0-1: SEC red-team pass (VA-12)\nadversarial PII/secret set"] -->|"100% catch"| C
    B -->|"any miss"| B2["Hard block go-live\nfix gate, re-test"]
    B2 --> C
    C["Week 1: Concierge wizard-of-oz wave\n(VA-1, VA-2, VA-5, VA-7)\nRobert hand-merges duplicates"]
    D["Week 1 (parallel, independent): Digest concierge\n3-day hand-written digest to Tomas (VA-2/P-5)"]
    C --> E["Week 1-2: Count answers-worth-persisting (VA-3)\nstart token/cost counters (VA-8)"]
    E --> F["Week 2: Write-back shadow test\nscratch corpus ON vs OFF cohort (VA-3, VA-10)"]
    F --> G["Week 2+: Fidelity audit sample (VA-10)\nreview bottleneck check (VA-9)"]
    G --> H["Decision gate: PRD Section 32\nbuild full loop / pivot per Section 31 / kill"]
    C --> H
    D --> H

5. Decision map

Reusing and extending the PRD’s own build/pivot/kill framing (§31, §32), tied to the tests above:

Numbers observed	Decision	Source
Dup rate >= 15% AND answers-worth-persisting >= 5/week AND re-ask rate falling	BUILD the full loop (dedup + write-back + fan-out for real)	PRD §32, VA-1/VA-3
Dup rate < 15% alone, or answers-worth-persisting < 5/week alone	WATCH, do not act on one leg alone	PRD §32
All three legs fail together after 2 waves	KILL dedup + write-back + digest; keep INFORMATION_REQUEST + queue + Temporal resume only (plan-001 v1)	PRD §32
North Star (interrupts/merged unit) flat or rising after 4 weeks, no fixable cause	HARD KILL the whole loop, revert to GChat relay/bare triage	PRD §32
Confirmed corpus-poisoning (VA-10 silent misapplication) OR confirmed sensitive-data leak (VA-12) at any time	SAFETY KILL, halt write-back, audit gate, zero tolerance	PRD §32
Resume smoke test fails at any N (VA-6)	Switch to N-completion-calls fallback before any further build	FR-14 fallback
Search-before-ask compliance < 40% (VA-4)	Make search a forced pre-step in the skill/proxy, remove the choice	New, this plan
Write-back review > 20/week or crowding Robert’s time (VA-9)	Pivot to sampled/batched review	PRD P-1
Any confirmed fuzzy false-merge (not applicable to v1’s exact-key-only auto-merge, but tracked if fuzzy-suggest ships post-MVP)	Pivot to exact-key-only, tighten threshold	PRD P-2
Digest ignored across 3-day concierge test	Delete the digest concept, zero cost	PRD P-5
Out-of-band answer rate rising, especially after a correction (VA-2, VA-11)	Fix queue latency/UX before adding scope	PRD P-4
Loop cost per prevented re-ask > modeled cost per interrupt (VA-8), sustained 2+ weeks	Narrow or kill LLM-drafted write-back, keep deterministic parts	New, this plan

6. Instrumentation gaps surfaced by this plan

Two counters do not exist in the PRD’s §17 analytics list and should be added before wave 1, not retrofitted after:

Loop cost/token counter (VA-8). No line in §17 tracks what the loop itself spends (write-back drafting is the one LLM-in-the-loop point per C5). Without it, viability is unmeasurable even if every other number comes back green. This is also a plain instance of the program’s own “no silent spend” rule for manual/spawned LLM runs; Atlas is exactly this kind of run and currently has no evidence line.
Downstream fidelity/correctness counter (VA-10). §17’s re-ask rate and activation event measure whether an agent stopped asking, not whether the answer it consumed was applied correctly. A stale or wrong-scope answer silently misapplied produces zero interrupts and looks identical to success in every existing counter. This gap should be closed with a sampled fidelity audit (§2 VA-10) before “search-before-ask” is treated as an unqualified win condition.

Summary of open questions this plan does not resolve

Whether a true ON/OFF randomized comparison is possible at all on a single fleet, or whether the program must accept cohort-split evidence as the ceiling (§3.2 caveat).
The MODELED cost-per-interrupt number used in VA-8 is a placeholder; it needs a real conversion (architect decision-time value) before the viability gate means anything numeric.
Whether Tomas’s trust, once tested by a real correction event (VA-11), behaves as literature-analogy predicts on this specific two-human program, is inherently a wait-and-see, not a thing this plan can pre-test.