Study session · 2026-10-23
Software Factories
What they are, what they are not, how one is built, what people argue about, and whether they pay.
Human Agent Machine / CI Gate Artifact (spec, test, doc)
These colors mean the same thing on every slide. Press N for speaker notes, ← → to move.
Opening. This deck is built from a survey of about 130 sources (blogs, company engineering posts, OSS repos, HN and X threads) collected in October 2026. The color legend is used everywhere, so point it out now: orange is a human, blue is an agent, grey is a machine or CI, purple is a gate, teal dashed is an artifact.
Agenda
Six parts, from definition to my own view
1 · Define What it is and is not
2 · Blueprint 11 stations and their tech
3 · Cases StrongDM, Stripe, Spotify, OpenAI…
4 · Debates Dark or light? Who reviews?
5 · Stack Sandboxes, CI, harnesses
6 · My view Tacit knowledge, lanes, money
SURVEY OF OTHERS
OUR POSITION
Parts 1–5 report what the field says, with sources on every slide. Part 6 is opinion, marked as such.
Parts 1–5 are reporting. Part 6 is our own view, and I'll flag it clearly when we get there.
1
Define: what a software factory is, and what it is not
One definition, two births of the word, and four neighbours it gets confused with: Claude Code, Ralph loops, CI/CD and "dark" automation.
Part 1 sets shared vocabulary. The word "software factory" is used loosely on HN and X, and threads like "Personal software factories are a waste of time" (Oct 2026) argue mostly about the definition. So we fix one definition first, then contrast it with the things people mix it up with: a single agent harness like Claude Code, a Ralph-style while-true loop, and ordinary CI/CD and platform engineering. We end with Shapiro's five levels, the light/twilight/dark vocabulary, and why this became possible in 2025–26.
Human sets intent what to build, why, limits
Human accepts outcome owns risk, approves evidence
Spec plan, scenarios, acceptance tests
Queue split into tasks, schedule workers
Agent loop · sandbox 1
Agent loop · sandbox 2
Agent loop · sandbox n
Verify tests, CI, judges, hidden scenarios
Gate merge rule by risk
Ship & observe verified software in production
FAIL → RETRY
FEEDBACK: INCIDENTS, FAILED RUNS → SPEC
Governance token budgets · agent identity with a human owner · permissions · delivery metrics
Osmani's short form: "an org chart made of loops" . The agent is one part. Queue, sandboxes, verification, gates and feedback make it a factory.
Where the definition comes from. There is no standard. The sentence on the title is our synthesis of three sources. Addy Osmani gives the most usable one: many harnessed loops running at once, fed by a queue of work and drained through a review gate into production, with humans above the system owning it. Dan Shapiro's Level 5 calls the end state "a black box that turns specs into software". StrongDM's manifesto (Feb 2026) made the term popular with two rules: humans don't write code and humans don't review code; they verify with holdout scenarios kept outside the repo. Dex Horthy (HumanLayer) grounds the term in the pre-AI 2022 pipeline: decide, tracker, build, PR review, ship, monitor, feedback into the tracker. In the agent version, "build" becomes agents and review becomes the bottleneck. The 11-station version of this line is Part 2. Color legend: orange human, blue agent, grey machine, purple gate, teal dashed artifact, amber dashed loops.
1 · Define · History
The factory was born twice: 1968 to standardize human work, 2026 to run without it
Born once: make humans produce like a factory
Born again: agents produce
1968 Bemer coins "software factory"; NATO, McIlroy: reusable parts
1969 Hitachi Software Works: first software-only plant
1991 Cusumano, Japan's Software Factories: reuse, QA, standards
2004 Microsoft Software Factories: models and DSLs generate
2017–22 Kessel Run, DoD DevSecOps: 29 factories = CI/CD + in-house teams
2026 Jan Shapiro: "dark factory" Feb StrongDM manifesto Jul AIEWF main-stage theme; Osmani, HumanLayer
Problem solved: variance and reuse. The worker is a person.
Problem: unattended output
Same word, different problem. Osmani traces the term to Bob Bemer's 1968 paper "The economics of program production" (we did not read the original). The 1968 NATO conference in Garmisch coined "software engineering", and McIlroy argued for mass-produced, catalogued components. Hitachi's own history says its Software Works became independent in 1969 as the world's first software-dedicated facility; Toshiba, NEC and Fujitsu followed. Cusumano's 1991 book described these as "flexible mass production": standards, reuse, quality control. Microsoft's 2004 Software Factories (Greenfield and Short) aimed at economies of scope with models and DSLs, an ancestor of today's spec-driven development. From 2017, the US Air Force's Kessel Run and later the DoD (29 factories by 2022, per Breaking Defense) used "software factory" for a DevSecOps platform plus in-house agile teams. Every earlier wave moved craft into a system but kept people as the workers. In 2026 the worker becomes an agent, so the question becomes how much can run unattended. Wavect's essay judges that the Japanese factories stayed company-specific wins; the same question applies to AI factories.
1 · Define · Is / is not
A factory is the system around the agents: queue, checks, gates and feedback
A software factory is
Many agent loops running at once, fed by a queueA line from intent to production, end to end
Machine verification as the main check on every change
Humans setting intent and owning outcomes and risk
A feedback loop from production back into specs and context
It is not
One smarter agent, a bigger prompt, or one long chat
A while true loop on its own
CI/CD with a chatbot added
"No humans" by definition: dark is one setting of it
Proof of quality: green tests can hide unreadable code
Test for any claimed factory: who decides a change is done, and what happens when that check is wrong?
The left column is the common core across Osmani, StrongDM, Shapiro and Horthy. The right column lists the confusions we saw in the HN threads, where people asked whether a factory is just a bundle of AGENTS.md files or different from Claude Cowork. Osmani is explicit: a factory "is not a bigger agent; it is an org chart made of loops". "No humans" is a design choice, StrongDM's choice, and Mollick argues it is the wrong goal for most organizations. The last row comes from HumanLayer: in July 2025 they went lights-off, and by around the third outage in November a cofounder rewrote the code by hand over two weeks. Their explanation is that RL rewards score pass/fail only, so nothing penalizes eroding maintainability. The question at the bottom is the one to ask any vendor or internal team calling something a factory.
Software factory = all of this
Intake tickets
Spec plan, tests
Context AGENTS.md
Orchestr. task graph
Sandbox VM, devbox
Harness model+tools
Verify CI, judges
Review merge gate
Ship canary, flags
Feedback incidents
Governance: cost, identity, permissions, metrics
Claude Code / Codex: this one station
Spotify kept its Fleet Management pipeline (targeting, PRs, review, merge) and swapped only the transformation step for an agent, later Claude Code.
Tool versus plant. A coding agent is, in Thorsten Ball's words, an LLM, a loop and enough tokens; he builds one in under 400 lines. Claude Code adds tools, subagents, hooks, experimental agent teams, a headless mode and an SDK, which is why it sits inside many factories: Gas Town, Fabro, Ralph loops and Spotify's Honk all run on it. Codex CLI has a non-interactive exec mode and is the worker in OpenAI's Symphony. A factory is everything else: what starts work, where the spec lives, how each agent gets an isolated environment, who checks the result, what merges, and how incidents flow back. Spotify is the cleanest proof that the parts are separable: they kept the plumbing and replaced only the transformation step, after their Maven updater script had grown past 20,000 lines. They moved from Goose, Aider and a homegrown loop to Claude Code without changing the pipeline. The 11 stations here are the blueprint of Part 2; "Orchestr." is orchestration.
Ralph loop (Huntley, 2025-07)
Software factory
Human tunes prompt
Agent run fresh context each time
REPEAT
PROMPT.md, specs/ plan, git = state
while :; do cat PROMPT.md | claude-code ; done
Spec + task queue
Loop A
Loop B
Loop n
Verify against scenarios
Holdout scenarios
Merge gate
Human owns gate + intent
FAILED RUN → NEW TASK
Loop engineering is a building block. Huntley's Ralph, in its purest form, is the Bash loop on the slide: the same prompt every time, a fresh context each run, and state kept on disk in specs, a plan file and git. He does one thing per loop and budgets about 170k tokens of context. It is monolithic on purpose: one process in one repo, with subagents for search and writing but only one running build and test. Huntley calls it deterministically bad in an undeterministic world, with failures fixed by tuning the prompt. Osmani's vocabulary helps: a loop is one agent doing one job on repeat, a harness is the walls around a loop, a factory is many harnessed loops behind a review gate. The factory adds what a single loop lacks: a queue of different tasks, verification the agent cannot game (StrongDM keeps scenarios outside the repo like a holdout set because agents can rewrite tests), a merge gate, and failures that become new work. Carlini's C compiler sits between the two: 16 parallel loops, about 2,000 sessions and under $20,000, coordinated only by lock files in git, with GCC as the near-perfect oracle.
Claude Code session Ralph loop CI/CD platform Software factory
Unit of work One chat, one task One prompt, run again and again A commit or merge request A ticket, through to a deployed change Who writes code Agent , human watchingAgent , unattendedHumans Many agents in parallelWho decides "done" The human at the terminal The loop's exit check Tests pass, a human approves Machine checks plus a gate sized to risk Where humans are Every turn Between runs, tuning the prompt Writing and reviewing code Intent, gates, accepting outcomes What limits scale One person's attention One repo, one process Human coding hours How fast changes can be verified
The new product is the check: specs, scenarios and verifiers decide what an unattended agent may ship.
What is actually new. CI/CD, DevOps and platform engineering already gave us the pipeline: tracker, build, PR, review, ship, monitor. The 2019 DoD DevSecOps reference design calls that a software factory. Agents plug into the same pipeline, so the plumbing is old. What changes is who sits in the build seat. When the worker is an agent that never tires but also never feels doubt, the binding constraint moves to verification. Osmani says verification, not generation, is the real constraint. Carlini's lesson from the compiler project: the verifier must be nearly perfect, otherwise Claude will solve the wrong problem. Stripe's Minions show the hybrid: deterministic nodes (lint, push) mixed with agentic nodes (implement, fix CI), at most two CI rounds per run, and still human-reviewed before merge. The table's "What limits scale" row is our summary of these sources, not a measured result.
COMPANY PLACEMENT = OUR READING OF THEIR POSTS
L0 Manual you write it all
L1 AI intern autocomplete, tasks
L2 Junior buddy pairing; ~90% of devs
L3 Manager you review the diffs
L4 Product mgr you write specs, check outcomes
L5 Dark factory specs in, code out; no human reads code
Stripe Minions Ramp Inspect Uber Cloudflare HumanLayer (after lights-off)
Spotify Honk OpenAI harness AWS Kiro Crew Airbnb migration Ralph loops
StrongDM Yegge's Gas Town Carlini C compiler
"ALMOST EVERYONE TOPS OUT HERE"
The ladder. Dan Shapiro (Glowforge CEO) borrowed the self-driving levels. L0 manual. L1 AI intern: autocomplete and small tasks. L2 junior buddy: pairing, where he puts about 90% of developers. L3: you become a reviewer, "Your life is diffs", and almost everyone tops out here. L4: you write specs and skills, leave for hours, then check whether tests pass; he says he is here. L5 borrows "dark factory" from FANUC's robot-staffed plant: a black box that turns specs into software; he says only a few teams of under five people work there. Placement is our reading. Stripe merges over 1,000 minion-produced PRs a week but still human-reviews them, so L3. Ramp's PRs are opened with the user's own token so nobody approves their own agent's change. Uber attributes over 70% of PRs to agents. Spotify Honk and OpenAI's harness work are spec- and verifier-driven with humans checking outcomes, so L4; OpenAI's team says no line of the ~1M-line product was written by hand. StrongDM's rules are no human writes or reviews code. Yegge says Gas Town is 100% vibe coded and he does not look at the code, so we put it at L5, though his later Gas City post says it keeps the lights on. Carlini's compiler ran 16 agents with a GCC oracle and he says he is uneasy deploying code nobody verified.
Light: humans read the code
Twilight: agents call humans in
Dark: no human reads code
MORE HUMAN READING LESS HUMAN READING
HumanLayer (Horthy)
Went lights-off July 2025; outages,
then a hand rewrite. Heading:
"Turning the lights back on"
Osmani
Keep humans on auth, billing,
public APIs
Mollick, "Twilight Factory"
Agents do most of the work.
A facilitator agent decides when
to bring a person in: approval,
missing expertise, new ideas.
Says StrongDM-style minimal human
involvement is wrong for most orgs.
StrongDM
"Code must not be written by humans."
Humans don't review it either.
Shapiro L5
"Dark factory", from FANUC's
lights-out robot plant
Osmani's rule: run a loop dark only if its check is cheap, hard to fake, and stable.
The vocabulary. "Dark factory" is a manufacturing term: a plant where machines do so much that you can turn off the lights; Shapiro cites FANUC. StrongDM is the radical primary example: humans neither write nor review code; holdout scenarios, a probabilistic "satisfaction" score and a Digital Twin Universe of cloned services (Okta, Jira, Slack) do the checking, and they say spending under $1,000 of tokens per engineer per day means room for improvement. Dex Horthy's essay says plainly that the lights-off factory does not work: HumanLayer went full lights-off in July 2025 and by about the third outage in November a cofounder rewrote it by hand; his section heading is "Turning the lights back on", and he aims for a safe 2–3x instead of 10–100x. Mollick's "Twilight Factory" sits between: agents do most of the work but proactively reach out to humans, with a facilitator agent deciding when. Osmani turns this into a design rule per loop, and warns that going dark builds comprehension debt while tests stay green. Yegge's later Gas City post also says it keeps the lights on and maximizes observability. Part 4 returns to this debate.
1 · Define · Why now
Long-running agents and cheap sandboxes arrived in 2025; PR volume followed
Models hold long tasks METR: doable task length doubles ~every 7 months. OpenAI: single Codex runs of 6h+.
Sandboxes are cheap and fast Stripe: a devbox in ~10 s. Ramp: full stack per agent on Modal.
Harnesses run headless Claude Code SDK, Codex exec: a pipeline calls the agent like a build step.
1,300+
Stripe: minion-produced PRs merged per week
70%+
Uber: PRs attributed to local or cloud agents
~50%
Ramp: merged PRs started by its Inspect agent
~1M
OpenAI: lines in 5 months, none written by hand
Three enablers. (1) Long-horizon models: METR measures the human-time length of tasks models finish 50% of the time; it has doubled about every 7 months (reportedly faster lately, unverified). Karpathy called Claude Code the first convincing LLM agent. (2) Cheap isolation: Stripe spins up a devbox in about 10 seconds from a pre-warmed pool with full permissions inside; Ramp runs full environments on Modal with VNC and Chromium for UI checks, rebuilt every 30 minutes. (3) Headless harnesses with SDKs, hooks and subagents, so a scheduler can call the agent like any other build step. Numbers, with caveats. Stripe: 1,300+ minion-produced PRs merged per week in Part 2 (1,000+ in Part 1), still human-reviewed. Uber: 70%+ of PRs attributed to agents (Aug 2026), up from 11% opened by agents in March per Pragmatic Engineer; attribution is not autonomy. Ramp: about 50% of merged PRs started by Inspect per Modal (about 30% "written" in Ramp's own earlier post). OpenAI: about 1M lines and about 1,500 PRs by initially three engineers. These are self-reported by the companies. The productivity evidence (METR's RCT showing a 19% slowdown, DORA's stability findings) is in Part 4.
2
Blueprint: the whole line, then each station
Eleven stations. For each: what it does, which technology runs it, one research finding, and what is still open.
This part first shows the whole map on one slide, then walks the line station by station. A small locator strip at the top of every slide shows where we are on the line. The station list comes from our survey of about 130 sources; it is a synthesis, not a standard that any one company publishes.
Human Agent Machine / CI Gate Artifact Feedback loop COLOR = WHO TYPICALLY RUNS THE STATION WHAT TO BUILD BUILD CHECK AND SHIP 01 HUMAN Intake tickets Slack threads prod alerts 02 ARTIFACT Spec requirements design task list 03 ARTIFACT Context AGENTS.md docs, MCP skills 04 MACHINE Orchestrate task graph loops fan-out 05 MACHINE Sandbox VM / devbox worktree service fakes 06 AGENT Harness model + tools Claude Code Codex, goose 07 MACHINE Verify tests, CI LLM judge scenarios 08 GATE Review merge gate human or AI risk tiers 09 MACHINE Ship canary feature flags telemetry 10 Feedback incidents, errors, failed runs production signals new tickets fix the factory 11 Govern under every station: agent identity with a human owner · permissions · token budgets · delivery metrics
This is the anchor diagram; every later slide in this part has the same strip at the top. Read it left to right. Intake turns a request into a task; spec and context are the documents the agent works from; orchestrate, sandbox and harness are the build machinery; verify, review and ship decide what reaches users. The amber loop is the part most teams skip: production signals become new tickets, and failures become fixes to the factory itself (AGENTS.md, a lint rule, a test). Govern sits under everything: who each agent acts for, what it may touch, what it costs, and how we measure it. Colors show the typical owner across the cases we read, which is our reading; real factories vary. Osmani describes the same shape as many harnessed loops fed by a queue and passed through a review gate. HumanLayer traces the pre-agent version (tracker, build, PR, review, ship, monitor, feedback) and notes that once build becomes an agent, review becomes the bottleneck.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern L3 human reviews diffs 6 cases 6/6 2/6 n/c 1/6 n/c 0/6 1/6 6/6 1/6 0/6 n/c L4 human writes specs 5 cases 5/5 5/5 n/c 0/5 n/c 0/5 0/5 2/5 0/5 2/5 n/c L5 no human reads code 3 cases 2/3 2/3 n/c 0/3 n/c 0/3 0/3 0/3 0/3 0/3 n/c human in most cases in some in none n/c = not coded in the case swimlanes. Counts are our reading of each case write-up.
Method. For each of 14 cases we drew a swimlane from the public write-up and marked who runs each station: human, agent, machine, or not described. A cell counts a case if a human runs that station, alone or with an agent. Levels follow Shapiro: L3 human as manager reviewing diffs, L4 human as PM writing specs, L5 dark factory. Placing a company on a level is our reading, not their claim. L3 (Stripe, Ramp, Dropbox, Uber, Cloudflare, HumanLayer): humans start every run and approve every merge. L4 (OpenAI, Huntley Ralph, Spotify, Kiro Crew, Airbnb): every case has a human writing the spec, and review is human in only 2 of 5. L5 (StrongDM, Gas Town, Carlini compiler): humans write the spec or scenarios and do nothing downstream; StrongDM states humans do not review code. Harness is never human: in every case the agent writes the code. Context, sandbox and govern were not coded per case, so they show n/c. Caveat: three L5 cases is a tiny sample, and the write-ups describe the happy path.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern 01 INTAKE: WHAT STARTS A RUN 02 SPEC AND PLAN Slack @bot Stripe, Ramp Linear ticket OpenAI Symphony Prod alert Datadog Bits Fleet change Spotify, Uber Task small, clear Requirements WHAT Design / plan HOW Task list STEPS TOOLS GitHub Spec Kit · AWS Kiro (EARS) HumanLayer ACE · StrongDM NLSpec Human writes or reviews the spec; the agent implements from it. What it does Turns requests into small tasks, then writes what to build, and how, as files.
Key finding Thoughtworks sorts tools into spec-first, spec-anchored and spec-as-source; heavy spec tools are overkill for small fixes.
Open question How detailed? Too thin and agents guess; too thick and it is just slower code.
Intake. Four common triggers: tag a bot in Slack and get a PR back (Stripe Minions, Ramp Inspect); an orchestrator watches a Linear board and starts one run per ticket (OpenAI Symphony); production errors open draft PRs with no human request (Datadog Bits, which still never auto-merges); a platform team defines one change and fans it out to many repos (Spotify Fleetshift, Uber). The pre-AI idea here is Toyota pull: make only what is needed, so work is not pushed in blindly. Spec. Spec Kit runs constitution, specify, plan, tasks, implement. Kiro is an IDE built on requirements (EARS notation), design and tasks. HumanLayer ACE is research, plan, implement, with a human reviewing the plan. StrongDM publishes the natural-language spec itself and lets any agent build it. Böckeler (Thoughtworks) warns that spec-as-source risks combining the flaws of model-driven development and LLMs. Open problem: splitting messy human requests into agent-sized tasks is still mostly manual.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern AGENTS.md ~100-line map of the repo ALWAYS docs/ in the repo design, decisions, kept fresh by lint WHEN RELEVANT MCP tools Stripe Toolshed ~500 · Uber 1,000+ WHEN CALLED Skills instructions + scripts · Uber 3,600+ ON DEMAND CONTEXT WINDOW 40% 60% target Compaction: summarize research into a file, then start a fresh window (ACE). What it does Gives the agent what a new hire would need, loaded only when relevant.
Key finding OpenAI: give Codex "a map, not a 1,000-page instruction manual"; the docs folder is the source of truth.
Open question Docs and AGENTS.md drift from the code, and nobody can cheaply tell which context is stale.
Four layers, from always-on to on-demand. OpenAI keeps AGENTS.md to about 100 lines that point into a structured docs/ directory, kept fresh by linters, CI and a doc-gardening agent. Mitchell Hashimoto adds a line to AGENTS.md each time an agent makes a mistake. Cloudflare generated short AGENTS.md files for about 3,900 repos. Tools come through MCP gateways: Stripe Toolshed has about 500 tools, Uber reports 1,000+ behind one gateway. Skills are folders of instructions and scripts loaded only when needed; Uber reports 3,600+. The gauge is the HumanLayer ACE rule of thumb: keep context use around 40 to 60 percent and compact often. Huntley makes the same point for Ralph: the less of the window you use, the better the result. Anthropic frames the window as a scarce budget, filled by just-in-time retrieval, compaction and notes. Open problem: context drifts from code silently.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern Loop Huntley Ralph Coordinator Gas Town + Beads Claude Code teams Graph runner Stripe Blueprints StrongDM Attractor Agent same prompt, fresh context each turn spec + plan + git Mayor W1 W2 W3 task graph in git lint implement test fix CI code owns the flow; agents are boxed nodes No orchestrator at all: 16 agents claim tasks with lock files in git (Carlini). What it does Splits work into a task graph and keeps agents running on it.
Key finding Osmani, "The Orchestration Tax": adding agents is easy; human attention does not parallelize, so it becomes the bottleneck.
Open question How much control flow to give the model versus fixing it in code is still taste, not evidence.
Three shapes. The Ralph loop is a bash while loop that feeds the same prompt to the agent again and again; state lives in the spec, the plan and git, not in the context window. Huntley says it is for greenfield code only. The coordinator shape is Gas Town: a Mayor agent reads workers' output and talks to you, and tasks live in Beads, a git-backed dependency graph. Claude Code subagents and agent teams use a lead agent with a shared task list. The graph runner puts control flow in code or a DOT graph; Stripe Blueprints mix deterministic nodes (lint, push) with agent nodes (implement, fix CI), and StrongDM Attractor runs a Graphviz pipeline. A fourth, very common shape is kanban plus worktrees: one card, one worktree, one agent, the human manages the board (Vibe Kanban, Conductor). Carlini ran 16 parallel agents with no orchestrator, claiming tasks by writing lock files. Anthropic's "Building effective agents" advises the simplest pattern that works.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern Sandbox, one per agent Agent repo branch Postgres, Redis, browser Digital twins of Okta, Slack, Jira behavior clones of outside services (StrongDM) full permissions, no prompts inside (Stripe) Production Secrets Open internet blocked by default CHEAP, LOCAL STRONGER, MORE REALISTIC git worktree Conductor devbox, ~10 s Stripe full-stack VM Ramp on Modal microVM Firecracker What it does Gives each agent a safe copy of the dev environment, plus fakes of outside services.
Key finding Stripe names the key trait as "the absence of a supervisory human": isolation is what makes that safe.
Open question Keeping sandboxes realistic (data, services, secrets) without making them dangerous or costly.
Why isolation comes first. Without it you either approve every command, or risk an agent touching production and secrets. With it, Stripe runs agents with full permissions and no confirmation prompts inside pre-warmed devboxes that start in about 10 seconds and are cut off from production and the internet. Ramp Inspect runs on Modal: a VM with Postgres, Redis, Temporal, RabbitMQ, a browser and VNC, from a repo image rebuilt every 30 minutes, so the agent can check its own UI with screenshots. StrongDM built behavioral clones of six outside services (Okta, Jira, Slack, Google apps and others), refined against the real ones "until we stop finding behavioral differences." At the cheap end, git worktrees give one folder and branch per agent on a laptop (Conductor). At the strong end, Firecracker microVMs from AWS start fast and isolate strongly; many sandbox products build on them. Jake Saunders shows a small self-hosted version that needs no platform team.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern task + context Model picks next step TOOLS read, search edit files run commands run tests call result done? diff + evidence Claude Code / Agent SDK · Codex CLI · goose (forked by Stripe) OpenHands · mini-swe-agent: about 100 lines, bash only What it does The coding agent itself: a model that reads, edits, runs and tests code in a loop.
Key finding SWE-agent (Princeton): designing the agent's commands for search, view and edit raised results without changing the model.
Open question Training rewards tests passing; "there is no penalty for eroding codebase maintainability" (HumanLayer).
The harness is thinner than people expect. Thorsten Ball (Amp) builds a working agent in under 400 lines: a model, a loop and a few tools. mini-swe-agent does it in about 100 lines with bash as the only tool, a useful baseline. Most factories put an off-the-shelf harness at this station: Claude Code or its Agent SDK (headless mode, hooks, subagents), Codex CLI (the worker in OpenAI Symphony), goose (Stripe Minions run a fork), OpenHands. Spotify moved from Goose, Aider and a homegrown loop to Claude Code. SWE-agent coined the agent-computer interface: the shape of the commands and their output matters, not only the model. OpenAI's "harness engineering" widens the term to linters and structural tests around the agent. The open problem is in the model, not the harness: HumanLayer argues RL rewards and benchmarks score pass or fail only, so models drift toward code that passes but is hard to maintain, and there is no fast oracle for maintainability yet.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern Format, lint, types Tests + CI, max 2 rounds (Stripe) LLM judge vs task (Spotify) Holdout scenarios (StrongDM) Proof: CI, video COSTLY, FEW, HARD TO GAME Hidden scenarios kept outside the repo Agent fixes and retries fail retry CHEAP, RUN ON EVERY CHANGE What it does Checks the agent's work by machine before any human looks at it.
Key finding Carlini (Anthropic): make sure "the task verifier is nearly perfect, otherwise Claude will solve the wrong problem."
Open question Tests prove behavior, not maintainability or intent, and an LLM judge can share the agent's blind spots.
Read the stack bottom up. Formatters, linters and type checks are cheap and run on every change. Tests and CI come next; Stripe lets the agent fix CI failures but caps it at two rounds, because returns diminish and tokens cost money. Spotify Honk runs verifiers in a stop hook before any PR and then an LLM judge compares the diff to the task; Spotify reported the judge vetoed about a quarter of sessions, but we could not verify that figure, and a later report says the judge was removed. StrongDM calls them scenarios, not tests, because agents can rewrite tests to match the code; scenarios live outside the repo like a holdout set and give a probabilistic satisfaction score. OpenAI Symphony asks for proof of work on every PR: CI results, review notes, a video. Carlini used GCC as a known-good oracle for his C compiler, and wrote test output for the agent: a few lines on screen, details in grep-able log files. Pre-AI ideas that apply: mutation testing (test your tests) and Toyota jidoka (stop on an abnormality so defects do not flow downstream). Cognition's Frontier Code benchmark tries to measure whether a maintainer would merge, beyond pass or fail.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern PR Risk triage trivial · lite · full Cloudflare Human approves Stripe, Google, Datadog AI review + human Cloudflare 100%, Uber uReview No human review StrongDM: scenarios instead Merge Ramp: PRs open with the requester's own token, so nobody approves their own agent's change. What it does Decides which changes merge, with stricter rules for riskier code.
Key finding Google's report on AI-driven code migrations finds human review becomes the bottleneck even when generation works.
Open question AI reviewers raise the floor, not the ceiling (HumanLayer). No review UX fits agent volume yet.
Three positions on the gate. Stripe merges over 1,000 fully agent-written PRs a week, and every one is still human-reviewed; Google and Datadog also keep human approval and no auto-merge. Cloudflare runs an AI code reviewer on 100% of repos on its standard CI: it sorts each merge request into trivial, lite or full risk and sends it to specialist agents for quality, security, docs and performance. Uber scores its uReview on precision and recall against real PRs with known bugs. StrongDM is the dark end: humans neither write nor review code, and trust comes from holdout scenarios. Ryan Carson's Code Factory recipe chooses checks by risk and requires machine-checkable evidence. Osmani warns that going dark builds comprehension debt while tests stay green, and suggests keeping humans on auth, billing and public APIs. Faros AI reports that when review is skipped under AI volume, incidents per PR rise sharply (their figure is +242.7%, with 31.3% of PRs merged without review); we could not verify these numbers ourselves.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern 09 SHIP: SMALL BLAST RADIUS Merged passed the gate Feature flag merge dark, ramp up Canary small share first Full rollout all traffic Telemetry errors, latency Stop the line roll back, call a human 10 FEEDBACK Error to draft PR Datadog Bits, Sentry (Ramp) to 01 intake Fix the factory AGENTS.md, skill, lint, test to 03 context Andon: any abnormal signal stops the line and pulls in a human. What it does Ships merged changes safely, watches production, and routes failures back upstream.
Key finding DORA 2025: AI use goes with higher throughput but lower delivery stability.
Open question Who is on call for code no human wrote or read, and when should an agent stop the line?
Ship. At ten times the change volume, production is where bad changes show up first, so blast radius must be small: merge behind a feature flag (Ramp uses LaunchDarkly), send a small share of traffic to the new version, compare, then widen (Google SRE canarying). Charity Majors argues that at 10 to 100x change rates, rigor has to move into production observability; telemetry is the spec that survived. OpenAI reports agents that watch CI, deploy and monitor, with review by risk. DORA 2025: AI correlates positively with throughput and negatively with delivery stability; "AI doesn't fix a team; it amplifies what's there." Feedback. Two paths. Errors in Sentry or Datadog trigger an agent to propose a fix (Datadog Bits never auto-merges). And when an agent fails, fix the factory: Hashimoto adds a line to AGENTS.md for each failure; Tessl Agent mines PRs and session logs to ship improved workflows. Google SRE blameless postmortems are the pre-AI habit. Mollick proposes a facilitator agent that knows when to call a human.
Intake Spec Context Orchestrate Sandbox Harness Verify Review Ship Feedback Govern Agent identity one per agent Human sponsor accountable owner Ramp Risk tier autonomy by domain less where errors cost Token budget cost per agent (Uber) $1,000/eng/day (StrongDM) Metrics throughput + stability DORA, SPACE Uber spend = users × sessions × turns × requests × tokens × price What it does Controls who each agent acts for, what it may touch, what it costs, and how it is measured.
Key finding METR RCT: 16 experienced developers were 19% slower with AI, while believing they were 20% faster.
Open question PR counts and "AI code share" use different definitions, and few report cost or quality.
Four controls around each agent. Ramp gives every agent its own identity with a responsible human sponsor. Risk tiers allow more autonomy where errors are cheap and less where they are costly. Uber breaks spend into users, sessions, turns, requests, tokens and price, and puts a lever on each factor (model routing, compaction at 400K tokens, a one-hour cache TTL, code mode); it reports cost per session down 52% from its June 2026 peak and tracks value per agent, not per seat. StrongDM treats under $1,000 of tokens per engineer per day as room to improve, which Simon Willison questions. Cloudflare routed 241 billion tokens through its AI Gateway in 30 days. Measure delivery with DORA (throughput and stability together) and SPACE (no single metric, activity counts mislead). METR's 2025 RCT (16 developers, 246 tasks) is the warning about self-reported gains; a 2026 follow-up was judged unreliable because people refused to work without AI. Kiro Crew advises audit logs and permission control from day one.
2 · Blueprint · Summary
Cheap generation moves the jam upstream to spec and downstream to review
INTAKE SPEC CONTEXT BUILD VERIFY REVIEW SHIP Before agents humans write code BOTTLENECK With agents generation is cheap BOTTLENECK BOTTLENECK agents in parallel BOTTLENECK BOTTLENECK Upstream: is this the right thing? Downstream: is it right? HumanLayer: aim for a safe 2–3x, not 10–100x, and invest where the line is slowest.
This is the one-line summary of Part 2. Before agents, writing code was the slow station and everything else waited on it. Once build is an agent that runs in parallel, the line jams at both ends. Downstream, someone has to decide whether the output is right: HumanLayer says review becomes the bottleneck in the agentic factory, and Osmani says verification, not generation, is the real constraint. Upstream, someone has to decide what to build and give the agent enough context: Spotify says the constraint moved to human decisions on what to build and review, and Maggie Appleton describes the bottleneck moving from implementation to design. Kiro Crew's five stages of solo work show the bottleneck moving at each stage. HumanLayer's advice is theory of constraints: a safe 2 to 3x beats chasing 10 to 100x, so spend effort on the jammed stations, not on more generation. Part 6 takes the same idea one step further, beyond engineering. Note: the Spotify post was not independently verified in our survey.
3
Cases: how real factories are wired
Eight landmark factories drawn as swimlanes, six more at a glance, then what they share.
Part 3 is reporting, not opinion. Each case slide uses the same swimlane: lanes are who does the work (human, agent, machine), boxes are colored by role, amber dashed arrows are loops (retry or feedback). The title of each slide says what that factory is in one sentence. The level (L3, L4, L5) on each slide is our reading of the published sources, using Dan Shapiro's Five Levels. All numbers are self-reported by the companies or authors unless the notes say otherwise.
L3 human reviews diffs
L4 human writes specs
L5 nobody reads the code
LESS HUMAN READING
Big-company
platform
Startup /
small team
Individual
Research lab
Stripe Minions
Ramp Inspect
Uber
Dropbox Nova
Cloudflare
OpenAI harness + Symphony
Spotify Honk
Kiro Crew (AWS)
Airbnb test migration
HumanLayer, 2026
StrongDM
Ralph loop (Huntley)
Gas Town (Yegge)
C compiler (Anthropic)
HumanLayer, Jul 2025 trial
lights back on after outages
Blue dot: has its own slide next. Grey dot: in the table at the end. Levels follow Shapiro's Five Levels.
How to read it. x is Shapiro's level: L3 means a human reviews the AI's diffs, L4 means the human writes specs and checks outcomes, L5 is the dark factory where no human writes or reads code. y is who built the factory. The placement is our reading of each public write-up, not a label the companies gave themselves. Every big-company platform we found keeps a human or a strict machine gate before merge, so they sit at L3 or L4. The three L5 cases are a three-person team at StrongDM, one person (Steve Yegge), and a research demo (Nicholas Carlini at Anthropic). That matches Shapiro's own remark that only a handful of teams under five people work at Level 5. HumanLayer is the counter-example: they ran lights-off from July 2025, then after about the third outage in November a cofounder spent two weeks rewriting by hand, and they moved back to a reviewed process. Shapiro's model itself is not a case and is left off the map.
Human
Agents
(Attractor)
Digital twins
+ LLM judge
Writes
NLSpec
Holdout
scenarios
Two models
plan the sprint
Writes code,
iterates
Runs on
digital twins
LLM judge:
satisfaction
hidden from agents
low score → iterate
STACK
Attractor Graphviz DOT pipelines Digital Twin Universe cxdb Claude Code / Codex / …
$1,000
tokens per engineer per day, their floor
6
services cloned: Okta, Jira, Slack, Google
3
people on the team (Oct 2025)
“Code must not be written by humans.”factory.strongdm.ai
What it is. StrongDM's AI team (Justin McCarthy, Jay Taylor, Navan Chauhan, founded July 2025) runs on two rules: humans do not write code, and humans do not review code. Flow. A human writes a natural-language spec (NLSpec) and end-to-end user-story scenarios. The scenarios live outside the repo like an ML holdout set, because agents can rewrite tests to match their code; that is why they say "scenario" and not "test". Sprint planning is a "consensus" of two models whose independent plans an LLM merges. Attractor, their non-interactive agent, runs Graphviz DOT pipelines whose nodes are LLM tasks, human gates or fan-outs, checkpointed after each node. The software runs against the Digital Twin Universe, Go-binary clones of Okta, Jira, Slack and Google Docs, Drive and Sheets, refined "until we stop finding behavioral differences", at thousands of scenarios per hour. An LLM judge estimates satisfaction: the share of observed runs that likely satisfy the user. A public Weather Report tracks which model they use for each role. Numbers. If you spend under $1,000 of tokens per human engineer per day, they say the factory has room to improve. The Willison write-up drew 304 points and 459 comments on HN. Caveats. No output or satisfaction figures are published, so results cannot be checked from outside. Willison and HN critics doubt that LLM judges and agent-written specs can fully verify intent, and HN reviewers found weak Rust in the released cxdb. A StrongDM engineer said on HN the company was under an acquisition agreement. L5 is our reading of their stated rules.
Engineer
Minion
(goose fork)
Devbox + CI
Tags bot
in Slack
Warm devbox
in ~10 s
Fixed step:
load context
Implements,
runs lints
CI, at most
2 rounds
PR from
template
Human
reviews PR
push
fail → one retry
re-prompt
STACK
goose (internal fork) Toolshed MCP EC2 devboxes Slack rule files Stripe CI
1,300+
minion PRs merged per week
~10 s
devbox from a warm pool
Blueprints mean “putting LLMs into contained boxes.”Stripe, Minions part 2
What it is. A fleet of unattended agents from Stripe's Leverage team (Alistair Gray). The defining trait is "the absence of a supervisory human": no confirmation prompts, full permissions, but only inside an isolated devbox. Flow. An engineer tags the Slack app in a thread (CLI, web, docs, feature-flag and ticket UIs and a flaky-test button also work). A pre-warmed EC2 devbox with a QA environment, no production data and no internet is ready in about 10 seconds. Deterministic MCP calls through Toolshed load the linked docs and tickets before the agent starts. The goose-fork agent implements the task, guided by scoped rule files, and runs local linters in a loop. On push, CI runs, autofixes apply, and remaining failures go back to the agent for one more try. A PR is opened from Stripe's template, and a human reviews, re-prompts or edits it. Blueprint. Each run mixes deterministic nodes (lint, push) with agentic nodes (implement, fix CI). CI is capped at two rounds because of diminishing returns and the token, compute and time cost. Numbers. Over 1,000 merged PRs a week were "completely minion-produced" in part 1, 1,300+ in part 2; Toolshed grew from 400+ to about 500 tools. Stripe CI has over 3 million tests. Caveats. The models behind the goose fork are not named. PR counts are self-reported and say nothing about PR size or revert rate. Every PR is still reviewed by a human, so this is L3, not a dark factory.
Migration
owner, team
Claude Code
in a pod
Fleet
Management
One prompt
per change
Fleetshift
picks repos
Claude Code
applies it
Verifiers:
build, test
LLM judge
Opens PR;
safe → merge
Team reviews
the rest
per repo
pass
fail → fix
veto → correct
STACK
Claude Code Claude Agent SDK Kubernetes Fleetshift Backstage MCP
1,500+
agent PRs merged (Nov 2025)
~1 in 4
sessions vetoed by the LLM judge
60–90%
time saved on migrations
Prompts should “describe the end state”, one change each.Spotify Engineering, part 2
What it is. Fleet Management (Fleetshift) already did fleet-wide changes with deterministic scripts; the Maven updater script alone had grown past 20,000 lines. Spotify kept the plumbing (targeting, PRs, reviews, merging) and swapped only the transform step for an agent. They moved from Goose, Aider and a homegrown loop to Claude Code. Flow. A migration owner defines the change as a prompt in Fleet Management, or someone mentions Honk in Slack or GitHub. Fleetshift picks in-scope repos (Backstage lineage pages, Codesearch) and schedules the rollout. Honk runs Claude through the Agent SDK in a Kubernetes pod with minimal tools: verification via MCP, restricted git, allow-listed bash. Verifiers (format, build, test) and an LLM judge must pass in a stop hook, otherwise no PR is opened. The judge vetoes about a quarter of sessions, and the agent course-corrects about half the time. Safe maintenance PRs auto-merge; others go to the owning team, which reviews (and for data pipelines tests by hand) and merges. Progress shows per repo in Backstage. Prompts. Describe the end state, one change per prompt, kept static and version-controlled. Numbers. Fleet Management has automated about half of Spotify's PRs since mid-2024; a June 2026 talk summary reports 2.5M+ automated maintenance PRs. The claim there: coding is no longer the bottleneck; human decisions on what to build and review are. Caveats. In their words they flew "mostly by intuition" on prompt quality, and the judge had no evals. InfoQ reported the LLM judge was later removed; we did not check that. The 2.5M figure and the auto-merge share come from a talk summary with no named author. L4 is our reading, because most maintenance PRs merge with no human.
Human
Codex
Symphony +
repo harness
Writes Linear
ticket
Symphony
claims ticket
Reads map,
reproduces
Implements,
opens PR
Custom lints
+ CI
Agent
reviews
Approves
proof of work
land skill
merges
WORKFLOW.md
error says the fix
until satisfied
STACK
Codex app-server Symphony (Elixir) Linear AGENTS.md + docs/ git worktrees custom linters
~1M
lines in 5 months, none by hand
~1,500
PRs merged, started with 3 engineers
3.5
PRs per engineer per day
“Humans steer. Agents execute.”OpenAI, "Harness engineering"
What it is. Ryan Lopopolo's Codex team built a product of about a million lines in five months with no line written by hand, including tests, CI and docs. The work went into the repo as a harness: something Codex can read, run and police. Symphony is the daemon that turns tickets into Codex runs. Map, not manual. A roughly 100-line AGENTS.md points into a structured docs/ directory, kept fresh by linters, CI and a doc-gardening agent. Architecture and taste are enforced mechanically: fixed layers (Types to UI) checked by Codex-written custom lints whose error messages tell the agent how to fix the problem. Flow. A human writes a Linear ticket with acceptance criteria. Symphony polls the board and claims it (10 agents by default), creates a per-issue workspace and renders WORKFLOW.md, the in-repo prompt and config, into the first turn. Codex reads AGENTS.md and docs/, reproduces the bug in an app booted per worktree with Chrome DevTools and local logs and metrics (LogQL, PromQL), implements, records before and after videos, opens a PR, and loops on its own and other agents' reviews until they are satisfied. The ticket moves to Human Review with proof of work (CI, review feedback, complexity analysis, walkthrough video). After approval the agent runs the land skill to merge. Background Codex tasks scan for drift from "golden principles" and open small refactor PRs, mostly automerged. Single runs often last 6 hours or more, while humans sleep. Symphony is shipped mainly as a language-agnostic SPEC.md you ask an agent to implement. Caveats. OpenAI warns the result depends on this repo's tooling and long-term coherence is unknown. The product is unnamed. The team later grew to seven. openai.com blocked direct fetch, so the post was read through a text mirror. Talk-only figures were left out. L4 is our reading: humans write tickets and approve outcomes.
Human
16 Claudes
in Docker
Git repo +
test harness
Goal + test
harness
Loop spawns
container
Claims task
by lock file
Fixes
failing tests
Tests +
GCC oracle
Merge, push
upstream
Specialists:
dedupe, perf
the verifier
fail → fix
next session
STACK
Claude Opus 4.6 Claude Code Docker git lock files GCC torture tests Rust
~2,000
sessions over two weeks
<$20k
API cost (2B in, 140M out tokens)
100k
lines; boots Linux 6.9
Make sure “the task verifier is nearly perfect, otherwise Claude will solve the wrong problem”.Carlini, Anthropic
What it is. Nicholas Carlini (Anthropic Safeguards) ran 16 Claude Code loops in Docker containers sharing one git repo, with no orchestrator: "I don't use an orchestration agent." The test harness and GCC as a reference oracle did the steering. Flow. The human set the goal (from scratch, GCC-compatible, builds Linux, SSA IR) and built the test harness and CI. An infinite loop spawns a fresh Claude Code session in a new container that clones the bare upstream repo. The agent orients from READMEs and progress files and claims the next obvious task by writing a lock file into current_tasks/; git resolves races. It fixes failing tests. For the Linux kernel the harness compiles most files with GCC and a random subset with Claude's compiler to isolate bugs. Then it pulls, merges others' work, pushes, releases the lock, and the loop starts the next session. Specialist agents run alongside: dedupe code, compiler speed, output code quality, Rust design critique, documentation. Harness advice. Write it for Claude, not yourself: print a few lines, log details to files with grep-able ERROR lines, offer a deterministic --fast 1% or 10% sample. Result. About 100,000 lines of std-only Rust that boots Linux 6.9 on x86, ARM and RISC-V, for under $20,000 of API cost. Caveats. This is a research capability demo on a well-specified greenfield problem with a perfect oracle, not a production team. Output code is slower than unoptimized GCC, and near the end new features kept breaking old ones. Carlini himself is uneasy that passing tests can look like a finished job. The productized Claude Code agent teams feature differs from this bare harness. L5 is our reading: no human reviewed the code.
Human
Claude
(one loop)
Repo files
+ tests
Talks out
requirements
specs/ +
fix_plan.md
Same prompt,
fresh context
Picks one
item, builds
Tests +
type checks
Commit,
update plan
Adds signs,
resets
LLM writes specs
every loop
fail → fix
drop context, restart
tune the prompt
STACK
Claude Code or Amp PROMPT.md specs/* fix_plan.md AGENTS.md git
~170k
tokens of context; use less
1
process runs build and tests
1,768
stars on the playbook repo
while :; do cat PROMPT.md | claude-code ; done Huntley, "Ralph"
What it is. "In its purest form, Ralph is a Bash loop": the same PROMPT.md is fed to the agent again and again, each time with a fresh context. Specs and a plan file on disk are the memory, and tests and type checks are the backpressure. Flow. Hold a long conversation with the LLM about requirements, then have it write one spec file per topic into specs/. A planning loop has subagents compare specs against src/ and write a prioritized fix_plan.md (TODOs, placeholders, gaps). Each building loop reads specs and plan, picks the most important item, searches first ("don't assume not implemented"), and implements. Tests and type checks for that unit run; fix_plan.md and AGENT.md get updated with learnings. When green: commit, push, tag; the context is thrown away and the loop restarts. The human watches the stream, adds a "sign" to the prompt for each bad pattern, and resets with git reset --hard or a new plan when it derails. Design choices. One thing per loop. Of about 170k tokens of context, the less you use the better. No multi-agent mesh: one process in one repo; up to 500 subagents may search or write, but only one runs build and tests. Limits, in his words. Senior engineers are still needed, and it is a greenfield tool: "There's no way in heck would I use Ralph in an existing code base." Caveats. No measured outcomes beyond anecdotes (the CURSED language, a contractor's ROI). The how-to-ralph-wiggum README reads as a community playbook rather than Huntley's own text. L4 is our reading: the human writes and tunes the spec and prompt, not the code. Carlini and Symphony run this same loop in parallel; Gas Town on the next slide scales it into a town.
Human
Mayor +
Polecats
Beads ledger
+ Refinery
Tells Mayor
the goal
Mayor splits
the work
Beads tickets
in git
Polecats in
worktrees
Witness,
Deacon watch
Refinery
merge queue
Merged
to main
dispatch
gt done
logged to the ledger
stuck → escalate
STACK
Gas Town (Go, gt) Beads (bd, Dolt) git worktrees tmux Claude Code default
20–30
agents it claims to run comfortably
17 days
to build it: 75k LOC, 2,000 commits
$30k/yr
SaaS tool a customer rebuilt on it
A dark factory “just means background work”.Yegge, "Welcome to Gas City"
What it is. Steve Yegge's "town" of 20 to 30 Claude Code-style agents. You chat only with the Mayor, which reads the workers' output and talks to you like a chief of staff, so you read less. Flow. The Mayor turns your request into beads: git-backed issues with a dependency graph, bundled into Convoys; reusable workflows are TOML formulas. Polecats, workers with a persistent identity but short-lived sessions, are dispatched, each in a git-worktree "hook" that survives crashes. The Witness watches the polecats of one project, the Deacon patrols all of them, and Dogs do maintenance; blocked agents escalate by severity to the Mayor or the human. Finished work goes via gt done to the Refinery, a Bors-style bisecting merge queue that runs verification gates and merges to main. Every step is logged on the Beads and Dolt ledger, so later agents can "seance" their predecessors for context. The durable idea is the task ledger: work state, dependencies and the reason for each piece of work live in git and Dolt, not in agent memory. Beads is "the Why". Yegge also says never let one agent manage infrastructure alone; use crews of two or three, since reliability is a dial. Numbers. Beads and Gas Town have 28,129 and 18,485 GitHub stars (2026-10-23). The 17-day build figure is Yegge's, as cited by Appleton. Caveats. The original January 2026 post returned 403, so claims like "100% vibe-coded, never read the code" come via Appleton and HN. Appleton's counterpoint: once agents write fast, design and taste become the bottleneck. A 2026-04 issue alleged Gas Town used users' LLM credits for self-improvement (closed). L5 reflects Yegge's own no-code-reading stance; whether users read their code varies. Its successor Gas City "keeps the lights on" and maximizes observability.
Engineer +
reviewer
Claude Code
Triage by
task size
Drafts
the specs
Product +
arch review
Program
design review
Builds 1–3
slices
Reads code,
resteers
medium, large
~40% one-shot
redraft
next slice
STACK
Claude Code research / plan / implement prompts sub-agents Mermaid diagrams
2–3x
safe speedup, instead of 10–100x
~40%
tasks one-shot to the agent
2 weeks
hand rewrite after lights-off
“Read the dang code”Horthy, "Why Software Factories Fail"
What it is. The most cited failure report of a lights-off factory. In July 2025 HumanLayer went fully lights-off; by about the third outage in November they rewrote by hand, a cofounder spending two weeks on it. Agents still do the building, but humans align on product, architecture and program design first, steer in small vertical slices, and read the code. Why it failed, in their reading. RL rewards and benchmarks score pass or fail only: "there is no penalty for eroding codebase maintainability". Flow. Triage by size: about 40% of tasks are one-shot to the agent; medium tasks get one combined plan doc; large tasks get every phase. Product review: a short doc on the user problem and success signal, with rough HTML mockups. System architecture: services, endpoints and schemas as sequence diagrams, contract shapes and SQL. Program design: call-stack trees, file-tree diffs, key types and signatures; the model drafts, the human argues with it. The PR's future reviewer goes through the specs with the author first (author opt-in, async doc comments). The agent then builds one to three vertical slices at a time (API with mock data, then UI, then services, then DB), and a human reviews and resteers each slice. Earlier method (ACE). Research, plan, implement, with frequent intentional compaction to keep context use around 40-60%; with it they shipped 35k lines of features into BAML, a 300k-line Rust codebase, in about 7 hours. Caveats. Outage and rewrite details are self-reported with no public post-mortem, and the author sells tooling for this approach (disclosed). HN critics reply that the July 2025 trial predates newer models. L3 is our reading.
Case Trigger Factory shape Gate Headline number Level
Uber Engineer, CI failure, alert, migration Model and MCP gateways, DevPods, any harness uReview on every PR, then a human 70%+ of PRs attributed to agents L3 Ramp Inspect Slack, web, Chrome, PR comment Full dev env in a Modal sandbox Agent self-verifies; a teammate reviews ~30% of merged PRs, later ~50% L3 Dropbox Nova UI, CLI, Slack, flaky-test bot Any agent on a repo snapshot, one branch CI loop, up to 5 tries; owner reviews 100+ CI runs to accept a flaky fix L3 Cloudflare Engineer, keyless OpenCode login Own products turned inward Risk-tiered AI reviewer on every MR Weekly MRs 5,600 → 8,700+ L3 Kiro Crew Human sets a goal Coordinator agent, 30–50 sessions CI and review before merge 1,000+ PRs in 7 days, 3 devs L4 Airbnb Migrate ~3.5K test files Per-file state machine with retries Jest, lint and tsc must pass 1.5-year estimate → 6 weeks L4
Uber. Shared platform layers under many harnesses (Claude Code, Codex, Cursor, OpenCode). Every model call goes through a Model Gateway (identity, PII anonymizer, cache, router) and every tool call through an MCP Gateway; agents use an AI Context Graph and 3,600+ skills. Managed agents fire on CI failures, alerts and migrations. They break spend into users x sessions x turns x requests x tokens x price, with a lever per factor; cost per session is down 52% from its June 2026 peak. Watch the metric: "70%+ attributed to agents" (Aug 2026) and "11% opened by agents" (Mar 2026, Pragmatic Engineer) measure different things. Ramp Inspect. OpenCode in a Modal sandbox with Postgres, Redis, Temporal and RabbitMQ, an image rebuilt every 30 minutes. It "closes the loop on verifying its work" with tests, Sentry, Datadog, LaunchDarkly and screenshots. PRs use the requester's GitHub token, so nobody can approve their own agent's change. The ~50% figure and "80%+ of its own code" come from Modal's post. Dropbox Nova. The platform, not the agent, owns the loop: CI is triggered by the workflow, the agent is called back only on failure, one branch per session, publishing happens outside the agent. No adoption numbers; L3 is inferred. Cloudflare. Keyless access through Access SSO and a proxy Worker; the GitLab CI reviewer sorts each MR into trivial, lite or full and fans out to specialist agents; open-weight models take about 15% of reviewer traffic. 3,683 internal users. Single self-reported source that doubles as marketing; no defect data. Kiro Crew. The coordinator does five human chores: decompose, assign, patrol, accept, sequence. The repo is their own, and the post does not say whether review is human or agent, nor models or cost. Airbnb. Each Enzyme test file moves through refactor, fix Jest, fix lint and tsc; the LLM retries with the errors until it passes or hits a limit. 75% done in 4 hours, 97% after 4 days of sample-tune-sweep, the last ~100 files by hand. Choosing related files mattered more than prompt wording. All levels are our reading; all numbers are self-reported.
Pattern StrongDM Stripe Spotify OpenAI Carlini Ralph Gas Town HumanLayer A sandbox per task twins devbox K8s pod worktree Docker — worktree — A machine check before any human LLM judge CI, 2 rounds verifiers + judge lints, CI, agent review tests + GCC tests + types Refinery — Human effort moves into the spec or check hidden scenarios — end-state prompt tickets, lints test harness specs/ — reviewed specs State lives in files, not agent memory cxdb — versioned prompts docs/ lock + progress files fix_plan.md Beads ledger plan docs The final gate: where they split none: a score human PR review auto-merge if safe human approves proof tests only human watches merge queue human reads code
“—” means the source does not describe it. Gate color: orange is a human, grey is a machine.
1. A sandbox per task. Almost every factory gives each task its own isolated place to run: Stripe's network-isolated devbox, Spotify's locked-down Kubernetes pod, OpenAI's per-issue workspace and worktree, Carlini's fresh Docker container, Gas Town's worktree hooks. StrongDM isolates the outside world instead, with digital twins of Okta, Slack, Jira and Google. Ramp (Modal sandbox), Dropbox (repo snapshot) and Uber (DevPods) do the same. Ralph is the exception: one process in one repo. Isolation is what lets agents run without confirmation prompts. 2. A machine check before any human. Nobody shows a human raw agent output. Most of these checks are deterministic (CI, lints, tests, a GCC oracle); StrongDM and Spotify add an LLM judge. Stripe caps CI at 2 rounds, Dropbox at 5 tries, Airbnb at a retry limit. 3. Human effort moves into the spec or the check. StrongDM hides scenarios from agents like a holdout set; Carlini says the verifier must be nearly perfect; Ralph's specs/ are reloaded every loop; Spotify wants prompts that describe the end state; OpenAI encodes taste as custom lints; HumanLayer puts its human review on product, architecture and program design docs. 4. State lives in files. Agents forget, so plans, locks, tickets and the reason for each change are kept on disk or in git: fix_plan.md, current_tasks/ lock files, Beads, docs/ as the system of record. 5. The gate is where they split. Stripe, OpenAI and HumanLayer keep a human approval; Spotify auto-merges routine PRs; StrongDM and Carlini stop at a score or test suite; Gas Town uses a merge queue. That last choice decides the level on the map. The cell contents are our summary of each source; HumanLayer's machine checks and Stripe's spec practice are simply not described.
4
Debates: dark or light, who reviews, and does it pay?
Six open arguments, both sides with sources, ending with what each position needs technically.
Part 4 reports the arguments, not our verdict. For each debate I show the strongest case on each side with a source. The three questions people asked most: should a factory go dark, how is review done, and what technology do you need. The last slide in this part maps each position to the capabilities it requires, so you can pick a side based on what you can actually build.
4 · Debates · Map
On all six debates, builders lean "go further" while measurers and maintainers lean "slow down"
← FOR / GO FURTHER
AGAINST / SLOW DOWN →
Should humans read code? dark ↔ light / twilight
Where does rigor go? machine checks ↔ design + production
Real productivity gains? big counts ↔ measured slowdowns
Spec-driven, or MDD again? specs first-class ↔ MDD flaws again
Will history repeat? wave 4 works ↔ stays internal
Is the token bill worth it? value moves up ↔ cost too high
StrongDM
Yegge
Shapiro
Cherny
Willison
Ronacher
Mollick
Osmani
HumanLayer
OpenAI
Cloudflare
Spotify
Carlini
Fowler
Osmani
Majors
HumanLayer
Stripe
Google
Uber
Ramp
AWS Kiro
DORA
Faros
METR 2025
Spec Kit
Kiro
BMAD
Harper Reed
StrongDM
Böckeler
DoD 2022
TrueFoundry
Wavect
Kessel Run pivot
a16z
Sequoia
Factory
Tessl
Garry Tan
Gas Town #3649
Peter P.
pro cautious critical
Dot color = the source's stance in our survey. Horizontal position is our reading, not a measurement.
Read this as a seating chart. Each row is one debate from our survey; the left pole is "go further", the right pole is "slow down". Green dots are sources whose stance we coded as pro, amber cautious, red critical. A pattern shows up: practitioners shipping volume (StrongDM, Stripe, Uber, OpenAI, Spec Kit) sit left, while people who measure outcomes or maintain codebases (METR, DORA, Faros, HumanLayer, Böckeler) sit right. Two caveats. Placement within a side is our reading. Some people sit in two places: Carlini built a compiler with no human-written code, yet says deploying never-verified software worries him; HumanLayer ran a lights-off factory before arguing against it. The next slides take the rows one at a time, starting with the first: should humans read the code at all.
For dark: verify behavior, not code
Agents can rewrite tests to fit the code. So StrongDM keeps scenarios outside the repo as a holdout and scores "satisfaction" against clones of Okta, Jira and Slack.
Shapiro: a few teams under 5 people already work at Level 5
Yegge: 20-30 parallel agents, code openly unread
Carlini: 100K-line C compiler that builds Linux, GCC as oracle
Cherny: no hand-written code since Nov 2025 (self-report)
Against: tests pass while design rots
HumanLayer went lights-off in July 2025, hit outages from code nobody could read, and rewrote by hand in November. Cause: "there is no penalty for eroding codebase maintainability."
Osmani: comprehension debt grows while tests stay green
Willison: LLM error rates, and $1,000/day of tokens
Mollick: the wrong goal for most organizations
Carlini: never-verified software is a real concern
For. StrongDM's two rules are that humans neither write nor review code, and they say a factory spending under $1,000 of tokens per engineer per day has room to improve. Their key move is calling tests "scenarios": scenarios live outside the codebase like an ML holdout set so the agent cannot tune to them, and success is a probabilistic satisfaction score, run against a Digital Twin Universe of six cloned SaaS services refined until no behavioral differences are found. Shapiro, who coined "dark factory" from FANUC's robot plant, says only a handful of teams under five people work at Level 5. Yegge's Gas Town claims 20-30 coordinated agents. Carlini's 16 parallel agents built a 100K-line Rust C compiler for under $20,000. Cherny says he has not hand-written code since November 2025, which is self-reported.
Against. Dex Horthy's essay reports that HumanLayer went fully lights-off in July 2025; around the third outage in November a cofounder spent two weeks rewriting by hand. His diagnosis is the training signal: RL rewards and benchmarks score pass or fail, and maintainability has no fast oracle, so its cost shows up months later. He also says review agents raise the floor but not the ceiling, and recommends aiming for a safe 2-3x. Osmani calls the hidden cost comprehension debt. Willison admires StrongDM but questions skipping review given LLM error rates and the token cost. Mollick calls a StrongDM-style factory the wrong goal for most organizations. A counterpoint raised on HN: HumanLayer's trial predates later model improvements.
4 · Debates · Twilight
Twilight: automate only loops with trustworthy checks, and keep people on intent, design and risk
A loop to automate
Is its check cheap,
hard to fake, stable?
Osmani's test for which
loops may run dark
yes
no
Lights off: agents + machine checks
• Maintenance PRs: most auto-merge, no human (Spotify)
• Format, lint, type and test fixes inside CI loops
• Test migrations retried until green (Airbnb, 3.5K files)
Lights on: a human stays at the gate
• Product intent, specs and scenarios
• Architecture and program design, before code (HumanLayer)
• Auth, billing and public APIs (Osmani)
Andon: the agent stops and calls a human (Mollick)
Most cautious voices do not argue for going back to hand-written code. They argue for a "twilight" factory, Mollick's word. Osmani's rule is to automate a loop only when its check is cheap, hard to fake and stable, and to keep humans on auth, billing and public APIs. Mollick adds a facilitator agent whose job is to decide when to bring a person in: the agent has to know when to look up. HumanLayer keeps humans on product review, system architecture and program design before code exists, then reviews code in 100-200 line vertical slices. Spotify says most of its maintenance PRs auto-merge with no human in the loop, which is the clearest public example of a dark loop inside a lit factory. The Airbnb example is an agent retry loop ("retry steps multiple times until they passed") that took about 3.5K test files from a 1.5-year estimate to about 6 weeks. The andon arrow is the Toyota idea we return to on the history slide. Which items sit in which box is our reading of these sources.
4 · Debates · Review
Six review strategies are in use, and they differ mainly in how much code a human still reads
Diff review
Human reads every diff before merge
Stripe Minions: 1,300+ PRs/week, human-reviewed
Stripe; Shapiro L3 ≈ L3
Design first
Human reviews product, system, program design,
then 100-200 line code slices
HumanLayer L3 → L4
AI reviewer
Agents review every change, a human signs off
Cloudflare: 100% of standard CI
Cloudflare, Uber L3-L4
Risk tiers
Review depth set by risk: trivial, lite or full
People keep auth, billing, APIs
Cloudflare, Fowler L4
Proof of work
Human checks the evidence: CI, review notes,
video and screenshots, then approves
Symphony, Ramp L4
Scenarios only
No one reads code. Hidden scenarios scored
for satisfaction on digital twins
StrongDM L5
human reading
READS EVERY LINE
READS NO CODE
Left to right, the human reads less code. Diff review: Stripe merges over 1,000 (later 1,300+) fully minion-produced PRs a week, and every one is still reviewed by a human; Shapiro's Level 3, "Your life is diffs." Design first: HumanLayer moves human reading earlier: product doc and mockup, then system architecture (sequence diagrams, schemas), then program design (types, signatures, call-stack tree, file-tree diff), then working slices of 100-200 lines, which Horthy says are much cheaper to check and re-steer. AI reviewer: Cloudflare runs an AI reviewer on 100% of standard-CI repos; Uber scores its uReview on precision, recall and F1 against real PRs with known bugs. Ramp opens PRs with the requester's own token so nobody approves their own agent's change. Risk tiers: Cloudflare sorts each MR into trivial, lite or full and routes it to specialist agents; Fowler's retreat proposes risk tiering; Osmani keeps humans on auth, billing and public APIs. Proof of work: Symphony moves a ticket to Human Review with CI results, review feedback, complexity analysis and a walkthrough video; Ramp's agent attaches tests, telemetry and UI screenshots. Scenarios only: StrongDM. The level labels are our reading. Many companies combine several of these.
4 · Debates · Rigor
Rigor moves into specs, tests, scenarios and production; nothing yet checks maintainability fast
Writing code now cheap: agents do most of it
Specs spec → plan → todo (Harper Reed); Spec Kit, Kiro
Tests and oracles GCC as oracle; test quality decided it (Carlini)
Holdout scenarios outside the repo, scored as satisfaction (StrongDM)
Lints, structure custom lints enforce layers and taste (OpenAI)
Design review review the design before code exists (HumanLayer)
Production telemetry is "the surviving spec" (Majors)
Gap no fast oracle for maintainability
The Thoughtworks retreat summarized by Fowler asked where rigor goes when humans stop writing code; this slide lists the answers in our survey. Specs: Harper Reed's spec.md, prompt_plan.md, todo.md, and tools like Spec Kit and Kiro. Tests and oracles: Carlini says the task verifier must be nearly perfect, otherwise Claude solves the wrong problem; GCC served as the oracle. Holdout scenarios: StrongDM keeps them away from the agent. Mechanical architecture: OpenAI enforces fixed layers with custom lints whose error messages tell the agent how to fix the code. Design review: HumanLayer's staged review. Production: Charity Majors argues the retreat's list missed production; at 10-100x change rates, observability and fast feedback are the verification. The red box is Horthy's central point: every one of these checks behavior, and none gives a fast signal that the codebase is getting harder to change. Cloudflare and Spotify mechanize review with AI reviewers and LLM judges; Spotify's judge reportedly vetoed about a quarter of sessions (InfoQ later reported the judge was removed; we did not verify this).
4 · Debates · Evidence
Company numbers count AI output; controlled studies find smaller, mixed and sometimes negative effects
Volume, self-reported by companies
1,300+
PRs/week merged, fully agent-produced, still human-reviewed (Stripe)
70%+
of Uber PRs attributed to agents; only 11% agent-opened (Mar 2026)
~75%
of Google's new code AI-generated, per its CEO
~50%
of Ramp's merged PRs started by its agent (Modal)
Measured: trials and telemetry
+19%
task time with AI, METR 2025 RCT (16 devs); they felt 20% faster
−18%
task time in METR's 2026 rerun, which METR calls an "unreliable signal"
↑ ↓
DORA 2025: AI goes with higher throughput and lower stability
+242.7%
incidents per PR at high AI adoption (Faros, 22K devs)
Counts mix "touched", "authored", "generated" and "merged", so they are not comparable across companies.
Left column is what companies report about their own systems: Stripe 1,000+ then 1,300+ merged minion PRs a week; Uber 70%+ of PRs attributed to local or cloud agents, while Pragmatic Engineer's March 2026 numbers put agent-opened PRs at 11%; Google about 75% of new code AI-generated, up from 25% in 2024; Ramp about half of merged PRs. These count output, not outcome, and use different definitions.
Right column. METR's 2025 randomized trial: 16 experienced open-source developers, 246 real tasks, AI-allowed tasks took 19% longer (CI +2% to +39%); developers forecast a 24% speedup and still believed afterwards they had been 20% faster. Fairly, METR now says those results are out of date. Its February 2026 rerun estimated −18% task time for returning developers (CI −38% to +9%) and −4% for new recruits, but METR calls the data an unreliable signal: many developers refused to work without AI and withheld tasks, so the true speedup could be higher. DORA 2025: about 90% adoption, AI correlates positively with throughput and negatively with delivery stability. Faros AI telemetry from 22K developers across 4K teams, comparing high and low AI adoption: incidents per PR +242.7%, bugs per developer +54%, median review time about 5x, deployments per week −11.7%. Faros is a vendor; we read the page but not the full report.
4 · Debates · Specs
The closer spec-driven work gets to spec-as-source, the more it inherits the problems of MDD
Spec-first
spec written, then used for the task
Kiro, Spec Kit (her placement)
Spec-anchored
spec kept to evolve the feature
Tessl aims here
Spec-as-source
spec is the source; code untouched
where MDD (UML/DSL → generator) sat
SPEC GUIDES ONE TASK
HUMANS NEVER TOUCH CODE
Why it may be different this time
LLMs remove the DSL and custom-generator overhead that sank MDD
Specs are plain language, so product people can write them
Specs now also drive tests and holdout scenarios (StrongDM, Symphony SPEC.md)
Why it may not be
MDD "never took off for business applications" (Böckeler)
Spec-as-source risks "inflexibility and non-determinism" together
Verbose Markdown: she would rather review code than these files
Birgitta Böckeler of Thoughtworks sorts spec-driven development into three levels. Spec-first: a well-thought-out spec is written, then used for the task. Spec-anchored: the spec is kept after the task and used to evolve the feature. Spec-as-source: the spec is the main source file and the human never touches the code. She places Kiro and Spec Kit at spec-first (GitHub aspires to anchoring, but each spec lives on its own branch), and Tessl as the only one explicitly aiming at spec-anchored and exploring spec-as-source. StrongDM's Attractor, published as natural-language specs instead of code, and OpenAI's Symphony SPEC.md lean toward spec-as-source; that placement is our reading.
4 · Debates · History
Earlier software factories raised quality but stalled; Toyota's stop-the-line rule is worth borrowing
1969 Hitachi
Japan's factories: standards, reuse, QA
Quality up, but company-specific
1975 SDC
First US attempt: tools + process
Stalled quickly
2004 Microsoft
DSLs, models, product lines
Never mainstream
2017 Kessel Run
DoD: DevSecOps platform + teams
2025: pivot to vendor-managed
2026 AI factories
Agent loops + verification gates
Open question
Toyota: jidoka and andon
A machine stops itself on an abnormality,
so defects never flow to the next step.
Andon: stop, and call for help at once.
Agent factory equivalent (our angle)
A failing check or low confidence stops
the loop before more PRs pile up,
and pulls a human in (Mollick's facilitator).
The "factory" label has been tried at least four times before AI. Hitachi says its Software Works became independent in 1969 as the world's first software-dedicated facility; Cusumano's 1991 book compares Hitachi, Toshiba, NEC, Fujitsu and SDC and describes standardization, reuse and quality control. Wavect's 2026 essay judges that Japan's factories delivered quality and productivity but stayed company-specific and lost the package and platform race. SDC's mid-1970s US attempt stalled quickly; Cusumano attributes this to organization and long-term investment more than tools. Microsoft's 2004 Software Factories bet on DSLs and model-driven product lines and never went mainstream. DoD's 2022 strategy named a department-wide software factory ecosystem as a goal, and Kessel Run, the first Air Force factory (2017), pivoted in 2025 to a government-led, vendor-managed model, criticized as a retreat. The optimistic reading is that each wave moved craft into systems and AI is the fourth reinterpretation; the pessimistic one is that AI factories may also end as internal efficiency. The bottom row is our angle: our survey found English-language essays that cite FANUC's lights-out plant but none that connect AI factories to Toyota's jidoka and andon, where the line stops on a defect and a person is called. That maps closely to Mollick's facilitator agent and to "stop the loop, don't pile up PRs".
4 · Debates · Money
The token bill can rival a salary, and the evidence of payoff is so far mostly spend and valuations
Cost
$1,000
tokens per engineer per day: StrongDM's floor. About $250K a year at 250 workdays (our arithmetic)
~6x
Uber's AI cost vs 2024
−52%
Uber cost per session vs its June 2026 peak, via model routing, compaction and caching
Value signals
$4.0B
enterprise spend on AI coding in 2025, 55% of departmental AI spend (Menlo)
$3T
a year of developer labor a16z sees as the market (30M × $100K)
<$20K
API cost of Carlini's 100K-line C compiler
Push-back
Garry Tan (YC): "Stop building Foxconn factories for your agents"
Gas Town: alleged use of users' LLM credits for self-improvement (HN 253 pts)
"Personal software factories are a waste of time" (Oct 2026)
Value may flow to the harness layer: Factory raised $150M at $1.5B
StrongDM says a factory spending under $1,000 of tokens per human engineer per day has room for improvement. At 250 workdays that is roughly $250K a year per engineer, our arithmetic, and the HN thread pushed back that this exceeds many salaries. Uber's AI cost rose about 6x versus 2024 (Pragmatic Engineer, March 2026); Uber's own August post breaks spend into users × sessions × turns × requests × tokens × price and reports cost per session down 52% from the June 2026 peak. On the value side, Menlo counts $4.0B of enterprise AI coding spend in 2025, 55% of departmental AI spend, and a16z frames the market as 30M developers × $100K = $3T a year. Carlini's compiler cost under $20,000 in API fees for 100K lines. Who captures the value is open: Sequoia and a16z treat the harness or factory layer as its own value layer, and Factory raised $150M at a $1.5B valuation (a later $5B round is mentioned on its site, date unknown). Critics: Garry Tan reflects on generating 500K lines of Rails and warns against mass-production factories; Gas Town faced an allegation of spending users' credits on self-improvement; the October 2026 "waste of time" post drew an HN thread noting that distribution, not building, is now scarce. None of these are audited returns on the token spend; we did not find any in our survey.
Capability Light · L3 Twilight · L4 Dark · L5 Seen at
Sandboxes, CI and lints, audit logs, scoped permissions need need need Stripe, OpenAI, AWS Small PRs and fast diff-review tooling need helps – HumanLayer, Uber Design docs reviewed before code helps need – HumanLayer AI reviewer with measured precision and recall helps need need Uber, Cloudflare Risk classification of every change helps need need Cloudflare Proof-of-work artifacts: video, screenshots, logs helps need need Symphony, Ramp Andon: agent escalates to a human helps need need Mollick (proposal) Holdout scenarios and digital twins of dependencies – helps need StrongDM Production telemetry, flags, fast rollback helps need need Honeycomb, Ramp Token budget and cost per session tracking helps need need Uber
This is a checklist, and the need/helps grading is our synthesis of the sources, not anyone's published standard. The first two rows are table stakes at every level: Stripe runs agents in isolated devboxes with full permissions inside the box; AWS's Kiro Crew post says audit logs and permission control belong in place from day one because trust comes from traceable history, not agent self-reports; OpenAI enforces architecture with custom lints. Light factories live or die on review throughput: HumanLayer's 100-200 line slices and Uber's Code Inbox and uReview fight review load. Twilight adds the things that let a human skip reading some code safely: design review up front, a measured AI reviewer (Uber scores uReview by precision, recall and F1 on real PRs with known bugs), risk tiers (Cloudflare's trivial, lite, full), proof-of-work artifacts, and an escalation path. Dark adds what replaces the human entirely: StrongDM's holdout scenarios and Digital Twin Universe, plus production telemetry as the last line of verification (Majors). Practical reading: if you cannot build the dark-column items, you are choosing light or twilight whether you say so or not.
Part 5
5
The stack
The layers a factory is built from, the products people use for each today, and which layers start to matter more as agents open hundreds of PRs.
This part is an engineering survey, before any opinion. For each layer we name real products and only state features we read in vendor docs, engineering blogs or changelogs (checked 2026-10-23). The last slide, Now to next, is our own reading and is labeled as such. Vendor speed claims (start times, snapshot times) are the vendors' own numbers, not our benchmarks.
5 · The stack · Layers
A factory is eight layers of tooling, with governance cutting across all of them
LAYER STATION NAMED PRODUCTS / PRACTICES
Model
harness Claude Opus / Sonnet · GPT-5.x · Kimi K2.5 (open)
Harness
harness Claude Code / Agent SDK · Codex CLI · goose · OpenCode
Context and tools
context AGENTS.md · MCP gateways (Stripe Toolshed) · Skills
Orchestration
orchestrate Symphony · Gas Town · Temporal · CF Workflows
Sandbox / compute
sandbox Modal · E2B · Cloudflare Sandbox · Stripe devboxes
Verification / CI
verify Bazel remote cache · Buildkite MCP · LLM judges
Review / merge
review GitHub merge queue · AI reviewers (Cloudflare, Uber)
Deploy / observe
ship · feedback Vercel previews · Sentry, Datadog · OTel
Identity, permissions, cost
govern · every layer
Cloudflare Access
AI Gateway (keys, cost)
Ramp Agent Keys
MCP OAuth 2.1
Uber spend tiers
Read top to bottom as the path of one task: a model inside a harness, fed context and tools, scheduled by an orchestrator, run in a sandbox, checked by CI and verifiers, reviewed and merged, then deployed and observed. Governance is drawn as a side bar because identity, permissions and cost apply to every layer, not one step. The products are examples we verified in sources, not a recommended list. Examples by company: Ramp Inspect runs OpenCode in Modal sandboxes with Cloudflare Durable Objects for session state; Stripe runs a goose fork with its Toolshed MCP server (400+ tools in Part 1, about 500 in Part 2) on EC2 devboxes; Cloudflare routes all model traffic through AI Gateway behind Access; Uber routes 1,000+ MCP servers through one gateway. Kimi K2.5 is the open-weight model Cloudflare runs on Workers AI for some review work. The next slides take the layers one by one.
5 · The stack · Sandbox
Sandbox vendors now compete on state: snapshot, fork and resume, not only isolation
A FACTORY NEEDS: one box per task fast start snapshot / fork hours, not minutes network + secret policy many in parallel
Option Isolation State: snapshot / fork / resume Stands out for
Modal Sandboxes gVisor, or VM runtime FS, dir, memory (exp.) snapshots; fork many up to 24 h; Ramp Inspect runs on it E2B Firecracker microVM pause saves filesystem + memory, kept until killed open source; 24 h on Pro Daytona own kernel per sandbox snapshots; VM type can snapshot memory "under 90ms" start (vendor claim) Cloudflare Sandbox container, each a microVM backup/restore; VM snapshots announced egress proxy injects credentials Vercel Sandbox Firecracker microVM persistent by default; snapshots no CPU billed while model thinks Fly Sprites Firecracker microVM checkpoint in ~300 ms, incl. packages stateful by design; L3 egress policy Morph Infinibranch VM snapshot, branch, restore "under 250ms" one state, many parallel tries Stripe devboxes EC2, no prod, no internet pre-warmed pool, ready in ~10 s same box human engineers use
Highlighted: keeps memory state or forks. Speeds are vendor claims. Firecracker = AWS microVM; gVisor = Google user-space kernel.
The six needs come from the factory cases: Stripe gives every minion its own devbox so parallel runs need no git worktrees; Ramp rebuilds a repo image every 30 minutes so a sandbox starts close to current. Isolation is now table stakes; the differences are in state. Modal offers filesystem, directory and experimental memory snapshots (memory snapshots kept 7 days), and many sandboxes can start from one filesystem snapshot; default lifetime is 5 minutes, configurable to 24 hours. E2B pause keeps filesystem and memory; runtime is up to 1 hour on Hobby and 24 hours on Pro. Vercel's limit is 45 minutes (Hobby) or 24 hours (Pro) per session, and the limit resets when a persistent sandbox stops and resumes; its docs cite research that 71 to 98 percent of agent runtime is waiting on the model, which is why it bills only active CPU. Cloudflare's April GA post says VM-level snapshots were "rolling out in the coming weeks"; we did not confirm the later release. Fly argues for stateful sandboxes; Thoughtworks put Sprites in "Assess" in April 2026. Morph's 250 ms figure is from its own page with no benchmark conditions. We left out prices on purpose. Fly Machines (the general VM product) start a pre-built machine in "well under a second" but are not agent-specific.
5 · The stack · Verify
CI becomes a tool the agent calls in its loop, so it must be fast, cached and capped
Agent edits in its own sandbox
Local checks lint on push under 5 s
Remote CI selected tests, remote cache
Merge queue PR + latest base + PRs ahead
Preview env URL per branch and per commit
Human review requester can't approve own PR
fail: autofix if known, else back to agent · at most 2 CI rounds (Stripe)
Queueing
Hundreds of PRs fight for CI and for the main branch.
Merge queue tests changes together; build concurrency is set from 1 to 100 (GitHub).
Flaky tests
A random failure sends the agent chasing noise.
Dropbox needed 100+ CI runs to validate one flaky-test fix; its Deflaker caps fixes at 5.
Cost and time
Each retry spends tokens, compute and wall-clock time.
Cap rounds, reuse build outputs, give the agent a one-call failure summary (Buildkite MCP).
Human-paced CI assumes a few pushes per person per day and a person who reads the failure. An agent pushes many times, reads every log, and retries; so CI turns into a tool call inside the agent loop. Stripe's pattern is the clearest: a local executable picks and runs lints on each push in under five seconds, remote CI runs selected tests from a suite of over three million, known failures are autofixed, and the minion gets at most two CI rounds because of diminishing returns. Dropbox Nova triggers CI from the workflow itself and only brings the agent back when a failure needs inspection; it uses Bazel remote execution. Bazel's remote cache only works if outputs from one machine are safe to reuse on another, which means hermetic, reproducible builds become a prerequisite. Buildkite added get_build_failure_summary (July 2026) and wait_for_build so agents get a bounded summary instead of raw logs. GitHub's merge queue tests each PR on a temporary branch with the latest base and the PRs ahead of it, and removes failing PRs. For Copilot cloud agent, GitHub by default does not run Actions workflows on agent code until a user with write access approves, and the requester cannot approve the PR.
NEED WHAT THEY SHIP (2025-09 TO 2026-04)
Run agent code
Dynamic Workers: isolates
Sandboxes, Containers GA
State per agent
Durable Objects
Agents SDK, Think
Workflows
Tools
MCP Server Portal
Code Mode: model writes code
Models and cost
AI Gateway
Workers AI (open-weight)
Identity, secrets
Access
egress proxy adds credentials
Code storage
Artifacts: git-compatible storage for agents
USED ON THEMSELVES · 30 DAYS · 2026-04
3,683 AI coding users, 93% of R&D
241.37B tokens through AI Gateway
100% MRs on standard CI get AI review
5.6k → 8.7k+ weekly merge requests (4-wk avg)
Their own line: "isolates are the right primitive", yet "Coding agents genuinely need containers".
Cloudflare's Agents Week thesis is that agents are "one-to-one": every user and task needs its own execution environment, so per-agent compute must be cheap and dense. Their answer is two runtimes: isolates (Dynamic Workers, a few milliseconds to start, open beta since March 2026, "around 100x faster" than a typical container by their estimate) and containers (Sandboxes and Containers, GA April 2026, with credential injection at an egress proxy so the agent never holds the secret). State lives in Durable Objects, which the Agents SDK and the new Think framework build on; Workflows add durable steps. Code Mode (September 2025) turns MCP tools into a TypeScript API and lets the model write one program instead of many tool calls. Internally, a proxy Worker injects API keys so no keys live on laptops, 13 MCP servers (182+ tools) sit behind one portal collapsed by Code Mode into search and execute tools, about 3,900 repos got short AGENTS.md files, and a multi-agent reviewer sorts each MR into trivial, lite or full risk. Our reading: the front-runner sees value less in the model or the agent UI and more in the runtime per agent and in the gateways (models, tools, identity) that every call passes through. They say themselves that "None of these pieces are especially novel on their own"; the value is the integration. Next on their list is cloud background agents on Durable Objects, the Agents SDK and Sandbox containers.
5 · The stack · Harness
Every harness is the same loop; the choice is where it runs and whether you build on it
SANDBOX / WORKSPACE
Model
Hooks + permissions allow / deny / ask
Tools read, edit, bash, MCP
Context AGENTS.md skills, memory session history compaction
Subagents own context window, report a summary back
loop until done
INTERACTIVE, ON A LAPTOP BACKGROUND, IN THE CLOUD
USE AS A PRODUCT
Claude Code CLI Codex CLI Aider (auto git commits) goose desktop / CLI
USE AS A PRODUCT
Codex cloud Cursor cloud agents Copilot cloud agent Devin
BUILD ON IT · YOU HOST
Claude Agent SDK OpenAI Agents SDK OpenHands SDK goose, OpenCode (OSS)
IN-HOUSE OR HOSTED
Stripe Minions: goose fork Spotify Honk: Agent SDK Ramp Inspect: OpenCode Claude Managed Agents
A harness is a model in a loop with tools and context; Thorsten Ball showed one in under 400 lines and mini-swe-agent does it in about 100. What separates harnesses is the control around the loop. Claude Code's PreToolUse hook can allow, deny, ask or rewrite a tool call before it runs, and a Stop hook is where Spotify runs its build and test verifiers before any PR. The Claude Agent SDK gives the same tools, loop, hooks, subagents, permissions and sessions as Claude Code in Python and TypeScript; Claude Managed Agents (beta) hosts that harness with sessions in an Anthropic cloud sandbox or a self-hosted one. The OpenAI Agents SDK has agents, handoffs, guardrails, sessions, tracing and sandbox agents. On the product side, Cursor's cloud agents (formerly Background Agents) run in isolated cloud VMs and can be started from Slack, Linear, a PR comment or the API; GitHub renamed its coding agent to Copilot cloud agent; Codex cloud runs tasks from a published environment while your computer sleeps. goose is Apache-2.0 and part of the Agentic AI Foundation at the Linux Foundation. The pattern in the large cases is the bottom-right box: companies take an open or SDK harness and wrap it with their own sandbox, tools and verifiers. Spotify moved from Goose, Aider and a homegrown loop to Claude Code.
No orchestrator lock files + git races Carlini: 16 Claudes, one C compiler
Coordinator agent Gas Town Mayor: 20–30 Kiro Crew: 30–50 sessions on 6–10 machines
Ticket daemon Symphony: each Linear issue gets an isolated Codex run
Fixed blueprint code runs the steps, agents fill bounded nodes (Stripe, Dropbox Nova)
MODEL OWNS CONTROL FLOW CODE OWNS CONTROL FLOW
Durable execution underneath Temporal · Inngest · Cloudflare Workflows
1 · plan result checkpointed
2 · implement result checkpointed
3 · run CI fails: retry this step
4 · wait for approval waitForEvent, then merge
Completed steps return saved results on replay, so a crash or a failed step
does not redo hours of agent work. Runs can wait minutes to weeks.
Left to right, control moves from the model into code. Carlini ran 16 Claude loops with no orchestrator at all: agents claim tasks by writing lock files into current_tasks/ and git resolves races. Gas Town has a Mayor agent that splits work into Beads tickets for ephemeral workers in git worktrees and lands them through a merge queue (the Refinery); it claims to handle 20 to 30 agents. Kiro Crew runs a coordinator over 30 to 50 sessions on 6 to 10 machines, and says one coordinator hits its limit at about 50 sessions. OpenAI's Symphony polls Linear and turns each issue into an isolated Codex run that ends in a PR with proof of work; it ships mainly as a SPEC.md plus an experimental Elixir reference. Stripe's blueprints and Dropbox Nova keep control flow in code: deterministic nodes for lint, push and CI, agent nodes for implement and fix. Under all of these, long agent runs need durable execution: Inngest stores each step's result and on retry reruns only the failed step; Cloudflare Workflows steps retry automatically, can sleep, and can wait for an external event such as an approval (their example waits up to 24 hours); Temporal markets itself for agents with retries until an LLM returns valid output. The step names in the strip are our illustration, not from one source.
5 · The stack · Verify
Green tests are the cheapest check; factories stack harder-to-fake checks on top
Deterministic lint, types, tests Stripe: lint < 5 s, 3M+ test suite
Stop-hook checks format, build, test before any PR (Spotify, via MCP)
LLM judge vetoes ~1/4 of sessions (Spotify) Uber uReview scored on precision/recall
Browser checks Playwright MCP: accessibility tree Ramp: screenshots for frontend
Hidden scenarios kept outside repo, scored as "satisfaction" (StrongDM)
Digital twins clones of Okta, Slack, Jira, Google to test at scale (StrongDM DTU)
CHEAP · FAST · EASY TO GAME COSTLY · SLOW · HARD TO GAME
Proof of work ties them together: Symphony attaches CI status, review feedback and walkthrough videos to each PR.
Agents solve the problem the verifier defines, so the weakest check sets the quality ceiling. Deterministic checks are cheap and run first. Spotify runs format, build and test verifiers through MCP in a stop hook so the agent cannot end the session with a broken build; an LLM judge then compares the diff to the task, vetoes about a quarter of sessions, and the agent corrects course about half the time. Uber scores its reviewer, uReview, against real PRs with known bugs on precision, recall, F1, cost and latency, which is a way to verify the verifier. Browser checks close the loop for UI: Playwright MCP gives the model the accessibility tree instead of pixels; Ramp's Inspect takes screenshots for frontend work and OpenAI's harness engineering repo exposes Chrome DevTools to Codex. At the top, StrongDM keeps end-to-end scenarios outside the codebase like a holdout set because agents can rewrite tests to match code, and tests against behavioral clones of Okta, Slack, Jira and Google apps refined "until we stop finding behavioral differences." Carlini's C compiler used GCC as a known-good oracle. Caveat from parts research: LLM judges can share the agent's blind spots, and none of these measure maintainability.
5 · The stack · Govern
Each agent gets a human sponsor, short-lived scoped tokens and a cost model per session
IDENTITY AND PERMISSIONS
Human sponsor agent never gets more than this role
Agent identity Agent Key, expires by default (Ramp)
Short-lived token OAuth2-PKCE, about an hour, revocable
Gateway MCP gateway: auth + policy in one place
Tools, APIs secret added at egress, not in agent
Audit: "Sarah (via Codex)" · PRs use the requester's token, so no one approves their own agent's PR
COST · UBER'S EQUATION, ONE LEVER PER FACTOR
users
× sessions
× turns
× requests
× tokens
× price
context graph, skill tuning
compact at 400K, 1 h cache, code-mode
route by benchmark, cheaper subagents
Budgets: spend tiers, Slack nudges at 50 / 80 / 100%, manager sign-off to upgrade (Uber).
Opposite stance: StrongDM treats under $1,000 of tokens per engineer per day as room to improve.
Identity: Ramp's model ties every agent to a human sponsor ("On Behalf Of User"); the agent can never have more access than the sponsor, its Agent Key is exchanged through OAuth2-PKCE for short-lived access tokens (a leaked token is "bad for an hour" and can be revoked), and agent actions flow into the same audit log as human ones with attribution such as "Sarah (via Codex)". Ramp's Inspect opens PRs with the requesting engineer's own GitHub token so the requester cannot approve their own agent's PR; GitHub enforces the same rule for Copilot cloud agent. The MCP authorization spec makes the gateway pattern standard: OAuth 2.1, tokens bound to one MCP server via resource indicators, and no token passthrough to downstream APIs. Secrets stay out of the agent: Cloudflare's Sandbox egress proxy adds credentials by hostname, and its internal proxy Worker injects model API keys so none live on laptops. Uber routes 1,000+ MCP servers through one gateway with central auth and policy. Cost: Uber splits spend into users × sessions × turns × requests × tokens × price and assigns levers to the last three factors; it reports cost per session down 52% from its June 2026 peak, and it deliberately avoids hard caps. StrongDM argues the opposite end: spend more. Stripe's two-round CI cap is also a cost control.
Layer Common practice now Becomes important (our reading) Seen at
Model one frontier model per tool route per workload; open-weight for easy work Uber, Cloudflare Harness interactive CLI on a laptop same harness, headless, via SDK or hosted Spotify, Stripe, Ramp Context, tools hand-written AGENTS.md, ad-hoc MCP one MCP gateway; Code Mode; agent-kept docs Cloudflare, Uber Orchestration a human juggling tabs and worktrees ticket-driven runs on durable workflows Symphony, Kiro Crew Sandbox laptop, container or worktree per-task microVM: snapshot, fork, egress policy Ramp/Modal, Stripe CI human-paced, retried by hand CI as an agent tool: test selection, cache, round caps Stripe, Dropbox Verification "tests pass" means done hidden scenarios, twins, judges, proof-of-work bundles StrongDM, Spotify Review, merge a human reads every diff risk tiers: AI review, merge queue, humans on risky Cloudflare, Uber Govern per-seat licenses, shared keys agent identity + sponsor; keys at gateway; cost/session Ramp, Uber
Highlighted rows: where we expect the bottleneck to sit as PR volume grows. This column is interpretation, not a survey result.
This slide is our reading of the evidence in Part 5, not a finding from any single source. The "Seen at" column names where the next-practice already exists in production, so each row is something that already exists in a front-runner, not a prediction from nothing. Our main point: models and harnesses are converging and easy to swap (Spotify swapped Goose and Aider for Claude Code; StrongDM's Attractor can run on Claude Code, Codex, OpenCode, Amp or Cursor), so the lasting engineering investment is in the sandbox, CI, verification and review layers that sit between the agent and the main branch. Cloudflare's reviewer already sorts every MR into trivial, lite or full risk, and Uber runs uReview on all PRs; that is the template for risk-tiered review. Caveat: these are large companies with platform teams; at INDX's size, a hosted sandbox plus a hosted background agent may cover the first three rows without building anything. Prices were left out on purpose; check vendor pricing at the time of a decision.
6 · Our view · Opinion
6
Our view: tacit knowledge decides whether a factory pays
Parts 1 to 5 reported what others say. This part is our own opinion, with research added only where it supports a point.
From here on this is the presenter's own view, not a survey. Six points: (1) most customer knowledge is tacit, (2) build factories as lanes that grow with demand, (3) small drift compounds, so correct on a cadence, (4) shared documents act as a common language, (5) whether a factory makes money depends on where the bottleneck moves, (6) the cornerstones we need. Where a slide cites research, the source line says so; everything else is our opinion.
WATERLINE
Manuals
Specs
Existing data
On site
Who actually uses it
What they struggle with every day
How they think while using it
Which tasks are most frequent
What they talk about with their customers
~20% documented
the factory can read this
~80% tacit
lives with people on site
80/20 is our rough estimate, not a measurement
Our view (presenter's opinion)
A factory can build "an attendance clock-in button" or "a CRM" if you ask for it. But each customer carries a large amount of tacit knowledge: who uses the system, what kind of customers they serve, what data exists. Some of it is written down in manuals or already sits in data. Much more of it is only on site: who actually uses it, what they struggle with every day, how they think while using it, which tasks are most frequent, what they talk about with their own customers. It feels like an 80/20 rule: roughly 80% is buried, and the documented part is only the tip of the iceberg. The 80/20 split is an intuition, not a measured number.
What the customer knows
Who uses it
What kind of customers they have
What data exists
Daily struggles on site
How users think while working
Most frequent tasks
Conversations with their customers
mostly tacit, mostly unwritten
The request
"a clock-in button"
"a CRM" + docs
Factory
agents build
Generic app
works, but not
for this site
everything left of here is lost
The factory only sees what passes through the narrow pipe: the request and whatever documents exist. Everything the customer knows but never wrote down is lost on the way in, so the factory builds a technically working but generic answer. Supporting research: HumanLayer's prescription starts with a product review, a short document on the user problem and the success signal with rough HTML mockups of the screens, before any architecture or code. We read that as one way of widening the pipe. The diagram itself is our view.
START MORE USE MORE KINDS OF TASKS
One general lane agents do everything
Frontend lane design + engineer agents
Backend lane own agents
Frontend lane
Backend lane
Next lane added when demand appears
Cognitive load per lane owner
target: flat as the factory grows
= lane owner (human)
The more you use a factory, the more kinds of tasks it has to handle. Instead of one lane that does everything, build specialized lanes: a frontend lane with its own design and engineer agents, a backend lane with its own lane owner, and so on, adding a new lane only when demand for it shows up. Each lane has one human owner, so the number of people grows with the lanes and no single person has to hold the whole factory in their head. The goal is software engineering that keeps human cognitive load low. Supporting framing: the vendor Tessl markets "build your software factory, one workflow at a time", which matches the incremental idea; we have not evaluated their product.
6 · Our view · Drift
A 1% gap in understanding compounds the longer the factory runs unattended
far off
on target
steps the factory runs without a human check
Unattended: 1.01²⁰⁰ ≈ 7.3×
harder to recover, AI slop piles up
Checked every 25 steps: 1.01²⁵ ≈ 1.28×
Illustration, not measured data.
Purple dots = review checkpoints.
Small tacit-knowledge gaps, say 1%, grow as automation runs. The further the factory runs unattended, the harder it is to come back, and AI-specific slop accumulates. The curve is arithmetic, not data: 1.01 to the 200th is about 7.3, while resetting every 25 steps keeps the gap near 1.28. Supporting research: HumanLayer went fully lights-off in July 2025 and, by their own account, hit outages from code nobody could read and rewrote by hand in November; they argue training rewards pass/fail with "no penalty for eroding codebase maintainability". Addy Osmani calls the same effect "comprehension debt" when a factory goes dark. Both are self-reported or opinion pieces.
6 · Our view · Drift
So review, develop and get feedback on a fixed cadence, and stop the line when something looks wrong
Review humans check output
Develop agents build a slice
Feedback from users on site
Every cycle,
on a schedule
SUPPORT · TOYOTA
Jidoka
A machine stops itself on an
abnormality, so defects never
flow to the next step.
Andon
A signal to stop and call for
help at once. For us: when an
agent should escalate.
Because drift compounds, the answer is regular correction: review regularly, develop regularly, get feedback regularly, on a cadence rather than when someone remembers. Supporting idea from manufacturing: Toyota's jidoka means a machine stops itself on an abnormality so defects never flow downstream, and the andon is the signal to stop and call for help at once. In a software factory, that maps to an agent stopping and escalating to a human instead of building on top of a wrong assumption. Our survey found that the 2026 English-language debate cites lights-out plants like FANUC (for example OpenHands' dark-factory post) but almost never Toyota's jidoka or andon, so this is an angle a Japanese team can contribute.
WHAT WE DO NOW
UX flow who does what, in order
review
Design doc how the system works
review
Implementation plan steps the agents follow
review
Agents build
Each document lowers cognitive load: a reviewer reads one short layer at a time.
FOR COMPARISON · HUMANLAYER (RESEARCH)
Product review problem, mockups
System architecture services, schemas
Program design call trees, types
Vertical slices 1-3 at a time, reviewed
Same shape: agree on intent, then structure, then code-level plan, before agents build.
Our current practice: three document types, each reviewed before the next is written. The UX flow fixes who does what and in what order; the design doc fixes how the system works; the implementation plan fixes the steps the agents will follow. They lower cognitive load because a reviewer only has to judge one layer at a time, and they become the common language between people and agents. Supporting research: HumanLayer arrived at a similar staged sequence after their lights-off failure: product review, system architecture, program design with call-stack trees and type signatures, then vertical slices of 1 to 3 at a time. They also triage by size; about 40% of tasks are one-shot without the full sequence. HumanLayer sells tooling for this approach.
Factory Apartment block Skyscraper Submarine Rocket
One drawing the single source everyone builds from
Experiments in it prototypes, 3D-printed tests
Check documents derived from the drawing
Review meetings and more checks
Any 100 builders build the right thing, correctly
Why it works The drawing does not drift, so the result does not depend on who builds it.
For a software factory The three documents play the role of the drawing. Agents are the 100 builders.
Our view (analogy, no external source)
Why can thousands of people build a big factory, an apartment block, a skyscraper, a submarine or a rocket? Because everyone works from one drawing that has been properly reviewed. Experiments live inside it: prototypes and 3D-printed test parts. Check documents are derived from it, and review meetings and other checks sit on top. Because the drawing does not drift, the right building gets built correctly, by any 100 people. For a software factory, the reviewed documents from the previous slide play the role of the drawing, and the agents are the interchangeable builders. This is the presenter's analogy, not a cited claim.
Human tacit knowledge
what the customer meant
how the site really works
why a choice was made
AI tacit knowledge
what the model assumed
defaults it picked silently
code nobody has read yet
Review
over shared docs
both sides surface
what they assumed
The examples in each circle are our reading.
Reviewing more and harder is not sufficient on its own. There are two kinds of tacit knowledge: the human kind (what the customer meant, how the site really works, why a decision was taken) and the AI kind (what the model assumed, defaults it chose without saying so, code nobody has read). Review is where the two meet, and it works best over shared documents where each side writes down what it assumed. The examples inside the circles are our interpretation of "AI tacit knowledge". Supporting research: HumanLayer's program-design step has the model draft and the human argue with it, and their closing advice is "Read the dang code".
↓
Cost cheaper to build and change
↑
Features more shipped per month
?
Usability and outcomes no clear link to feature count
The first two follow from cheaper code. The third depends on knowing what users need, which is the tacit part.
For a single product, a factory clearly lowers the cost of building and grows the number of features. What we doubt is that more features lead to better usability or better outcomes for the customer. Those depend on understanding the user, which is the below-the-waterline knowledge from the iceberg slide. Supporting voice: Garry Tan, after generating about 500K lines of Rails, warned against volume-first "Foxconn" factories for agents. That is his opinion, not a measurement.
Co. A
Co. B
Co. C
Co. D
Co. E
Co. F
Co. G
Co. H
Co. I
Co. J
Teal layers = context accumulated in that company's factory (illustrative).
Use on site
Data and context pile up
Better output next time
Our view (no external source)
For enterprise customers, building factories for 10 companies means absorbing each company's tacit knowledge. So we expect to end up with 10 dedicated factories rather than one shared factory. Each one accumulates data and context from its company and gets better the more it is used, which is the compounding effect on the right. The heights of the teal stacks are illustrative only. This is our expectation, not something we have measured yet.
Coding
now easy
Everywhere:
sales · customer success · planning · consulting
Enterprise
Sales Internal approval (ringi) Embedding in the org Rollout on site
SMB
Sales Marketing Price competition Differentiation
Japan, especially
Onboarding Customer success Evangelism-style marketing Branding
Amber = new constraint. The Japan column is highlighted because it is our home market.
In the end, a software factory is a bottleneck shift. When coding becomes easy, the constraint moves to sales, customer success, planning and consulting. For enterprise: sales, internal approval (the Japanese ringi process), embedding the system in the organization, and rollout on site. For SMB: sales, marketing, price competition and differentiation. In Japan in particular: onboarding, customer success, and evangelism-style marketing and branding. Supporting research: Spotify's 2026 talk says coding is no longer the constraint, with the constraint moving to human decisions on what to build and review (we have not read that page in full). In the HN discussion of peterp's October 2026 post, commenters noted that distribution, not building, is now scarce. The segment lists are the presenter's view.
Coding is cheap
for everyone
?
A · All in on branding
Win on trust and recognition
that cheap code cannot copy.
B · Follow the scarcity
Keep moving to wherever the
bottleneck goes next.
Our view (open question, no external source)
If coding becomes cheap for everyone, the question is where to put our effort. Option A is to go all in on branding. Option B is to keep following where scarcity and the bottleneck move: sales, onboarding, customer success, and whatever comes after. The presenter has not decided; this is posed as an open question for discussion, and we come back to it in the discussion questions at the end.
AI builds the right app, accurately, for every department
Context
collected, kept, reused
Factory design
the skill to design software factories
Customer intent
grasp what they actually mean
Templates
turn it into templates and roll them out
A factory where context accumulates the more it builds
Our view (no external source)
The long-term question: if we build a factory where context accumulates the more it builds, can AI eventually make the right application, accurately, for every department? To get there we need four cornerstones. First, context itself, collected and kept. Second, the ability to design software factories. Third, the ability to grasp what customers actually mean, which is the iceberg problem again. Fourth, the ability to turn what we learn into templates and roll them out to the next department or customer. Pillar colors follow the role legend: context is an artifact, design and intent are human skills, templated rollout is where agents scale the work.
Closing · Summary
Five things to take away from today
1 · Definition A software factory is agents doing the building inside a pipeline of specs, checks and review gates. 2 · Field Designs range from lights-on with human review to dark, with no human reading code, and HumanLayer turned the lights back on. 3 · Tacit knowledge Most of what a customer knows is not written down. A factory only builds from what it is given (our view). 4 · Practice Grow lanes with demand, correct on a cadence, and share reviewed documents: UX flow, design doc, implementation plan (our view). 5 · Money Cheap code moves the bottleneck to sales, adoption and rollout. That is where the value goes (our view).
Lines 1 and 2 summarize the survey parts: Osmani frames a factory as many agent loops wrapped in review gates, Shapiro's levels run from autocomplete to the dark factory, and HumanLayer is the best-known report of a lights-off factory that went back to human review. Lines 3 to 5 are the presenter's own view from Part 6: tacit knowledge is the limit, the practical answer is growing lanes plus cadence plus shared documents, and the business question is where the bottleneck moves once coding is cheap.
Closing · Discussion
Five questions to discuss before we build our own factory
1 · Below the waterline For one current customer, what do we know only from being on site? How would we write it down?
2 · First lane Which lane should we build first, and who owns it?
3 · Cadence How often do we review, and what should make an agent stop and call a human?
4 · Our drawing Are UX flow, design doc and implementation plan enough? What is missing?
5 · Brand or scarcity If coding is cheap for everyone, do we bet on brand, or keep moving to the next bottleneck?
Our view
Use these to open the discussion. Question 1 tests the iceberg claim on a real customer. Question 2 makes the lane idea concrete. Question 3 asks for a review cadence and an andon rule. Question 4 challenges our three-document pipeline; HumanLayer, for comparison, also writes a system-architecture document and works in vertical slices. Question 5 is the open business question from slide F12.
Closing · Reading list
Twelve sources worth reading after this session If you read only two: HumanLayer's "Why Software Factories Fail" for the cautious side and StrongDM's factory site for the radical side; Willison's write-up is the easiest entry to StrongDM. Shapiro gives the shared vocabulary of levels. Osmani gives the cleanest light versus dark framing. OpenAI, Stripe and Spotify are first-party engineering reports. OpenHands and Wavect connect software factories to manufacturing and to Hitachi's 1969 Software Works. DORA and METR are the main counter-evidence to self-reported speedups. All twelve links come from the verified source list in our research data.