The machine behind the output

Anyone can get an agent to write code

An agent, here, means an AI tool given a task and allowed to act on its own for a while — reading files, writing code, running commands — rather than answering one question at a time. Getting one to produce code is easy; knowing when to believe the result is not, and that does not come from better prompting. It comes from structure: written contracts, provenance requirements, limits enforced in code, and independent verification.

What follows is the architecture I run, not a description of best practice.

One orchestrator, many narrow workers

A single orchestrator running the strongest model available holds the plan and makes the judgement calls. It does not do the work. It delegates almost everything, and the reason is the single most important design constraint in the whole system.

Context discipline is the design constraint, not an optimisation

An agent's attention is finite, and as its context fills, quality degrades gradually and invisibly. No error, no warning, just steadily worse judgement — the one failure mode you cannot catch by watching, so it has to be designed against up front.

So the orchestrator is instructed to keep its own context light. It delegates reading rather than reading into itself, and subagents return conclusions instead of file dumps. An orchestrator that has absorbed forty files is blunt. One that has absorbed forty summaries is still sharp, and can still hold the whole plan.

Context files as durable infrastructure

An agent that rediscovers a codebase every session wastes most of its budget doing it, so each repository and agent role carries a context file: where things live, the conventions, the hard rules, what must never be touched. The repository-structure map is a maintained artifact, not a byproduct.

Two of these are real and public in effect. One is the operating contract for my knowledge base, described below. The other is the rules file in the repository for this website, which carries a table of six figures that were retired as wrong — so that a future agent editing this site cannot quietly reintroduce them. The constraint lives where it will be enforced, not where it will be remembered.

Dispatch by difficulty

Not every task deserves the strongest model. Renames, greps, boilerplate, mechanical refactors and formatting go to fast cheap models at low reasoning effort. Ordinary implementation goes to the middle tier. The strongest model at the highest effort is reserved for architecture, ambiguous debugging and adversarial review. The orchestrator makes that call per task rather than the whole session running at one setting.

The economics are not subtle: running everything at maximum is slow and expensive, and a rename does not benefit from deeper reasoning.

Parallelism, with barriers only where they are earned

Independent work fans out concurrently. The instinct is then to wait for all of it before the next stage, and that instinct is usually wrong. A barrier is only justified when a stage genuinely needs every prior result at once: deduplicating across a complete set, or exiting early because there was nothing to find.

Otherwise the work pipelines, so one item can be in its third stage while another is still in its first. The wall-clock cost is then the slowest single chain rather than the sum of the slowest step in each stage. Where agents would edit the same files, each gets an isolated copy of the repository so they cannot collide.

Permissions scoped per agent

Every agent class gets the narrowest scope that still lets it work. Read-only explorers cannot write. Implementers can write but cannot touch version control. Anything irreversible or public — a push, a deploy, a deletion, a published document — stops at an explicit human gate.

This website was built across five rounds on a branch never pushed without my review. Not caution theatre — the control that makes the autonomy safe to grant.

Every claim carries its source, and "not found" is a legal answer

Anything the agent writes into my knowledge base is tagged with its source: a dated attribution, a version-control log, a file path. Unconfirmed claims are marked as such.

The rule doing the real work is that the agent must never invent a fact, a number or a date, and that NOT FOUND is a valid and preferred answer. Hallucination is not prompted against; it is designed out by making the absence of a fact something the output format can express.

The contract the knowledge base runs under

I produce thought; the agent owns structure. It files, links and propagates, and never asks me where something goes. Autonomy is stated up front as file freely, report after: placement is the agent's call and I veto afterwards. Ambiguous items stay queued rather than placed — an unfiled note is recoverable, a misfiled one vanishes — so the agent is told which way to err. Routing keys on what kind of fact something is rather than its topic, which makes it decidable. Questions are batched to the end and capped at five.

It also names its own anti-patterns. One tracking file is capped at five rows, because I once over-built a nineteen-item registry with a dashboard and retired it; if it wants to grow, that is a signal to raise with me. The tag vocabulary is closed at five: one that grows without resistance stops separating anything. Blast radius is bounded too — two directories frozen, two files append-only, no version-control access.

Enforce it in code, because a prompt is a hope

The clearest example is the market-data portal. Telling a model not to leak internal material is a hope; a build gate that fails the build when it happens is a guarantee. That gate exists because two real leaks got past an earlier exclusion operating at the rendering layer rather than the build layer.

tools/check_no_leaks.mjs — the gate that fails the build

let leaks = 0;
for await (const file of walk(distDir)) {
  const text = await readFile(file, 'utf8');
  for (const { pattern, why } of FORBIDDEN) {
    const match = text.match(pattern);
    if (match) {
      console.error(`LEAK  ${relative('.', file)}\n      matched "${match[0]}" — ${why}`);
      leaks++;
    }
  }
}
if (leaks) {
  console.error(`\n${leaks} leak(s). Internal material must not ship. Fix before deploying.`);
  process.exit(1);
}

tools/extract_sample.py — refusing to guess

    raise NotImplementedError(
        "Archive layout unknown — see docs/schema.md 'File layout' [VERIFY]. "
        "Do not guess a partition scheme; confirm it first."
    )

Eight functions in that file raise rather than return. A half-built extractor emitting a plausible sample is worse than one that does not run: the sample is what a buyer checks the dataset against.

Decide what counts as success before the run

In the research work this is literal pre-registration: a written specification committed to version control, with a timestamp, before any result exists. It is what let me report that my own model failed its primary test rather than re-cutting the metric until something passed.

The same device works on agent output. Decide what done means after seeing the output and you will grade it on whatever it happened to do. This is worth one sentence of context, because it is not obvious from outside: reporting a result that went against you is rare, and the reason is an incentive. When something comes out flat, the quiet option is to adjust the measurement and try again until it looks convincing, and nobody can tell afterwards. Fixing the target in advance removes that option.

Fan out, then reconcile

I surveyed 53 project directories across parallel agents, each entry recording the surveying agent, a confidence rating, and a flag for credentials or personal data. What mattered was the reconciliation afterwards: twelve ambiguities resolved by hand — which checkout was current, whether two repositories were one project, three similarly-named directories that proved unrelated. Fan-out is the cheap half; skipping reconciliation is how you get a confident, internally inconsistent answer.

Propagation, not filing

The agent's real task is checking new facts against what is already recorded. Two catches, both against me.

Gates that refuse an unproven claim

The contract above is written down and published rather than re-improvised per task. default-fail is git hooks, subagent definition files, criteria files and shell scripts — nothing imports it and nothing runs a supervisor process. Every criterion starts at fail, and only a citation that resolves moves it: a file:line I can open, or a command with its real output. The grader is a read-only subagent working from a context that never saw the work being produced. Until the hooks are installed none of it is enforced, and the repository says so in those words rather than implying otherwise.

The part worth reading is not the workflow, though. It is that the instruments get tested too. A claim-checker I had written to catch fabricated numbers returned ok on the exact fabrication it was built to catch: its mask for version-like tokens was \d+\.\d+, which also matches 0.66 and 0.001 — every p-value in the corpus. Masked tokens are stripped before comparison, so it deleted the entire class of number it existed to check, then reported success having examined nothing. It printed tokens it checked: [], which nobody read.

That one is written up with the rest, including the gates found broken while the proof suite was still green. The repository carries the detail; this page does not try to.

default-fail on GitHub →

This website, and the six errors the pipeline caught

This site was built by the system described above, across five rounds. The point of the list below is not that the agent was right — it is that these were found, by checks that existed before the work started.

• A verification harness that lied. The mobile-overflow check reported failures that were not real, because headless Chrome on Windows clamps the window size and crops the screenshot instead of reflowing the page. Rebuilt on the browser's debugging protocol so the viewport is genuinely emulated.
• Five of eight figures silently broken. The chart generator emitted unclosed elements — harmless while the markup sat inside a page, fatal once the same file was loaded as an image and parsed strictly. Caught in a dark-mode screenshot showing alt text where a chart should have been. The generator now validates every figure before writing it.
• Identifiers leaking on an embargoed page. Shortened market identifiers are effectively unique within the corpus, so publishing them alongside per-market statistics would have identified specific markets. Pseudonymised, and the page now forbids pasting the raw documents in.
• A contrast failure no amount of tuning could fix. A colour-shaded table could not hold legible text in both light and dark themes at any shade. Replaced with a value-and-bar encoding at 18:1.
• Two numbers that disagreed. Prose said one figure was 27 times another; the published data divided to 28. One had been computed on averages and the other on medians. A reader could have checked it in seconds.
• Content hidden behind JavaScript. A scroll animation hid every below-fold section and handed visibility back via script — what the rules forbid. Caught by the check that loads every page with scripting disabled.

Check what the agent says it did

Gates run before anything merges: contrast across every colour pair in both themes, layout at two widths, prohibited-content sweeps, and idempotence checks on every generator so a stale figure is detectable rather than silent. The agent proposes; I merge.

Just as important, the agent reports what it did not verify: no Lighthouse score appears on this site, because Lighthouse was never run.

Where I am taking it next