Measuring an agent orchestration pattern

What caught things was independent reading

Start with the part the design got wrong. This pattern leans on enforcement: written contracts an agent has to satisfy, gates it has to pass, checks that run whether or not anyone asks for them. That machinery is what I would have defended if challenged, and across the two synthetic beds and the five research projects it produced approximately one mechanical refusal.

That count is approximate on purpose, and it is worth being precise about why. It is a tally of refusals I happened to notice, in an unblinded record, with no mechanism anywhere in the pipeline that would have logged a counter-instance. So it is a ratio between things I saw, not a rate over things that occurred, and it should be read as an order of magnitude rather than as a measurement. What it is good enough to support is the comparison: the enforcement machinery was not where the catches came from.

They came from reading. A session belonging to an unrelated project, and an auditor with no stake in the outcome — each handed the situation rather than a checklist. The absence of the checklist is the active ingredient rather than an oversight. A checklist can only carry what somebody already anticipated, and the failures worth catching are the ones nobody did. The enforcement is not worthless: it is cheap, it runs unattended, and it holds a floor. It is simply not what found the problems, and I had the emphasis backwards.

The seal taught the same lesson at a lower layer

Every dataset in the two beds was generated with a known true answer, and that answer was sealed away from the agent behind permission rules — the whole point of the exercise being that a result you can check beats a result you assert. An audit tried 18 routes against those rules, and every one of the 18 succeeded: literal paths, tilde and absolute forms, paths assembled from shell variables at runtime, globs, upward traversal, find, other interpreters — Python, Perl, Node — and a tar of the whole tree. The audit reported reaching the file inventory and stopped there without reading the files.

The mechanism is not subtle once seen. The rules matched on command text, and a shell is not command text: no rule that reads a string can evaluate one, so any construction that defers the path until runtime is invisible to it. What held was an operating-system boundary — a root-owned directory the agent's account could not read at all. The tool layer is where the rule was easy to write. The kernel is where it was true.

Both failures are the same failure. A rule that lists what it expects catches only what was expected of it.

A dataset where I already knew the answer

Seven projects over roughly four weeks in August 2026: five ordinary research projects, and two synthetic beds built to make the pattern measurable rather than describable.

The distinction between those two groups is the one that matters, and softening it would wreck the piece. The five research projects have no sealed answer. Whatever an agent produced on them can be reviewed, argued with and improved, but it cannot be scored, because there is no key to score it against. Only the two beds had one.

A bed is a synthetic dataset generated for the purpose, so that the true relationship between the columns and the outcome is a thing I chose rather than a thing to be discovered. Into each one I planted a trap: a column that looks powerfully predictive and is inadmissible, because its value does not exist at the moment the prediction is for. This is target leakage — information about the outcome smuggled into the inputs — and it is the single most common way a modelling result is quietly wrong. A model that uses it scores brilliantly and is worth nothing, because at the moment a real decision has to be made the column is blank.

Two things were then sealed away from the agent: the generating process, and the answer key. The consequence is what makes the exercise worth running. The agent could not compute how good a model could possibly be, so it could not grade itself, tune toward the key, or argue with the score afterwards. I could compute all of that from the sealed side, with a script the bed never had access to. The results below are checkable rather than asserted, and that property comes entirely from the boundary rather than from anything the agent did.

The first bed reported a number above its own ceiling

The agent used the planted trap and reported a headline of 0.7802. Excluding the trap, the honest score for that same work is 0.1994. The best score any model could have achieved on that bed without the trap — measured afterwards from the sealed side — is 0.367.

So the reported headline was more than twice the ceiling of the problem it claimed to have solved. That is not a subtle statistical error. It is arithmetically impossible on its face: a number above the ceiling is proof that information from after the outcome is in the feature set, and it needs no domain knowledge to spot, only the ceiling. Nothing in the pipeline remarked on it — not the contracts, not the gates, not the automated checks, and not the agent, which had no ceiling to compare against because I had sealed it away.

That last clause is the honest reading, and it cuts both ways. The seal is what makes the measurement trustworthy and it is also what removed the one check that would have caught the error instantly. The revision after this bed was to stop expecting enforcement to notice, and to put an independent reader in the loop instead.

The second bed excluded the trap on the reasoning that settles it

The second bed ran on 28 August 2026, on a card-fraud problem: 833,391 transactions from 2024 to train on, 104,160 from the first quarter of 2025 to score against, sharing no customers at all. Fraud is 2.62% of the training file and 2.06% of the scoring file.

Before fitting anything, the agent was required to ask of each of the 34 columns a single question: at what moment does this column's value become known, relative to the moment of the transaction being predicted? Thirty could be placed as knowable then. Two could not be placed at all and were recorded as unresolved rather than waved through. Two could be ruled out — one is the outcome itself, and the other is the trap: the difference between the amount authorised and the amount finally settled. Settlement happens after authorisation. The column is a fact about what came next, so it cannot be an input to a decision made at the transaction.

The agent excluded it on that reasoning, from the column's name alone, before measuring anything. It then measured what the exclusion cost, as three separately fitted models rather than one subtraction: adding the trap back to the best model, and nothing else, raises average precision — a ranking score where higher is better — from 0.3971 to 0.6194, a difference of +0.2231 (95% paired bootstrap interval +0.2071 to +0.2383). Every improvement the search found across its whole budget, added together, is a small fraction of that one number.

What the honest score is worth

The best admissible model scored 0.3971. From the sealed side, the best achievable average precision on those same admissible columns is 0.4198. So the search reached 94.6% of the achievable score — a figure I can state only because I held the key, and one the report itself leaves standing as PENDING because it was sealed from the material needed to compute it.

Two comparisons from the same sealed measurement put that in proportion. An out-of-the-box gradient boosting fit on the identical 27 admissible columns scores 0.2706, so the search bought +0.1265 over simply running the library — the tuning was worth something real. And the absolute bound using every column, traps included, is 0.5560: any score above that line is proof on its face that post-outcome information has reached the feature set. A line of that kind is the cheapest check in this whole programme, and it is the one bed one had no access to.

Three changes carried the search

The search is the less interesting half of that number, and it is worth seeing whole. Across its budget it made 14 distinct changes. Three produced a gain whose interval excludes zero, three produced a real loss, and the remaining 8 bought nothing that can be told apart from noise — including four consecutive attempts at regularisation, and the constructed ratios the final model leans on hardest.

Read the full technical report for the second bed (HTML, 1.0 MB) — the agent's own account of every run, generated end-to-end from its committed result files with no number typed by hand. It is reproduced as it was written, including the sections where it declines to answer.

One inference short

The second bed carried a second trap, and what happened to it is more interesting than either catching it or missing it would have been.

The column is a per-merchant risk score whose name states no time window — no "as of", no period. The agent investigated it properly. It established that the score is constant within each merchant and identical for the same merchant across both files. It then computed the two quantities that settle the question: the score correlates 0.943 with that merchant's fraud rate over the training period and 0.838 with the same merchant's fraud rate over the scoring period, while the two periods' own rates correlate only 0.620 with each other.

Read those three numbers together and they say something specific. A score fixed before the data begins should track each period about as well as the periods track one another. This one tracks both periods far better than they track each other, which is the signature of a quantity computed across both — that is, of a summary that has seen the outcomes it is being used to predict.

The agent computed both correlations, wrote them down, recorded the column unresolved, and excluded it from every reported model. What it did not do is take the last step and say what the pattern means. Its own report states the two readings are "consistent with a standing merchant attribute fixed in advance and with a summary computed across both files, and the two files cannot tell them apart" — a conclusion that is defensible on the evidence it allowed itself, and one step short of the evidence it had already produced.

The behaviour was right and the inference was missing. That is a far more useful failure mode to know about than a miss, because a miss looks like inattention and this looks like caution.

One prediction held and one was wrong

Both were written down and sealed before the second bed ran. Publishing the one that turned out wrong is what makes the rest of this page worth reading.

A prediction I got wrong is worth more here than the one I got right, for two reasons. The first is evidential: written predictions are only worth anything if the falsified ones get published, and a page reporting two hits would be indistinguishable from a page where the predictions were written afterwards. The second is that being wrong is what located the near-miss at all. I expected a miss, found an investigation that stopped one inference short, and that gap is the most specific thing this programme learned about where the pattern needs work — an agent that reasons carefully and then declines to conclude will not be caught by any gate that checks whether the work was done.

Limits of the evidence

This was measured twice. The measurement is the contribution, and it is not a proof that the pattern works.

• It is a measurement, not a benchmark. The traps were planted by a session I directed, sealed by me, and scored by me. Two beds, one operator, one model family. Nothing here is a controlled comparison against an alternative, and the numbers should not be read as though it were.
• Only two of the seven projects had sealed answers. The other five are ordinary research work with no key to score against. They inform the refusal count and nothing else on this page.
• The refusal count is a ratio, not a rate. An unblinded record, with no mechanism that would have logged a counter-instance, cannot produce a denominator. It supports the direction of the comparison and no figure derived from it.
• The second bed's headline is optimistic by an amount nothing can bound. No untouched holdout was sealed, by design, and the scoring rows were consulted 583 times over the course of the search before the reported number was computed on them. The 94.6% is a ratio between two figures, one of which carries that optimism.
• Applying a rule is not the same as noticing a problem. The second bed was instructed in advance, and with emphasis, to check every column's availability. It complied, and complying is all it demonstrates. Whether an unprompted session would have found the trap is exactly the question this design cannot answer, and the agent's own report says so three separate times rather than parking it in a footnote.
• The search found a plateau rather than a winner. Eight of the second bed's configurations are statistically indistinguishable at the top, and of the fourteen distinct changes the search made, three produced a gain whose interval excludes zero. Calling any one of them "the best" overstates what was measured.

← The architecture this measures