Predicting card fraud in a fixed search budget
A record of how a search with nothing fixed in advance spent
25 runs — what was tried, what each attempt scored, and why each one was
chosen against the one before it. Project synthetic-fraud-2. The data is synthetic:
it was generated for this project, and how it was generated is not available here.
The three things this report has to say before anything else.
1. What it establishes. How a free search allocated 25 attempts at a modelling problem. Nothing was decided before the first attempt; each configuration was chosen with the previous result in front of us, and the written reason for each choice was recorded before the attempt was run, so that it is a record of intent rather than of hindsight. Every one of the 25 appears below, including 1 run that produced no score at all, and the many that changed nothing.
2. What it does not establish. Whether anyone would have noticed that one of the columns is unusable without being told to look. A column can only be an input to a prediction if its value existed at the moment the prediction is for, and this project was instructed, in advance and with emphasis, to check that for every column and write the answer down before fitting anything. It did, it found a column that fails the test, and excluding that column turns out to matter enormously — but applying a rule you were handed is not the same achievement as spotting an unstated problem, and nothing here is evidence about the second. That distinction is repeated wherever the column check is discussed rather than parked in a footnote.
3. What the final number is an estimate of. It was computed on all 104,160 rows of the validation file — and the search consulted those same rows 583 times before computing it, because several runs read their score at dozens of intermediate points. There is no untouched holdout in this project, so no figure in this report estimates performance on data the search has not touched. That sentence is meant literally: the headline number is optimistic by an amount this project cannot measure.
What happened, in plain language
The task was to predict which card transactions are fraudulent, from a file of 833,391 transactions covering 2024-01-01 to 2024-12-31, and to score the result on a second file of 104,160 transactions covering 2025-01-01 to 2025-03-31. Fraud is rare — it is 2.62% of the training file and 2.06% of the validation file — and the two files share no customers at all (20,000 customers in one, 10,000 in the other, 0 in common), so the score measures generalisation to a later period and to people the model has never seen.
Before fitting anything, every one of the 34 columns was asked a single question: at what moment does this column's value become known, relative to the moment of the transaction being predicted? 30 columns could be placed as knowable at that moment. 2 could not be placed at all and were recorded as unresolved rather than waved through. 2 could be ruled out — one of them is the target itself, and the other is the difference between the amount authorised and the amount finally settled, which is a fact about what happened after the transaction and therefore cannot be an input to a decision made at it. That left 27 columns the models were allowed to use.
Excluding that one column is the largest single effect measured in this project. Adding it back, and nothing else, to the best model raises average precision — a score explained below, where higher is better and 0.0206 is what pure guessing gets — from 0.3971 to 0.6194. That is the settlement column on its own changed average precision by +0.2231 (95% paired bootstrap interval +0.2071 to +0.2383). Every improvement the search found across all 25 runs, added together, is a small fraction of that one number.
The two columns that could not be placed are worth far less: adding both of them changed average precision by +0.0169 (95% paired bootstrap interval +0.0127 to +0.0209). So the expensive exclusion was the one that needed no measurement — its name settled it — and the two genuinely uncertain ones cost comparatively little. This is a demonstration that a rule handed to us in advance was applied to every column rather than to the interesting ones. It is not a demonstration that we would have thought to ask.
On the admissible columns the search moved the score from 0.2505 to 0.3971, and then stopped moving. Of the 14 distinct changes the search made, exactly 3 produced a gain whose uncertainty does not include zero: switching from a linear model to a tree ensemble, halving the learning rate once, and giving the model each merchant's own fraud rate instead of an arbitrary identifier. Everything else — more capacity, less capacity, several different regularisers, re-weighting the classes, dropping the apparently useless columns, adding constructed ratios — landed inside the noise. The search plateaued, and the plateau is the result.
The stopping reason, named as exactly one of the three this project allows itself: declared budget exhausted. All 25 runs were spent; run 26 would have been refused.
Everything below this line is the technical account, for a reader who wants to check the work rather than take it.
1. What each model actually does
A logistic regression adds things up
A logistic regression assigns one weight to each column, multiplies each column's value by its weight, adds the results, and squashes the total into a probability. That is the whole model. Its consequence is a hard limit on shape: because the contribution of each column is added on independently, the contribution of amount is the same whatever the merchant category is. The boundary it draws between "fraud" and "not fraud" is a single flat surface, and there is exactly one of them.
A boosted tree carves out boxes
A decision tree asks a sequence of threshold questions — "is the amount above 200?", then "is the merchant category electronics?" — and each answer narrows the region it is talking about. The region a leaf describes is a box: a combination of ranges across several columns at once. Because the second question is asked inside the answer to the first, the effect of amount is allowed to differ by merchant category. Boosting means building many small trees in sequence, each one trained to correct what the previous ones got wrong, and adding their contributions together.
That difference is measurable here and it is the largest thing the search found: replacing the linear model with the tree ensemble on identical raw columns changed average precision by +0.1039 (95% paired bootstrap interval +0.0859 to +0.1218). It is not an artefact of the tree getting better inputs: when the linear model was later given the tree's own improved inputs, it did improve — the better representation changed average precision by +0.0436 (95% paired bootstrap interval +0.0316 to +0.0560) — but the tree still led it by essentially the original margin, with both models on identical inputs changed average precision by +0.1023 (95% paired bootstrap interval +0.0869 to +0.1175).
2. The mathematics, only where it changes what you understand
What both models minimise
Both families here are fitted by minimising log loss, the average of −log(predicted probability of the outcome that actually happened):
In words: the model is punished in proportion to how confidently it was wrong, and the punishment grows without limit as a confident prediction approaches certainty on the wrong side. The reason this matters here is that it is not what we score. Log loss asks the model to get every transaction's probability right, so it is dominated by the 97.9% of rows that are not fraud. That mismatch is what run 19 tested by re-weighting the classes, and re-weighting made things slightly worse: weighting each fraudulent transaction far more heavily changed average precision by -0.0073 (95% paired bootstrap interval -0.0140 to -0.0000).
Average precision — the score used here, and what it is blind to
Rank every transaction by predicted probability, walk down the list, and at each position record precision (of everything flagged so far, what fraction really was fraud) and recall (of all the fraud there is, what fraction has been caught so far). Average precision is the area under the curve those two trace out. A ranker that orders transactions at random scores the base rate — here 0.0206 — so the best model found, at 0.3971, ranks fraud roughly 19 times better than chance.
What it is blind to: the actual probabilities. Average precision depends only on the order, so a model that says every transaction is 90% likely to be fraud, in the right order, scores identically to one that is perfectly calibrated. If you need to act on the number — to set a threshold, or to price a decision — average precision has told you nothing about whether the number means what it says.
ROC AUC — reported alongside, and why it is not steered on
ROC AUC is the probability that a randomly chosen fraudulent transaction is ranked above a randomly chosen legitimate one. What it is blind to is class imbalance, and here that blindness is not theoretical — it is visible in the data:
Concretely: in run 5, average precision peaked at 0.3592 at 130 rounds and had fallen to 0.3202 by the end, a loss of 0.0390 — while ROC AUC over the same rounds went from 0.8912 to 0.9336 and ended at 0.9332. Because almost every transaction is legitimate, adding a few thousand false alarms barely moves the false positive rate but wrecks precision. ROC AUC is reported at every run because a curve that is never scored cannot honestly be discussed, but it is not what the search steered on.
3. Every tuning knob: what it was aimed at, and what it actually changed
Including the ones that did nothing, which is most of them.
| Knob and range tried | What it controls | What was expected | What actually happened |
|---|---|---|---|
| Model family logistic regression against a boosted tree ensemble | Whether the model can represent a relationship that depends on two columns at once. | A large gain for the tree if the problem is interaction-shaped; nothing much if it is not. | The single largest gain in the search, and it survived giving the linear model the tree's own improved inputs: on identical inputs the tree still led, by 0.1023. measured: +0.1039, 95% paired bootstrap interval +0.0859 to +0.1218 |
| Boosting rounds 100 → 400, then read out at every round count from one fit | How many small trees are added, one correcting the last. | More rounds to help, since 100 is a default chosen for nobody in particular. | More rounds made it WORSE. The curve rises, peaks and then falls away, which is the shape of a model that has started memorising rather than one that has run out of capacity. |
| Learning rate 0.1 → 0.05 → 0.025, rounds raised each time to match | How large a correction each new tree is allowed to make. | Smaller steps to help, being the standard remedy for an ensemble that overshoots. | The first halving paid. The second did not: halving it again changed average precision by +0.0037 (95% paired bootstrap interval -0.0007 to +0.0078, an interval that straddles zero — which is to say the change bought nothing that can be told apart from noise). The axis was exhausted and the search left it. measured: +0.0205, 95% paired bootstrap interval +0.0132 to +0.0271 |
| Leaves per tree 31 → 63, and 31 → 15 | How many columns a single tree can chain together along one path. | More leaves to help if the remaining error is made of interactions between more columns than the trees can currently reach. | Nothing in either direction — halving them instead changed average precision by -0.0010 (95% paired bootstrap interval -0.0076 to +0.0055, an interval that straddles zero — which is to say the change bought nothing that can be told apart from noise). A score that does not move when capacity is halved OR doubled is not capacity-limited, and that is what sent the rest of the budget to the features. measured: -0.0003, 95% paired bootstrap interval -0.0062 to +0.0059 — straddles zero |
| Merchant identity encoding arbitrary integer code → that merchant's own fraud rate | Whether the tree can use which merchant this is, rather than only splitting an arbitrary numbering into ranges. | A real gain, because merchant identity is not noise and the code form was unusable. | The only change to the model's inputs that paid. It also halved the number of rounds the model needed, which is why run 10 measured it past its own optimum and run 11 had to refit. measured: +0.0187, 95% paired bootstrap interval +0.0095 to +0.0281 |
| Minimum transactions per leaf 20 → 100 | How much data a leaf must rest on before the tree may create it. | A gain, by forbidding rules built on two or three fraudulent cases. | No change in the score, but a real change in shape: the curve stopped falling away after its peak and stayed flat to the ceiling. A regulariser fixing a problem that was not costing anything. measured: +0.0007, 95% paired bootstrap interval -0.0022 to +0.0036 — straddles zero |
| Derived ratios 5 ratios between columns already in the model | Gives the model a comparison it cannot construct, since a tree splits one column at a time. | A gain, because the closest thing to this was one of the columns that had to be excluded. | Nothing measurable — and yet the top-ranked ratio is the column the final model leans on hardest. Run 25 resolved the apparent contradiction; see below. measured: +0.0022, 95% paired bootstrap interval -0.0044 to +0.0084 — straddles zero |
| Column subsampling every column at each split → a random 60% | Makes the trees disagree with each other, so the ensemble averages over more distinct views. | A gain, this being a different regularising mechanism from the ones already tried. | Nothing measurable. This is the configuration that scored highest, but its lead over the ones without it is not distinguishable from noise. measured: +0.0008, 95% paired bootstrap interval -0.0041 to +0.0061 — straddles zero |
| Class re-weighting off → each fraudulent transaction weighted about 37 times | Points the objective at the rare class instead of at the overwhelming majority. | Possibly a gain, since the objective and the score measure different things. | It HURT, slightly, by an amount whose interval reaches exactly to zero. measured: -0.0073, 95% paired bootstrap interval -0.0140 to -0.0000 |
| Leaf-value shrinkage 0 → 10 | How loudly a leaf may speak once it exists, as opposed to where it may be placed. | Little, by this point, but the report has to say what every knob did. | Nothing measurable. The fifth consecutive knob to land inside noise. measured: -0.0008, 95% paired bootstrap interval -0.0049 to +0.0033 — straddles zero |
| Feature set all 27 admissible columns → the 15 that showed class separation | Whether columns that look uninformative on their own are costing anything. | A gain, or at worst nothing: 12 columns that separate nothing are 12 chances per split to fit noise. | It HURT. A column whose fraud and non-fraud distributions look identical on its own is not therefore useless to a model that can combine it with five others — which is exactly what the profile written before the first fit could not see. measured: -0.0056, 95% paired bootstrap interval -0.0098 to -0.0009 |
| Encoding fold split seed 0 → seed 7 | Nothing about the model. It measures how much of the encoding's gain is an accident of a random 5-way split. | A small movement if the gain is real, a large one if it is an artefact. | Small, and inside noise. The encoding's advantage is a property of the encoding, not of the split. measured: -0.0018, 95% paired bootstrap interval -0.0060 to +0.0022 — straddles zero |
The one apparent contradiction, and its resolution. The chart below ranks
amount_over_avg30d — this transaction's size relative to what the customer normally
spends — as the column the best model leans on hardest. Yet adding the constructed ratios changed
the score by nothing measurable (the 5 ratios together changed average precision by +0.0022 (95% paired bootstrap interval -0.0044 to +0.0084, an interval that straddles zero — which is to say the change bought nothing that can be told apart from noise)).
Those two facts are both true and they are not in conflict: importance measures reliance,
not value. The tree reaches for the ratio first because it is the most convenient single
column, but deprived of it the tree can rebuild a staircase approximation from amount
and avg_amount_30d, both of which are still present. The last run of the budget tested
this by removing those two raw columns and keeping only the ratios:
dropping the two columns the top-ranked ratio is built from changed average precision by -0.0036 (95% paired bootstrap interval -0.0088 to +0.0011, an interval that straddles zero — which is to say the change bought nothing that can be told apart from noise). The ratio carries what the pair carried.
4. How to read the two curves
Reading it. The horizontal axis is recall: of all the fraud in the file, what fraction this model has caught by the time you have gone this far down its ranked list. The vertical axis is precision: of everything flagged by that point, what fraction really was fraud. A good curve stays high as it moves right — catching more fraud without drowning in false alarms. The dotted horizontal line is where a random ranker sits, at the fraud rate itself (0.0206).
What it cannot tell you. It cannot tell you where to set a threshold, because it does not show which predicted probability each point corresponds to. It cannot tell you what a false alarm costs relative to a missed fraud — that is a business fact, not a curve. And every point on it is computed on rows this search consulted 583 times, so the curve is optimistic in a way it cannot itself display.
Reading it. The horizontal axis is the fraction of legitimate transactions wrongly flagged; the vertical axis is the fraction of fraud caught. A good curve climbs steeply towards the top-left corner. The diagonal is a random ranker.
What it cannot tell you. With 97.9% of rows legitimate, moving the false positive rate by one percentage point means flagging about 1,020 extra legitimate transactions — so a change that looks negligible on this axis can be the difference between a usable and an unusable system. The two models furthest apart on the precision-recall plot are nearly superimposed here.
5. The ceiling comparison
The natural question about any score is how much of the achievable performance it captured — how far 0.3971 is from the best any model could do on these columns, and what the remaining gap is made of. That gap has two possible components: error that no model could remove because the columns simply do not determine the outcome, and error a better model would remove.
Best achievable average precision on the admissible columns: PENDING — not computable from the material available to this project
Composition of the gap between 0.3971 and that ceiling: PENDING — not computable from the material available to this project
Computing a ceiling requires knowing the true probability of fraud for each transaction, which is a property of how the data was generated. That is not available here and this project is forbidden from estimating it or from reasoning about the generating process. The section is left standing with its structure intact rather than dropped, because an absent section reads as an oversight and a filled-in guess would be worse than either.
6. Where Juan was required
Nowhere. Across the whole session — the inventory, the column-availability declaration, all 25 runs and this report — nothing was escalated and the project sat blocked for zero minutes. This is recorded as a finding rather than left as an absence, so that "he was not needed" can be told apart from "nobody wrote it down".
Two points came close enough to name. The first was whether to seal off part of the validation file so that one honest, unbiased number could be reported at the end; that was decided without asking, because this project's governing document states the no-holdout design as a fact rather than an option, so there was nothing to ask. The cost of that design is the third of the three statements at the top of this report. The second was whether a single fit could be read out at 40 different round counts, buying an entire curve for one run at the price of consulting the validation rows 40 times instead of once; that was decided without asking because choosing what to run next is the task itself. Both are written up with their rejected alternatives in the project's decision log.
7. Every run, in the order it was run
Rows shaded pink and marked ✕ used a column this project excluded on availability grounds. Those are not candidate models and are not eligible to be the reported best; they are measurements of what the exclusions cost. The "against" column names the run each configuration was chosen to answer, which is the same run its written justification was written against, before it was run.
| # | What changed, and why | Average precision | Against its predecessor |
|---|---|---|---|
| 1 | Logistic regression on the admissible columns The floor. A model that can only add up a weighted contribution from each column. | 0.2505 95% interval 0.2332 to 0.2701 | no predecessor |
| 2 | Boosted tree at library defaults Errored before producing a score: the library refuses a categorical column with more than 255 levels and the merchant identifier has 600. The row was consumed anyway. | errored — no score | — |
| 3 | Boosted tree at library defaults, merchant identifier passed as an integer code The repair of run 2, changing only what the error forced. | 0.3542 95% interval 0.3328 to 0.3769 | ▲ +0.1039 interval +0.0859 to +0.1218 against run 1 |
| 4 | Four times the boosting rounds Testing whether run 3 had finished learning. | 0.3202 95% interval 0.3011 to 0.3419 | ▼ -0.0340 interval -0.0437 to -0.0248 against run 3 |
| 5 | Run 4 repeated, instrumented to record the score at every tenth round count One fit bought the whole round-count curve instead of one point on it, and doubled as a check that the harness is deterministic. | 0.3202 95% interval 0.3011 to 0.3419 | ≈ +0.0000 interval +0.0000 to +0.0000 — straddles zero against run 4 |
| 6 | Learning rate halved to 0.05, round ceiling raised to match Smaller steps, more of them, because the curve showed the ensemble overshooting. | 0.3748 95% interval 0.3534 to 0.3963 | ▲ +0.0205 interval +0.0132 to +0.0271 against run 3 |
| 7 | Learning rate halved again to 0.025 Testing whether the learning-rate axis was still paying. | 0.3784 95% interval 0.3572 to 0.4001 | ≈ +0.0037 interval -0.0007 to +0.0078 — straddles zero against run 6 |
| 8 | Leaves per tree doubled to 63 Letting each tree combine one more column along any path. | 0.3745 95% interval 0.3530 to 0.3973 | ≈ -0.0003 interval -0.0062 to +0.0059 — straddles zero against run 6 |
| 9 | Leaves per tree halved to 15 The other direction, so that a flat result means the capacity axis is flat rather than that 31 happened to be optimal. | 0.3737 95% interval 0.3524 to 0.3969 | ≈ -0.0010 interval -0.0076 to +0.0055 — straddles zero against run 6 |
| 10 | Merchant identity given as that merchant's own fraud rate rather than an arbitrary code Measured at the previous run's round ceiling, which turned out to be well past this configuration's own optimum. | 0.3817 95% interval 0.3603 to 0.4049 | ≈ +0.0070 interval -0.0036 to +0.0179 — straddles zero against run 6 |
| 11 | The same encoding, refitted at its own best round count So a paired interval could be computed from stored predictions rather than read off a curve. | 0.3934 95% interval 0.3719 to 0.4160 | ▲ +0.0187 interval +0.0095 to +0.0281 against run 6 |
| 12 | Minimum transactions per leaf raised from 20 to 100 Forbidding rules supported by two or three fraudulent cases. | 0.3941 95% interval 0.3731 to 0.4170 | ≈ +0.0007 interval -0.0022 to +0.0036 — straddles zero against run 11 |
| 13 | 5 ratios between columns added A tree splits one column at a time and can never form a ratio, only approximate one with a staircase. | 0.3963 95% interval 0.3745 to 0.4187 | ≈ +0.0022 interval -0.0044 to +0.0084 — straddles zero against run 12 |
| 14 | ALL THREE EXCLUDED COLUMNS ADDED BACK Not a candidate model. A measurement of what the availability discipline cost. | 0.6344 95% interval 0.6139 to 0.6553 | ▲ +0.2382 interval +0.2220 to +0.2537 against run 13 |
| 15 | Only the two columns whose reference window could not be established Not a candidate model. Splits the previous measurement. | 0.4131 95% interval 0.3914 to 0.4357 | ▲ +0.0169 interval +0.0127 to +0.0209 against run 13 |
| 16 | Only the column known to postdate the transaction Not a candidate model. Completes the split as three measured points rather than two and an inference. | 0.6194 95% interval 0.5987 to 0.6404 | ▲ +0.2231 interval +0.2071 to +0.2383 against run 13 |
| 17 | Logistic regression given the tree's improved inputs Testing whether the tree's advantage was the model family or just the representation. | 0.2940 95% interval 0.2738 to 0.3148 | ▲ +0.0436 interval +0.0316 to +0.0560 against run 1 |
| 18 | Each split may see only 60% of the columns The one strong regulariser that had not been tried. | 0.3971 95% interval 0.3754 to 0.4194 | ≈ +0.0008 interval -0.0041 to +0.0061 — straddles zero against run 13 |
| 19 | Fraudulent transactions re-weighted to count far more in the objective The objective itself, which no earlier run had touched. | 0.3890 95% interval 0.3681 to 0.4109 | ▼ -0.0073 interval -0.0140 to -0.0000 against run 13 |
| 20 | Merchant identity removed entirely, in both the forms it had taken Measuring the same thing as run 11 from the other side. | 0.3316 95% interval 0.3111 to 0.3549 | ▼ -0.0647 interval -0.0790 to -0.0510 against run 13 |
| 21 | The 12 columns that showed no separation between the classes dropped Named from a profile written before the first fit, not chosen after seeing a score. | 0.3907 95% interval 0.3692 to 0.4134 | ▼ -0.0056 interval -0.0098 to -0.0009 against run 13 |
| 22 | A penalty on how large a value each leaf may predict The last untried mechanism in the library. | 0.3955 95% interval 0.3735 to 0.4177 | ≈ -0.0008 interval -0.0049 to +0.0033 — straddles zero against run 13 |
| 23 | The encoding's random fold split changed, and nothing else Measuring a source of uncertainty that a bootstrap over rows cannot see. | 0.3944 95% interval 0.3725 to 0.4167 | ≈ -0.0018 interval -0.0060 to +0.0022 — straddles zero against run 13 |
| 24 | Run 18 repeated, instrumented to measure what the model leans on A second determinism check, and the source of the importance figure. | 0.3971 95% interval 0.3754 to 0.4194 | ≈ +0.0000 interval +0.0000 to +0.0000 — straddles zero against run 18 |
| 25 | The two raw columns the top-ranked ratio is built from, removed Testing whether importance was measuring reliance or value. | 0.3935 95% interval 0.3721 to 0.4159 | ≈ -0.0036 interval -0.0088 to +0.0011 — straddles zero against run 24 |
8. The best configuration found, and precisely what its number is an estimate of
The highest-scoring admissible configuration is run 18, at an average precision of 0.3971 (95% bootstrap interval 0.3754 to 0.4194), ROC AUC 0.9406. It is a boosted tree ensemble with a learning rate of 0.05, 600 rounds, 31 leaves per tree, a minimum of 100 transactions per leaf, 60% of columns visible at each split, merchant identity encoded as that merchant's cross-fitted fraud rate, and 5 constructed ratios.
It is a tie, and calling it "the best" overstates it. Runs 11, 12, 13, 22, 23, 24, 25 are all within noise of it — every one of those differences has an interval that includes zero. Run 18 is the highest number in a group of 8 configurations that cannot be told apart, and the honest statement is that the search found a plateau rather than a winner.
What the number is an estimate of, in full. It was computed on all 104,160 rows of the validation file — 2,149 of which are fraudulent — covering 2025-01-01 to 2025-03-31, from a model fitted on all 833,391 rows of the training file covering 2024-01-01 to 2024-12-31. By the time it was computed, that same set of validation rows had been consulted 583 times over the course of the search — 25 headline scores plus the intermediate round counts read out of the runs that were instrumented to produce curves. The number of boosting rounds used by the final configuration was itself chosen by looking at those curves. There is no untouched holdout in this project, so this figure does not estimate performance on data the search has not touched, and it is optimistic by an amount nothing here can bound.
The interval quoted is a bootstrap over validation rows — it answers "how much would this number move if we had drawn a different sample of transactions of the same size". It does not cover randomness in the fitting procedure itself. That was measured separately, once, by changing the encoding's random fold split and nothing else: a different fold split changed average precision by -0.0018 (95% paired bootstrap interval -0.0060 to +0.0022, an interval that straddles zero — which is to say the change bought nothing that can be told apart from noise).
9. What could not be established
- Whether an unprompted session would find the inadmissible column. The instruction to check every column's availability was given in advance and with emphasis. Following it correctly demonstrates compliance with a stated rule and nothing more.
- An unbiased estimate of performance on unseen data. No holdout was sealed, by design; see the third opening statement.
- Where the two unresolved columns actually belong. Neither
amount_to_median_rationormerchant_risk_scorehas a name that states a reference window, and their values did not settle it. The merchant score is constant within each merchant and identical across both files; it correlates 0.943 with that merchant's fraud rate over the training period and 0.838 over the validation period, while the two periods' own rates correlate 0.620 with each other. Those numbers are consistent with a standing merchant attribute fixed in advance and with a summary computed across both files, and the two files cannot tell them apart. - How much of the plateau is the data and how much is the search. The ceiling is not computable here, so "the model has extracted what these columns contain" is a hypothesis this project cannot test.
- Whether the files hold each customer's complete history.
hours_since_prev_txnreconstructs exactly from the timestamps in the file (100% of rows match within a hundredth of an hour), but the other rolling-window columns do not, which is consistent with the files holding a sample of each customer's transactions. That was not established either way.
What's done, what happens next, what's waiting
- Done. All 25 runs of the budget are spent and recorded, each with the reason it was chosen written before it ran; every column's availability was declared before the first fit; the cost of the availability exclusions is measured as three separately fitted models; and this report is generated end-to-end from the committed result files, with no number typed by hand.
- What happens next without asking: nothing. Run 26 is refused by the budget, so the project stops here. The stopping reason is declared budget exhausted.
- Waiting on Juan: nothing. No question was escalated during the project and none is open now.