Your model just did something strange. Maybe it flagged a routine transaction as fraud, or answered a question with a confident, oddly specific mistake. Staring at the weights, the learned numbers inside the model, will not tell you why. The weights are just where the training data ended up: whatever the model believes, some specific set of training examples put it there. So when it surprises you, there is exactly one honest place to point. Which training examples taught it that?
This series is about the family of methods that try to answer that question. They go by names like influence functions, TracIn, EK-FAC (Grosse et al. 2023), datamodels, and TRAK; treat those as labels for now, since the series introduces each one when its turn comes. They show up anywhere someone needs to connect a model’s behavior back to its data: debugging bad predictions, curating training sets, studying memorization, and increasingly, arguing about what a model owes to the data it learned from. Hammoudeh and Lowd survey the whole landscape if you want the map before the territory (Hammoudeh and Lowd 2024).
Before we get to any of the clever methods, though, we need to pin down what the question even means, and what a perfect answer would look like. The perfect answer exists. You can compute it. By the end of this post you will have computed it in your browser, sixty times. You will also see exactly why nobody computes it at scale.
The counterfactual question
Here is the setup for the whole series. We have a training set of \(n\) examples \(z_1, \dots, z_n\), where each \(z_i = (x_i, y_i)\) pairs an input with a label. A learning algorithm turns that set into trained parameters \(\hat\theta\): the model’s internal numbers, with the hat marking the specific values training settled on. And we have one test example \(z_{\text{test}}\) whose prediction we care about.
Asked directly, “which training example taught the model that?” is vague. Machine learning has a standard move for making vague questions precise: turn them into counterfactuals. Philosophy got there first: the classical account of causation says A caused B when, had A not occurred, B would not have occurred (Lewis 1973). Applied here: what would the model have predicted if example \(i\) had never been in the training set?
That question has an exact answer. Delete example \(i\), retrain from scratch, and compare. If retraining without \(z_i\) makes the prediction on \(z_{\text{test}}\) worse, then \(z_i\) was helping: it taught the model something it needed for this prediction. If removing \(z_i\) makes the prediction better, then \(z_i\) was hurting it: noise, an outlier, maybe a mislabeled point. This remove-and-retrain procedure is called leave-one-out (LOO) retraining, and it is the ground truth that every method in this series is trying to approximate (Koh and Liang 2017).
The picture to hold onto is two parallel universes. Same data, same training recipe, same everything, except one universe never saw example \(i\):
THE COUNTERFACTUAL, DRAWN
The counterfactual, drawn. Both universes run the identical training procedure; the only difference is one erased example. Universe B’s boundary tilts, and the star’s prediction moves. That movement, measured in test loss (a penalty on the star’s answer, defined in Fig. 2), is what the rest of this post computes. The numbers are real, not decorative: this is the sixty-point dataset the next section introduces, the gold star is the test example \(z_{\text{test}}\), and the erased example is its single most influential point.
Sixty students, one question
Meet the dataset the whole series will use, drawn in Fig. 1 below. I call it the classroom: sixty training points in two dimensions, thirty per class, and a logistic regression, one weight per coordinate plus a bias, that learns a straight boundary between them. Each point is a student, and everything the model knows, it learned from these sixty.
Logistic regression here is three numbers: two weights, one per coordinate of \(x\), and a bias. Given a point, the model forms a weighted sum of its coordinates, adds the bias (one extra number that slides the boundary without rotating it), then squashes that sum through a sigmoid function \(\sigma\), which turns any real number into a probability between 0 and 1. The decision boundary in the classroom figure below (Fig. 1) is just the place where that probability equals exactly 50%.
The whole model is three dials. Training turns them until the classroom’s answers look right; that is all “learning” means here, and you can do the turning yourself in the figure below. If you want the full derivation, both textbooks live free online: Hastie, Tibshirani, and Friedman devote a chapter to it (Hastie, Tibshirani, and Friedman 2009, chap. 4), and so does Murphy (Murphy 2022).
TURN THE DIALS YOURSELF
The model is three numbers, live. Drag a knob and the boundary follows; the readout tracks what the star gets. Scramble the dials to wreck the model, then click “Train from here” and watch gradient descent, a step-by-step downhill march toward lower penalty, turn the knobs back by itself. Training is nothing more than automatic knob-turning.
Filled navy circles in Fig. 1 are class 1, and open circles are class 0. The gold star is the test point, the one question we will keep asking the class. The trained model puts the star in class 1 with about 68% confidence. The star’s true label is class 1, so the model has it right, but not comfortably. The navy shading behind everything is that confidence drawn everywhere at once.
Nothing in Fig. 1 is a mock-up. The dataset, the boundary, and every influence value were computed offline by a small Python script. The page ships the same training recipe (the step size, the number of steps: the settings the field calls hyperparameters), so when you click a point your browser deletes it and retrains the identical model, live.
FIG. 1 · CLICK A POINT, RETRAIN LIVE
The classroom. The shading is the model’s confidence in class 1, drawn everywhere in the plane; the boundary is its 50/50 contour. Students whose removal barely matters wear no ring at all. The rest carry a ring and halo that grow with their leave-one-out effect on the gold star: green means helpful (removing them hurts the prediction), orange-red means harmful (removing them helps). Click any point to remove it and retrain the model in your browser; click again to put it back. Only one student is ever out at a time; clicking a new point sends the previous one back to class. The readout’s “test loss” is the penalty score defined just below in Fig. 2. The “Confidence shading” toggle above turns the shading off.
Three things to notice while you play:
- Most points do not matter. Remove a point far from the boundary and the line barely twitches. The prediction for the star moves by a rounding error.
- The points that matter sit near the action. The big halos cluster around the boundary, where a single example can still tip the balance of the fit.
- Helping and hurting are both real. Some points pull the boundary toward the star’s correct side, others push it away. Removing a harmful point improves this prediction.
The halo sizes and colors are an answer key: I precomputed all sixty removals offline. But you do not have to trust me. Pick the biggest halo, click it, and watch the boundary move exactly as advertised. The “Remove most helpful” and “Remove most harmful” buttons above the figure jump straight to the extremes.
The number we watch
Before slowing anything down, we need the score we will be watching. When the model answers the star, it does not just pick a class; it commits to a probability \(p\) that the star belongs to class 1. The test loss turns that confidence into a penalty: \(L = -\log p\). Confidently right costs almost nothing, since \(-\log p\) falls toward zero as \(p \to 1\). Confidently wrong is catastrophic, since the penalty grows without bound as \(p \to 0\).
This is the ordinary cross-entropy loss used everywhere in statistical learning (Hastie, Tibshirani, and Friedman 2009; Murphy 2022), and the \(p\) it penalizes is the same one the shading in Fig. 1 draws.
Everything that follows boils down to watching this one number move. Feel out the curve first:
FIG. 2 · THE SCORE WE WATCH
The loss curve. \(L = -\log p\) as a function of \(p\), the model’s probability on the correct label. The gold marker is the star’s score under the fully trained model; move your cursor along the curve to explore it, or jump to a preset: the same easy and on-the-boundary star positions that Fig. 5 returns to.
One removal, frame by frame
Fig. 1 already lets you click a point and watch the boundary jump. That jump is really four separate things happening at once: the model trains on everything, one example gets deleted, the model retrains from scratch without it, and the two results get compared. Bundled together, that is easy to watch and hard to describe. Unbundled, each step has a name, and the last one is exactly the definition the next section writes down.
The figure below walks through those four beats one at a time, for the student with the biggest effect on the star’s prediction. Step forward through train, delete, retrain, and compare, and watch the retraining beat closely: the boundary does not teleport to its new position. It walks there the same way the original training run did, by the same step-by-step gradient descent from the same starting point, just with one fewer student in the class. The two numbers in the readout are old friends now: \(p\), and its penalty \(L\) from Fig. 2.
FIG. 3 · ONE REMOVAL, FRAME BY FRAME
One removal, frame by frame. Four beats: train on everything, delete one example, retrain from scratch, then compare the test loss before and after. Use “Try another point” to swap a different student into whichever step you are on; step back to the start to watch their four beats from the top.
Naming what you just did
Each click deserves a formal name. Write \(\hat\theta\) for the parameters trained on all sixty points (here, the three dials), and \(\hat\theta_{-i}\) for the parameters trained on everything except example \(i\). Write \(L(z_{\text{test}}, \theta)\) for the loss on the test point: a number that is small when the model predicts it well and large when it does not. The leave-one-out influence of training example \(z_i\) on \(z_{\text{test}}\) is the change in that loss:1
\[ \Delta_i \;=\; L\!\left(\class{eq-test}{z_{\text{test}}},\, \class{eq-model}{\hat\theta_{-i}}\right) \;-\; L\!\left(\class{eq-test}{z_{\text{test}}},\, \class{eq-model}{\hat\theta}\right) \]
FIG. 4 · THE DEFINITION, TERM BY TERM
The definition, term by term. The colors are the same cast as every figure in this series: gold is the test point (the star), navy is the trained model (the boundary), and green versus orange is the sign of the answer, matching the halos in Fig. 1.
Read it left to right: train without example \(i\), measure the test loss, subtract the loss you started with. The \(L\) inside it is exactly the curve from Fig. 2: the same penalty, measured twice, once with everyone present and once without student \(i\). That is all the math this post needs. One definition, two loss evaluations, one subtraction.
The hard part is not the formula. It is the \(\hat\theta_{-i}\) sitting inside it, because every distinct \(i\) demands its own full training run.
Deletion diagnostics are not a machine learning invention. In 1977, R. Dennis Cook measured exactly this kind of effect for linear regression: what deleting one observation does to the fitted model, now called Cook’s distance (Cook 1977). For linear regression there is even a closed form, no retraining required, because the effect of deleting a point can be read off the original fit directly (Cook and Weisberg 1982). Decades earlier still, the jackknife built a whole theory of statistical inference on systematically leaving observations out, one at a time (Quenouille 1956; Efron 1982). Part 2 imports exactly this spirit into machine learning.
The full answer sheet
Click all sixty points in Fig. 1 and you have answered the attribution question completely, for this one test point: a ranked list of the whole classroom, from most helpful to most harmful. That list is worth staring at, so here it is as a chart. And because the list is an answer to a specific question, it changes when the question does. Drag the star.
FIG. 5 · ALL SIXTY, RANKED
The full leave-one-out answer sheet. Left: the classroom, with the same halo encoding as Fig. 1 and a draggable test point. Right: all sixty \(\Delta_i\) values, ranked. The values are exact, not approximated: training never involves the star, so the same sixty retrained models answer every question, and each \(\Delta_i\) is just two loss evaluations for wherever you drag the star. No new retraining happens. The axis is fixed; bars that exceed it pin to the rail with a chevron. Click “Run all 60 retrainings” to replay the whole experiment: each bar pops in as its retraining finishes, in the order the experiment actually ran, and the list re-sorts into the ranking at the end.
This figure holds the three lessons that motivate everything later in the series:
- Attribution is top-heavy. The longest bar moves the test loss more than six times as much as the median one, and the bottom half of the classroom put together matters less than the top three students. “Which examples taught the model that?” has a short answer at the top of a long, fast-fading list.
- Attribution is query-specific. Drag the star deep into class 1 territory. The model’s confidence climbs past 99% and every bar collapses: the star now sits so far from the boundary that no single student’s absence can move its answer. An easy question has no strong teachers. There is no such thing as “the most influential training point,” only the most influential for a given prediction.
- Unstable questions have unstable answers. Drag the star along the boundary, out toward the edge of the data. Bars slam into the rails of the chart. Where the model is barely constrained by data, single examples swing the answer wildly. Hold that thought: it returns in part 3.
Staring at the classroom, it is tempting to shortcut the whole procedure: find the training points that look most like the test point and call them the teachers. That shortcut answers a real question, but a different one.
Retrieval by similarity finds evidence that supports an output, the way a citation does, and the evidence does not even need to come from the training set; researchers who found image-generating models (diffusion models) emitting near-duplicates of specific training images were doing exactly this kind of search (Somepalli et al. 2023). Whether an example actually changed the model is a causal claim, and the two can come apart in both directions: a look-alike example the model never needed, or an unassuming one that moved the boundary. The literature keeps these apart as corroborative versus contributive attribution (Worledge et al. 2023). Fig. 5 is the contributive kind, and so is the rest of this series.
The bill
So leave-one-out retraining answers the question exactly. Why isn’t this the end of the series?
Count the training runs. The answer sheet above needed sixty of them, one per training point. My laptop ran all sixty offline in about a second, which is why this post can exist. But the total cost scales badly twice over: \(n\) training points means \(n\) retraining runs, and each run’s own cost also grows with \(n\), because every pass of training has to visit every example it keeps. Total work grows like \(n^2\).
The figure below makes that concrete: it times one real retraining of the classroom on your device, right now, and then does the arithmetic.
FIG. 6 · THE BILL
The cost of exact leave-one-out. The card above the chart multiplies the two numbers out explicitly as the slider moves: \(n\) runs times the cost of one run. The line assumes the per-run training time measured in your browser scales linearly with \(n\), and that LOO needs \(n\) runs. The anchors are real training set sizes: MNIST has 60,000 training images (LeCun et al. 1998), ImageNet-1k has about 1.28 million (Russakovsky et al. 2015), and the text collections that large language models (LLMs) are first trained on run to a billion documents and beyond.
Slide \(n\) to MNIST and the classroom’s milliseconds become hours or days. Slide it to a billion documents and exact LOO for this three-parameter toy model runs to hundreds of thousands of years of browser time on fast hardware, and millions on typical hardware. A real model makes it incomparably worse: a single frontier LLM pretraining run takes weeks on thousands of GPUs, and LOO asks you to repeat it once per training document.
The gold standard is not merely expensive. At the scale of any modern model it is out of the question by many orders of magnitude.
People still pay approximations of this bill, because the question is worth answering even at a discount:
- Datamodels trained four million networks on subsets of CIFAR-10 and a satellite-imagery dataset (FMoW) to approximate exactly this kind of counterfactual, and shipped the predictions (Ilyas et al. 2022).
- Feldman and Zhang estimated memorization by retraining on random subsets: 2,000 trials for ImageNet, 4,000 for MNIST and CIFAR-100, each trial a full training run (Feldman and Zhang 2020).
- Data Shapley faces the same wall from a different direction: a Shapley value splits credit by averaging each example’s contribution over every subset of classmates it could join, so an exact answer needs performance measured over exponentially many subsets, and every practical version samples instead (Ghorbani and Zou 2019).
What we can’t do yet
Where does that leave us? With a definition, and no algorithm. Leave-one-out retraining tells us what “this example taught the model that” means, and on sixty points it even tells us the answer. It tells us nothing about how to get that answer from a model we cannot afford to retrain.
One honest caveat before moving on. For the small model in this post the training penalty, viewed as a landscape over the three dials, is convex: a single bowl with one lowest point, so gradient descent always rolls to the same bottom no matter where it starts, and LOO is a crisp ground truth. For deep networks, even the ground truth gets murky: retrain twice from different random seeds and you get two different models, so “the effect of removing one example” becomes a noisy quantity, and any method that approximates it inherits that noise. That mess, and what it does to how much you should trust an influence score, is where this series is headed in part 3 (Basu, Pope, and Feizi 2021).
A second caveat is deeper, and no amount of compute fixes it. One-at-a-time removal has a blind spot: redundancy. Picture two students who teach the class the same fact. Send either one home and the class still knows it; the test score does not move, and \(\Delta_i\) hands both students a zero, even though the fact was plainly taught.2 Keep that asterisk next to everything in this series. The counterfactual is precise, but “removing it changes nothing” and “it taught the model nothing” are not the same sentence.
But first, the obvious challenge: can we estimate all \(n\) leave-one-out effects from the single model we already trained, without retraining anything? In 2017, Pang Wei Koh and Percy Liang imported a tool from classical statistics that does exactly that (Koh and Liang 2017), using only gradients (which way each dial should turn to lower the loss) and the local curvature of the loss (how sharply the penalty bends where training stopped). On this classroom, its estimates correlate with the true leave-one-out values you just computed at better than 0.999.
That tool is the influence function. It is the subject of part 2.
Further reading
The paper this series follows
- Koh and Liang, “Understanding Black-box Predictions via Influence Functions” (Koh and Liang 2017). The paper part 2 covers in full: how to estimate every \(\Delta_i\) from one trained model, no retraining required.
The map of the field
- Ilyas, Park, Engstrom, Georgiev, and Mądry, “Data Attribution at Scale” (Ilyas et al. 2024). The ICML 2024 tutorial whose breadth this series roughly follows, with slides, video, and four readable text chapters.
- Hammoudeh and Lowd, “Training Data Influence Analysis and Estimation: A Survey” (Hammoudeh and Lowd 2024). A broad survey of attribution methods, useful for seeing where any one technique sits relative to the rest.
- Grosse et al., “Studying Large Language Model Generalization with Influence Functions” (Grosse et al. 2023). What this question looks like at the scale of a large language model, with EK-FAC standing in for exact retraining.
- Worledge et al., “Unifying Corroborative and Contributive Attributions in Large Language Models” (Worledge et al. 2023). The similarity-versus-causation distinction from the callout above, made precise.
The history
- Cook, “Detection of Influential Observation in Linear Regression” (Cook 1977), and Cook and Weisberg, Residuals and Influence in Regression (Cook and Weisberg 1982). Deletion diagnostics for linear models, with a closed form.
- Quenouille, “Notes on Bias in Estimation” (Quenouille 1956), and Efron, The Jackknife, the Bootstrap and Other Resampling Plans (Efron 1982). The statistical theory behind systematic leave-one-out.
The bills people actually pay
- Ilyas et al., “Datamodels: Predicting Predictions from Training Data” (Ilyas et al. 2022). The full version of the retraining-at-scale program summarized in The bill.
- Feldman and Zhang, “What Neural Networks Memorize and Why” (Feldman and Zhang 2020). The memorization study behind the subset-retraining bullet in The bill.
- Ghorbani and Zou, “Data Shapley: Equitable Valuation of Data for Machine Learning” (Ghorbani and Zou 2019). The game-theoretic route to the same wall.
Free textbooks, if you want the derivations
- Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning (Hastie, Tibshirani, and Friedman 2009). Free PDF; chapter 4 covers logistic regression in full.
- Murphy, Probabilistic Machine Learning: An Introduction (Murphy 2022). Free online; the same model from a probabilistic angle.
References
Reproducibility. Every interactive figure on this page runs the real model, and every number, in the diagrams and the prose alike, comes from the same seeded experiment. The dataset, the exact LOO retraining, and the static fallbacks are generated by a seeded script in the site repository (scripts/influence-data/), which also ships the training hyperparameters to the page so the in-browser retraining matches the offline computation exactly.
Acknowledgements. The counterfactual framing of attribution and the overdetermination caveat in this post follow Louis Jaburi’s lecture notes “Data (Attribution) for Alignment” (Iliad Intensive, April 2026).