Here is an experiment that should keep every user of influence scores, the numbers that claim to say which training examples caused a prediction, up at night.
Security researchers took a CIFAR-10 image classifier whose training set had been salted with adversarial examples built from MNIST digits, training images crafted so the model learns the attacker’s behavior. They pointed four popular attribution methods at a handwritten “0” from the test set and asked the obvious question: which training examples are responsible for this prediction? The planted MNIST digits, the images actually driving the behavior, were sitting right there in the training set. Every method returned photographs instead: airplanes, animals, ordinary CIFAR images, ranked as the most influential examples for a handwritten digit. Influence functions failed. So did TracIn (a trajectory method we meet properly in part 4), its checkpoint variant, and representer-point methods (Hammoudeh and Lowd 2022).
On part 2’s classroom, this series’ running toy with sixty training points and a model small enough to retrain on demand, the influence function tracked sixty ground-truth retrainings, one per deleted training point, at a correlation of 0.9992. On a real network it cannot reliably tell a digit from an airplane. Nothing in between those two sentences is magic: the formula did not change. The assumptions under it did. This post walks through what actually breaks, one cause at a time, passing through the modern reframe that rescues the whole enterprise by changing the question, and ends at a fork: two roads the rest of the series will travel.
The autopsy, in four causes
Part 2’s derivation leaned on a staircase of assumptions, and it helps to name every step before watching them fail (Ilyas et al. 2024):
The first three causes are this post’s work. And the right place to watch the first one is a model so small we can see its entire loss surface at once.
The valley floor
Meet the smallest deep network in the world: two parameters, \(a\) and \(b\), and the model’s output is their product \(u = ab\). Fit it to a handful of observed target values \(y_1, \dots, y_k\) and the loss cares only about the product, never about \(a\) and \(b\) separately. That symmetry has a geometric consequence: the set of best fits is not a point. It is a curve, the hyperbola \(ab = \bar{y}\), where \(\bar{y}\) is just the average of the observed values, an entire valley floor of models with identical loss. Double \(a\), halve \(b\): same product, same loss, different parameters.
This is the smallest honest model of what deep networks do everywhere. Scale one stage of a network up and the next stage down, or swap two internal units along with their wiring: modern architectures are stuffed with reparameterizations that leave the function unchanged, so their minima come in valleys, walls, and plateaus, not points (Ilyas et al. 2024). And a valley floor is exactly the situation part 2’s formula cannot survive: along the valley the loss has no curvature, so the Hessian reads zero in that direction (a zero eigenvalue: one direction in parameter space, one way of turning the model’s dials, is perfectly flat), and \(H^{-1}\) asks you to divide by that zero.
The standard first aid is damping: replace \(H^{-1}\) with \((H + \lambda I)^{-1}\), which always exists. Here \(I\) is the identity matrix, the “same amount in every direction” matrix, so adding \(\lambda I\) props a small floor of curvature \(\lambda\) under every direction, including the perfectly flat one; nothing reads zero anymore, and the division always goes through. The figure below shows what that patch buys, and what it costs.
FIG. 1 · SAME VALLEY, DIFFERENT ANSWERS
The whole pathology in one picture. Every point on the navy hyperbola is an equally good model: same loss, same predictions. Drag the marker along the floor and watch the damped influence of one fixed example change, even though nothing about the model’s behavior does. The dashed bar is the flat direction (the Hessian’s null space). The small colored arrow at the marker shows the direction the probe example pushes the model; its length and color track the size and sign of the damped influence. The strip below tracks the influence readout across the entire valley. Smaller damping makes the answers larger and wilder; larger damping calms them by quietly changing the question.
Stare at what the readout is doing. The influence of a training example is supposed to be a fact about the model and the data. On the valley floor it has become a fact about which of many identical models gradient descent happened to park in, times an arbitrary knob \(\lambda\). Two training runs that land at different spots on the same hyperbola, identical in every prediction they make, report different influence scores for the same example. Nothing is broken numerically; the number just no longer means what we wanted it to mean.
There is a second reading of \(\lambda\) worth planting now: mathematically, damping is the same as demanding that the imagined retrained model stay on a leash near the model you already have, with \(\lambda\) setting the leash’s stiffness. That leash is the question the next section grades the formula on.
This is also where cause two lands: which spot on the floor a run parks in is decided by the seed and the shuffle, so “retrain and see” stops being one number and becomes a lottery draw.
At a saddle or ridge, some Hessian directions curve down, and the formula’s geometry stops making sense: part 2’s argument asked how the bottom of the bowl shifts when a point is removed, and along a downhill direction you are standing on a crest, so “the bottom” points at nothing. The standard fix is to swap \(H\) for the Gauss-Newton Hessian (GNH), a stand-in whose curvature is never negative in any direction, built from how the network’s outputs (rather than its loss) respond to the parameters (Schraudolph 2002; Martens 2020).
Between damping and the GNH, the patched formula always runs. Whether it still tracks retraining is an empirical question, so let us run the experiment.
Depth against the formula
Every number in the next figure is computed fresh for this post, on the same sixty students as always, the sixty training points of the series’ classroom. The model deepens step by step: part 2’s convex classroom first, then small neural networks of one, two, and three hidden layers (a hidden layer is a bank of part 1-style dial-and-squash units wedged between input and output, each bank feeding the next; every extra bank is one step of “depth”). At each depth we pay for the truth, sixty full retrainings, and ask how well the damped Gauss-Newton influence estimates track it.
FIG. 2 · THE 0.9992 DOES NOT SURVIVE DEPTH
The same estimator, the same sixty students, deeper models. Each dot is a full experiment: sixty exact retrainings against sixty damped Gauss-Newton influence estimates. Click a dot to see that depth’s scatter. The convex model’s 0.9992 erodes as the loss surface grows valleys and saddles. Basu and colleagues documented the at-scale version of this decay, along with its sensitivity to the strength of the regularizer (weight decay) and to architecture choices (Basu, Pope, and Feizi 2021).
This is a toy-scale rendition of what the field found the hard way. Basu, Pope, and Feizi measured influence estimates against true retraining across depths and architectures and watched the correlation collapse, especially without strong regularization (Basu, Pope, and Feizi 2021). At the scale of a ResNet on CIFAR-10, the field’s standard benchmark places the plain influence function’s predictive power near 0.05, barely above noise (Ilyas et al. 2024). Whatever those scores were measuring, it was not “what happens if you retrain without this example.”
The yardstick
That yardstick deserves a sentence, because a version of it grades every method from here on. The linear datamodeling score (LDS) asks an estimator to predict what models retrained on random half-subsets of the data will do (the prediction is simply the sum of the estimator’s per-example scores over the examples removed), then correlates prediction with reality:
In part 6 the LDS becomes the axis every method is judged on, and gets its full definition; for today, one number suffices: plain influence functions score about 0.05 on it at ResNet scale.
Wrong answer, or wrong question?
Now for the twist that reorganized how the field thinks about all of this. In 2022, Bae, Ng, Lo, Ghassemi, and Grosse asked a question with a built-in trap: if influence functions are the answer, then what is the question? (Bae et al. 2022).
Their finding, on real networks: cold-start correlations were weak and inconsistent in sign across architectures, while correlations against the proximal Bregman response function (PBRF), a warm-start response that asks “retrain without this example, while staying near the model you already have,” were consistently high, mostly above 0.8.
Read that again, because it is the most important nuance in this series so far. The influence function is not a broken estimator. It is a precise estimator faithfully answering a question nobody consciously asked. The figure below runs a simplified cousin of the PBRF on our own classroom network so you can see the shape of the result yourself. The star is the single gold test point every influence number in this series is measured against.
FIG. 3 · THE SAME ESTIMATES, JUDGED AGAINST TWO QUESTIONS
Sixty influence estimates for the three-hidden-layer classroom network, judged twice. Left: against cold-restart leave-one-out retraining (delete the example, retrain from scratch, measure the change), the counterfactual this series started from in part 1. Right: against a warm-start (proximal) retraining that stays near the trained model, a simplified cousin of Bae et al.’s PBRF; their exact construction adds a Bregman divergence term, a made-to-measure distance penalty that keeps the retrained model’s predictions close to the original’s (Bae et al. 2022). Green marks a student who was helping the star under cold-restart retraining, orange-red one who was hurting it. Same estimates, different questions, very different verdicts. Hover a point to track it across both panels.
Why would the formula answer the proximal question? Look back at how part 2 built it: a tangent line at the trained model, using curvature at the trained model. Every ingredient is local. The formula never saw the far-away places a cold restart can wander to, across the valley floors of Fig. 1 or into a different basin entirely; it can only speak about what happens near here. On a convex classroom, “near here” and “anywhere” coincide, which is why part 2 got away with it. On a deep network they part ways, and the formula keeps faithfully describing the local question while we keep grading it on the global one.
Influence is a moving target
Causes one and two were about where training ends. The last one is about when. Part 2’s formula, and everything else built on the endpoint, treats the trained model as a finished fact. But training is a process, and an example’s influence is a signal that unfolds, and can reverse, across it.
The cleanest place to see this is a deep linear network, the fruit-fly of training dynamics: layers stacked deep with no nonlinearity between them (no squashing step like part 1’s sigmoid; each layer just multiplies and adds), simple enough to solve, deep enough to keep the behavior that matters here. Such networks famously learn their task one “mode” at a time, strong patterns first, fine distinctions later (Saxe, McClelland, and Ganguli 2014). Lee and coworkers showed what this staging does to attribution: an example’s influence on the star peaks when the mode connecting them is being learned, and can flip sign when a later mode arrives that they disagree about (Lee et al. 2025).
Their motivating example makes it concrete: a dog photo helps a cat test image early, while the network is learning the coarse “animal” mode the two share. It hurts the same test image later, once the fine dog-versus-cat mode starts separating them.
The figure below reproduces the phenomenon at classroom scale: two runs of the same tiny network, identical except that one training example is removed, with the gap between them tracked across every checkpoint.
FIG. 4 · WATCH ONE EXAMPLE CHANGE SIDES MID-TRAINING
The exact leave-one-out effect of one training example on the star, recomputed at every checkpoint of training (two real runs, identical seeds, one example apart). Early, removing the example hurts the star: it is helping while the coarse mode is learned. Later the curve crosses zero: once the fine mode arrives, the same example is hurting. The strip below shows the two modes switching on in sequence; the fine mode overshoots before settling to its final strength. Scrub, or press play.
The uncomfortable summary: “the influence of example \(i\)” is a time slice of a moving signal, and endpoint methods answer with whatever the signal happens to read at the moment training stopped. Stop earlier or later and the sign of the answer can differ. Recall part 1’s dials, the model’s own parameters as knobs training turns: the modes are dials that turn on one at a time, and an example can be pushing a dial in the helpful direction while it is turning and stand in its way afterward.
Two roads out
So the wreckage is real, but it is organized wreckage, and each pile points somewhere.
Fix the question. Damping, the Gauss-Newton Hessian, and the PBRF reframe all accept the influence function’s local nature and make the target honest about it: a well-posed, near-the-model counterfactual that the formula answers, with the pieces that were never defined (flat directions, cold restarts) removed from the contract. That road leads to the methods that run at LLM scale, where the remaining enemy is the fourth cause, the sheer size of the Hessian. That is part 5.
Take time seriously. Fig. 4 says the training trajectory carries structure the endpoint cannot see. So differentiate the trajectory: treat training itself as the computation and push credit back through it, step by step, with the same backward bookkeeping training already uses to decide which dial to turn (backpropagation), now applied to the whole run instead of a single prediction. That idea, unrolling, and the family of methods it spawns, TracIn, SOURCE, MAGIC, is part 4, next.
One warning to carry forward, from this post’s opening exhibit: the CIFAR/MNIST sanity check failed for the trajectory methods too (Hammoudeh and Lowd 2022). Neither road exempts anyone from checking their answers. The classroom keeps every method in this series honest because sixty retrainings are always affordable there; at scale, honesty takes more ingenuity, and the LDS is what it looks like.
Further reading
The wreckage, documented
- Basu, Pope, and Feizi, “Influence Functions in Deep Learning Are Fragile” (Basu, Pope, and Feizi 2021). The systematic study of how depth, width, and regularization degrade influence estimates.
- Hammoudeh and Lowd, “Identifying a Training-Set Attack’s Target Using Renormalized Influence Estimation” (Hammoudeh and Lowd 2022). Source of the opening exhibit, and a repair (renormalization) this post did not cover.
The reframe
- Bae, Ng, Lo, Ghassemi, and Grosse, “If Influence Functions are the Answer, Then What is the Question?” (Bae et al. 2022). The PBRF paper: cold-start correlations were weak and inconsistent in sign across architectures, while PBRF correlations were consistently high.
- Schraudolph, “Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent” (Schraudolph 2002), and Martens, “New Insights and Perspectives on the Natural Gradient Method” (Martens 2020). The Gauss-Newton Hessian and why it is the curvature you can actually trust.
The dynamics
- Lee, Smith, Adam, and Hoogland, “Influence Dynamics and Stagewise Data Attribution” (Lee et al. 2025). Influence as a time-varying signal, sign flips included.
- Saxe, McClelland, and Ganguli, “Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks” (Saxe, McClelland, and Ganguli 2014). Why deep linear networks learn one mode at a time, which is what makes Fig. 4 constructible.
The wider survey
- Hammoudeh and Lowd, “Training Data Influence Analysis and Estimation: A Survey” (Hammoudeh and Lowd 2024), and the ICML 2024 tutorial “Data Attribution at Scale” (Ilyas et al. 2024), whose four-cause framing this post follows.
References
Reproducibility. Every figure on this page is a real experiment: the valley-floor surface and its closed-form influence readouts, the depth sweep (sixty exact retrainings at each of four depths), the proximal-versus-cold comparison, and the two-run sign-flip trace are all generated by a seeded script in the site repository (scripts/influence-data/), which asserts the depth-0 correlation matches part 2’s 0.9992 before shipping the JSON this page loads. The at-scale LDS value of 0.05 and the qualitative summary of Bae et al.’s architecture-specific cold-start and PBRF correlations are drawn from the cited papers, not reproduced here.
Acknowledgements. The causes-first organization of this post, the valley-floor toy, and the reading of damping as a question-changing choice follow Louis Jaburi’s lecture notes “Data (Attribution) for Alignment” (Iliad Intensive, April 2026).