Part 1 ended on a dare. We defined the influence of a training example counterfactually, as the change \(\Delta_i\) in test loss when you delete example \(i\) and retrain, and we paid for that definition in full: sixty full training runs for sixty answers, with a bill that grows like \(n^2\). Then came the dare: a formula from Koh and Liang (Koh and Liang 2017), computed from the single model we already trained, matches all sixty answers at a correlation (a straight-line agreement score where 1 is perfect) better than 0.999.
If you are joining here, the running example is part 1’s toy classroom: sixty training points (the students) feed a tiny two-feature logistic-regression classifier, a model that draws a straight boundary through the scores and ramps its predicted probability smoothly across that boundary. We watch the loss on one held-out test point.
This post builds that formula. Not by pulling it out of a hat, but the way it was actually built: starting from a 1974 idea about contaminated samples, climbing through a trick regression statisticians were using in 1981, and arriving at the modern formula, which turns out to be the 1981 trick with one term deliberately thrown away. By the end you will know exactly which term, why throwing it away is usually fine, and when it is not.
Start fifty years back
Forget models with parameters for a moment. Take the simplest estimator in statistics (an estimator is any rule that turns a dataset into a number): the mean of a sample. Now ask the attribution question about it. If I slip one new observation \(z\) into your sample, how much does your estimate move?
You do not need to re-average to answer that. The new mean is a weighted blend of the old mean and the newcomer, so the shift is proportional to \(z - \bar{x}\): how far the newcomer sits from where the estimate already was. That expression, the response of an estimator to a small contamination at \(z\), is what Frank Hampel named the influence function in 1974, as a tool for building estimators that resist outliers (Hampel 1974).
For the mean, influence is a straight line in \(z\): contaminate near the mean and nothing happens, contaminate far away and you drag the estimate proportionally.
Drag the gold point and watch both halves of that sentence at once.
FIG. 1 · TURN THE 1974 IDEA WITH YOUR OWN HANDS
Hampel’s influence function for the mean. Top: twelve fixed observations (navy), their mean, and one contaminant you can drag (gold). Bottom: the influence of the contaminant, \(I(z) = z - \bar{x}\), is just a line, and your current position slides along it. No refitting happens anywhere: the response is a formula.
The point of the warm-up is the shape of the idea: for a well-behaved estimator, the effect of adding, removing, or reweighting one observation can be read off the fitted estimate directly, no refitting required. Removing a training example is just contamination with a minus sign. What Koh and Liang did in 2017 was carry this shortcut from means and regressions over to machine learning models (Koh and Liang 2017). The word “influence” in every method this series covers is borrowed from exactly this line of work.
The mathematical machinery is older than the name, and the lineage reads at a glance:
Machine learning did not invent this tool; it inherited it. The calculus is von Mises’s (Mises 1947), the infinitesimal jackknife is Jaeckel’s read-the-effect-off-the-fit shortcut phrased as an infinitesimal reweighting of one observation (Jaeckel 1972), the name is Hampel’s (Hampel 1974), the working diagnostics are Cook’s and Pregibon’s (Cook 1977; Pregibon 1981), and modern statistics supplied finite-sample guarantees, error bounds for actual datasets rather than only infinite-data limits (Giordano et al. 2019; Rahnama Rad and Maleki 2018).
Three rungs to the formula
Here is the climb from “exact but special” to “general but approximate.” Each rung trades a little exactness for a lot of generality, and the last rung is the influence function. The staircase labels Newton step and leverage before the paragraphs below define them.
Rung 1: linear regression, where deletion is free. Fit a line by least squares (pick the line that makes the total of the squared vertical misses smallest), and the effect of deleting any point has a closed form (a formula you can evaluate directly to get the exact answer, with no iterative refitting). Instead of redoing the expensive matrix work without row \(j\), a shortcut patches the answer you already have with one small, cheap correction for the removed row (the Sherman-Morrison identity).
The whole leave-one-out shift (the change from deleting that one point and refitting) then collapses to a fraction: the point’s residual (how wrong the fit is about it) on top, one minus the point’s leverage (how unusual its position is) on the bottom. This is the identity behind Cook’s classic regression diagnostics, deletion effects with no retraining required (part 1 mentioned it in passing) (Cook 1977; Cook and Weisberg 1982).
Rung 2: logistic regression, one Newton step. Our classroom model, part 1’s logistic regression, draws a straight boundary through the two-dimensional scores while the predicted probability ramps smoothly across it. It has no closed form: delete a student and the refit is an iterative optimization, which is why part 1 had to rerun training. Pregibon’s 1981 move is to approximate the leave-one-out training objective by its quadratic expansion around the optimum you already found (replace the true penalty surface with the bowl shape that best matches it right where you stand), and solve that in one shot (Pregibon 1981). The bottom of a bowl can be computed in one algebraic hop, with no iteration. That one-hop move is what optimizers call a Newton step.
Geometrically, take a single Newton step away from the trained model. The answer keeps the same residual-over-one-minus-leverage shape, with the model’s local curvature standing in for rung 1’s table of data coordinates. It is exact whenever the loss really is quadratic, because then the bowl is the true surface. This is exactly why rung 1 was exact: a sum of squared misses is a quadratic.
Rung 3: any smooth model, and one deliberate deletion. One fact unlocks this rung: the training penalty is a sum of one term per student. Give each student’s term its own volume knob, normally set to 1. Turning student \(i\)’s knob to 0 deletes that student, and anywhere in between counts the student partially.
For a general loss, differentiate the optimum with respect to that volume knob, then Taylor-expand (approximate the response by its straight tangent line at the point where you already stand). The result is the influence function estimator, and comparing it to rung 2 reveals something worth staring at: it is the same formula with the leverage denominator deleted. The residual-shaped numerator survives; the \(1 - h\) correction does not. That word, leverage, is the number this post’s subtitle promised, and the rest of this post is the story of what deleting it costs.
FIG. 2 · THE SAME IDEA, THREE TIMES; THE LAST RUNG DROPS ONE TERM
The climb. Rung 1: deleting a point from a linear fit has an exact closed-form answer. Rung 2: for the classroom’s logistic model, the one-Newton-step boundary (dotted) lands almost exactly on the true retrained boundary (dashed ink), one step away from the full model’s solid line. Rung 3: the influence function (dashed gold) is the Newton step minus its leverage correction, and for this well-behaved point it barely matters. The point where it does matter gets its own figure below.
The rung-3 derivation looks more principled, but on rung 1’s own turf it is worse: applied to linear regression, the influence function does not reproduce the exact closed form. Only the Newton-step estimator does, because only it keeps the leverage denominator. Generality was bought with a real coin.
The formula, term by term
Time to write rung 3 down properly. The influence-function estimate of example \(i\)’s effect on the test loss is
\[ \widehat{\Delta}_i \;=\; \tfrac{1}{n}\,\class{eq-test}{\nabla_\theta L(z_{\text{test}}, \hat\theta)}^{\!\top}\; \class{eq-model}{H^{-1}}\; \class{eq-data}{\nabla_\theta \ell(z_i, \hat\theta)} \]
where \(H\) is the Hessian of the training objective at the trained parameters: the curvature of the loss surface you finished on.
Read the rest of the symbols plainly. Theta-hat, \(\hat\theta\), is the trained setting of the dials. A nabla, \(\nabla\), in front of a loss is its gradient: the list of directions, one per dial, in which that loss climbs fastest. Capital \(L\) is the star’s test loss, while lowercase \(\ell\) is student \(i\)’s training loss. The transpose mark and products combine the three pieces by multiplying matching entries and adding them, an operation called a dot product. The \(1/n\) is bookkeeping: each student is one of \(n = 60\) in the class average, so deleting one changes the recipe by a one-sixtieth share.
FIG. 3 · THREE FACTORS, EACH WITH ONE JOB
Read it right to left, in the order training feels it. Example \(i\) pulls the parameters in some direction. The inverse Hessian converts that pull into actual movement: a small tug along a flat valley moves the optimum a long way, while the same tug against a stiff wall barely moves it. Then the test gradient measures how much of that movement the star, the gold-star test point this whole series scores against, actually notices.
Part 1 pictured the model’s three parameters, two feature weights and an offset, as dials, and training as turning them until the training loss, the penalty being minimized, stops improving. The Hessian is the stiffness of each dial at the resting point: how sharply the penalty climbs if you nudge it, plus how nudging one dial changes the pull on the other two. That is why it is a 3-by-3 grid of numbers rather than just three. The formula says influence is a conversation between three quantities: how hard the example pushes, how yielding the dials are, and whether the test point lives in the direction things moved. An example can push hard and still not matter if it pushes where the star does not look.
One Hessian, sixty answers
The second thing this formula buys is the economics of the whole field: the expensive factor does not depend on \(i\). The Hessian is one object, computed once from the trained model. After you have it (or rather, after you can multiply by its inverse), each additional training example costs one gradient and one dot product. Part 1’s bill was \(n\) retrainings for \(n\) answers; the influence function’s bill is one Hessian for all answers.
That exchange rate is why every scaling method in parts 4 through 6 (TracIn, EK-FAC, TRAK) is at bottom a scheme for approximating this one matrix, or for dodging it.
Leverage, or where the tangent lies
Now for the term rung 3 threw away. The influence function is a first-order estimate: it is the tangent line, at “example fully present,” of the curve that describes what happens as you dial the example’s weight down to zero. Tangent lines are honest near the point of tangency and lie more the farther you travel, and full removal is the far end of the trip.
How big is the lie? For regression there is an exact answer, and it is beautiful: the true removal effect exceeds the influence-function estimate by a factor of \(\frac{1}{1-h_i}\), where \(h_i\) is the example’s leverage: a number between 0 and 1 measuring how unusual the example’s position is, how much of a direction it has to itself. Ordinary points, leverage near zero, tangent barely lies. Unusual points, leverage near one, the truth can dwarf the estimate.
FIG. 4 · DIAL ONE STUDENT’S WEIGHT FROM 1 TO 0
The influence function is the tangent at weight 1 of exactly this curve: the dial refits the model with one student’s weight at \(\beta\) while the class average stays divided by sixty, the counterfactual the formula differentiates. Left: an ordinary, low-leverage student; the tangent tracks the curve to within 0.00013 across the whole dial. Right: a high-leverage student; the curve bends away from the tangent, missing by 0.00198 at full removal. The open ring at weight 0 is part 1’s true removal, the fifty-nine-student retrain: it sits another 0.0014 past each curve’s endpoint, because real removal also renormalizes the class average from sixty to fifty-nine, a small shift the frozen average hides. The toggle applies the \(1/(1-h)\) correction, the term the Newton step keeps on rung 1’s linear toy. It closes most of the high-leverage bend and barely moves the low-leverage line, which needed no correcting.
Here is the uncomfortable part. Which points have high leverage? The unusual ones: the outliers, the points holding down an under-sampled region alone. Which points do you most want to attribute when a model misbehaves? The same ones. First-order attribution is weakest exactly where the question is most interesting.
This theme runs through the whole series. One of part 1’s closing lessons, that unstable questions have unstable answers, was leverage wearing a costume. Part 3 is what happens when this tension meets deep networks.
Sixty estimates, one trained model
Cash the dare. Part 1 computed all sixty \(\Delta_i\) the honest way, sixty retrainings. The influence function computes all sixty from one trained model: one Hessian (a 3-by-3 matrix here, this is a toy), sixty gradients, sixty dot products.
FIG. 5 · THE FORMULA vs. SIXTY RETRAININGS
Every training point, estimate against truth, with the identity line, where estimate equals truth, for reference. Green points were helping the star, and orange-red points were hurting it. The toggle swaps in the one-Newton-step estimate from Fig. 2, which keeps the leverage correction: watch the biggest misses, out in the far corners, snap toward the line. Hover a point to find it in the classroom.
On this classroom the correlation is 0.9992, and the Newton step is tighter still, at 0.9998. The tangent’s lies from the last section are still here in the biggest misses, which the Newton toggle in Fig. 5 pulls toward the identity line as the overall fit tightens from 0.9992 to 0.9998. That is the promise from part 1 kept: for a small, single-bowl model like this one, the counterfactual you cannot afford is recoverable from the model you already have, essentially for free.
Where the fine print lives
The climb rested on assumptions that were all true here and are all false for the models you actually care about.
The derivation assumed training finds the minimum: unique, and with curvature you can invert. Our classroom qualifies: its loss has one lowest point and carries a regularizer (a small penalty on extreme weights that keeps the bowl from flattening out), so both assumptions hold.
A deep network’s loss surface has no unique minimum, entire flat valleys of them, and its Hessian is not invertible at all: some directions have no stiffness whatsoever, and \(H^{-1}\) divides by that zero. The derivation also assumed the tangent line is taken at a converged optimum, but real training stops early, at a checkpoint that is still moving.
Every one of these failures has a name, a paper, and a repair bill, and they are not edge cases: at deep-network scale, the estimator this post built stops tracking retraining almost entirely (Basu, Pope, and Feizi 2021; Ilyas et al. 2024). The climb this post made ends in wreckage at scale.
The same formula, pointed the other way
It is easy to hear “fragile at scale” as “harmless.” Koh and Liang’s original paper used influence functions as more than a diagnostic lens: they used them as a weapon, crafting visually unremarkable training images whose presence flips a model’s predictions on chosen test examples: a data-poisoning attack steered by exactly the formula in Fig. 3 (Koh and Liang 2017). In their demo, an imperceptible tweak to a single training image flipped the classifier’s verdict on over half of a set of test photos of one dog, the first author’s.
Attribution methods are dual-use: a tool that says “this example moved the model most” is also a targeting system for anyone who gets to add examples. Part 7 returns to poisoning as a first-class application.
- One trained model prices all sixty counterfactuals: test gradient, times inverse Hessian, times training gradient.
- The estimate is a tangent line, and leverage measures where the tangent lies most: exactly the unusual points you most want to attribute.
- The guarantee rests on a convex bowl (one with one lowest point) and a converged optimum, and deep networks have neither.
Now, the wreckage. Part 3 takes the estimator to deep networks, watches it stop tracking retraining, and lands on the surprising verdict that the influence function keeps being a good estimator of a different question.
Further reading
The paper this post orbits
- Koh and Liang, “Understanding Black-box Predictions via Influence Functions” (Koh and Liang 2017). The 2017 import of influence functions into deep learning: the estimator, the approximations that make it run, and the poisoning demo from the close of this post.
The statistics it came from
- Hampel, “The Influence Curve and Its Role in Robust Estimation” (Hampel 1974). The 1974 paper that named the idea; Fig. 1 is its first example.
- Pregibon, “Logistic Regression Diagnostics” (Pregibon 1981). Rung 2: the one-Newton-step deletion diagnostic, leverage correction included.
- Jaeckel’s Bell Labs memo introduced the infinitesimal jackknife (Jaeckel 1972), with the functional calculus going back to von Mises (Mises 1947).
The modern guarantees
- Giordano, Stephenson, Liu, Jordan, and Broderick, “A Swiss Army Infinitesimal Jackknife” (Giordano et al. 2019). Finite-sample error bounds for exactly the estimator built here.
- Rahnama Rad and Maleki, “A Scalable Estimate of the Extra-Sample Prediction Error via Approximate Leave-One-Out” (Rahnama Rad and Maleki 2018). The approximate-LOO line of work in high-dimensional statistics.
The wider map
- Ilyas, Park, Engstrom, Georgiev, and Mądry, “Data Attribution at Scale” (Ilyas et al. 2024). The ICML 2024 tutorial: this post is roughly its “theoretical foundations” chapter with sliders.
References
Reproducibility. Every figure on this page is live and every number is real. The dataset is part 1’s classroom, regenerated bit-for-bit by a seeded script in the site repository (scripts/influence-data/) that also computes the influence-function estimates, the Newton-step estimates, the leverage values, and the exact weight-dialing curves in Fig. 4, and asserts they agree with part 1’s sixty retrainings before shipping the JSON this page loads.
Acknowledgements. The three-factor reading of the influence formula and the treatment of leverage as the gap between the influence function and the exact removal response follow Louis Jaburi’s lecture notes “Data (Attribution) for Alignment” (Iliad Intensive, April 2026).