In 2023, researchers at Anthropic decided to point influence functions at a real large language model: which training sequences move a 52-billion-parameter model’s answers? (Grosse et al. 2023). Take part 2’s formula, which scores a training example by pushing its gradient through the inverse of the Hessian and against the test point’s gradient, and try to write down its bill for that model. The Hessian is a square matrix with one row and one column per parameter, each entry recording how a pair of parameters jointly bends the loss: \(52\text{B} \times 52\text{B}\), roughly \(2.7 \times 10^{21}\) entries. Stored in half precision (two bytes per entry), that is around five million petabytes, eight orders of magnitude beyond a big hard drive. Storing it would be hard enough. The formula wants it inverted.
Part 3 catalogued four ways the formula’s assumptions fail; sheer size was the fourth cause, and it got postponed. No more postponing. This cause has three answers with different assumptions and costs. By the end of this post, influence functions run on real language models, and the series’ toy classroom will have checked each idea’s arithmetic.
Road one: factor it
The Hessian is impossibly large, but most of it was never going to matter. The factoring road makes two structural bets, each of which collapses the object by orders of magnitude (Martens and Grosse 2015; George et al. 2018):
Bet one: layers don’t interact. Treat curvature between parameters in different layers as zero. The giant matrix becomes a chain of blocks along the diagonal (block-diagonal, in matrix speak), one per layer: still huge, but now each block only spans one layer’s parameters.
Bet two: within a layer, curvature factorizes. A linear layer’s gradient has a special shape: it is built from two smaller pieces, what came into the layer and what flowed back through it, multiplied together (an outer product: every entry of one piece times every entry of the other).
K-FAC (Kronecker-factored approximate curvature) bets that the layer’s curvature block inherits that shape: approximately the Kronecker product (the matrix version of the same trick: every entry of the first small matrix scales a full copy of the second, so two small tables generate one enormous one) of two small matrices, one summarizing the inputs, one the backward signals (Martens and Grosse 2015). A block spanning millions of parameters becomes two matrices of hundreds to tens of thousands of rows, depending on the layer’s width. The huge block is generated when needed rather than stored, and inverting it only requires inverting its two small factors.
EK-FAC, the eigenvalue-corrected version of K-FAC, sharpens the bet by rescaling the factored approximation along its own natural axes so its entries better match the true block. The correction stays inside the same cheap factored form (George et al. 2018). Combined with part 3’s Gauss-Newton stand-in for the curvature (a rebuilt Hessian guaranteed to have no negative directions), that is what actually ran at 52B (Grosse et al. 2023).
The study presents this EK-FAC influence estimate as an approximation to part 3’s reframed question, the proximal Bregman response function (PBRF), which asks how the learned function responds after a training example is nudged and the model is allowed to settle again (Grosse et al. 2023).
Watch the object collapse:
FIG. 1 · SQUEEZING 10²¹ ENTRIES INTO MEMORY, TWO BETS AT A TIME
The factoring road, with live arithmetic. Pick a model size and apply the two bets in sequence: full matrix, block-diagonal by layer, Kronecker-factored per block. The readout is real storage math for that architecture, from our sixty-student classroom network up to frontier scale. The bets are approximations, not theorems; the next figure prices them.
Bets need auditing, and the classroom is where this series audits. Our part 3 network is small enough to compute the full Gauss-Newton Hessian exactly, so we can run influence with the true matrix and with its K-FAC factorization and see what the factorization costs:
FIG. 2 · WHAT THE FACTORING BETS COST, MEASURED
Sixty influence estimates computed twice on the part 3 classroom network: once with the exact Gauss-Newton Hessian, once with its K-FAC factorization. The toggle grades both against the sixty true retrainings (drop one student, retrain from scratch, and measure how the star’s answer moves). The factorization is an approximation with a measurable, modest price here; at real scale the same trade is what makes the computation exist at all.
What the 52B run found. Two results worth carrying (Grosse et al. 2023).
Influence over a large corpus is heavy-tailed and spread out: typically no single smoking-gun document, but a long tail of related sequences each contributing a little. That finding should permanently retire the fantasy that every model behavior has one responsible training example.
And influence abstracts with scale: for small models the influential sequences share surface wording with the query; for larger models they increasingly share themes and structure, influence tracking meaning rather than string overlap.
Road two: never form it
The second road is newer. Everything so far treats training as producing one model, a single point \(\hat\theta\). But you could instead think in distributions: of all the parameter settings compatible with the data, weighted by how well they fit, which is a posterior. And in that view, a training example’s influence on a test observable (any number you measure on the test point; here, its loss) is (minus) the covariance, over that distribution, between the example’s loss and the observable (Kreer et al. 2025).
Sit with how strange and how practical that is. No Hessian appears. No inversion appears.
A covariance is something you estimate by sampling: jiggle the parameters around the trained model (stochastic-gradient Langevin dynamics does exactly this: gradient steps plus noise calibrated so the walk visits parameter settings in proportion to their posterior weight), record each sample’s loss on example \(i\) and on the test point, and take the covariance of the two columns. If they move together across the posterior, models that fit example \(i\) better also do better on the test point: that is influence, seen without ever touching curvature.
The classical influence function drops out as the leading term of this quantity under a Gaussian (Laplace) approximation, which replaces the posterior near the trained model with a bell-shaped distribution (Kreer et al. 2025). That placement explains part 3 better than part 3 could explain itself: the flat valley floors of the loss surface, the near-zero-curvature directions where part 3 watched influence functions break, are exactly where a Gaussian approximation of the posterior is most wrong, and a sampler just walks the valley and averages over it.
The classroom can show the whole mechanism live:
FIG. 3 · INFLUENCE WITHOUT A HESSIAN: JIGGLE AND CORRELATE
The Bayesian influence function (BIF) on the classroom, live: press “Replay the sampling” to watch the estimates build. Left: posterior samples around the trained model (a precomputed Langevin chain, replayed). Right: for three students, the running covariance between their loss and the star’s loss, sharpening as samples accumulate, with part 2’s influence-function answers as reference lines.
Notice the curves converge to roughly 0.4x the magnitude of each reference line, on the same side of zero (below the line for the positive, helpful case and above it for the two negative cases), while ranking the students identically (correlation 0.999 across all sixty). The sampler’s temperature (how much noise each Langevin step injects, relative to the loss’s pull) sets the covariance’s units, so the BIF recovers the influence function’s geometry, not its raw numbers.
The honest ledger for road two: sampling replaces a one-time matrix computation with compute you can dial (more samples, better estimates). It inherits decades of craft and caveats from Markov chain Monte Carlo (MCMC), the statistics of estimating quantities from exactly this kind of correlated random walk, and at LLM scale it is young.
Related tools based on susceptibilities (the same jiggle-and-correlate idea, measuring how a model’s behavior shifts when part of the training distribution is reweighted) are already being used to dissect what small language models learned (Baker et al. 2025). But road two is the only road whose definition of influence already knows about part 3’s valleys, and it extends the endpoint story (the family of methods that only ever look at the final trained model) rather than patching it.
Road three: let training do the damping
The third road is short here only because part 4 already built it. Unrolling-family methods never form the Hessian either: they differentiate the training run, and the chains of per-step factors they multiply together (part 4’s transport chains) automatically implement the filter \((1 - e^{-\eta K \sigma})/\sigma\), with \(\eta\) the learning rate, \(K\) the number of steps, and \(\sigma\) the curvature in one direction.
The filter agrees with \(1/\sigma\) where curvature is healthy and levels off at the training effort \(\eta K\) (learning rate times number of steps) where it vanishes. The damping constant (the small number practitioners add to the curvature so its flat directions don’t make the inverse explode) everyone else tunes by hand, the trajectory sets for free.
At language-model scale this family runs too: TracIn-style checkpoint sums (adding up gradient agreement at saved checkpoints along the run), SOURCE’s segmented transport (part 4’s coarse-grained unrolling) (Bae et al. 2024), and influence-function variants built directly on training-run structure for LLM-scale data valuation (Choe et al. 2024). A 2025 extension even chains EK-FAC influence through multiple training stages, tracing a fine-tuned model’s behavior back to its pretraining data (Bao et al. 2025).
Choosing a road
Which one should you actually reach for? The benchmark answer depends on the data type, such as text or images, and the budget.
- Road one is the tested workhorse: in the head-to-head evaluations gathered for the field’s 2024 ICML tutorial, the factored endpoint method EK-FAC led on language modeling while surrogate methods (train many small stand-in models and fit influence from them; part 6 builds them properly) led on vision, with everything priced in GPU-hours on a shared axis (Ilyas et al. 2024).
- Road two is the youngest and the most theoretically self-aware: reach for it when part 3’s valleys are your actual problem and you can spend samples instead of linear algebra.
- Road three owns the questions where when matters: which checkpoint, stage, or phase of training a behavior came from.
Part 6 draws this map properly, with our own measured numbers on the classroom next to the field’s published ones.
But notice what all three roads share. Each one stopped demanding what part 2’s derivation demanded, a converged optimum with invertible curvature, and started respecting what actually exists: a finite training run, an uncertain posterior, and an architecture with structure the math can exploit. The formula survived scale by giving up its idealizations, which is the most transferable lesson in this series.
Further reading
Road one
- Grosse et al., “Studying Large Language Model Generalization with Influence Functions” (Grosse et al. 2023). The 52B run: EK-FAC, the query-batching tricks, and the heavy-tail and abstraction findings.
- Martens and Grosse, “Optimizing Neural Networks with Kronecker-factored Approximate Curvature” (Martens and Grosse 2015), and George et al., “Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis” (George et al. 2018). The two bets, in their original habitat (optimization).
Road two
- Kreer, Wu, Adam, Furman, and Hoogland, “Bayesian Influence Functions for Hessian-Free Data Attribution” (Kreer et al. 2025). Influence as posterior covariance; the classical IF as its Laplace leading term.
- Baker, Wang, Hoogland, and Murfet, “Structural Inference: Interpreting Small Language Models with Susceptibilities” (Baker et al. 2025). The susceptibility view put to interpretive work.
Road three
- Bae, Lin, Lorraine, and Grosse, “Training Data Attribution via Approximate Unrolling” (Bae et al. 2024), and Choe et al., “What is Your Data Worth to GPT?” (Choe et al. 2024). Part 4’s family, at language-model scale.
References
Reproducibility. The storage arithmetic in Fig. 1 uses a simplified weights-only model of each architecture. It omits bias and normalization parameters, treats embedding lookup tables as generic dense layers, and treats the matrix that projects each feed-forward block back to the model width the same way. The result is accurate to the stated order of magnitude, not a literal parameter-by-parameter accounting.
Figs. 2 and 3 are real experiments on the series’ classroom, generated by a seeded script in the site repository (scripts/influence-data/): the K-FAC comparison computes both the full Gauss-Newton Hessian and its Kronecker factorization exactly, and the Bayesian-influence demo ships an actual Langevin chain with its running covariances, asserted against part 2’s influence values before shipping. The 52B-scale claims are cited, not reproduced.
Acknowledgements. The three-roads organization of this post and the framing of the Bayesian influence function against the Laplace approximation follow Louis Jaburi’s lecture notes “Data (Attribution) for Alignment” (Iliad Intensive, April 2026).