The series has spent most of its time making attribution cheaper. Part 1 retrained the model after removing each example. Part 2 replaced those retrainings with an influence function, a local prediction based on gradients and curvature. Part 4 followed gradients through the training run. Parts 5 and 6 introduced curvature approximations, datamodels, and TRAK for larger models. Each step gave us another way to produce one score per training example.
That progression can sound like a ladder, with each new method taking a higher rung. A ladder needs an ordering, though, and we have never supplied one. Which scores should we trust? A cheap answer is to compare each new method with the previous one. Agreement only tells us that two estimators share an opinion. It cannot tell us whether either opinion predicts retraining.
This epilogue builds a ruler from retraining and gives every scorer the same exam. Two deliberately simple baselines sit at the same table. The result is less tidy than a champion’s podium. That is useful: a method name alone tells you very little until you know what its scores were checked against.
Build the ruler before ranking the methods
The linear datamodeling score from part 6 asks a counterfactual question many times. Pick a subset of the training set. Retrain the model from scratch on the points that remain. Record how much the gold-star query’s loss changed. Then ask an attribution scorer to predict that same change by adding the scores of the removed points. Repeat across many fixed subsets and compare the two rankings with Spearman correlation. That correlation is the LDS.
The bake-off uses part 3’s small nonlinear network, a depth-2 tanh MLP, because this is where the methods separate. On a logistic classroom with only 3 parameters, the larger approximations collapse toward their simplest forms and the comparison turns into a tie. The ground truth here comes from the MLP’s own 60 leave-one-out retrainings and 200 subset retrainings. Every subset keeps 48 of the 60 classroom points, so each predicted change sums the scores of the 12 dropped points.
Fig. 1 makes the ruler visible. Each dot on the right is one subset. Its horizontal position is the scorer’s predicted loss change; its vertical position is the change from a real retraining. A useful scorer puts the dots in the same rank order as reality. The diagonal gives a stricter visual reference for calibration, although LDS itself checks rank order rather than distance from that line.
FIG. 1 · BUILD THE RULER FROM FIXED RETRAININGS
Each dot is one of the same 200 fixed subsets used throughout this post. Hover or focus a dot to see its 12 dropped students fade in the classroom, then switch to the deliberately degraded scorer and watch the Spearman LDS fall from 0.7562 to 0.5596. This figure establishes what a higher LDS looks like. It does not judge the seven bake-off scorers yet, and the diagonal is a calibration reference rather than the quantity Spearman measures.
The trustworthy view uses the damped Gauss-Newton influence estimate from part 3. The degraded view multiplies its per-point scores by fixed random noise before the subset sums are formed. It is a synthetic teaching control, included so a worse ranking has an observable signature before any method carries a familiar name. The actual retraining results never change when you toggle. Only the predictions move.
The assumption inside the ruler
LDS asks each scorer to predict a group removal by summing single-point scores. That move assumes additivity: the effect of removing a group equals the sum of its members’ individual effects. Part 7’s unlearning experiment showed where that assumption strains. Points can back one another up, compete for the same role, or change the fitted representation together. Once that happens, the group has an effect that the separate scores cannot express.
Our subsets remove about 20% of the classroom at once. That makes LDS a demanding test of the scores and a test of their linear form at the same time. A low LDS can mean that a scorer estimated single-point effects poorly. It can also mean that useful single-point scores failed to add over this removal size. The ruler is still informative, but its reading belongs to this subset distribution.
There is another boundary. Spearman checks whether predicted and actual changes have the same rank order. A scorer can therefore receive a high LDS while compressing every magnitude or shifting the whole cloud away from the diagonal. That choice matches part 6’s definition and the data-attribution literature (Ilyas et al. 2022). It answers a ranking question, which fits selection and retrieval uses, while leaving calibration as a separate audit.
Seven scorers enter on equal terms
Every scorer sees the same trained MLP, the same gold-star query at \((0.35, -0.15)\), and the same 200 subset masks. The leaderboard begins alphabetically so the plotted position carries no hint about the result.
Exact LOO x60 retrains once without each classroom point, then sums those measured single-point changes. It is the strongest single-point information available here, although its group prediction still assumes additivity.
Damped GNH IF is part 2’s influence idea on the MLP, using part 3’s damped Gauss-Newton Hessian to turn training gradients into local loss-change estimates.
K-FAC IF is part 5’s scale move: it replaces the full curvature matrix with layerwise Kronecker factors, then solves the influence system through that structured approximation (Grosse et al. 2023).
TracIn is part 4’s path-based scorer. It adds gradient alignment across saved training checkpoints, so it asks how a point and the query pushed together during optimization (Pruthi et al. 2020).
TRAK-mini is part 6’s projected datamodel recipe on this classroom MLP. It projects the gradients to 16 dimensions and applies a correction designed to predict retraining behavior (Park et al. 2023).
Grad dot takes the raw inner product between each training gradient and the query gradient. It has no curvature correction and no checkpoint history.
Query-blind ranks each training point by its own loss. It never looks at the gold-star query, which makes it a useful check on whether query-specific machinery beats a generic hard-example ranking.
The final two methods are deliberately plain, but Fig. 2 gives them the same bar treatment as the others. A baseline loses its value if the visual design tells the reader to ignore it before the measurement does.
FIG. 2 · THE SAME EXAM, SEVEN SCORERS
All seven scorers predict the loss changes for the same 200 subsets and are compared with the same real retrainings. The default order is alphabetical; sort by score to reveal the ranking. Exact LOO leads this toy at 0.8052, damped GNH IF follows at 0.7562, and TRAK-mini reaches 0.6321. This ranks subset predictions for one fixed MLP and query. It does not establish a general ranking for other models, queries, or subset distributions.
Sorting produces an immediate surprise at the bottom. The raw gradient dot product reaches 0.2601, close to TracIn’s 0.2641 on this exam. K-FAC IF scores 0.3374, while the query-blind ranking falls to -0.1623. The extra machinery changes the result, but method complexity does not determine the order.
The top of the chart also explains why exact leave-one-out is a demanding reference rather than a complete oracle. Exact LOO has the best LDS at 0.8052, yet it remains below a perfect rank correlation. Its single-point measurements are exact for this model and query. Their sums cannot capture every interaction among the 12 points removed together. The gap is additivity made visible.
One score can hide seven different answers
The leaderboard evaluates subset-level prediction. A practitioner often asks a point-level question: which training examples mattered most for this output? Those views can diverge. Two scorers may preserve a similar ordering over 200 group removals while assigning the largest individual scores to different students.
Fig. 3 returns to the classroom for that question. Each panel marks the five points with the largest absolute score for one method. Green means the score predicts that removing the point raises the query loss, so the point is helpful. Vermillion means the predicted removal lowers the loss, so the point is harmful. Mark size records rank within the panel. Hover, focus, or tap any position to trace the same student through all seven opinions.
FIG. 3 · SEVEN ANSWERS TO ONE QUERY
Every panel contains the same 60 students and the same gold-star query; only the scorer changes. The top-five sets overlap without becoming identical, and some shared points change sign. This shows that a scalar LDS can conceal disagreement about which examples matter. It does not declare a panel correct, except that exact LOO supplies the closest available single-removal ground truth for this fixed model and query.
Some agreement is reassuring. Student positions 35, 24, and 50 in one-based classroom numbering recur across several panels, although their signs and ranks move. The disagreements are equally informative. TRAK-mini elevates positions that several other scorers leave in the background. K-FAC and TracIn can emphasize the same location with opposite signs. A downstream debugging tool would show different evidence depending on which panel supplied its ranking.
This is why a leaderboard needs a second view. LDS asks whether a scorer predicts group retraining over a chosen subset distribution. The top-five panels ask whether scorers tell the same local story about one query. Those questions share data and still have different failure modes. A method can be useful for choosing subsets while unstable as an explanation interface, or steady on the top few points while mediocre over the full collection of subset effects.
Draw the line around the toy
A 60-point classroom and one query are an instrumented example, not a benchmark for language models. The experiment can establish that these seven scorers disagree on a nonlinear model, that a query-blind baseline fails this particular exam, and that raw gradient similarity can sit near a path-based method. It cannot establish which ordering survives a change of architecture, training run, corpus, task, or query.
The model is also much smaller than the systems K-FAC and TRAK were designed to make tractable. Their approximations have room to matter here because the MLP is nonlinear, yet the computational pressure that motivated them is absent. Exact LOO is affordable. Curvature matrices fit in memory. Repeating the training run is routine. At language-model scale, the methods face approximation choices and systems constraints that this classroom removes.
So the claim from Figs. 1 through 3 stops at the boundary of the generated JSON: one depth-2 tanh MLP, one query, 200 shared subset retrainings, and 60 shared leave-one-out retrainings. The field needs application-level evidence beyond that line.
The real bake-off changes the winner
DATE-LM provides that broader check. It evaluates attribution methods and simple non-attribution baselines on training-data selection, toxicity and bias filtering, and factual attribution for language models (Jiao et al. 2025). Its tasks use different metrics and experimental designs, so the qualitative panel below uses low, medium, and high bands within each task. The bands should never be read as one common numeric scale.
FIG. 4 · THE REAL BAKE-OFF, QUALITATIVE REPRODUCTION
Qualitative reproduction of DATE-LM’s documented finding across its three task categories (Jiao et al. 2025). Task-specific baselines can match or beat attribution on selection and filtering, while similarity-based attribution is stronger in the benchmark’s counterfactual factual-attribution setup. The bands summarize low, medium, and high performance within each task; they are not DATE-LM’s numeric results, and they cannot be compared as a shared unit across columns.
The pattern is the point: no family stays high in every column. DATE-LM also shows why evaluation design matters. In factual attribution, lexical overlap can let a retrieval baseline look strong for the wrong reason. Breaking that overlap changes the ordering. In toxicity filtering, safety-aligned examples that resemble unsafe text make detection harder. The benchmark measures the application, including these confounders, instead of assuming one generic attribution metric transfers unchanged.
That conclusion matches a broader review of influence-based attribution: open obstacles include choosing a target quantity and validating approximations under distribution and scale changes. The review also asks how point scores connect to the decisions they are meant to support (Zhu and Cangelosi 2025). Better linear algebra cannot settle a poorly specified evaluation.
What to ask when someone shows you an attribution score
This series began with a strange prediction and a concrete question: which training example taught the model that? Exact retraining gave an answer and exposed the cost. Influence functions made a local estimate. Training-path methods changed what was differentiated. Curvature approximations made larger models possible. Datamodels changed the unit from a parameter perturbation to a prediction over datasets. Applications turned the scores into choices about data.
The bake-off adds the missing audit. Ask what the attribution number was checked against. Ask whether every method saw the same subsets, query, model, and training run. Inspect a simple baseline with the same attention as the named method. Then look past the summary score and compare the examples each method actually elevates.
There is no crown at the end of the series. DATE-LM found no method that dominates across its tasks, and this classroom found different answers even where leaderboard values were close. That leaves a stronger habit than a winner would: treat attribution as a prediction, state the counterfactual it claims to approximate, and demand a check that can prove it wrong.
Further reading
Benchmarks and evaluation
- Jiao et al., “DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models” (Jiao et al. 2025). The application-driven benchmark behind Fig. 4 and its no-single-winner finding.
- Zhu and Cangelosi, “Revisiting Data Attribution for Influence Functions” (Zhu and Cangelosi 2025). A recent account of the open technical and evaluation obstacles.
Methods behind the bake-off
- Park et al., “TRAK: Attributing Model Behavior at Scale” (Park et al. 2023), and Pruthi et al., “Estimating Training Data Influence by Tracing Gradient Descent” (Pruthi et al. 2020). The projected-datamodel and training-path branches.
- The ICML tutorial “Data Attribution at Scale” (Ilyas et al. 2024) and Ilyas et al., “Datamodels” (Ilyas et al. 2022). The evaluation framing and the linear-datamodel view used by LDS.
References
Reproducibility. Every toy result comes from scripts/influence-data/09_bakeoff_lds.py with SEED=7. The script trains the depth-2 tanh MLP from a fixed initialization, uses the chosen query \((0.35, -0.15)\), performs 200 retrainings on fixed subsets that each keep 48 of 60 points, and performs 60 leave-one-out retrainings. All scorers share the same subset masks. LDS is Spearman rank correlation. The query-selection check kept the series’ existing gold-star query, so no replacement query was needed. The JSON, static fallbacks, and featured image are generated together; Fig. 4 is a cited qualitative reproduction and contains no DATE-LM numeric values.