Abstract

We use training-data attribution as an interpretable tool for capability discovery, mapping which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in OLMo3-7B. Training-data attribution measures how strongly each training document influences a model’s predictions on a benchmark, but document-level scores are too noisy to identify which corpus regions support which capabilities. We compute gradient-based attribution (TrackStar via Bergson) over a working set drawn from the de-duplicated Dolma3 mix, aggregate influence across WebOrganizer’s 24-format × 24-topic taxonomy (576 bins), and contrast benchmark pairs in a 2×2 design that varies domain (social vs. STEM) and capability type (reasoning vs. knowledge): SocialIQA and MMLU Social Sciences against ARC-Challenge and MMLU STEM. Social and STEM reasoning draw on qualitatively distinct corpus regions, and the contrast is sharper at the reasoning level than at the knowledge level. Targeted machine unlearning provides partial causal validation: forgetting high-attribution topics (e.g., Literature for SocialIQA) degrades the aligned benchmark more than within-topic random baselines. We validate on two other open-data models, Comma v0.1 7B-2T (Common Pile) and DCLM-Baseline-7B (DataComp-LM): causal selectivity holds on both models, while the provenance map is ecosystem-specific. We open-source all code, data artifacts, influence scores, and checkpoints at https://github.com/eilab-gt/capabilibara.

At a Glance

• Gradient-based training-data attribution as a discovery tool: we run TrackStar via Bergson over a working set drawn from the de-duplicated Dolma3 mix, so corpus-scale influence becomes a readable map of where a capability comes from rather than a pile of noisy per-document scores
• Structured taxonomy, not top-k documents: influence is aggregated into WebOrganizer’s 24-format × 24-topic taxonomy (576 bins), which turns attribution into a principled instrument you can inspect before, not after, picking exemplars
• A 2×2 contrastive design: SocialIQA and MMLU Social Sciences against ARC-Challenge and MMLU STEM separate domain (social vs. STEM) from capability type (reasoning vs. knowledge); the two reasoning tasks draw on qualitatively distinct corpus regions, and the split is sharper at the reasoning level than at the knowledge level
• Targeted unlearning as causal validation: forgetting high-attribution topics (e.g., Literature for SocialIQA) degrades the aligned benchmark more than within-topic random baselines, so the identified regions are load-bearing rather than merely correlated
• Replication across two more open-data ecosystems: re-running the whole pipeline on Comma v0.1 7B-2T (Common Pile) and DCLM-Baseline-7B (DataComp-LM) keeps it causally selective, while the recovered provenance map stays specific to each corpus

Cite This Paper

@inproceedings{matlin2026capability,
  title     = {Capability Provenance in Language Models: A Case Study in Social Reasoning},
  author    = {Matlin, Glenn and Chakraborty, Chandreyi and Eom, Saehee and Okamoto, Mika and Castilla, Rayan and Jaburi, Louis and Deng, Alvin and Min, Taywon and Quirke, Lucia and Biderman, Stella and Riedl, Mark},
  booktitle = {Conference on Language Models (COLM)},
  year      = {2026}
}