Abstract

Language Models (LMs) have demonstrated impressive capabilities with core Natural Language Processing (NLP) tasks. The effectiveness of LMs for highly specialized knowledge-intensive tasks in finance remains difficult to assess due to major gaps in the methodologies of existing evaluation frameworks, which have caused an erroneous belief in a far lower bound of LMs’ performance on common Finance NLP (FinNLP) tasks. To demonstrate the potential of LMs for these FinNLP tasks, we present the first holistic benchmarking suite for Financial Language Model Evaluation (FLaME). We are the first research paper to comprehensively study LMs against ‘reasoning-reinforced’ LMs, with an empirical study of 23 foundation LMs over 20 core NLP tasks in finance. We open-source our framework software along with all data and results.

At a Glance

• 23 foundation language models evaluated, including reasoning-reinforced variants
• 20 core finance NLP tasks spanning knowledge, reasoning, and compliance
• Open framework, data, and results — the full pipeline is reproducible via the public repository
• Published in the Findings of ACL 2025 (pp. 22633–22679)

How the framework works

What the benchmark covers. FLaME spans the parts of finance where a wrong answer is expensive: financial knowledge and terminology, numerical and multi-step reasoning, regulatory and compliance understanding, market and portfolio judgment, and robustness under perturbed or adversarial inputs. The 20 core tasks were chosen to reflect the work practitioners actually hand to a model, not generic language benchmarks that happen to use financial text.

How the evaluation is designed. Every task uses standardized construction, so comparisons across the 23 models are like-for-like, with metrics that keep accuracy and calibration separate and make failure cases legible instead of averaging them away. The pipeline controls for the usual ways a finance benchmark leaks, and each stage is modular, so a new model or a new task suite drops into the same harness without rewriting the evaluation.

What researchers can build on. Because the framework, data, and results are all open, the benchmark supports work beyond the paper: comparing frontier and open-weight models on finance-specific tasks, measuring how much fine-tuning or prompting actually moves the numbers, probing safety and misuse risk in a high-stakes domain, and grounding domain-adaptation claims in reproducible evidence.

Cite This Paper

@inproceedings{matlin-etal-2025-financial,
    title = "Financial Language Model Evaluation ({FL}a{ME})",
    author = "Matlin, Glenn  and
      Okamoto, Mika  and
      Pardawala, Huzaifa  and
      Yang, Yang  and
      Chava, Sudheer",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.1164/",
    doi = "10.18653/v1/2025.findings-acl.1164",
    pages = "22633--22679",
    ISBN = "979-8-89176-256-5"
}