We introduce FinNLI, a benchmark dataset for Financial Natural Language Inference (FinNLI) across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts. Our dataset framework ensures diverse premise-hypothesis pairs while minimizing spurious correlations. FinNLI comprises 21,304 pairs, including a high-quality test set of 3,304 instances annotated by finance experts. Evaluations show that domain shift significantly degrades general-domain NLI performance. The highest Macro F1 scores for pre-trained (PLMs) and large language models (LLMs) baselines are 74.57% and 78.62%, respectively, highlighting the dataset's difficulty. Surprisingly, instruction-tuned financial LLMs perform poorly, suggesting limited generalizability. FinNLI exposes weaknesses in current LLMs for financial reasoning, indicating room for improvement.
View on arXiv@article{magomere2025_2504.16188,
title={ FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking },
author={ Jabez Magomere and Elena Kochkina and Samuel Mensah and Simerjot Kaur and Charese H. Smiley },
journal={arXiv preprint arXiv:2504.16188},
year={ 2025 }
}