Standard Benchmarks Fail - Auditing LLM Agents in Finance Must Prioritize Risk

Standard benchmarks fixate on how well large language model (LLM) agents perform in finance, yet say little about whether they are safe to deploy. We argue that accuracy metrics and return-based scores provide an illusion of reliability, overlooking vulnerabilities such as hallucinated facts, stale data, and adversarial prompt manipulation. We take a firm position: financial LLM agents should be evaluated first and foremost on their risk profile, not on their point-estimate performance. Drawing on risk-engineering principles, we outline a three-level agenda: model, workflow, and system, for stress-testing LLM agents under realistic failure modes. To illustrate why this shift is urgent, we audit six API-based and open-weights LLM agents on three high-impact tasks and uncover hidden weaknesses that conventional benchmarks miss. We conclude with actionable recommendations for researchers, practitioners, and regulators: audit risk-aware metrics in future studies, publish stress scenarios alongside datasets, and treat ``safety budget'' as a primary success criterion. Only by redefining what ``good'' looks like can the community responsibly advance AI-driven finance.

@article{chen2025_2502.15865,
  title={ Standard Benchmarks Fail - Auditing LLM Agents in Finance Must Prioritize Risk },
  author={ Zichen Chen and Jiaao Chen and Jianda Chen and Misha Sra },
  journal={arXiv preprint arXiv:2502.15865},
  year={ 2025 }
}