Small changes in how a prompt is worded can lead to meaningful differences in the behavior of large language models (LLMs), raising concerns about the stability and reliability of their evaluations. While prior work has explored simple formatting changes, these rarely capture the kinds of natural variation seen in real-world language use. We propose a controlled paraphrasing framework based on a taxonomy of minimal linguistic transformations to systematically generate natural prompt variations. Using the BBQ dataset, we validate our method with both human annotations and automated checks, then use it to study how LLMs respond to paraphrased prompts in stereotype evaluation tasks. Our analysis shows that even subtle prompt modifications can lead to substantial changes in model behavior. These results highlight the need for robust, paraphrase-aware evaluation protocols.
View on arXiv@article{chataigner2025_2505.03563,
title={ Say It Another Way: A Framework for User-Grounded Paraphrasing },
author={ Cléa Chataigner and Rebecca Ma and Prakhar Ganesh and Afaf Taïk and Elliot Creager and Golnoosh Farnadi },
journal={arXiv preprint arXiv:2505.03563},
year={ 2025 }
}