RankLLM: A Python Package for Reranking with LLMs

The adoption of large language models (LLMs) as rerankers in multi-stage retrieval systems has gained significant traction in academia and industry. These models refine a candidate list of retrieved documents, often through carefully designed prompts, and are typically used in applications built on retrieval-augmented generation (RAG). This paper introduces RankLLM, an open-source Python package for reranking that is modular, highly configurable, and supports both proprietary and open-source LLMs in customized reranking workflows. To improve usability, RankLLM features optional integration with Pyserini for retrieval and provides integrated evaluation for multi-stage pipelines. Additionally, RankLLM includes a module for detailed analysis of input prompts and LLM responses, addressing reliability concerns with LLM APIs and non-deterministic behavior in Mixture-of-Experts (MoE) models. This paper presents the architecture of RankLLM, along with a detailed step-by-step guide and sample code. We reproduce results from RankGPT, LRL, RankVicuna, RankZephyr, and other recent models. RankLLM integrates with common inference frameworks and a wide range of LLMs. This compatibility allows for quick reproduction of reported results, helping to speed up both research and real-world applications. The complete repository is available atthis http URL, and the package can be installed via PyPI.

@article{sharifymoghaddam2025_2505.19284,
  title={ RankLLM: A Python Package for Reranking with LLMs },
  author={ Sahel Sharifymoghaddam and Ronak Pradeep and Andre Slavescu and Ryan Nguyen and Andrew Xu and Zijian Chen and Yilin Zhang and Yidi Chen and Jasper Xian and Jimmy Lin },
  journal={arXiv preprint arXiv:2505.19284},
  year={ 2025 }
}