ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 2412.01269
92
0

CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search

2 December 2024
Kaixin Wu
Yixin Ji
Ziyang Chen
Qiang Wang
Cunxiang Wang
Hong Liu
Baijun Ji
Jia Xu
Zhongyi Liu
Jinjie Gu
Yuan Zhou
Linjian Mo
    KELM
    CLL
ArXivPDFHTML
Abstract

Relevance modeling between queries and items stands as a pivotal component in commercial search engines, directly affecting the user experience. Given the remarkable achievements of large language models (LLMs) in various natural language processing (NLP) tasks, LLM-based relevance modeling is gradually being adopted within industrial search systems. Nevertheless, foundational LLMs lack domain-specific knowledge and do not fully exploit the potential of in-context learning. Furthermore, structured item text remains underutilized, and there is a shortage in the supply of corresponding queries and background knowledge. We thereby propose CPRM (Continual Pre-training for Relevance Modeling), a framework designed for the continual pre-training of LLMs to address these issues. Our CPRM framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, a novel approach where LLMs are pre-trained on a sequence of related queries or items, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information (e.g., generating summaries and corresponding queries) to further strengthen LLMs. Results on offline experiments and online A/B testing demonstrate that our model achieves convincing performance compared to strong baselines.

View on arXiv
@article{wu2025_2412.01269,
  title={ CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search },
  author={ Kaixin Wu and Yixin Ji and Zeyuan Chen and Qiang Wang and Cunxiang Wang and Hong Liu and Baijun Ji and Jia Xu and Zhongyi Liu and Jinjie Gu and Yuan Zhou and Linjian Mo },
  journal={arXiv preprint arXiv:2412.01269},
  year={ 2025 }
}
Comments on this paper