ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 2506.00726
30
0

Structured Gradient Guidance for Few-Shot Adaptation in Large Language Models

31 May 2025
Hongye Zheng
Yichen Wang
Ray Pan
Guiran Liu
Binrong Zhu
Hanlu Zhang
ArXiv (abs)PDFHTML
Main:4 Pages
5 Figures
Bibliography:1 Pages
1 Tables
Abstract

This paper presents a gradient-informed fine-tuning method for large language models under few-shot conditions. The goal is to enhance task adaptability and training stability when data is limited. The method builds on a base loss function and introduces two gradient-related regularization terms. The first enforces gradient direction consistency to guide parameter updates along task-relevant directions and prevent drift. The second controls gradient magnitude to avoid abnormal updates. Together, these components support a more efficient and stable optimization path. To further improve cross-task generalization, the method incorporates a gradient alignment mechanism. This mechanism measures the consistency between optimization directions of the source and target tasks. It enhances fine-tuning performance in multi-task and cross-domain scenarios. Across various natural language understanding tasks, the method outperforms existing fine-tuning strategies in average accuracy, gradient stability, and directional alignment. Empirical evaluations under different sample sizes and domain-specific tasks confirm the method's robustness and broad applicability in low-resource environments. In particular, the method shows clear advantages in controlling parameter update paths. The results demonstrate that a gradient-based fine-tuning framework can effectively leverage the representational power of large language models. It ensures training stability while reducing dependence on large volumes of labeled data.

View on arXiv
@article{zheng2025_2506.00726,
  title={ Structured Gradient Guidance for Few-Shot Adaptation in Large Language Models },
  author={ Hongye Zheng and Yichen Wang and Ray Pan and Guiran Liu and Binrong Zhu and Hanlu Zhang },
  journal={arXiv preprint arXiv:2506.00726},
  year={ 2025 }
}
Comments on this paper