QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm

The attention operator remains a critical performance bottleneck in large language models (LLMs), particularly for long-context scenarios. While FlashAttention is the most widely used and effective GPU-aware acceleration algorithm, it must require time-consuming and hardware-specific manual implementation, limiting adaptability across GPU architectures. Existing LLMs have shown a lot of promise in code generation tasks, but struggle to generate high-performance attention code. The key challenge is it cannot comprehend the complex data flow and computation process of the attention operator and utilize low-level primitive to exploit GPU performance.To address the above challenge, we propose an LLM-friendly Thinking Language (LLM-TL) to help LLMs decouple the generation of high-level optimization logic and low-level implementation on GPU, and enhance LLMs' understanding of attention operator. Along with a 2-stage reasoning workflow, TL-Code generation and translation, the LLMs can automatically generate FlashAttention implementation on diverse GPUs, establishing a self-optimizing paradigm for generating high-performance attention operators in attention-centric algorithms. Verified on A100, RTX8000, and T4 GPUs, the performance of our methods significantly outshines that of vanilla LLMs, achieving a speed-up of up to 35.16x. Besides, our method not only surpasses human-optimized libraries (cuDNN and official library) in most scenarios but also extends support to unsupported hardware and data types, reducing development time from months to minutes compared with human experts.

@article{zhou2025_2506.12355,
  title={ QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm },
  author={ Qirui Zhou and Shaohui Peng and Weiqiang Xiong and Haixin Chen and Yuanbo Wen and Haochen Li and Ling Li and Qi Guo and Yongwei Zhao and Ke Gao and Ruizhi Chen and Yanjun Wu and Chen Zhao and Yunji Chen },
  journal={arXiv preprint arXiv:2506.12355},
  year={ 2025 }
}