A Semi-Supervised Text Generation Framework Combining a Deep Transformer and a GAN

This paper introduces a framework that connects a deep generative pre-trained Transformer language model with a generative adversarial network for semi-supervised text generation. In other words, the proposed model is first pre-trained unsupervised on a large and diverse text corpus with 24 layers. Then a simple GAN architecture for synthetic text generation is introduced, and Gumbel-Softmax is applied to handle the discreteness of tokens. The paper also shows a semi-supervised approach where real data is augmented with GAN samples, which is further used to fine-tune the Transformer model on the merged dataset. Detailed theoretical derivations are also included, outlining the proof of the min-max objective function, and an extensive discussion of the Gumbel-Softmax reparameterization trick.

@article{wang2025_2502.05937,
  title={ A Semi-Supervised Text Generation Framework Combining a Deep Transformer and a GAN },
  author={ Shengquan Wang },
  journal={arXiv preprint arXiv:2502.05937},
  year={ 2025 }
}