Discrete Diffusion for Text Infilling

Generative modeling of text is a fundamental challenge in natural language processing. While autoregressive models have achieved remarkable success, they face limitations in parallelizability and flexible control. Discrete diffusion models offer a promising alternative paradigm, leveraging iterative refinement and potentially enabling bidirectional context use, parallel generation, and flexible prompting. However, existing discrete text diffusion models typically assume fixed token positions, hindering their application to tasks requiring dynamic sequence lengths, such as unconstrained text infilling where ground-truth positional information is absent. vspace{baselineskip} This thesis introduces textbf{D}iscrete textbf{D}iffusion with textbf{O}ptimal textbf{T}ransport Position Coupling (DDOT) to overcome this critical limitation. DDOT is presented as the first discrete diffusion framework capable of handling flexible-length text infilling. At its core, DDOT employs a novel diffusion process that jointly models discrete token identities and continuous token positions. To maintain sequence coherence during the iterative generation process, a sample-level optimal transport (OT) coupling is integrated, ensuring consistent relative ordering of tokens. vspace{baselineskip} The methodology developed in this thesis is designed to be compatible with various underlying discrete diffusion techniques and pretrained denoising models. Comprehensive experimental validation on challenging constrained text generation benchmarks demonstrates DDOT's effectiveness. Results show that DDOT achieves performance competitive with state-of-the-art non-autoregressive methods, nears the quality of autoregressive models, and provides significant gains in training efficiency and flexibility for position-aware generation tasks. This research thus advances the capabilities of discrete diffusion models for complex text generation scenarios.

Name thesis:degree_name

Master of Science

Level thesis:degree_level

masters

Discipline thesis:degree_discipline

Computer Science & Applications

Department dc:contributor.department

Computer Science and#38; Applications

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2025

Author dc:creator

• Zhang, Andrew Xinghua

Chair dc:contributor.committeechair

• Thomas, Christopher Lee

Committee members dc:contributor.committeemember

• Wang, Xuan
• Yanardag Delul, Pinar

• Discrete Diffusion
• Text Modeling
• Text Infilling
• Masked Diffusion

Statement dc:rights

• Creative Commons Attribution 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by/4.0/

Language dc:language.iso

en

Dc Identifier Other

vt_gsexam:44258

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/135961