arXiv:2609.18565
Variational Quantum Transformer Architecture for Synthetic Language Generation
Julian Hager, Michael Kölle, Gerhard Stenzel, Tobias Rohe, Jonas Stein, Claudia Linnhoff-Popien
Abstract
We propose a compact NISQ-compatible quantum transformer architecture for synthetic QNLP sequence modelling. The model preserves the autoregressive next-token interface of a classical transformer, but replaces attention and feed-forward sublayers with variational quantum encoder blocks, connector circuits, decoder blocks and a direct two-qubit measurement readout. Token contexts are angle-encoded into small quantum registers, processed by parallel variational heads and encoder integration circuits and conditioned through decoder ancillae to produce a distribution over a four-token vocabulary. We evaluate several architecture variants on deterministic and lexicographic grammar-generation tasks against a compact classical transformer baseline. The quantum models are trainable end-to-end and learn nontrivial grammar structure, including perfect deterministic generation in individual runs and high lexicographic validity in the strongest variant. The classical baseline remains more accurate and stable and the quantum models are sensitive to initialization. The contribution is therefore not a claim of quantum advantage, but a concrete architecture and evaluation of transformer-inspired QNLP sequence modelling under near-term quantum constraints.
AI Summary
Can a compact variational quantum circuit implement transformer-inspired autoregressive sequence modelling? This paper constructs an encoder–connector–decoder architecture that maps four-token contexts to next-token probabilities over $\{A,B,C,\texttt{space}\}$. Tokens are angle-encoded, parallel variational heads and integration circuits compress context into two-qubit states, and decoder circuits condition a direct two-qubit measurement readout. The architecture uses at most six active qubits and 124–364 trainable parameters; it does not implement scaled dot-product attention. Evaluation uses exact simulation on two synthetic grammars: the periodic cycle `AAA BBB CCC` and nondecreasing three-letter words. Models train on 25-word texts and generate 100 tokens greedily, with results averaged over three seeds. On the deterministic task, the highest mean quantum grammar score is $0.755\pm0.213$, with token accuracy $0.557\pm0.384$; individual runs achieve perfect continuation. On the lexicographic task, the strongest quantum variant reaches grammar score $0.828\pm0.299$ and token accuracy $0.380\pm0.156$. A 700-parameter classical transformer achieves perfect grammar scores on both tasks and deterministic token accuracy of 1. Quantum and classical training schedules differ substantially: 20 versus 800 epochs. The stated limitations include initialization-sensitive optimization, synthetic tasks that do not test semantics or natural-language generalization, and no shot-based or noisy-hardware evaluation. Lexicographic validity can reflect repetitive outputs rather than distribution learning, while the two-qubit readout restricts the vocabulary to four tokens. The results establish small-scale feasibility, not quantum advantage.
The authors used OpenAI ChatGPT for language editing.
Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.
- Comments
- Accepted for publication in the QNLPAI 2026 proceedings (Springer Lecture Notes in Computer Science, LNCS). 10 pages, including references and appendix, 2 figures