Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2507.13354

Physical models realizing the transformer architecture of large language models

Chen, Zeqian

Published May 21, 2025• 0 citations
arXivHTMLPDF

Abstract

The introduction of the transformer architecture in 2017 marked the most striking advancement in natural language processing. The transformer is a model architecture relying entirely on an attention mechanism to draw global dependencies between input and output. However, we believe there is a gap in our theoretical understanding of what the transformer is, and how it works physically. From a physical perspective on modern chips, such as those chips under 28nm, modern intelligent machines should be regarded as open quantum systems beyond conventional statistical systems. Thereby, in this paper, we construct physical models realizing large language models based on a transformer architecture as open quantum systems in the Fock space over the Hilbert space of tokens. Our physical models underlie the transformer architecture for large language models.

DOI
10.48550/arxiv.2507.13354

Related papers

Mathematical Formalism and Physical Models for Generative Artificial Intelligence

Zeqian Chen

This paper presents a mathematical formalism for generative artificial intelligence (GAI). Our starting point is an observation that a “histories” approach to physical systems agrees with the compositional nature of deep neural networks. Mathematically, we define a GAI system as a family of sequential joint probabilities associated with input texts and temporal sequences of tokens (as physical event histories). From a physical perspective on modern chips, we then construct physical models realizing GAI systems as open quantum systems. Finally, as an illustration, we construct physical models realizing large language models based on a transformer architecture as open quantum systems in the Fock space over the Hilbert space of tokens. Our physical models underlie the transformer architecture for large language models.

Foundations•Jun 24, 2025•0 citations

Quantum-Aware Large Language Models: Taxonomy, Architectures, and Future Applications

Reyhan Dede

The convergence of quantum computing and large language models (LLMs) represents a major milestone in computational intelligence, introducing novel paradigms for information processing, complexity modeling, and scientific advancement. This survey provides a comprehensive overview of the emerging interdisciplinary field that integrates quantum mechanics with natural language processing. We begin by presenting the foundational principles of quantum computing and LLMs, then examine their integration through automated quantum code generation, hybrid transformer architectures, and quantum natural language processing (QNLP) frameworks. State-of-the-art models, such as quantum-augmented attention mechanisms and circuit embedding strategies, are evaluated across domain-specific datasets and benchmarks. We also investigate the role of LLMs as quantum research assistants, educational tools, and simulation agents, particularly in resource-constrained environments. Furthermore, the survey explores challenges related to scalability, interpretability, and quantum data encoding, and outlines future research directions-especially the vision for fully quantum-native LLMs. Our goal is to guide researchers through the current landscape and to encourage continued exploration at the intersection of quantum technologies and advanced language models.

International Conference on Electrical, Communication, and Computer Engineering (ICECCE)•Aug 27, 2025•0 citations

Quantum-Inspired Transformers for Scalable Natural Language Processing Applications in Future AI

Priya Vij, Parmanand Yadav

Quantum-Inspired Transformers are a new class of approaches for natural language processing (NLP) that use principles inspired by physical quantum mechanics to encode and process linguistic information. Paper aims to improve both scalability and efficiency for large-scale NLP tasks in the next generation of artificial intelligence (AI). However, traditional transformers can struggle to encode complex semantic relationships across long sequences and can require high levels of compute resources, hence limiting scalability for large datasets and multilingual tasks. In addition, traditional embeddings do not represent multiple contextual meanings simultaneously, which decreases model expressiveness. To address these challenges and maximize the exploration of potential quantum effects in AI, this paper presents a Quantum State Embedding Attention (QSEA) framework that encodes tokens as quantum inspired states with complex-valued amplitudes. Attention, which is based on entangled states, is meant to efficiently capture long-range dependencies while reducing compute and increasing scalability, all without losing a rich understanding of context. Whereas multilingual translation needs to encode both the source similarity and the target language within a smaller semantic representation space. Our experimentation shows that QSEA consistently provides increased accuracy in decoding, measures attention across long sequences, and yields enduring scalability over large multilingual datasets. The evidence demonstrates the potential for QSEA to be the new layer in the future of AI systems, a layer that is scalable and a highperforming NLP application.

Procedia Computer Science•Jan 1, 2026•1 citation

Quantum Transformer: Accelerating model inference via quantum linear algebra

Naixu Guo, Yu, Zhan, Choi, Matthew +5 more

Powerful generative artificial intelligence from large language models (LLMs) harnesses extensive computational resources for inference. In this work, we investigate the transformer architecture, a key component of these models, under the lens of fault-tolerant quantum computing. We develop quantum subroutines to construct the building blocks in the transformer, including the self-attention, residual connection with layer normalization, and feed-forward network. As an important subroutine, we show how to efficiently implement the Hadamard product and element-wise functions of matrices on quantum computers. Our algorithm prepares an amplitude encoding of the transformer output, which can be measured for prediction or use in the next layer. We find that the matrix norm of the input sequence plays a dominant role in the quantum complexity. With numerical experiments on open-source LLMs, including for bio-informatics applications, we demonstrate the potential of a quantum speedup for transformer inference in practical regimes.

arXiv:2402.16714•Feb 26, 2024•52 citations

Quantum-Enhanced Attention Mechanism in NLP: A Hybrid Classical-Quantum Approach

S. M. Yousuf Iqbal Tomal, Abdullah Al Shafin, Bhattacharjee, Debojit +2 more

Recent advances in quantum computing have opened new pathways for enhancing deep learning architectures, particularly in domains characterized by high-dimensional and context-rich data such as natural language processing (NLP). In this work, we present a hybrid classical-quantum Transformer model that integrates a quantum-enhanced attention mechanism into the standard classical architecture. By embedding token representations into a quantum Hilbert space via parameterized variational circuits and exploiting entanglement-aware kernel similarities, the model captures complex semantic relationships beyond the reach of conventional dot-product attention. We demonstrate the effectiveness of this approach across diverse NLP benchmarks, showing improvements in both efficiency and representational capacity. The results section reveal that the quantum attention layer yields globally coherent attention maps and more separable latent features, while requiring comparatively fewer parameters than classical counterparts. These findings highlight the potential of quantum-classical hybrid models to serve as a powerful and resource-efficient alternative to existing attention mechanisms in NLP.

arXiv:2501.15630•Jan 26, 2025•14 citations

Quantum-Inspired Self-Attention in a Large Language Model

Nikita Kuznetsov, Niyaz Ismagilov, Ernesto Campos

Recent advances in Natural Language Processing have been predominantly driven by transformer-based architectures, which rely heavily on self-attention mechanisms to model relationships between tokens in a sequence. Similarly, the field of Quantum Natural Language Processing, which seeks to leverage quantum principles to address challenges in language understanding and generation tasks, has seen the recent development of quantum self-attention mechanisms. We propose a classical quantum-inspired self-attention (QISA) mechanism and integrate it into the full autoregressive language modeling pipeline of GPT-1. To the best of our knowledge, this is the first integration of this kind, as previous quantum self-attention mechanisms have been primarily tested on text classification. In our experiments, QISA achieves better performance when compared to standard self-attention on the metrics character error rate ($15.5\times$ better), word error rate ($4.7 \times $) and cross-entropy loss ($13 \times$). This is achieved while only requiring a $ 2.6\times$ longer inference time.

cs.CLCross-listedarXiv:2603.03318•Feb 10, 2026•2 citations
The introduction of the transformer architecture in 2017 marked the most striking advancement in natural language processing. The transformer is a model architecture relying entirely on an attention mechanism to draw global dependencies between input and output. However, we believe there is a gap in our theoretical understanding of what the transformer is, and how it works physically. From a physical perspective on modern chips, such as those chips under 28nm, modern intelligent machines should be regarded as open quantum systems beyond conventional statistical systems. Thereby, in this paper, we construct physical models realizing large language models based on a transformer architecture as open quantum systems in the Fock space over the Hilbert space of tokens. Our physical models underlie the transformer architecture for large language models.
This paper presents a mathematical formalism for generative artificial intelligence (GAI). Our starting point is an observation that a “histories” approach to physical systems agrees with the compositional nature of deep neural networks. Mathematically, we define a GAI system as a family of sequential joint probabilities associated with input texts and temporal sequences of tokens (as physical event histories). From a physical perspective on modern chips, we then construct physical models realizing GAI systems as open quantum systems. Finally, as an illustration, we construct physical models realizing large language models based on a transformer architecture as open quantum systems in the Fock space over the Hilbert space of tokens. Our physical models underlie the transformer architecture for large language models.
The convergence of quantum computing and large language models (LLMs) represents a major milestone in computational intelligence, introducing novel paradigms for information processing, complexity modeling, and scientific advancement. This survey provides a comprehensive overview of the emerging interdisciplinary field that integrates quantum mechanics with natural language processing. We begin by presenting the foundational principles of quantum computing and LLMs, then examine their integration through automated quantum code generation, hybrid transformer architectures, and quantum natural language processing (QNLP) frameworks. State-of-the-art models, such as quantum-augmented attention mechanisms and circuit embedding strategies, are evaluated across domain-specific datasets and benchmarks. We also investigate the role of LLMs as quantum research assistants, educational tools, and simulation agents, particularly in resource-constrained environments. Furthermore, the survey explores challenges related to scalability, interpretability, and quantum data encoding, and outlines future research directions-especially the vision for fully quantum-native LLMs. Our goal is to guide researchers through the current landscape and to encourage continued exploration at the intersection of quantum technologies and advanced language models.
Quantum-Inspired Transformers are a new class of approaches for natural language processing (NLP) that use principles inspired by physical quantum mechanics to encode and process linguistic information. Paper aims to improve both scalability and efficiency for large-scale NLP tasks in the next generation of artificial intelligence (AI). However, traditional transformers can struggle to encode complex semantic relationships across long sequences and can require high levels of compute resources, hence limiting scalability for large datasets and multilingual tasks. In addition, traditional embeddings do not represent multiple contextual meanings simultaneously, which decreases model expressiveness. To address these challenges and maximize the exploration of potential quantum effects in AI, this paper presents a Quantum State Embedding Attention (QSEA) framework that encodes tokens as quantum inspired states with complex-valued amplitudes. Attention, which is based on entangled states, is meant to efficiently capture long-range dependencies while reducing compute and increasing scalability, all without losing a rich understanding of context. Whereas multilingual translation needs to encode both the source similarity and the target language within a smaller semantic representation space. Our experimentation shows that QSEA consistently provides increased accuracy in decoding, measures attention across long sequences, and yields enduring scalability over large multilingual datasets. The evidence demonstrates the potential for QSEA to be the new layer in the future of AI systems, a layer that is scalable and a highperforming NLP application.
Powerful generative artificial intelligence from large language models (LLMs) harnesses extensive computational resources for inference. In this work, we investigate the transformer architecture, a key component of these models, under the lens of fault-tolerant quantum computing. We develop quantum subroutines to construct the building blocks in the transformer, including the self-attention, residual connection with layer normalization, and feed-forward network. As an important subroutine, we show how to efficiently implement the Hadamard product and element-wise functions of matrices on quantum computers. Our algorithm prepares an amplitude encoding of the transformer output, which can be measured for prediction or use in the next layer. We find that the matrix norm of the input sequence plays a dominant role in the quantum complexity. With numerical experiments on open-source LLMs, including for bio-informatics applications, we demonstrate the potential of a quantum speedup for transformer inference in practical regimes.
Recent advances in quantum computing have opened new pathways for enhancing deep learning architectures, particularly in domains characterized by high-dimensional and context-rich data such as natural language processing (NLP). In this work, we present a hybrid classical-quantum Transformer model that integrates a quantum-enhanced attention mechanism into the standard classical architecture. By embedding token representations into a quantum Hilbert space via parameterized variational circuits and exploiting entanglement-aware kernel similarities, the model captures complex semantic relationships beyond the reach of conventional dot-product attention. We demonstrate the effectiveness of this approach across diverse NLP benchmarks, showing improvements in both efficiency and representational capacity. The results section reveal that the quantum attention layer yields globally coherent attention maps and more separable latent features, while requiring comparatively fewer parameters than classical counterparts. These findings highlight the potential of quantum-classical hybrid models to serve as a powerful and resource-efficient alternative to existing attention mechanisms in NLP.
Recent advances in Natural Language Processing have been predominantly driven by transformer-based architectures, which rely heavily on self-attention mechanisms to model relationships between tokens in a sequence. Similarly, the field of Quantum Natural Language Processing, which seeks to leverage quantum principles to address challenges in language understanding and generation tasks, has seen the recent development of quantum self-attention mechanisms. We propose a classical quantum-inspired self-attention (QISA) mechanism and integrate it into the full autoregressive language modeling pipeline of GPT-1. To the best of our knowledge, this is the first integration of this kind, as previous quantum self-attention mechanisms have been primarily tested on text classification. In our experiments, QISA achieves better performance when compared to standard self-attention on the metrics character error rate (15.5×15.5\times15.5× better), word error rate ($4.7 \times $) and cross-entropy loss (13×13 \times13×). This is achieved while only requiring a $ 2.6\times$ longer inference time.