Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

QuantumGPTMini: A Hybrid Quantum-Classical Transformer for Enhanced NLP

Ramazan Amire, Amire Ramazan

Published Apr 28, 2025• 0 citations
Publisher

Abstract

Transformer architectures have driven advances in natural language processing, powering systems from question answering to creative text generation. Yet as model sizes have grown from hundreds of millions to hundreds of billions of parameters, the computational cost of these systems has become prohibitive, placing advanced language technology out of reach for many researchers. In this work, we introduce QuantumGPTMini, a hybrid quantum-classical transformer with just over one million classical parameters that leverages lightweight quantum circuits to perform its core attention and feed-forward computations. By encoding token embeddings both in conventional vectors and in the amplitudes of small variational quantum circuits, our model replaces the conventional softmax attention mechanism with an entanglement-driven alternative. When trained on the Wikipedia-2 corpus using PennyLane's lightning.qubit simulator on NVIDIA A100 GPUs, QuantumGPTMini matched the perplexity of a GPT-2 Small baseline while demonstrating superior pattern recognition and symbolic reasoning behaviors. We have made this system publicly accessible through a FastAPI endpoint and a Next.js web interface. While our initial results underscore the potential of quantum-augmented language models, they also highlight challenges in moving beyond simulation, including circuit depth limitations, decoherence, and the high overhead of parameter-shift gradient estimation.

DOI
10.22541/au.174587094.49802997/v1

Related papers

Quantum-Enhanced Attention Mechanism in NLP: A Hybrid Classical-Quantum Approach

S. M. Yousuf Iqbal Tomal, Abdullah Al Shafin, Bhattacharjee, Debojit +2 more

Recent advances in quantum computing have opened new pathways for enhancing deep learning architectures, particularly in domains characterized by high-dimensional and context-rich data such as natural language processing (NLP). In this work, we present a hybrid classical-quantum Transformer model that integrates a quantum-enhanced attention mechanism into the standard classical architecture. By embedding token representations into a quantum Hilbert space via parameterized variational circuits and exploiting entanglement-aware kernel similarities, the model captures complex semantic relationships beyond the reach of conventional dot-product attention. We demonstrate the effectiveness of this approach across diverse NLP benchmarks, showing improvements in both efficiency and representational capacity. The results section reveal that the quantum attention layer yields globally coherent attention maps and more separable latent features, while requiring comparatively fewer parameters than classical counterparts. These findings highlight the potential of quantum-classical hybrid models to serve as a powerful and resource-efficient alternative to existing attention mechanisms in NLP.

arXiv:2501.15630•Jan 26, 2025•14 citations

GroverAttention: Quantum-Enhanced Attention Mechanism for Effcient Transformers

Belkacem Chikhaoui

Abstract Transformer-based models have revolutionized natural language processing (NLP), but their computational complexity remains a bottleneck. This paper proposes a novel approach to integrating Quantum Attention into Transformers, leveraging Grover’s algorithm to optimize query-key matching. By replacing classical attention mechanisms with quantum-enhanced token selection, we achieve reduced computational cost and improved inference speed. Experimental results on real-world datasets demonstrate the effectiveness of our hybrid Quantum-Classical Transformer, showing an improvement in accuracy with reduced training overhead. Our findings suggest that quantum-assisted attention has the potential to significantly optimize deep learning architectures for NLP applications.

•Oct 31, 2025•0 citations

Quantum Natural Language Processing Using Hybrid AI Algorithms: A Variational Quantum–Classical Framework for Efficient Text Classification

P. Anitha

Quantum Natural Language Processing (QNLP) represents a novel interdisciplinaryparadigm that integrates Quantum Computing with Natural Language Processing toenhance semantic modeling and computational efficiency. Despite the remarkable successof transformer-based architectures such as BERT and GPT, classical NLP systems requirelarge-scale computational resources and extensive parameter tuning. This paper proposesa hybrid quantum–classical architecture employing parameterized quantum circuits (PQCs)integrated with classical embedding layers for text classification tasks. The frameworkutilizes classical preprocessing, quantum state encoding, variational optimization, andclassical post-measurement classification. Experimental results on benchmark sentimentanalysis datasets demonstrate competitive accuracy with significantly reduced parametercomplexity. The findings indicate that hybrid QNLP models provide enhanced semanticcompositionality and efficient representation in Hilbert space while remaining feasiblewithin Noisy Intermediate-Scale Quantum (NISQ) constraints

International Journal For Multidisciplinary Research•Mar 4, 2026•0 citations

A Hybrid Quantum-Classical Language Model for Text Generation: A Case Study on The Gift of the Magi

Aadrian Routh, Juhi M Singh

This paper presents a hybrid quantum-classical language model for text generation, evaluated through a detailed case study of O. Henry's classic short story, The Gift of the Magi. The model integrates a classical transformer with a quantum generator to enhance the extraction of abstract narrative features. Our system architecture includes stages for input tokenization, contextual embedding through multi-head attention, quantumenhanced feature extraction using a parameterized quantum circuit, and output refinement through a classical discriminator. Quantitative evaluations using the BLEU score, besides qualitative analysis, reveal that while the hybrid approach generates syntactically coherent text, it currently falls short in capturing the nuanced symbolic and emotional depth of the original narrative. The findings show the present limitations of quantum-enhanced language models and point towards future research directions in refining quantum circuit designs, reducing noise, and improving integration strategies to fully exploit the potential of quantum computing in natural language processing.

International Conference on Multidisciplinary Research and Innovations in Engineering (MRIE)•Jul 30, 2025•1 citation

A Hybrid Transformer Architecture with a Quantized Self-Attention Mechanism Applied to Molecular Generation

Anthony M. Smaldone, Yu Shee, Gregory W. Kyro +4 more

The success of the self-attention mechanism in classical machine learning models has inspired the development of quantum analogs aimed at reducing the computational overhead. Self-attention integrates learnable query and key matrices to calculate attention scores between all pairs of tokens in a sequence. These scores are then multiplied by a learnable value matrix to obtain the output self-attention matrix, enabling the model to effectively capture long-range dependencies within the input sequence. Here, we propose a hybrid quantum-classical self-attention mechanism as part of a transformer decoder, the architecture underlying large language models (LLMs). To demonstrate its utility in chemistry, we train this model on the QM9 dataset for conditional generation, using SMILES strings as input, each labeled with a set of physicochemical properties that serve as conditions during inference. Our theoretical analysis shows that the time complexity of the query-key dot product is reduced from O ( n 2 d ) in a classical model to O ( n 2 log ⁡ d ) in our quantum model, where n and d represent the sequence length and the embedding dimension, respectively. We perform simulations using NVIDIA’s CUDA-Q platform, which is designed for efficient GPU scalability. This work provides a promising avenue for quantum-enhanced natural language processing (NLP).

arXiv:2502.19214•Journal of Chemical Theory and Computation•May 7, 2025•22 citations

Hybrid Quantum-Classical Transfer Learning for Parameter-Efficient Code Generation

Su-Chang Lim, Jong-Chan Kim

The remarkable success of large language models (LLMs) in code generation has transformed software development, with systems like Codex, CodeGen, and StarCoder demonstrating strong capabilities in synthesizing functional code from natural language descriptions. However, this progress comes at a steep computational cost. Modern code generation models require hundreds of millions to billions of parameters—GPT-3 contains 175 billion parameters, CodeGen-16B uses 16 billion, and the trend continues toward ever-larger models. This scale creates fundamental challenges in training cost, inference latency, memory consumption, and environmental impact. We present a quantum transfer learning framework as an alternative approach to classical model compression for code generation. By replacing selected MLP layers with variational quantum circuits (VQCs), we exploit the exponential representational capacity of quantum systems to achieve $\Theta (D \log D)$ parameter scaling instead of classical $\Theta (D^{2})$ . Our hybrid classical-quantum architecture balances three competing objectives: leveraging existing pre-trained knowledge, achieving parameter compression through quantum representation, and maintaining compatibility with near-term quantum hardware constraints. We demonstrate our approach on CodeGen-350M, a production-scale code generation model, adopting a bottleneck architecture that substantially reduces parameters compared to the original MLP layer. Through a fair comparison against a classical bottleneck baseline with near-identical parameter count, we show that the VQC achieves superior representational quality—lower perplexity and higher code generation accuracy—validating the quantum layer’s expressivity advantage in the compressed latent space. Our design employs data re-uploading, where classical input features are re-encoded at each variational layer via trainable RY gates, enhancing the circuit’s expressive power on near-term devices.

IEEE Access•Jan 1, 2026•0 citations
Transformer architectures have driven advances in natural language processing, powering systems from question answering to creative text generation. Yet as model sizes have grown from hundreds of millions to hundreds of billions of parameters, the computational cost of these systems has become prohibitive, placing advanced language technology out of reach for many researchers. In this work, we introduce QuantumGPTMini, a hybrid quantum-classical transformer with just over one million classical parameters that leverages lightweight quantum circuits to perform its core attention and feed-forward computations. By encoding token embeddings both in conventional vectors and in the amplitudes of small variational quantum circuits, our model replaces the conventional softmax attention mechanism with an entanglement-driven alternative. When trained on the Wikipedia-2 corpus using PennyLane's lightning.qubit simulator on NVIDIA A100 GPUs, QuantumGPTMini matched the perplexity of a GPT-2 Small baseline while demonstrating superior pattern recognition and symbolic reasoning behaviors. We have made this system publicly accessible through a FastAPI endpoint and a Next.js web interface. While our initial results underscore the potential of quantum-augmented language models, they also highlight challenges in moving beyond simulation, including circuit depth limitations, decoherence, and the high overhead of parameter-shift gradient estimation.
Recent advances in quantum computing have opened new pathways for enhancing deep learning architectures, particularly in domains characterized by high-dimensional and context-rich data such as natural language processing (NLP). In this work, we present a hybrid classical-quantum Transformer model that integrates a quantum-enhanced attention mechanism into the standard classical architecture. By embedding token representations into a quantum Hilbert space via parameterized variational circuits and exploiting entanglement-aware kernel similarities, the model captures complex semantic relationships beyond the reach of conventional dot-product attention. We demonstrate the effectiveness of this approach across diverse NLP benchmarks, showing improvements in both efficiency and representational capacity. The results section reveal that the quantum attention layer yields globally coherent attention maps and more separable latent features, while requiring comparatively fewer parameters than classical counterparts. These findings highlight the potential of quantum-classical hybrid models to serve as a powerful and resource-efficient alternative to existing attention mechanisms in NLP.
Abstract Transformer-based models have revolutionized natural language processing (NLP), but their computational complexity remains a bottleneck. This paper proposes a novel approach to integrating Quantum Attention into Transformers, leveraging Grover’s algorithm to optimize query-key matching. By replacing classical attention mechanisms with quantum-enhanced token selection, we achieve reduced computational cost and improved inference speed. Experimental results on real-world datasets demonstrate the effectiveness of our hybrid Quantum-Classical Transformer, showing an improvement in accuracy with reduced training overhead. Our findings suggest that quantum-assisted attention has the potential to significantly optimize deep learning architectures for NLP applications.
Quantum Natural Language Processing (QNLP) represents a novel interdisciplinaryparadigm that integrates Quantum Computing with Natural Language Processing toenhance semantic modeling and computational efficiency. Despite the remarkable successof transformer-based architectures such as BERT and GPT, classical NLP systems requirelarge-scale computational resources and extensive parameter tuning. This paper proposesa hybrid quantum–classical architecture employing parameterized quantum circuits (PQCs)integrated with classical embedding layers for text classification tasks. The frameworkutilizes classical preprocessing, quantum state encoding, variational optimization, andclassical post-measurement classification. Experimental results on benchmark sentimentanalysis datasets demonstrate competitive accuracy with significantly reduced parametercomplexity. The findings indicate that hybrid QNLP models provide enhanced semanticcompositionality and efficient representation in Hilbert space while remaining feasiblewithin Noisy Intermediate-Scale Quantum (NISQ) constraints
This paper presents a hybrid quantum-classical language model for text generation, evaluated through a detailed case study of O. Henry's classic short story, The Gift of the Magi. The model integrates a classical transformer with a quantum generator to enhance the extraction of abstract narrative features. Our system architecture includes stages for input tokenization, contextual embedding through multi-head attention, quantumenhanced feature extraction using a parameterized quantum circuit, and output refinement through a classical discriminator. Quantitative evaluations using the BLEU score, besides qualitative analysis, reveal that while the hybrid approach generates syntactically coherent text, it currently falls short in capturing the nuanced symbolic and emotional depth of the original narrative. The findings show the present limitations of quantum-enhanced language models and point towards future research directions in refining quantum circuit designs, reducing noise, and improving integration strategies to fully exploit the potential of quantum computing in natural language processing.
The success of the self-attention mechanism in classical machine learning models has inspired the development of quantum analogs aimed at reducing the computational overhead. Self-attention integrates learnable query and key matrices to calculate attention scores between all pairs of tokens in a sequence. These scores are then multiplied by a learnable value matrix to obtain the output self-attention matrix, enabling the model to effectively capture long-range dependencies within the input sequence. Here, we propose a hybrid quantum-classical self-attention mechanism as part of a transformer decoder, the architecture underlying large language models (LLMs). To demonstrate its utility in chemistry, we train this model on the QM9 dataset for conditional generation, using SMILES strings as input, each labeled with a set of physicochemical properties that serve as conditions during inference. Our theoretical analysis shows that the time complexity of the query-key dot product is reduced from O ( n 2 d ) in a classical model to O ( n 2 log ⁡ d ) in our quantum model, where n and d represent the sequence length and the embedding dimension, respectively. We perform simulations using NVIDIA’s CUDA-Q platform, which is designed for efficient GPU scalability. This work provides a promising avenue for quantum-enhanced natural language processing (NLP).
The remarkable success of large language models (LLMs) in code generation has transformed software development, with systems like Codex, CodeGen, and StarCoder demonstrating strong capabilities in synthesizing functional code from natural language descriptions. However, this progress comes at a steep computational cost. Modern code generation models require hundreds of millions to billions of parameters—GPT-3 contains 175 billion parameters, CodeGen-16B uses 16 billion, and the trend continues toward ever-larger models. This scale creates fundamental challenges in training cost, inference latency, memory consumption, and environmental impact. We present a quantum transfer learning framework as an alternative approach to classical model compression for code generation. By replacing selected MLP layers with variational quantum circuits (VQCs), we exploit the exponential representational capacity of quantum systems to achieve Θ(Dlog⁡D)\Theta (D \log D)Θ(DlogD) parameter scaling instead of classical Θ(D2)\Theta (D^{2})Θ(D2) . Our hybrid classical-quantum architecture balances three competing objectives: leveraging existing pre-trained knowledge, achieving parameter compression through quantum representation, and maintaining compatibility with near-term quantum hardware constraints. We demonstrate our approach on CodeGen-350M, a production-scale code generation model, adopting a bottleneck architecture that substantially reduces parameters compared to the original MLP layer. Through a fair comparison against a classical bottleneck baseline with near-identical parameter count, we show that the VQC achieves superior representational quality—lower perplexity and higher code generation accuracy—validating the quantum layer’s expressivity advantage in the compressed latent space. Our design employs data re-uploading, where classical input features are re-encoded at each variational layer via trainable RY gates, enhancing the circuit’s expressive power on near-term devices.