Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2609.31066

Modeling quantum neural network gradient with reinforcement learning

Nhan Trong Luu, Duong Trung Luu, Nam Ngoc Pham, Thang Cong Truong

quant-phcs.ETcs.NESubmitted Sep 25, 2026• Announced Sep 28, 2026• 1 citation
arXivHTMLPDF

Abstract

Training quantum neural networks (QNNs) on near-term hardware remains hampered by two compounding difficulties: the exponential vanishing of gradient variance known as the barren plateau, and the $\mathcal{O}(L \cdot 2^n)$ time and memory cost of differentiating through an $n$-qubit, $L$-layer circuit. We propose RLQ-Grad, a reinforcement-learning-based optimizer in which a classical policy $π_φ$ (a spectrally-normalized PPO agent) learns to propose parameter updates directly, conditioned on the QNN's current parameters, loss, accuracy, and previous update. Because the surrogate gradient is emitted by a classical network rather than obtained by differentiating through the unitary $U(θ)$, its variance is not constrained by the barren plateau concentration bound, and its cost scales with the number of trainable parameters rather than the Hilbert-space dimension. We prove these properties formally and verify them on a hardware-efficient ansatz across four supervised benchmarks with up to $n=20$ qubits. RLQ-Grad preserves a near-flat gradient-variance curve where backpropagation, parameter-shift, and adjoint differentiation decay by 1 to 2 orders of magnitude. Accounting for the full training pipeline (PPO rollouts, actor-critic updates, and optimizer states), RLQ-Grad needs under 2 MB of memory and runs $2490\times$, $7876\times$, and $673\times$ faster per iteration than these three methods at $n=20$. It improves top-1 accuracy by up to $+10\%$ over gradient-based baselines on circuits of up to 12 qubits, and matches dedicated barren plateau mitigation methods on CIFAR-10 at 14 to 20 qubits, where evolutionary and gradient-free optimizers collapse to chance.

AI Summary

The paper asks whether a learned classical optimizer can reduce differentiation costs and improve supervised quantum neural network (QNN) training when circuit gradients become small. RLQ-Grad uses a PPO actor–critic with spectral normalization to propose circuit-parameter updates from current parameters, minibatch loss and accuracy, and the previous update. Classical input/output layers are trained by backpropagation; agents are trained separately for each dataset and circuit configuration. Experiments use noiseless statevector simulations of hardware-efficient ansätze on Breast Cancer, MNIST, Fashion-MNIST, and CIFAR-10. Across 2–12 qubits, reported validation accuracies exceed differentiation-based baselines, reaching 95.27% on MNIST and 87.62% on Fashion-MNIST at 12 qubits. At 20 qubits and 600 circuit parameters, CIFAR-10 accuracy is $53.02\pm2.83\%$, versus $40.13\pm2.91\%$ for backpropagation and $53.16\pm3.65\%$ for layerwise training. Resource benchmarks report 1.95 MB for the agent pipeline and CPU iteration speedups of $2490\times$ over backpropagation and $7876\times$ over parameter-shift; the complexity comparison excludes common circuit-forward costs. The theoretical argument establishes that policy-generated updates need not obey quantum-gradient variance concentration bounds; it does not establish that nonvanishing updates are useful descent directions. The paper explicitly acknowledges that concentrated losses also suppress reward information, so RLQ-Grad cannot remove exponential shot requirements in fully developed barren plateaus. Other stated limitations include poor local minima, vanishing gradients in classical preprocessing, small-circuit agent overhead, and lack of hardware, finite-shot, or noisy-device tests. Adjoint differentiation remains more memory-efficient in the reported benchmarks.

Likely largely AI-generatedOverclaimsPossible errorUnclear noveltyMissing comparison

Central claim may not hold: The barren-plateau theorems do not establish useful descent directions or efficient policy learning, and the paper itself acknowledges that concentrated quantum rewards retain the fundamental training obstruction. The purported end-to-end computational advantage omits necessary circuit forward evaluations from its complexity accounting, so Hilbert-space-independent policy computation does not establish Hilbert-space-independent training cost. The stated CIFAR-10 PPO rollout requires approximately 281 MB for float32 observations and actions alone at 20 qubits, contradicting the rollout-inclusive full-pipeline memory claim below 2 MB. Adjoint differentiation retains statevector-sized working buffers, so a constant buffer count does not justify O(1) additional memory in qubit count. The different barren-plateau variance slopes reported for mathematically equivalent exact differentiation methods are unexplained and do not validate the claimed bypass.

Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.

Comments
NeurIPS 2026 Main Track (Poster). OpenReview: https://openreview.net/forum?id=lfrvu8xfsh

Related papers

Quantum Surrogate-Driven Image Classifier: a Gradient-Free Approach to Avoid Barren Plateaus

Yichen Xie

Training deep quantum neural networks (QNNs) for image classification is notoriously difficult due to vanishing gradients (barren plateaus) and limited nonlinearity in purely unitary circuits. We propose a novel gradient-free surrogate-driven framework combined with mid-circuit measurement and reset of ancillary qubits to induce effective nonunitarity. Our approach uses a classical neural surrogate to predict measurement outcomes from circuit parameters to avoid direct gradients. Theoretical results prove that bypassing quantum gradients mitigates plateau issues. Experiments on MNIST, CIFAR-10, and CIFAR-100 with 15-qubit, 6-layer circuits using four resettable ancillas demonstrate superior accuracy compared to direct-gradient QNNs and classical baselines. Our method also serves as a potential for a generalized training framework applicable to various QNN architectures beyond image classification.

arXiv:2505.05249•IEEE International Conference on Quantum Control, Computing and Learning (qCCL)•Jun 25, 2025•6 citations

Policy gradients using variational quantum circuits

André Sequeira, Luís Paulo Santos, Luís Soares Barbosa

Abstract Variational quantum circuits are being used as versatile quantum machine learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to reinforcement learning, less is known. In this work, we considered a variational quantum circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a reinforcement learning agent. We show that an 𝜖-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly to typical classical neural networks used in standard benchmarking environments and quantum control, using only a fraction of the parameters. Moreover, we study the barren plateau phenomenon in quantum policy gradients using the Fisher information matrix spectrum.

arXiv:2203.10591•Quantum Machine Intelligence•Apr 11, 2023•42 citations

Multi-Agent Quantum Reinforcement Learning Using Evolutionary Optimization

Michael Kölle, Felix Topp, Thomy Phan +3 more

Multi-Agent Reinforcement Learning is becoming increasingly more important in times of autonomous driving and other smart industrial applications. Simultaneously a promising new approach to Reinforcement Learning arises using the inherent properties of quantum mechanics, reducing the trainable parameters of a model significantly. However, gradient-based Multi-Agent Quantum Reinforcement Learning methods often have to struggle with barren plateaus, holding them back from matching the performance of classical approaches. While gradient free Quantum Reinforcement Learning methods may alleviate some of these challenges, they too are not immune to the difficulties posed by barren plateaus. We build upon an existing approach for gradient free Quantum Reinforcement Learning and propose three genetic variations with Variational Quantum Circuits for Multi-Agent Reinforcement Learning using evolutionary optimization. We evaluate our genetic variations in the Coin Game environment and also compare them to classical approaches. We showed that our Variational Quantum Circuit approaches perform significantly better compared to a neural network with a similar amount of trainable parameters. Compared to the larger neural network, our approaches archive similar results using $97.88\%$ less parameters.

arXiv:2311.05546•International Conference on Agents and Artificial Intelligence (ICAART)•Jan 1, 2024•15 citations

Training Hybrid Quantum Policies with Classical Surrogates for Actor-Critic Reinforcement Learning

Jeremy Kulcsar, Giulio Giaconi, Jie Luo +2 more

Reading out a parametrized quantum circuit through finite-shot measurements introduces a discrete stochastic component into hybrid quantum-classical neural networks, making gradient-based training inefficient. In this work, we introduce a surrogate-assisted actor-critic method for training hybrid quantum policies that avoids differentiating the quantum circuit during backpropagation. A classical neural surrogate trained on the circuit's input-output pairs from a replay buffer provides a differentiable substitute for actor updates and critic-target computation, while forward interaction and evaluation retain the quantum circuit's measurements. Our experiments on standard MuJoCo continuous-control tasks demonstrate promising policy learning. Notably, on Humanoid, more runs reach high returns than with learned classical baselines. Moreover, the best average return in our ablation comes from a smaller circuit / surrogate, suggesting that strong performance does not depend on a large network. We also derive conditional upstream gradient-error bounds that separate surrogate fitting from the mean-feature relaxation. Together, the results support surrogate-assisted training as an effective and efficient approach to learning hybrid quantum policies.

arXiv:2503.09119•Mar 12, 2025•1 citation

Breaking Through Barren Plateaus: Reinforcement Learning Initializations for Deep Variational Quantum Circuits

Yifeng Peng, Xinyi Li, Zhemin Zhang +3 more

Variational Quantum Algorithms (VQAs) have gained prominence as a viable framework for exploiting near-term quantum devices in applications ranging from optimization and chemistry simulation to machine learning. However, the effectiveness of VQAs is often constrained by the so-called barren plateau problem, wherein gradients diminish exponentially as system size or circuit depth increases, thereby hindering training. In this work, we propose a reinforcement learning (RL)-based initialization strategy to alleviate the barren plateau issue by reshaping the initial parameter landscape to avoid regions prone to vanishing gradients. In particular, we explore several RL algorithms (Deterministic Policy Gradient, Soft ActorCritic, and Proximal Policy Optimization, etc.) to generate the circuit parameters (treated as “actions”) that minimize the VQAs cost function before standard gradient-based optimization. By pre-training with RL in this manner, subsequent optimization using methods such as gradient descent or Adam proceeds from a more favorable initial state. Extensive numerical experiments under various noise conditions and tasks consistently demonstrate that the RL-based initialization method significantly enhances both convergence speed and final solution quality. Moreover, comparisons among different RL algorithms highlight that multiple approaches can achieve comparable performance gains, underscoring the flexibility and robustness of our method. These findings shed light on a promising avenue for integrating machine learning techniques into quantum algorithm design, offering insights into how RL-driven parameter initialization can accelerate the scalability and practical deployment of VQAs. Opening up a promising path for the research community in machine learning for quantum, especially barren plateau problems in VQAs.

IEEE International Conference on Quantum Computing and Engineering (QCE)•Aug 30, 2025•6 citations

Adaptive Grover-driven optimization for quantum-inspired deep learning: A gradient-free training framework

Irsa Sajjad, Huda M. Alshanbari, Mohammed M. A. Almazah +2 more

Training deep neural networks remains difficult due to vanishing gradients, non-convex loss surfaces, and hyperparameter sensitivity. These obstacles are compounded by quantum machine learning, where barren plateaus, circuit depth, and hardware noise restrict the applicability of gradient-based approaches. To overcome these drawbacks, this study presents adaptive Grover-driven parallel quantum optimization (AG-PQO), a hybrid, gradient-free scheme that leverages Grover's quadratic search speedup, along with adaptive loss-aware discretization and fidelity-based regularization. In contrast to more classical optimizers, such as Adam or evolutionary strategies (ES), which are either sensitive to the adequacy of the gradient update or exhibit poor scaling behavior, AG-PQO optimizes by performing Grover-accelerated candidate exploration across layers and reuses high-quality solutions in quantum memory caching. Testing indicates that AG-PQO yields higher accuracy, 2%–3% above Adam and ES, and faster convergence with less end-value loss than Adam, ES, and quantum feedforward-backpropagation (QFB). It is worth noting that AG-PQO remains stable at the simulated noise level of NISQ and has the potential to scale to near-term quantum processors.

AIMS Mathematics•Jan 1, 2025•4 citations
Training quantum neural networks (QNNs) on near-term hardware remains hampered by two compounding difficulties: the exponential vanishing of gradient variance known as the barren plateau, and the O(L⋅2n)\mathcal{O}(L \cdot 2^n)O(L⋅2n) time and memory cost of differentiating through an nnn-qubit, LLL-layer circuit. We propose RLQ-Grad, a reinforcement-learning-based optimizer in which a classical policy πφπ_φπφ​ (a spectrally-normalized PPO agent) learns to propose parameter updates directly, conditioned on the QNN's current parameters, loss, accuracy, and previous update. Because the surrogate gradient is emitted by a classical network rather than obtained by differentiating through the unitary U(θ)U(θ)U(θ), its variance is not constrained by the barren plateau concentration bound, and its cost scales with the number of trainable parameters rather than the Hilbert-space dimension. We prove these properties formally and verify them on a hardware-efficient ansatz across four supervised benchmarks with up to n=20n=20n=20 qubits. RLQ-Grad preserves a near-flat gradient-variance curve where backpropagation, parameter-shift, and adjoint differentiation decay by 1 to 2 orders of magnitude. Accounting for the full training pipeline (PPO rollouts, actor-critic updates, and optimizer states), RLQ-Grad needs under 2 MB of memory and runs 2490×2490\times2490×, 7876×7876\times7876×, and 673×673\times673× faster per iteration than these three methods at n=20n=20n=20. It improves top-1 accuracy by up to +10%+10\%+10% over gradient-based baselines on circuits of up to 12 qubits, and matches dedicated barren plateau mitigation methods on CIFAR-10 at 14 to 20 qubits, where evolutionary and gradient-free optimizers collapse to chance.
The paper asks whether a learned classical optimizer can reduce differentiation costs and improve supervised quantum neural network (QNN) training when circuit gradients become small. RLQ-Grad uses a PPO actor–critic with spectral normalization to propose circuit-parameter updates from current parameters, minibatch loss and accuracy, and the previous update. Classical input/output layers are trained by backpropagation; agents are trained separately for each dataset and circuit configuration.
Experiments use noiseless statevector simulations of hardware-efficient ansätze on Breast Cancer, MNIST, Fashion-MNIST, and CIFAR-10. Across 2–12 qubits, reported validation accuracies exceed differentiation-based baselines, reaching 95.27% on MNIST and 87.62% on Fashion-MNIST at 12 qubits. At 20 qubits and 600 circuit parameters, CIFAR-10 accuracy is 53.02±2.83%53.02\pm2.83\%53.02±2.83%, versus 40.13±2.91%40.13\pm2.91\%40.13±2.91% for backpropagation and 53.16±3.65%53.16\pm3.65\%53.16±3.65% for layerwise training. Resource benchmarks report 1.95 MB for the agent pipeline and CPU iteration speedups of 2490×2490\times2490× over backpropagation and 7876×7876\times7876× over parameter-shift; the complexity comparison excludes common circuit-forward costs.
The theoretical argument establishes that policy-generated updates need not obey quantum-gradient variance concentration bounds; it does not establish that nonvanishing updates are useful descent directions. The paper explicitly acknowledges that concentrated losses also suppress reward information, so RLQ-Grad cannot remove exponential shot requirements in fully developed barren plateaus. Other stated limitations include poor local minima, vanishing gradients in classical preprocessing, small-circuit agent overhead, and lack of hardware, finite-shot, or noisy-device tests. Adjoint differentiation remains more memory-efficient in the reported benchmarks.
Training deep quantum neural networks (QNNs) for image classification is notoriously difficult due to vanishing gradients (barren plateaus) and limited nonlinearity in purely unitary circuits. We propose a novel gradient-free surrogate-driven framework combined with mid-circuit measurement and reset of ancillary qubits to induce effective nonunitarity. Our approach uses a classical neural surrogate to predict measurement outcomes from circuit parameters to avoid direct gradients. Theoretical results prove that bypassing quantum gradients mitigates plateau issues. Experiments on MNIST, CIFAR-10, and CIFAR-100 with 15-qubit, 6-layer circuits using four resettable ancillas demonstrate superior accuracy compared to direct-gradient QNNs and classical baselines. Our method also serves as a potential for a generalized training framework applicable to various QNN architectures beyond image classification.
Abstract Variational quantum circuits are being used as versatile quantum machine learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to reinforcement learning, less is known. In this work, we considered a variational quantum circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a reinforcement learning agent. We show that an 𝜖-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly to typical classical neural networks used in standard benchmarking environments and quantum control, using only a fraction of the parameters. Moreover, we study the barren plateau phenomenon in quantum policy gradients using the Fisher information matrix spectrum.
Multi-Agent Reinforcement Learning is becoming increasingly more important in times of autonomous driving and other smart industrial applications. Simultaneously a promising new approach to Reinforcement Learning arises using the inherent properties of quantum mechanics, reducing the trainable parameters of a model significantly. However, gradient-based Multi-Agent Quantum Reinforcement Learning methods often have to struggle with barren plateaus, holding them back from matching the performance of classical approaches. While gradient free Quantum Reinforcement Learning methods may alleviate some of these challenges, they too are not immune to the difficulties posed by barren plateaus. We build upon an existing approach for gradient free Quantum Reinforcement Learning and propose three genetic variations with Variational Quantum Circuits for Multi-Agent Reinforcement Learning using evolutionary optimization. We evaluate our genetic variations in the Coin Game environment and also compare them to classical approaches. We showed that our Variational Quantum Circuit approaches perform significantly better compared to a neural network with a similar amount of trainable parameters. Compared to the larger neural network, our approaches archive similar results using 97.88%97.88\%97.88% less parameters.
Reading out a parametrized quantum circuit through finite-shot measurements introduces a discrete stochastic component into hybrid quantum-classical neural networks, making gradient-based training inefficient. In this work, we introduce a surrogate-assisted actor-critic method for training hybrid quantum policies that avoids differentiating the quantum circuit during backpropagation. A classical neural surrogate trained on the circuit's input-output pairs from a replay buffer provides a differentiable substitute for actor updates and critic-target computation, while forward interaction and evaluation retain the quantum circuit's measurements. Our experiments on standard MuJoCo continuous-control tasks demonstrate promising policy learning. Notably, on Humanoid, more runs reach high returns than with learned classical baselines. Moreover, the best average return in our ablation comes from a smaller circuit / surrogate, suggesting that strong performance does not depend on a large network. We also derive conditional upstream gradient-error bounds that separate surrogate fitting from the mean-feature relaxation. Together, the results support surrogate-assisted training as an effective and efficient approach to learning hybrid quantum policies.
Variational Quantum Algorithms (VQAs) have gained prominence as a viable framework for exploiting near-term quantum devices in applications ranging from optimization and chemistry simulation to machine learning. However, the effectiveness of VQAs is often constrained by the so-called barren plateau problem, wherein gradients diminish exponentially as system size or circuit depth increases, thereby hindering training. In this work, we propose a reinforcement learning (RL)-based initialization strategy to alleviate the barren plateau issue by reshaping the initial parameter landscape to avoid regions prone to vanishing gradients. In particular, we explore several RL algorithms (Deterministic Policy Gradient, Soft ActorCritic, and Proximal Policy Optimization, etc.) to generate the circuit parameters (treated as “actions”) that minimize the VQAs cost function before standard gradient-based optimization. By pre-training with RL in this manner, subsequent optimization using methods such as gradient descent or Adam proceeds from a more favorable initial state. Extensive numerical experiments under various noise conditions and tasks consistently demonstrate that the RL-based initialization method significantly enhances both convergence speed and final solution quality. Moreover, comparisons among different RL algorithms highlight that multiple approaches can achieve comparable performance gains, underscoring the flexibility and robustness of our method. These findings shed light on a promising avenue for integrating machine learning techniques into quantum algorithm design, offering insights into how RL-driven parameter initialization can accelerate the scalability and practical deployment of VQAs. Opening up a promising path for the research community in machine learning for quantum, especially barren plateau problems in VQAs.
Training deep neural networks remains difficult due to vanishing gradients, non-convex loss surfaces, and hyperparameter sensitivity. These obstacles are compounded by quantum machine learning, where barren plateaus, circuit depth, and hardware noise restrict the applicability of gradient-based approaches. To overcome these drawbacks, this study presents adaptive Grover-driven parallel quantum optimization (AG-PQO), a hybrid, gradient-free scheme that leverages Grover's quadratic search speedup, along with adaptive loss-aware discretization and fidelity-based regularization. In contrast to more classical optimizers, such as Adam or evolutionary strategies (ES), which are either sensitive to the adequacy of the gradient update or exhibit poor scaling behavior, AG-PQO optimizes by performing Grover-accelerated candidate exploration across layers and reuses high-quality solutions in quantum memory caching. Testing indicates that AG-PQO yields higher accuracy, 2%–3% above Adam and ES, and faster convergence with less end-value loss than Adam, ES, and quantum feedforward-backpropagation (QFB). It is worth noting that AG-PQO remains stable at the simulated noise level of NISQ and has the potential to scale to near-term quantum processors.