Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2605.01716 · Machine Learning Science and Technology

Towards Real-time Control of a CartPole System on a Quantum Computer

Nguyen Truong Thu Ngo, Väinö Mehtola, Jérome Lenssen, Peiyong Wang, Francesco Cosco, Tien-Fu Lu, James Q. Quach

quant-phSubmitted May 3, 2026• Announced May 5, 2026• 1 citation
arXivHTMLPDF

Abstract

The application of quantum reinforcement learning (QRL) to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum-classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit (QPU). We demonstrate that a single-qubit agent acts as an effective learning model, solving the environment in substantially fewer episodes than a comparable classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on NISQ hardware by bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.

AI Summary

The paper investigates whether a minimal quantum reinforcement-learning policy can learn CartPole and approach the latency requirements of real-time control. It combines simulation-based training, deployment-robustness tests, and inference on a superconducting QPU. The hybrid actor–critic uses a single-qubit circuit with three state-encoding rotations and one trainable rotation, followed by small classical networks. Across 50 seeded runs, the analytical-gradient hybrid reaches the criterion of 100 consecutive perfect episodes in $161.6\pm56.3$ episodes, versus $429.4\pm117.9$ for a classical actor–critic with larger networks. Finite-shot parameter-shift training also converges faster than the classical baseline, although more slowly than analytical-gradient training. A noisy-simulator study sweeps training and inference frequencies from 20 to 100 Hz and inference budgets of 128–1024 shots. Higher inference frequency and shot count generally improve balancing; at 100 Hz, all tested shot budgets give near-maximal 10-second durations. Training frequency has no consistent effect. On hardware, directly programming control electronics rather than using the standard software stack increases inference rates from approximately 0.14 Hz to 2.71–6.23 Hz, an $18.8$–$43.3\times$ speedup. Offline simulated CartPole episodes reach average scores of approximately 143, 166, 474, and 500 at 128, 256, 512, and 1024 shots, respectively. The hardware tests do not demonstrate real-time physical control: environment updates assume 50 Hz while actual inference remains slower. Stated limitations include the single-qubit, unentangled architecture and lack of scaling tests; the learning result is not a complexity-theoretic quantum speedup.

Some signs of AI-generated textIncrementalMissing comparisonOverclaims

Central claim may not hold: The reported learning advantage is against a substantially different classical architecture, without a matched classical feature-map control that could establish whether quantum processing contributes to the improvement. The convergence experiment is described inconsistently as training for up to 1500 episodes with early stopping and as 500-episode runs with continued post-convergence performance. The stated native-gate decomposition does not implement the defined quantum policy under the paper's own PRX convention, leaving the identity of the trained and hardware-executed policies unresolved.

Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.

Related papers

Hybrid Quantum-Classical Policy Gradient for Adaptive Control of Cyber-Physical Systems: A Comparative Study of VQC vs. MLP

Aueaphum Aueawatthanaphisut, Nyi Wunna Tun

The comparative evaluation between classical and quantum reinforcement learning (QRL) paradigms was conducted to investigate their convergence behavior, robustness under observational noise, and computational efficiency in a benchmark control environment. The study employed a multilayer perceptron (MLP) agent as a classical baseline and a parameterized variational quantum circuit (VQC) as a quantum counterpart, both trained on the CartPole-v1 environment over 500 episodes. Empirical results demonstrated that the classical MLP achieved near-optimal policy convergence with a mean return of 498.7 +/- 3.2, maintaining stable equilibrium throughout training. In contrast, the VQC exhibited limited learning capability, with an average return of 14.6 +/- 4.8, primarily constrained by circuit depth and qubit connectivity. Noise robustness analysis further revealed that the MLP policy deteriorated gracefully under Gaussian perturbations, while the VQC displayed higher sensitivity at equivalent noise levels. Despite the lower asymptotic performance, the VQC exhibited significantly lower parameter count and marginally increased training time, highlighting its potential scalability for low-resource quantum processors. The results suggest that while classical neural policies remain dominant in current control benchmarks, quantum-enhanced architectures could offer promising efficiency advantages once hardware noise and expressivity limitations are mitigated.

arXiv:2510.06010•Oct 7, 2025•0 citations

Quantum Advantage Actor-Critic for Reinforcement Learning

Michael Kölle, Mohamad Hgog, Fabian Ritz +4 more

Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empirically test multiple quantum Advantage Actor-Critic configurations with the well known Cart Pole environment to evaluate our approach in control tasks with continuous state spaces. Our results indicate that the hybrid strategy of using either a quantum actor or quantum critic with classical post-processing yields a substantial performance increase compared to pure classical and pure quantum variants with similar parameter counts. They further reveal the limits of current quantum approaches due to the hardware constraints of noisy intermediate-scale quantum computers, suggesting further research to scale hybrid approaches for larger and more complex control tasks.

arXiv:2401.07043•International Conference on Agents and Artificial Intelligence (ICAART)•Jan 1, 2024•30 citations

Investigating Quantum Reinforcement Learning structure to the CartPole control task

Nguyen Truong Thu Ngo, Tien‐Fu Lu, James Q. Quach +1 more

In recent years, reinforcement learning (RL) has been proven to effectively provide solution to complex problems in various engineering problems such as self-driving cars, industry automation in production lines. These applications also extend to other fields including providing a platform for financial trading and healthcare. Reinforcement learning comes with the trade-offs between exploitation and exploration regardless of applied techniques and algorithms. In more complex engineering system, which often consists of large action and state-space. This space can increase exponentially, and most RL techniques fail to efficiently compute optimal policies to these problems. The inefficiency in RL models leads to the extensive requirement of computational process, which does not often come cheap and affordable. To tackle this challenge, quantum computational models were studied and achieved different levels of success. Since its first theory in the 1940s, quantum algorithm has been improved and advanced to provide exponential speed up compared to classical solutions. Quantum computation offers great potential improvements to traditional RL models due to its ability to create superpositions and entanglement. In past research, quantum variational circuits (QVC) were created as alternatives to neural networks commonly used in RL and experimented on several RL benchmarks in OpenAI Gym environments: CartPole, Acrobat and Lunar Landing. One of such QVCs were designed and tested to achieve greater efficiency in learning speed while offer reduced number of trainable parameter than classical RL. Our research aims to investigate the QVC to balance the CartPole problem running on both local PC simulator and quantum computer. The research signifies the first time a quantum RL could learn to obtain the optimal policy to acquire the maximum expected reward to a control problem and effectively apply the trainable parameter to balance the pole. The system is first tailored into a RL environment with state space and action space. For each time step, the input data from the environment (state space) is encoded into quantum states. Through the QVC, the algorithm learns to optimize the RL policy to calculate the probability of future action to obtain the optimal reward for each state-action pair. The trainable parameters from the QVC are again optimized using gradient descent. The QVC design is vital to the success of the RL model; therefore, we vary the QVC gates and investigate the model performance. It is expected that the Quantum RL model would outperform the classical RL in learning the optimal policy in term of speed and computational resources. The successful control of the pole on a quantum simulation would be proof that quantum models could offer real solution to future computation-intensive problems where classical solutions are unaffordable.

Asian Society for Precision Engineering and Nanotechnology (ASPEN 2022)•Jan 1, 2022•0 citations

Parameterized Quantum Circuits for Deep Reinforcement Learning

Hrvoje Kukina, Susanne F. Yelin

Deep reinforcement learning (DRL) is a standard approach to sequential decision-making under uncertainty, where value functions are often approximated with deep neural networks such as Deep Q-Networks (DQNs). Parameterized quantum circuits (PQCs) have emerged as a flexible modeling primitive for near-term quantum hardware, enabling hybrid quantum–classical predictors in which a PQC produces bounded expectation-value features that are processed by classical layers. A practical limitation of many quantum learning approaches, however, is the need for quantum hardware not only during training but also at deployment time. In this work, we study PQC-driven value-based agents in the CartPole control benchmark. Using a common DQN-style training loop, we compare classical DQN baselines, a PQC-based Q-network, and a hybrid workflow that leverages classical shadows for evaluation-only deployment. To probe near-term practicality, we further examine robustness by injecting additive Gaussian perturbations into the quantum feature vector across multiple noise amplitudes.

IEEE International Conference on Quantum Control, Computing and Learning (qCCL)•Jun 1, 2026•0 citations

First Experience with Real-Time Control Using Simulated VQC-Based Quantum Policies

Yize Sun, Mohamad Hagog, M. Weber +4 more

This paper investigates the integration of quantum computing into offline reinforcement learning and the deployment of the resulting quantum policy in a real-time control hardware realization of the cart-pole system. Variational Quantum Circuits (VQCs) are used to represent the policy. Classical model-based offline policy search was applied, in which a pure VQC with trainable input-output weights is used as a policy network instead of a classical multilayer perceptron. The goal is to evaluate the potential of deploying quantum architectures in real-world industrial control problems. The experimental results show that the investigated model-based offline policy search is able to generate quantum policies that can balance the hardware cart-pole. A latency analysis reveals that while local simulated execution meets real-time requirements, cloud-based quantum processing remains too slow for closed-loop control.

arXiv:2508.01690•IEEE International Conference on Quantum Computing and Engineering (QCE)•Aug 30, 2025•2 citations

Enhancement of Machine Learning for Adaptive Quantum Computing Control: The Optimization Performance and Prevention of Real Time Error

V. Saravanan, Ganesan M, R. Balasubramaniyan +3 more

Adaptive quantum computing control is important for the stability and efficiency of quantum systems, especially in the presence of dynamic environmental noise and hardware fluctuations. We present a unified strategy for machine learning (ML) techniques to improve adaptive quantum control in two main directions: optimization speed and out-of-sample error. Here, we propose a hybrid learning method combining reinforcement learning (RL), deep neural networks (DNNs), and quantum-inspired models to actively and adaptively update quantum gate parameters and error mitigation strategies on the fly. Embedding of quantum noise models directly into the learning allows the system to predict and correct for decoherence, gate errors and measurement inaccuracy before they affect computational fidelity. Using extensive simulations as well as experimental validation on noisy intermediate-scale quantum (NISQ) processors, our improved ML model achieves a substantially higher fidelity and robustness compared to standard model-free and static control approaches. We introduce a novel feedback-loop mechanism based on online learning algorithms enabling the system to adjust for changing noise profiles and hardware drift without retraining. The optimization part uses metaheuristic methods to quickly converge to control parameters that perform well, and the error prevention mechanism features anomaly detection and prediction correction layers to minimize the effects of cumulative decoherence. The results demonstrate that our adaptive control framework can be used to not only speed up the convergence in training quantum algorithms, but also sustain the performance of gates against error at a variety of error rates with the reduction of the real-time impact of the error up to 35 %. This work highlights the promise of machine learningaugmented quantum control for improving reliability and scalability in quantum computing and serves as a stepping stone for future incorporation with fault-tolerant designs.

International Conference on Electronics and Computing, Communication Networking Automation Technologies (ICEC2NT)•Sep 3, 2025•3 citations
The application of quantum reinforcement learning (QRL) to real-time control systems faces significant challenges regarding hardware latency, noise susceptibility, and learning convergence. This work presents an end-to-end investigation of a minimal hybrid quantum-classical agent applied to the CartPole benchmark, addressing the gap between idealized simulation and execution on a physical superconducting quantum processing unit (QPU). We demonstrate that a single-qubit agent acts as an effective learning model, solving the environment in substantially fewer episodes than a comparable classical actor-critic network even when the training of the hybrid agent is restricted to use parameter-shift for its quantum circuit component. To connect learning to deployment constraints, we map the inference-time trade-off between control-loop rate and measurement shot budget to provide guidance for an eventual real-time control demonstration. The resulting performance matrices show that both inference control frequency and shot count strongly affect balancing stability: higher inference frequencies consistently improve performance, and increasing the shot budget lowers the minimum inference frequency required to achieve near-maximal balancing. These results highlight the importance of finding an optimal medium between shot count and control frequency and developing circuits that are e.g. initial-state invariant. Lastly, we address the critical bottleneck of control latency on NISQ hardware by bypassing the standard high-level software stack and programming the Zurich Instruments readout electronics directly via command tables. These results quantify some of the current boundaries of quantum-assisted control and provide a start for achieving the tens-of-hertz throughput required for real-time closed-loop control feedback.
The paper investigates whether a minimal quantum reinforcement-learning policy can learn CartPole and approach the latency requirements of real-time control. It combines simulation-based training, deployment-robustness tests, and inference on a superconducting QPU.
The hybrid actor–critic uses a single-qubit circuit with three state-encoding rotations and one trainable rotation, followed by small classical networks. Across 50 seeded runs, the analytical-gradient hybrid reaches the criterion of 100 consecutive perfect episodes in 161.6±56.3161.6\pm56.3161.6±56.3 episodes, versus 429.4±117.9429.4\pm117.9429.4±117.9 for a classical actor–critic with larger networks. Finite-shot parameter-shift training also converges faster than the classical baseline, although more slowly than analytical-gradient training.
A noisy-simulator study sweeps training and inference frequencies from 20 to 100 Hz and inference budgets of 128–1024 shots. Higher inference frequency and shot count generally improve balancing; at 100 Hz, all tested shot budgets give near-maximal 10-second durations. Training frequency has no consistent effect.
On hardware, directly programming control electronics rather than using the standard software stack increases inference rates from approximately 0.14 Hz to 2.71–6.23 Hz, an 18.818.818.8–43.3×43.3\times43.3× speedup. Offline simulated CartPole episodes reach average scores of approximately 143, 166, 474, and 500 at 128, 256, 512, and 1024 shots, respectively.
The hardware tests do not demonstrate real-time physical control: environment updates assume 50 Hz while actual inference remains slower. Stated limitations include the single-qubit, unentangled architecture and lack of scaling tests; the learning result is not a complexity-theoretic quantum speedup.
The comparative evaluation between classical and quantum reinforcement learning (QRL) paradigms was conducted to investigate their convergence behavior, robustness under observational noise, and computational efficiency in a benchmark control environment. The study employed a multilayer perceptron (MLP) agent as a classical baseline and a parameterized variational quantum circuit (VQC) as a quantum counterpart, both trained on the CartPole-v1 environment over 500 episodes. Empirical results demonstrated that the classical MLP achieved near-optimal policy convergence with a mean return of 498.7 +/- 3.2, maintaining stable equilibrium throughout training. In contrast, the VQC exhibited limited learning capability, with an average return of 14.6 +/- 4.8, primarily constrained by circuit depth and qubit connectivity. Noise robustness analysis further revealed that the MLP policy deteriorated gracefully under Gaussian perturbations, while the VQC displayed higher sensitivity at equivalent noise levels. Despite the lower asymptotic performance, the VQC exhibited significantly lower parameter count and marginally increased training time, highlighting its potential scalability for low-resource quantum processors. The results suggest that while classical neural policies remain dominant in current control benchmarks, quantum-enhanced architectures could offer promising efficiency advantages once hardware noise and expressivity limitations are mitigated.
Quantum computing offers efficient encapsulation of high-dimensional states. In this work, we propose a novel quantum reinforcement learning approach that combines the Advantage Actor-Critic algorithm with variational quantum circuits by substituting parts of the classical components. This approach addresses reinforcement learning's scalability concerns while maintaining high performance. We empirically test multiple quantum Advantage Actor-Critic configurations with the well known Cart Pole environment to evaluate our approach in control tasks with continuous state spaces. Our results indicate that the hybrid strategy of using either a quantum actor or quantum critic with classical post-processing yields a substantial performance increase compared to pure classical and pure quantum variants with similar parameter counts. They further reveal the limits of current quantum approaches due to the hardware constraints of noisy intermediate-scale quantum computers, suggesting further research to scale hybrid approaches for larger and more complex control tasks.
In recent years, reinforcement learning (RL) has been proven to effectively provide solution to complex problems in various engineering problems such as self-driving cars, industry automation in production lines. These applications also extend to other fields including providing a platform for financial trading and healthcare. Reinforcement learning comes with the trade-offs between exploitation and exploration regardless of applied techniques and algorithms. In more complex engineering system, which often consists of large action and state-space. This space can increase exponentially, and most RL techniques fail to efficiently compute optimal policies to these problems. The inefficiency in RL models leads to the extensive requirement of computational process, which does not often come cheap and affordable. To tackle this challenge, quantum computational models were studied and achieved different levels of success. Since its first theory in the 1940s, quantum algorithm has been improved and advanced to provide exponential speed up compared to classical solutions. Quantum computation offers great potential improvements to traditional RL models due to its ability to create superpositions and entanglement. In past research, quantum variational circuits (QVC) were created as alternatives to neural networks commonly used in RL and experimented on several RL benchmarks in OpenAI Gym environments: CartPole, Acrobat and Lunar Landing. One of such QVCs were designed and tested to achieve greater efficiency in learning speed while offer reduced number of trainable parameter than classical RL. Our research aims to investigate the QVC to balance the CartPole problem running on both local PC simulator and quantum computer. The research signifies the first time a quantum RL could learn to obtain the optimal policy to acquire the maximum expected reward to a control problem and effectively apply the trainable parameter to balance the pole. The system is first tailored into a RL environment with state space and action space. For each time step, the input data from the environment (state space) is encoded into quantum states. Through the QVC, the algorithm learns to optimize the RL policy to calculate the probability of future action to obtain the optimal reward for each state-action pair. The trainable parameters from the QVC are again optimized using gradient descent. The QVC design is vital to the success of the RL model; therefore, we vary the QVC gates and investigate the model performance. It is expected that the Quantum RL model would outperform the classical RL in learning the optimal policy in term of speed and computational resources. The successful control of the pole on a quantum simulation would be proof that quantum models could offer real solution to future computation-intensive problems where classical solutions are unaffordable.
Deep reinforcement learning (DRL) is a standard approach to sequential decision-making under uncertainty, where value functions are often approximated with deep neural networks such as Deep Q-Networks (DQNs). Parameterized quantum circuits (PQCs) have emerged as a flexible modeling primitive for near-term quantum hardware, enabling hybrid quantum–classical predictors in which a PQC produces bounded expectation-value features that are processed by classical layers. A practical limitation of many quantum learning approaches, however, is the need for quantum hardware not only during training but also at deployment time. In this work, we study PQC-driven value-based agents in the CartPole control benchmark. Using a common DQN-style training loop, we compare classical DQN baselines, a PQC-based Q-network, and a hybrid workflow that leverages classical shadows for evaluation-only deployment. To probe near-term practicality, we further examine robustness by injecting additive Gaussian perturbations into the quantum feature vector across multiple noise amplitudes.
This paper investigates the integration of quantum computing into offline reinforcement learning and the deployment of the resulting quantum policy in a real-time control hardware realization of the cart-pole system. Variational Quantum Circuits (VQCs) are used to represent the policy. Classical model-based offline policy search was applied, in which a pure VQC with trainable input-output weights is used as a policy network instead of a classical multilayer perceptron. The goal is to evaluate the potential of deploying quantum architectures in real-world industrial control problems. The experimental results show that the investigated model-based offline policy search is able to generate quantum policies that can balance the hardware cart-pole. A latency analysis reveals that while local simulated execution meets real-time requirements, cloud-based quantum processing remains too slow for closed-loop control.
Adaptive quantum computing control is important for the stability and efficiency of quantum systems, especially in the presence of dynamic environmental noise and hardware fluctuations. We present a unified strategy for machine learning (ML) techniques to improve adaptive quantum control in two main directions: optimization speed and out-of-sample error. Here, we propose a hybrid learning method combining reinforcement learning (RL), deep neural networks (DNNs), and quantum-inspired models to actively and adaptively update quantum gate parameters and error mitigation strategies on the fly. Embedding of quantum noise models directly into the learning allows the system to predict and correct for decoherence, gate errors and measurement inaccuracy before they affect computational fidelity. Using extensive simulations as well as experimental validation on noisy intermediate-scale quantum (NISQ) processors, our improved ML model achieves a substantially higher fidelity and robustness compared to standard model-free and static control approaches. We introduce a novel feedback-loop mechanism based on online learning algorithms enabling the system to adjust for changing noise profiles and hardware drift without retraining. The optimization part uses metaheuristic methods to quickly converge to control parameters that perform well, and the error prevention mechanism features anomaly detection and prediction correction layers to minimize the effects of cumulative decoherence. The results demonstrate that our adaptive control framework can be used to not only speed up the convergence in training quantum algorithms, but also sustain the performance of gates against error at a variety of error rates with the reduction of the real-time impact of the error up to 35 %. This work highlights the promise of machine learningaugmented quantum control for improving reliability and scalability in quantum computing and serves as a stepping stone for future incorporation with fault-tolerant designs.