Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2210.16715 · Nature Communications

Realizing a deep reinforcement learning agent for real-time quantum feedback

Kevin Reuer, Jonas Landgraf, Thomas Fösel, James O’Sullivan, Liberto Beltrán, Abdulkadir Akın, Graham J. Norris, Ants Remm, Michael Kerschbaum, Jean-Claude Besse, Florian Marquardt, Andreas J. Wallraff, Christopher Eichler

Published Nov 6, 2023• 81 citations
arXivHTMLPDF

Abstract

Realizing the full potential of quantum technologies requires precise real-time control on time scales much shorter than the coherence time. Model-free reinforcement learning promises to discover efficient feedback strategies from scratch without relying on a description of the quantum system. However, developing and training a reinforcement learning agent able to operate in real-time using feedback has been an open challenge. Here, we have implemented such an agent for a single qubit as a sub-microsecond-latency neural network on a field-programmable gate array (FPGA). We demonstrate its use to efficiently initialize a superconducting qubit and train the agent based solely on measurements. Our work is a first step towards adoption of reinforcement learning for the control of quantum devices and more generally any physical device requiring low-latency feedback.

DOI
10.1038/s41467-023-42901-3

Related papers

Fast Quantum Gate Design with Deep Reinforcement Learning Using Real-Time Feedback on Readout Signals

Emily Wright, Rogério de Sousa

The design of high-fidelity quantum gates is difficult because it requires the optimization of two competing effects, namely maximizing gate speed and minimizing leakage out of the qubit subspace. We propose a deep reinforcement learning algorithm that uses two agents to address the speed and leakage challenges simultaneously. The first agent constructs the qubit in-phase control pulse using a policy learned from rewards that compensate short gate times. The rewards are obtained at intermediate time steps throughout the construction of a full-length pulse, allowing the agent to explore the landscape of shorter pulses. The second agent determines an out-of-phase pulse to target leakage. Both agents are trained on real-time data from noisy hardware, thus providing model-free gate design that adapts to unpredictable hardware noise. To reduce the effect of measurement classification errors, the agents are trained directly on the readout signal from probing the qubit. We present proof-of-concept experiments by designing X and square root of X gates of various durations on IBM hardware. After just 200 training iterations, our algorithm is able to construct novel control pulses up to two times faster than the default IBM gates, while matching their performance in terms of state fidelity and leakage rate. As the length of our custom control pulses increases, they begin to outperform the default gates. Improvements to the speed and fidelity of gate operations open the way for higher circuit depth in quantum simulation, quantum chemistry and other algorithms on near-term and future quantum devices.

arXiv:2305.01169•IEEE International Conference on Quantum Computing and Engineering (QCE)•Sep 17, 2023•6 citations

Reinforcement Learning with Neural Networks for Quantum Feedback

Thomas Fösel, Petru Tighineanu, Talitha Weiss +1 more

Machine learning with artificial neural networks is revolutionizing science.The most advanced challenges require discovering answers autonomously.In the domain of reinforcement learning, control strategies are improved according to a reward function.The power of neural-network-based reinforcement learning has been highlighted by spectacular recent successes such as playing Go, but its benefits for physics are yet to be demonstrated.Here, we show how a network-based "agent" can discover complete quantum-error-correction strategies, protecting a collection of qubits against noise.These strategies require feedback adapted to measurement outcomes.Finding them from scratch without human guidance and tailored to different hardware resources is a formidable challenge due to the combinatorially large search space.To solve this challenge, we develop two ideas: two-stage learning with teacher and student networks and a reward quantifying the capability to recover the quantum information stored in a multiqubit system.Beyond its immediate impact on quantum computation, our work more generally demonstrates the promise of neural-network-based reinforcement learning in physics.

arXiv:1802.05267•Physical Review X•Sep 27, 2018•279 citations

Deep reinforcement learning for quantum gate control

Zheng An, D. L. Zhou

How to implement multi-qubit gates efficiently with high precision is essential for realizing universal fault-tolerant computing. For a physical system with some external controllable parameters, it is a great challenge to control the time dependence of these parameters to achieve a target multi-qubit gate efficiently and precisely. Here we construct a dueling double deep Q-learning neural network (DDDQN) to find out the optimized time dependence of controllable parameters to implement two typical quantum gates: a single-qubit Hadamard gate and a two-qubit CNOT gate. Compared with traditional optimal control methods, this deep reinforcement learning method can realize efficient and precise gate control without requiring any gradient information during the learning process. This work attempts to pave the way to investigate more quantum control problems with deep reinforcement learning techniques.

arXiv:1902.08418•Europhysics Letters (EPL)•Jul 22, 2019•132 citations

Hardware-in-the-Loop Reinforcement Learning for Quantum Error Correction: A Systems-Engineering Approach to Latency-Aware Decoding

Hiroshi Tanaka, Kenji Yamamoto, Takeshi Saito

The scalability of fault-tolerant quantum computing is strictly bounded by the performance and latency of Quantum Error Correction (QEC) decoding systems. While Reinforcement Learning (RL) has emerged as a promising approach for deriving optimal decoding policies, current Deep Q-Network (DQN) agents are overwhelmingly trained using idealized software simulators. These simulators employ deterministic latency models that drastically fail to capture the stochastic load noise, memory contention, thermal throttling, and bus jitter inherent to real-world hardware deployment on Field-Programmable Gate Arrays (FPGAs). In this paper, we propose a comprehensive Hardware-in-the-Loop (HIL) Reinforcement Learning architecture that physically situates the FPGA in the RL environment feedback loop. By directly offloading syndrome processing jobs to the FPGA hardware during training, the DQN agent observes the exact microarchitectural execution latency as its environmental reward. We formulate a robust, systems-engineering oriented framework that captures true edge-deployment constraints. Our extensive HIL empirical results demonstrate that agents trained against true hardware telemetry develop distinct decoding strategies that actively avoid routing congestion in the FPGA fabric, yielding up to a $42\%$ reduction in long-tail latency events and improving the effective logical error rate under strict real-time decoding constraints.

•Jul 21, 2026•0 citations

Hardware-in-the-Loop Reinforcement Learning for Quantum Error Correction: A Systems-Engineering Approach to Latency-Aware Decoding

Yamamoto Kenji, Tanaka Hiroshi, Takeshi Saito

The scalability of fault-tolerant quantum computing is strictly bounded by the performance and latency of Quantum Error Correction (QEC) decoding systems. While Reinforcement Learning (RL) has emerged as a promising approach for deriving optimal decoding policies, current Deep Q-Network (DQN) agents are overwhelmingly trained using idealized software simulators. These simulators employ deterministic latency models that drastically fail to capture the stochastic load noise, memory contention, thermal throttling, and bus jitter inherent to real-world hardware deployment on Field-Programmable Gate Arrays (FPGAs). In this paper, we propose a comprehensive Hardware-in-the-Loop (HIL) Reinforcement Learning architecture that physically situates the FPGA in the RL environment feedback loop. By directly offloading syndrome processing jobs to the FPGA hardware during training, the DQN agent observes the exact microarchitectural execution latency as its environmental reward. We formulate a robust, systems-engineering oriented framework that captures true edge-deployment constraints. Our extensive HIL empirical results demonstrate that agents trained against true hardware telemetry develop distinct decoding strategies that actively avoid routing congestion in the FPGA fabric, yielding up to a $42\%$ reduction in long-tail latency events and improving the effective logical error rate under strict real-time decoding constraints.

•Jul 21, 2026•0 citations

Robust quantum control using reinforcement learning from demonstration

Shengyong Li, Yidian Fan, Xiangyang Li +6 more

Quantum control requires high-precision and robust control pulses to ensure optimal system performance. However, control sequences generated with a system model may suffer from model bias, leading to low fidelity. While model-free reinforcement learning (RL) methods have been developed to avoid such biases, training an RL agent from scratch can be time-consuming, often taking hours to gather enough samples for convergence. This challenge has hindered the broad application of RL techniques to larger and more complex quantum control issues, limiting their adaptability. In this work, we use Reinforcement Learning from Demonstration (RLfD) to leverage the control sequences generated with system models and further optimize them with RL to avoid model bias. By avoiding learning from scratch and starting with reasonable control pulse shapes, this approach can increase sample efficiency by reducing the number of samples, which can significantly reduce the training time. Thus, this method can effectively handle pulse shapes that are discretized into more than 1000 pieces without compromising final fidelity. We have simulated the preparation of several high-fidelity non-classical states using the RLfD method. We also find that the training process is more stable when using RLfD. In addition, this method is suitable for fast gate calibration using reinforcement learning.

arXiv:2503.21085•npj Quantum Information•Jul 25, 2025•26 citations
Realizing the full potential of quantum technologies requires precise real-time control on time scales much shorter than the coherence time. Model-free reinforcement learning promises to discover efficient feedback strategies from scratch without relying on a description of the quantum system. However, developing and training a reinforcement learning agent able to operate in real-time using feedback has been an open challenge. Here, we have implemented such an agent for a single qubit as a sub-microsecond-latency neural network on a field-programmable gate array (FPGA). We demonstrate its use to efficiently initialize a superconducting qubit and train the agent based solely on measurements. Our work is a first step towards adoption of reinforcement learning for the control of quantum devices and more generally any physical device requiring low-latency feedback.
The design of high-fidelity quantum gates is difficult because it requires the optimization of two competing effects, namely maximizing gate speed and minimizing leakage out of the qubit subspace. We propose a deep reinforcement learning algorithm that uses two agents to address the speed and leakage challenges simultaneously. The first agent constructs the qubit in-phase control pulse using a policy learned from rewards that compensate short gate times. The rewards are obtained at intermediate time steps throughout the construction of a full-length pulse, allowing the agent to explore the landscape of shorter pulses. The second agent determines an out-of-phase pulse to target leakage. Both agents are trained on real-time data from noisy hardware, thus providing model-free gate design that adapts to unpredictable hardware noise. To reduce the effect of measurement classification errors, the agents are trained directly on the readout signal from probing the qubit. We present proof-of-concept experiments by designing X and square root of X gates of various durations on IBM hardware. After just 200 training iterations, our algorithm is able to construct novel control pulses up to two times faster than the default IBM gates, while matching their performance in terms of state fidelity and leakage rate. As the length of our custom control pulses increases, they begin to outperform the default gates. Improvements to the speed and fidelity of gate operations open the way for higher circuit depth in quantum simulation, quantum chemistry and other algorithms on near-term and future quantum devices.
Machine learning with artificial neural networks is revolutionizing science.The most advanced challenges require discovering answers autonomously.In the domain of reinforcement learning, control strategies are improved according to a reward function.The power of neural-network-based reinforcement learning has been highlighted by spectacular recent successes such as playing Go, but its benefits for physics are yet to be demonstrated.Here, we show how a network-based "agent" can discover complete quantum-error-correction strategies, protecting a collection of qubits against noise.These strategies require feedback adapted to measurement outcomes.Finding them from scratch without human guidance and tailored to different hardware resources is a formidable challenge due to the combinatorially large search space.To solve this challenge, we develop two ideas: two-stage learning with teacher and student networks and a reward quantifying the capability to recover the quantum information stored in a multiqubit system.Beyond its immediate impact on quantum computation, our work more generally demonstrates the promise of neural-network-based reinforcement learning in physics.
How to implement multi-qubit gates efficiently with high precision is essential for realizing universal fault-tolerant computing. For a physical system with some external controllable parameters, it is a great challenge to control the time dependence of these parameters to achieve a target multi-qubit gate efficiently and precisely. Here we construct a dueling double deep Q-learning neural network (DDDQN) to find out the optimized time dependence of controllable parameters to implement two typical quantum gates: a single-qubit Hadamard gate and a two-qubit CNOT gate. Compared with traditional optimal control methods, this deep reinforcement learning method can realize efficient and precise gate control without requiring any gradient information during the learning process. This work attempts to pave the way to investigate more quantum control problems with deep reinforcement learning techniques.
The scalability of fault-tolerant quantum computing is strictly bounded by the performance and latency of Quantum Error Correction (QEC) decoding systems. While Reinforcement Learning (RL) has emerged as a promising approach for deriving optimal decoding policies, current Deep Q-Network (DQN) agents are overwhelmingly trained using idealized software simulators. These simulators employ deterministic latency models that drastically fail to capture the stochastic load noise, memory contention, thermal throttling, and bus jitter inherent to real-world hardware deployment on Field-Programmable Gate Arrays (FPGAs). In this paper, we propose a comprehensive Hardware-in-the-Loop (HIL) Reinforcement Learning architecture that physically situates the FPGA in the RL environment feedback loop. By directly offloading syndrome processing jobs to the FPGA hardware during training, the DQN agent observes the exact microarchitectural execution latency as its environmental reward. We formulate a robust, systems-engineering oriented framework that captures true edge-deployment constraints. Our extensive HIL empirical results demonstrate that agents trained against true hardware telemetry develop distinct decoding strategies that actively avoid routing congestion in the FPGA fabric, yielding up to a 42%42\%42% reduction in long-tail latency events and improving the effective logical error rate under strict real-time decoding constraints.
The scalability of fault-tolerant quantum computing is strictly bounded by the performance and latency of Quantum Error Correction (QEC) decoding systems. While Reinforcement Learning (RL) has emerged as a promising approach for deriving optimal decoding policies, current Deep Q-Network (DQN) agents are overwhelmingly trained using idealized software simulators. These simulators employ deterministic latency models that drastically fail to capture the stochastic load noise, memory contention, thermal throttling, and bus jitter inherent to real-world hardware deployment on Field-Programmable Gate Arrays (FPGAs). In this paper, we propose a comprehensive Hardware-in-the-Loop (HIL) Reinforcement Learning architecture that physically situates the FPGA in the RL environment feedback loop. By directly offloading syndrome processing jobs to the FPGA hardware during training, the DQN agent observes the exact microarchitectural execution latency as its environmental reward. We formulate a robust, systems-engineering oriented framework that captures true edge-deployment constraints. Our extensive HIL empirical results demonstrate that agents trained against true hardware telemetry develop distinct decoding strategies that actively avoid routing congestion in the FPGA fabric, yielding up to a 42%42\%42% reduction in long-tail latency events and improving the effective logical error rate under strict real-time decoding constraints.
Quantum control requires high-precision and robust control pulses to ensure optimal system performance. However, control sequences generated with a system model may suffer from model bias, leading to low fidelity. While model-free reinforcement learning (RL) methods have been developed to avoid such biases, training an RL agent from scratch can be time-consuming, often taking hours to gather enough samples for convergence. This challenge has hindered the broad application of RL techniques to larger and more complex quantum control issues, limiting their adaptability. In this work, we use Reinforcement Learning from Demonstration (RLfD) to leverage the control sequences generated with system models and further optimize them with RL to avoid model bias. By avoiding learning from scratch and starting with reasonable control pulse shapes, this approach can increase sample efficiency by reducing the number of samples, which can significantly reduce the training time. Thus, this method can effectively handle pulse shapes that are discretized into more than 1000 pieces without compromising final fidelity. We have simulated the preparation of several high-fidelity non-classical states using the RLfD method. We also find that the training process is more stable when using RLfD. In addition, this method is suitable for fast gate calibration using reinforcement learning.