Deep Reinforcement Learning (DRL) is a sub-field of Machine Learning (ML) that combines reinforcement learning (RL) with deep learning techniques. DRL has showcased promising results in gaming environments and real-world applications. However, DRL agents are susceptible to faulty observations due to sudden interference like blackouts, frozen observation, or adversarial interference in practical applications. These scenarios can hamper the learning and performance of DRL agents if they are not resilient. Drawing inspiration from causal inference, the Causal In-ference Q-Network (CIQ) explores a Deep Q-Network (DQN) framework that undergoes training with an additional task focusing on training for observational interference. Through an evaluation conducted in various benchmark DQN environments, the experimental findings demonstrate that the CIQ method exhibits enhanced performance and greater resilience against observational interference. In this paper, quantum networks and hybrid quantum-classical networks are tested for their resilience in noisy environments. A novel Quantum-Enhanced CIQ-Network (QCIQ) is proposed which could replicate the functioning of the CIQ using Variational Quantum Circuits (VQCs) and is benchmarked against the corresponding classical networks for its performance.
Deep Reinforcement Learning (DRL) is a sub-field of Machine Learning (ML) that combines reinforcement learning (RL) with deep learning techniques. DRL has showcased promising results in gaming environments and real-world applications. However, DRL agents are susceptible to faulty observations due to sudden interference like blackouts, frozen observation, or adversarial interference in practical applications. These scenarios can hamper the learning and performance of DRL agents if they are not resilient. Drawing inspiration from causal inference, the Causal In-ference Q-Network (CIQ) explores a Deep Q-Network (DQN) framework that undergoes training with an additional task focusing on training for observational interference. Through an evaluation conducted in various benchmark DQN environments, the experimental findings demonstrate that the CIQ method exhibits enhanced performance and greater resilience against observational interference. In this paper, quantum networks and hybrid quantum-classical networks are tested for their resilience in noisy environments. A novel Quantum-Enhanced CIQ-Network (QCIQ) is proposed which could replicate the functioning of the CIQ using Variational Quantum Circuits (VQCs) and is benchmarked against the corresponding classical networks for its performance.
In the realm of artificial intelligence, deep rein-forcement learning (RL) agents struggle with generalizability and require substantial computational resources, unlike humans who easily adapt and generalize across tasks. To address these challenges, we introduce Quantum Rainbow, a hybrid algorithm that leverages the neural mechanisms of human decision-making and the efficiency of quantum computing. Quantum Rainbow combines variational quantum circuits with the Rainbow Deep Q-Network (DQN) model to create a novel approach in rein-forcement learning that integrates quantum principles into deep learning paradigms. We evaluate our model using behavioral experiments through the Iowa Gambling Task and 4-Armed Bandit Task. Our investigations reveal a significant relationship between the architecture of quantum circuits and the performance of quantum RL agents. Specifically, using causal discovery methods, we demonstrate the critical role of quantum entanglement in enhancing model performance. These findings not only show promising results but also pave the way for future explorations into optimizing quantum circuit architectures for reinforcement learning applications. This study underscores the potential of quantum-enhanced algorithms to achieve “quantum advantage” by addressing fundamental limitations in conventional deep RL methods.
Quantum machine learning (QML) has been identified as one of the key fields that could reap advantages from near-term quantum devices, next to optimization and quantum chemistry. Research in this area has focused primarily on variational quantum algorithms (VQAs), and several proposals to enhance supervised, unsupervised and reinforcement learning (RL) algorithms with VQAs have been put forward. Out of the three, RL is the least studied and it is still an open question whether VQAs can be competitive with state-of-the-art classical algorithms based on neural networks (NNs) even on simple benchmark tasks. In this work, we introduce a training method for parametrized quantum circuits (PQCs) that can be used to solve RL tasks for discrete and continuous state spaces based on the deep Q-learning algorithm. We investigate which architectural choices for quantum Q-learning agents are most important for successfully solving certain types of environments by performing ablation studies for a number of different data encoding and readout strategies. We provide insight into why the performance of a VQA-based Q-learning algorithm crucially depends on the observables of the quantum model and show how to choose suitable observables based on the learning task at hand. To compare our model against the classical DQN algorithm, we perform an extensive hyperparameter search of PQCs and NNs with varying numbers of parameters. We confirm that similar to results in classical literature, the architectural choices and hyperparameters contribute more to the agents' success in a RL setting than the number of parameters used in the model. Finally, we show when recent separation results between classical and quantum agents for policy gradient RL can be extended to inferring optimal Q-values in restricted families of environments.
Deep reinforcement learning (DRL) is a standard approach to sequential decision-making under uncertainty, where value functions are often approximated with deep neural networks such as Deep Q-Networks (DQNs). Parameterized quantum circuits (PQCs) have emerged as a flexible modeling primitive for near-term quantum hardware, enabling hybrid quantum–classical predictors in which a PQC produces bounded expectation-value features that are processed by classical layers. A practical limitation of many quantum learning approaches, however, is the need for quantum hardware not only during training but also at deployment time. In this work, we study PQC-driven value-based agents in the CartPole control benchmark. Using a common DQN-style training loop, we compare classical DQN baselines, a PQC-based Q-network, and a hybrid workflow that leverages classical shadows for evaluation-only deployment. To probe near-term practicality, we further examine robustness by injecting additive Gaussian perturbations into the quantum feature vector across multiple noise amplitudes.
Deep Reinforcement Learning (RL) has considerably advanced over the past decade. At the same time, state-of-the-art RL algorithms require a large computational budget in terms of training time to converge. Recent work has started to approach this problem through the lens of quantum computing, which promises theoretical speed-ups for several traditionally hard tasks. In this work, we examine a class of hybrid quantum-classical RL algorithms that we collectively refer to as variational quantum deep Q-networks (VQ-DQN). We show that VQ-DQN approaches are subject to instabilities that cause the learned policy to diverge, study the extent to which this afflicts reproducibility of established results based on classical simulation, and perform systematic experiments to identify potential explanations for the observed instabilities. Additionally, and in contrast to most existing work on quantum reinforcement learning, we execute RL algorithms on an actual quantum processing unit (an IBM Quantum Device) and investigate differences in behaviour between simulated and physical quantum systems that suffer from implementation deficiencies. Our experiments show that, contrary to opposite claims in the literature, it cannot be conclusively decided if known quantum approaches, even if simulated without physical imperfections, can provide an advantage as compared to classical approaches. Finally, we provide a robust, universal and well-tested implementation of VQ-DQN as a reproducible testbed for future experiments.
Many challenges that occur with traditional reinforcement learning in uncertain environments, such as slow convergence and scalability, can be addressed with Quantum-Enhanced Reinforcement Learning (QERL). Specifically, this research is for adaptive decision-making in environments with rapid change and great uncertainty. A QERL framework that combines variational quantum circuits and deep Q-networks is introduced. This framework optimizes decision-making processes in real-time and is applicable in areas such as autonomous systems, smart manufacturing, and intelligent resource allocation. Training for the framework took place in simulated dynamic environments during 50,000 training episodes and was then evaluated alongside Deep Q-Learning and Proximal Policy Optimization models. The QERL self-model outperformed the baseline in policy accuracy (+18.6%) and cumulative reward loss (-21.4%). Decision latency decreased 16.2%, and convergence improved 23.8% with the use of a hybrid quantum architecture. The proposed analysis demonstrated improved average reward (78.4 to 94.7) with 95% CI, indicating model consistency. Ultimately, QERL improves decision-making in complex scenarios and provides robust adaptive learning. Also noted, QERL frameworks with hybrid quantum architecture are useful for real-world systems. Furthermore, the learning framework proved to be more stable and adaptable to rapid and stochastic environmental changes compared to classical agents, in environments with incomplete dynamic and rapidly changing environments.
The development of quantum computational techniques has advanced greatly in recent years, parallel to the advancements in techniques for deep reinforcement learning. This work explores the potential for quantum computing to facilitate reinforcement learning problems. Quantum computing approaches offer important potential improvements in time and space complexity over traditional algorithms because of its ability to exploit the quantum phenomena of superposition and entanglement. Specifically, we investigate the use of quantum variational circuits, a form of quantum machine learning. We present our techniques for encoding classical data for a quantum variational circuit, we further explore pure and hybrid quantum algorithms for DQN and Double DQN. Our results indicate both hybrid and pure quantum variational circuit have the ability to solve reinforcement learning tasks with a smaller parameter space. These comparison are conducted with two OpenAI Gym environments: CartPole and Blackjack, The success of this work is indicative of a strong future relationship between quantum machine learning and deep reinforcement learning.