Automatic index selection is a critical task for optimising query performance in distributed database systems. This problem is NP-hard, and existing approaches based on heuristic methods or optimisation problems struggle to generate good recommendations within a reasonable timeframe for large-scale database applications. While using classical deep reinforcement learning with Deep Q-Networks (DQN) has shown promise, its scalability in high-dimensional state spaces remains a challenge. In this paper, we explore the application of hybrid quantum-classical deep reinforcement learning, specifically a Quantum Deep Q-Network (Q-DQN), to the divergent index tuning problem. Using the TPC-H benchmark on a varying number of database replicas, we compare the performance of classical divergent index tuning against a hybrid quantum-classical algorithm using a Q-DQN implemented on the IBM Qiskit Aer simulator. We find that the hybrid quantum-classical algorithm exhibits remarkably higher sample efficiency, converging in significantly fewer action steps while maintaining a comparable query execution time. Overall, our findings suggest that variational quantum circuits provide an effective representation for deep reinforcement learning, achieving faster policy convergence, and that when appropriately configured, this improved sample efficiency can be obtained without loss in query execution performance.
Automatic index selection is a critical task for optimising query performance in distributed database systems. This problem is NP-hard, and existing approaches based on heuristic methods or optimisation problems struggle to generate good recommendations within a reasonable timeframe for large-scale database applications. While using classical deep reinforcement learning with Deep Q-Networks (DQN) has shown promise, its scalability in high-dimensional state spaces remains a challenge. In this paper, we explore the application of hybrid quantum-classical deep reinforcement learning, specifically a Quantum Deep Q-Network (Q-DQN), to the divergent index tuning problem. Using the TPC-H benchmark on a varying number of database replicas, we compare the performance of classical divergent index tuning against a hybrid quantum-classical algorithm using a Q-DQN implemented on the IBM Qiskit Aer simulator. We find that the hybrid quantum-classical algorithm exhibits remarkably higher sample efficiency, converging in significantly fewer action steps while maintaining a comparable query execution time. Overall, our findings suggest that variational quantum circuits provide an effective representation for deep reinforcement learning, achieving faster policy convergence, and that when appropriately configured, this improved sample efficiency can be obtained without loss in query execution performance.
Deep Reinforcement Learning (RL) has considerably advanced over the past decade. At the same time, state-of-the-art RL algorithms require a large computational budget in terms of training time to converge. Recent work has started to approach this problem through the lens of quantum computing, which promises theoretical speed-ups for several traditionally hard tasks. In this work, we examine a class of hybrid quantum-classical RL algorithms that we collectively refer to as variational quantum deep Q-networks (VQ-DQN). We show that VQ-DQN approaches are subject to instabilities that cause the learned policy to diverge, study the extent to which this afflicts reproducibility of established results based on classical simulation, and perform systematic experiments to identify potential explanations for the observed instabilities. Additionally, and in contrast to most existing work on quantum reinforcement learning, we execute RL algorithms on an actual quantum processing unit (an IBM Quantum Device) and investigate differences in behaviour between simulated and physical quantum systems that suffer from implementation deficiencies. Our experiments show that, contrary to opposite claims in the literature, it cannot be conclusively decided if known quantum approaches, even if simulated without physical imperfections, can provide an advantage as compared to classical approaches. Finally, we provide a robust, universal and well-tested implementation of VQ-DQN as a reproducible testbed for future experiments.
The power of classical computers is still insufficient for deep reinforcement learning (DRL) problems which has large state space. Thanks to entanglement and superposition, quantum computers have a high computational power. The concept of using this high computing power to solve problems that would be difficult for classical computers is fairly common. In this study, a hybrid approach is proposed to take advantage of the benefits of quantum computers. Deep Q-Network (DQN) algorithm for DRL, optimization operations, and storage operations are performed on the classical computer side of this hybrid approach. In the quantum side, a variational quantum circuit (VQC) is proposed. The proposed method is applied to a navigation problem. The proposed approach is evaluated in terms of target success rate, collision (going out) rate. The proposed approach is compared to DRL solutions in the literature for the navigation problem with classical computers. According to the number of parameters used, the proposed approach appears to be successful. As a result, the performance of the proposed approach has been validated.
With advances in digital technology in recent years, parallel computation utilizing GPUs has achieved remarkable efficiency, which has made it possible to use large-scale machine learning algorithms with large datasets, and deep learning, which is machine learning utilizing deep neural networks trained with millions to billions of data and parameters, has become increasingly popular and found its use for various tasks including facial recognition, language translation and combinatorial problems, etc., achieving considerably many record-breaking results.As a powerful generic tool to solve problems, deep learning has been applied to physical problems as well, and has become an important alternative tool for scientists to solve a number of important problems.With the development of experimental quantum technology over the past several decades, quantum control has attracted increasing attention due to the realization of controllable artificial quantum systems, which have opened up a new regime of complex quantum systems that can be engineered.Quantum control has found its use in controlled quantum-chemical processes and artificial quantum systems including quantum dots, superconducting qubits, trapped ions, and cavity optomechanical systems, etc., which are of considerable importance for future technology as candidates for sensors and quantum-computational devices.However, because quantum-mechanical systems are often too difficult to analytically deal with, heuristic strategies and generic numerical algorithms which search for proper control protocols are adopted.Therefore, deep learning, especially deep reinforcement learning, is a promising generic candidate solution for the control problems.Although there have already been a few successful examples of applications of deep reinforcement learning to quantum control problems, most of the existing reinforcement learning algorithms intrinsically suffer from instabilities and unsatisfactory reproducibility, and as a consequence, they typically require a large amount of fine-tuning and a large computational budget, both of which limit their applicability to quantum control problems and require expertise in machine learning.To resolve the issue of instabilities of the reinforcement learning algorithms, in this thesis, we first investigate the non-convergence issue of Q-learning, which is one of the most efficient reinforcement learning strategies.Then, we investigate the weakness of existing convergent approaches that have been proposed, and we develop a new convergent Q-learning algorithm, which we call the convergent deep Q network (C-DQN) algorithm, as an alternative to the conventional deep Q network (DQN) algorithm.We prove the convergence of the C-DQN algorithm, and since the algorithm is scalable and computationally efficient, we apply it to a standard reinforcement learning benchmark, the Atari 2600 benchmark, to demonstrate its effectiveness.We show that when the DQN algorithm fail, the C-DQN algorithm still learns successfully.Then, we apply the algorithm to the measurement-feedback cooling problems of a quantum-mechanical quartic oscillator i
The development of quantum computational techniques has advanced greatly in recent years, parallel to the advancements in techniques for deep reinforcement learning. This work explores the potential for quantum computing to facilitate reinforcement learning problems. Quantum computing approaches offer important potential improvements in time and space complexity over traditional algorithms because of its ability to exploit the quantum phenomena of superposition and entanglement. Specifically, we investigate the use of quantum variational circuits, a form of quantum machine learning. We present our techniques for encoding classical data for a quantum variational circuit, we further explore pure and hybrid quantum algorithms for DQN and Double DQN. Our results indicate both hybrid and pure quantum variational circuit have the ability to solve reinforcement learning tasks with a smaller parameter space. These comparison are conducted with two OpenAI Gym environments: CartPole and Blackjack, The success of this work is indicative of a strong future relationship between quantum machine learning and deep reinforcement learning.
The state-of-the-art machine learning approaches are based on classical von Neumann computing architectures and have been widely used in many industrial and academic domains. With the recent development of quantum computing, researchers and tech-giants have attempted new quantum circuits for machine learning tasks. However, the existing quantum computing platforms are hard to simulate classical deep learning models or problems because of the intractability of deep quantum circuits. Thus, it is necessary to design feasible quantum algorithms for quantum machine learning for noisy intermediate scale quantum (NISQ) devices. This work explores variational quantum circuits for deep reinforcement learning. Specifically, we reshape classical deep reinforcement learning algorithms like experience replay and target network into a representation of variational quantum circuits. Moreover, we use a quantum information encoding scheme to reduce the number of model parameters compared to classical neural networks. To the best of our knowledge, this work is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep Q-value function for decision-making and policy-selection reinforcement learning with experience replay and target network. Besides, our variational quantum circuits can be deployed in many near-term NISQ machines.
In the realm of artificial intelligence, deep rein-forcement learning (RL) agents struggle with generalizability and require substantial computational resources, unlike humans who easily adapt and generalize across tasks. To address these challenges, we introduce Quantum Rainbow, a hybrid algorithm that leverages the neural mechanisms of human decision-making and the efficiency of quantum computing. Quantum Rainbow combines variational quantum circuits with the Rainbow Deep Q-Network (DQN) model to create a novel approach in rein-forcement learning that integrates quantum principles into deep learning paradigms. We evaluate our model using behavioral experiments through the Iowa Gambling Task and 4-Armed Bandit Task. Our investigations reveal a significant relationship between the architecture of quantum circuits and the performance of quantum RL agents. Specifically, using causal discovery methods, we demonstrate the critical role of quantum entanglement in enhancing model performance. These findings not only show promising results but also pave the way for future explorations into optimizing quantum circuit architectures for reinforcement learning applications. This study underscores the potential of quantum-enhanced algorithms to achieve “quantum advantage” by addressing fundamental limitations in conventional deep RL methods.