In this article, a novel training paradigm inspired by quantum computation is proposed for deep reinforcement learning (DRL) with experience replay. In contrast to the traditional experience replay mechanism in DRL, the proposed DRL with quantum-inspired experience replay (DRL-QER) adaptively chooses experiences from the replay buffer according to the complexity and the replayed times of each experience (also called transition), to achieve a balance between exploration and exploitation. In DRL-QER, transitions are first formulated in quantum representations and then the preparation operation and depreciation operation are performed on the transitions. In this process, the preparation operation reflects the relationship between the temporal-difference errors (TD-errors) and the importance of the experiences, while the depreciation operation is taken into account to ensure the diversity of the transitions. The experimental results on Atari 2600 games show that DRL-QER outperforms state-of-the-art algorithms, such as DRL-PER and DCRL on most of these games with improved training efficiency and is also applicable to such memory-based DRL approaches as double network and dueling network.
In this article, a novel training paradigm inspired by quantum computation is proposed for deep reinforcement learning (DRL) with experience replay. In contrast to the traditional experience replay mechanism in DRL, the proposed DRL with quantum-inspired experience replay (DRL-QER) adaptively chooses experiences from the replay buffer according to the complexity and the replayed times of each experience (also called transition), to achieve a balance between exploration and exploitation. In DRL-QER, transitions are first formulated in quantum representations and then the preparation operation and depreciation operation are performed on the transitions. In this process, the preparation operation reflects the relationship between the temporal-difference errors (TD-errors) and the importance of the experiences, while the depreciation operation is taken into account to ensure the diversity of the transitions. The experimental results on Atari 2600 games show that DRL-QER outperforms state-of-the-art algorithms, such as DRL-PER and DCRL on most of these games with improved training efficiency and is also applicable to such memory-based DRL approaches as double network and dueling network.
Reinforcement learning (RL) encounters significant challenges related to scalability and computational efficiency, particularly in complex decision-making environments. Traditional RL algorithms often struggle with large-scale tasks, necessitating innovative approaches to enhance adaptability and performance. The quantum circuit-based priority replay (QCPR) algorithm was introduced in this study, leveraging quantum-inspired techniques to enhance learning efficiency and decision-making quality through quantum computing (QC). The QCPR algorithm implements two key mechanisms for prioritizing experiences: the magnitude and direction (MaDi) based priority, which assesses the significance and directionality of experiences, and QCPR-specific methods that ensure efficient experience replay. These mechanisms are seamlessly integrated into the Q-learning (QL) framework to optimize the RL process for superior performance. QCPR was evaluated against four well-established RL algorithms-QL, deep Q-networks (DQN), prioritized experience replay (PER), and dueling DQN (DDQN)-using standard decision-making benchmarks. The algorithm was further tested in various simulation and real-world environments, including Atari games, Qiskit integration, and Raspberry Pi hardware and software setups. These evaluations demonstrated QCPR’s adaptability and robustness, showcasing its capability for dynamic, large-scale applications. The results revealed that QCPR significantly outperforms its counterparts across key performance metrics, including accuracy, precision, recall, F1-score, and cumulative rewards. Specifically, QCPR achieved a 15.29% improvement in accuracy over QL, a 9.98% increase over DQN, a 6.33% improvement over PER, and an 8.23% enhancement over DDQN. This study highlights the potential of quantum-inspired approaches to advance RL, offering scalable and efficient solutions for complex decision-making tasks.
Abstract: Quantum deep reinforcement learning is an integration of the principles of quantum computing with deep reinforcement learning in an effort to advance robot navigation tasks. Due to the explosion of the state and action spaces, traditionally DRL methods are limited in scalability and efficiency. QDRL, however, uses quantum superposition and entanglement for more efficient processing of big amounts of information-it enables robots to learn optimal policies for navigating complex environments. We introduce here a new QDRL framework in which quantum neural networks encode policy functions and value estimates. We demonstrate how quantum circuits can take advantage of multi-dimensional state representations in order to strengthen the power of a robot in changing its environment over time. Benchmark experiments on navigation tasks reflect phenomenal acceleration in learning performance along with better decision-making accuracy compared to classical DRL methods. We then consider integrating such quantum algorithms, as Grover's search, to improve exploration strategies in sparse reward settings. The experiments show that QDRL speeds up convergence rates and provides robots with the possibility to react more rapidly to unexpected obstacles. It discusses new applications for quantum computing of autonomous systems, potentially applied to robotics, autonomous vehicles, and other environments which require complex decision making. Some lines of work in the future are optimization of the resources needed for quantum computers and hybrid architectures for the resolution of classical problems in navigation
Deep reinforcement learning (DRL) has been recognized as a powerful tool in quantum physics, where DRL's reward design is nontrivial but crucial for quantum control tasks. To address the problem of over-reliance on human empirical knowledge to design DRL's rewards, we propose a DRL with a novel reward paradigm designed by the learning process information (DRL-LPI), where the learning process information (LPI) comprises the state information and the experiences. In DRL-LPI, the state information after being classified by a fidelity threshold, and the experiences are first stored simultaneously in the respective sequences, and this process is repeated until a similar-segment ends. Then, the stored state information is converted to the real value and used to design the reward value by applying a self-amplitude function. Next, the designed reward values are integrated with the stored experiences to compose transitions for DRL's training. Through comparisons to five representative reward schemes, the proposed DRL-LPI is evaluated on two typical quantum control tasks, i.e., the spin-(1/2) quantum state control and many-coupled qubits state control, and the experimental results show the superior learning efficiency and control performance of the proposed approach. More results show that DRL-LPI exhibits the ability to learn the control strategy with few control actions compared to stochastic gradient descent (SGD) and genetic algorithm (GA).
Deep Reinforcement Learning (DRL) is a sub-field of Machine Learning (ML) that combines reinforcement learning (RL) with deep learning techniques. DRL has showcased promising results in gaming environments and real-world applications. However, DRL agents are susceptible to faulty observations due to sudden interference like blackouts, frozen observation, or adversarial interference in practical applications. These scenarios can hamper the learning and performance of DRL agents if they are not resilient. Drawing inspiration from causal inference, the Causal In-ference Q-Network (CIQ) explores a Deep Q-Network (DQN) framework that undergoes training with an additional task focusing on training for observational interference. Through an evaluation conducted in various benchmark DQN environments, the experimental findings demonstrate that the CIQ method exhibits enhanced performance and greater resilience against observational interference. In this paper, quantum networks and hybrid quantum-classical networks are tested for their resilience in noisy environments. A novel Quantum-Enhanced CIQ-Network (QCIQ) is proposed which could replicate the functioning of the CIQ using Variational Quantum Circuits (VQCs) and is benchmarked against the corresponding classical networks for its performance.
The rapid expansion of large-scale data analytics (LSDA) has presented significant issues concerning computing efficiency, scalability, adaptive learning, and model interpretability. In recent years, quantum-inspired deep reinforcement learning (QIDRL) has emerged as a promising research paradigm that integrates quantum-inspired optimization (QIO) concepts with deep reinforcement learning (DRL) approaches to enhance intelligent decision-making and complicated data processing. The need for transparent and reliable artificial intelligence (AI) systems has heightened the significance of interpretable learning frameworks that can elucidate model behavior and prediction results. This work offers a thorough examination of interpretable QIDRL methods for extensive data analysis. The review methodically examines current advancements in QIO models, interpretable RL architectures, policy explanation mechanisms, and scalable deep learning (DL) frameworks used for high-dimensional and dynamic datasets. The report examines the use of these methodologies across many sectors, including healthcare analytics, cybersecurity, smart transportation, industrial automation, financial forecasting, and intelligent Internet of Things (IoT) ecosystems. The study examines the benefits of QIDRL techniques in improving convergence speed, adaptive decision-making, feature optimization, and interpretability inside extensive analytical systems. Furthermore, significant research obstacles like computational complexity, training instability, scalability limitations, insufficient benchmark datasets, and the absence of defined criteria for interpretability assessment are thoroughly examined. The study emphasizes new research trends and prospects for creating robust, efficient, and transparent QIDRL frameworks for advanced intelligent DA applications.
With the evolution of technology, the need to solve complex computational problems like machine learning and deep learning has shot up. However, even the most powerful classical supercomputers find it difficult to execute these tasks. Advancements in quantum computing are leading researchers and tech-giants who strive for better quantum circuits to do machine learning tasks. Current works on Quantum Machine Learning (QML) ensure less memory consumption and reduced model parameters. However, it is strenuous to simulate the classical deep learning approach on existing quantum computing devices due to the inflexibility of Deep quantum circuits. Consequently, designing viable quantum algorithms for QML for noisy intermediate-scale quantum (NISQ) devices is essential. The proposed work aims to explore Variational Quantum Circuits (VQC) for Deep Q network-based Reinforcement Learning by remodeling the target network and experience replay into a representation of VQC. In addition, to the reduction in model parameters, quantum information encoding schemes are used to achieve better results than classical neural networks. VQCs are employed for policyselection and decision-making reinforcement learning by approximating the deep Q-value function with target network and experience replay.