Valeria Saggio, Beate E. Asenbeck, A. Hamann, Teodor Strömberg, Peter Schiansky, Vedran Dunjko, Nicolai Friis, Nicholas C. Harris, M. Hochberg, Dirk Robert Englund, Sabine Wölk, Hans J. Briegel, Philip John Walther
In reinforcement learning, a key question for applications is how fast agents can learn. By introducing an agent capable of interacting classically as well as quantum-mechanically with its environment, we experimentally prove that a speed-up in the agent’s learning time is possible.
As the field of artificial intelligence is pushed forward, the question arises of how fast autonomous machines can learn. Within artificial intelligence, an important paradigm is reinforcement learning, where agents - learning entities capable of decision making - interact with the world they are placed in, called an environment. Thanks to these interactions, agents receive feedback from the environment and thus progressively adjust their behaviour to accomplish a given goal. An important question in reinforcement learning is how fast agents can learn to fulfill their tasks. To answer this question we consider a novel reinforcement learning framework where quantum mechanics is used. In particular, we quantize the agent and the environment and grant them the possibility to also interact quantum-mechanically, that is, by using a quantum channel for their communication. We demonstrate that this feature enables a speed-up in the agent's learning process, and we further show that combining this scenario with classical communication enables the evaluation of such an improvement. This learning protocol is implemented on an integrated re-programmable photonic platform interfaced with photons at telecommunication wavelengths. Thanks to the full tunability of the device, this platform proves the best candidate for the implementation of learning protocols, where a continuous update of the learning process is required.
Combining quantum computing techniques in the form of amplitude amplification with classical reinforcement learning has led to the so-called “hybrid agent for quantum-accessible reinforcement learning”, which achieves a quadratic speedup in sample complexity for certain learning problems. So far, this hybrid agent has only been applied to stationary learning problems, that is, learning problems without any time dependency within components of the Markov decision process. In this work, we investigate the applicability of the hybrid agent to dynamic RL environments and thus to more realistic scenarios. To this end, we enhance the hybrid agent by introducing a dissipation mechanism and, with the resulting learning agent, perform an empirical comparison with a classical RL agent in an RL environment with a time-dependent reward function. Our findings suggest that the modified hybrid agent can adapt its behavior to changes in the environment quickly, leading to a higher average success probability compared to its classical counterpart.
Recent work on quantum machine learning has demonstrated that quantum computers can offer dramatic improvements over classical devices for data mining, prediction and classification. However, less is known about the advantages using quantum computers may bring in the more general setting of reinforcement learning, where learning is achieved via interaction with a task environment that provides occasional rewards. Reinforcement learning can incorporate data-analysis-oriented learning settings as special cases, but also includes more complex situations where, e.g., reinforcing feedback is delayed. In a few recent works, Grover-type amplification has been utilized to construct quantum agents that achieve up-to-quadratic improvements in learning efficiency. These encouraging results have left open the key question of whether super-polynomial improvements in learning times are possible for genuine reinforcement learning problems, that is problems that go beyond the other more restricted learning paradigms. In this work, we provide a family of such genuine reinforcement learning tasks. We construct quantum-enhanced learners which learn super-polynomially, and even exponentially faster than any classical reinforcement learning model, and we discuss the potential impact our results may have on future technologies.
In this thesis, we investigate the speedup we can achieve with quantum enhanced reinforcement learning compared to classical reinforcement learning. Reinforcement learning is a classical main model in machine learning and is based on the concept of trial and error, as well as receiving rewards for a correct behavior to learn speciffic tasks. In this thesis a quantum enhanced agent is implemented to play the game of tic-tac-toe. The example of tic-tac-toe is used to analyze the learning performance of the agent. The main goal of this thesis is to speed-up the learning time of the quantum enhanced agent in this setting, which means to reduce the number of games it has to play until it has learned. Within the process, the learning time of the quantum enhanced agent is compared to a completely classical agent. The comparison between those types of agents is made trough a simulation on a classical computer.
In recent years, quantum-enhanced machine learning has emerged as a particularly fruitful application of quantum algorithms, covering aspects of supervised, unsupervised and reinforcement learning. Reinforcement learning offers numerous options of how quantum theory can be applied, and is arguably the least explored, from a quantum perspective. Here, an agent explores an environment and tries to find a behavior optimizing some figure of merit. Some of the first approaches investigated settings where this exploration can be sped-up, by considering quantum analogs of classical environments, which can then be queried in superposition. If the environments have a strict periodic structure in time (i.e. are strictly episodic), such environments can be effectively converted to conventional oracles encountered in quantum information. However, in general environments, we obtain scenarios that generalize standard oracle tasks. In this work, we consider one such generalization, where the environment is not strictly episodic, which is mapped to an oracle identification setting with a changing oracle. We analyze this case and show that standard amplitude-amplification techniques can, with minor modifications, still be applied to achieve quadratic speed-ups. In addition, we prove that an algorithm based on Grover iterations is optimal for oracle identification even if the oracle changes over time in a way that the "rewarded space" is monotonically increasing. This result constitutes one of the first generalizations of quantum-accessible reinforcement learning.
Reinforcement learning is a machine learning paradigm where an agent learns how to optimize its behavior solely through its interaction with the environment. It has been extensively studied and successfully applied to complex problems of many different domains in the past decades, i.e., robotics, games, scheduling. However, the performance of these algorithms becomes limited as the complexity and dimension of the state-action space increases. Recent advances in quantum computing and quantum information have sparked interest in possible applications to machine learning. By taking advantage of quantum mechanics, it is possible to efficiently process immense quantities of information and improve computational speed. In this work, we combined quantum computing with reinforcement learning and studied its application to a board game to assess the benefits that it can introduce, namely its impact on the learning efficiency of an agent. From the results, we concluded that the proposed quantum exploration policy improved the convergence rate of the agent and promoted a more efficient exploration of the state space.