In recent years, the utility of parameterized quantum circuits as function approximators has been widely studied. In the context of reinforcement learning, this approach has led to variational quantum algorithms such as quantum Q-learning. While these methods show promising empirical results, and can provide provable advantages for artificial problems, it remains unclear whether they can provide a provable quantum advantage over classical approaches for problems of practical relevance. A natural way to investigate this question is through the lens of dequantization: The construction of efficient classical algorithms capable of matching the performance of quantum variational methods. Building on recent kernel-based dequantization results for supervised learning, we take steps towards extending this surrogate-based dequantization program to reinforcement learning. Specifically, we study the simplified setting of reinforcement learning with a uniform generative model in which uniformly random state-action samples are available, which models the regime of sampling from a large experience replay buffer after sufficient exploration. Within this setting, we provide finite sample guarantees for classical kernelized Fitted Q-Iteration, with classical kernels designed to match the inductive bias of particular parameterized quantum circuits. Using these results, we then provide a set of sufficient conditions, on the data-encoding strategy of a parameterized quantum circuit, the corresponding classical kernel, and the problem structure, under which kernelized Fitted Q-Iteration provides a meaningful dequantization of quantum Q-learning, in this simplified setting. Apart from providing rigorous dequantization guarantees when these conditions are met, these results also motivate the use of kernelized fitted Q-iteration as a dequantization heuristic when these sufficient conditions cannot be verified.
In recent years, the utility of parameterized quantum circuits as function approximators has been widely studied. In the context of reinforcement learning, this approach has led to variational quantum algorithms such as quantum Q-learning. While these methods show promising empirical results, and can provide provable advantages for artificial problems, it remains unclear whether they can provide a provable quantum advantage over classical approaches for problems of practical relevance. A natural way to investigate this question is through the lens of dequantization: The construction of efficient classical algorithms capable of matching the performance of quantum variational methods. Building on recent kernel-based dequantization results for supervised learning, we take steps towards extending this surrogate-based dequantization program to reinforcement learning. Specifically, we study the simplified setting of reinforcement learning with a uniform generative model in which uniformly random state-action samples are available, which models the regime of sampling from a large experience replay buffer after sufficient exploration. Within this setting, we provide finite sample guarantees for classical kernelized Fitted Q-Iteration, with classical kernels designed to match the inductive bias of particular parameterized quantum circuits. Using these results, we then provide a set of sufficient conditions, on the data-encoding strategy of a parameterized quantum circuit, the corresponding classical kernel, and the problem structure, under which kernelized Fitted Q-Iteration provides a meaningful dequantization of quantum Q-learning, in this simplified setting. Apart from providing rigorous dequantization guarantees when these conditions are met, these results also motivate the use of kernelized fitted Q-iteration as a dequantization heuristic when these sufficient conditions cannot be verified.
Variational quantum circuits have recently gained popularity as quantum machine learning models. While considerable effort has been invested to train them in supervised and unsupervised learning settings, relatively little attention has been given to their potential use in reinforcement learning. In this work, we leverage the understanding of quantum policy gradient algorithms in a number of ways. First, we investigate how to construct and train reinforcement learning policies based on variational quantum circuits. We propose several designs for quantum policies, provide their learning algorithms, and test their performance on classical benchmarking environments. Second, we show the existence of task environments with a provable separation in performance between quantum learning agents and any polynomial-time classical learner, conditioned on the widely-believed classical hardness of the discrete logarithm problem. We also consider more natural settings, in which we show an empirical quantum advantage of our quantum policies over standard neural-network policies. Our results constitute a first step towards establishing a practical near-term quantum advantage in a reinforcement learning setting. Additionally, we believe that some of our design choices for variational quantum policies may also be beneficial to other models based on variational quantum circuits, such as quantum classifiers and quantum regression models.
There is currently a huge effort to understand the potential and limitations of variational quantum machine learning (QML) based on the optimization of parameterized quantum circuits. Recent proposals toward dequantizing variational QML models for regression problems include approaches based on kernel methods with carefully chosen kernel functions, approximated via Random Fourier Features (RFF). In this short note we show that for a wide range of instances, this approach can be simplified. More specifically, we show that the kernels whose evaluation is approximated by RFF in this dequantization algorithm can in fact often be evaluated exactly and efficiently classically. As such, our results enhance the toolkit for kernel-based dequantization of variational QML.
Reinforcement learning is playing a crucial role in sustainable developments and enabling wide applications in our daily lives ranging from self-driving cars, robotics, automation, personalised product suggestions, user notifications and many more. Kernel-based reinforcement learning holds theoretical guarantees of convergence to a unique solution for decision function approximation. This has motivated the development of implementations that exploit the advantage of kernel methods whilst overcoming the ‘curse of dimensionality’. The theoretical properties of kernel-based methods also support the combination of learning agents in loosely coupled systems into a single combined model for composable learning. Learning speedup is achievable by introducing expert knowledge to the algorithm through the kernel, which reduces variance by increasing bias and thus learning speed. Despite the theoretical promise of kernel based reinforcement learning methods, and continued evolution of more efficient algorithms, practical implementation are not prevalent. Research in quantum machine learning has successfully applied quantum algorithms to achieve computational speed-ups for various machine-learning tasks, and has also identified potential quantum advantage through the identification of patterns not accessible to classical data mapping. To date, this has not been applied to kernel based reinforcement learning. Recent work in the field of quantum learning suggests an important duality between quantum machine learning and kernel machine learning methods. Therefore, this paper proposes a framework to complement kernel-based reinforcement learning with quantum generated feature maps for function approximation to improve learning speed. This has positive consequences for optimisation of reinforcement learning techniques already in use to support sustainability targets in applications such as industry innovation and infrastructure, power generation management, data centre cooling and self-driving cars. Our established framework enables the adoption of emerging quantum kernel methods for reinforcement learning applications. The achieved results show over 20 % improvement and positive impacts of quantum map selection on model performance for different problems.
This chapter develops quantum reinforcement learning (QRL) as a principled extension of classical reinforcement learning (RL) into the quantum domain. Building on the standard Markov decision process framework, it shows how policies and value functions can be represented by quantum states and parameterized quantum circuits, with action probabilities arising from Born-rule measurement statistics and value estimates obtained from expectation values of observables. In this formulation, RL is preserved at the algorithmic level while its representational substrate is shifted from classical vector spaces to Hilbert space. The chapter emphasizes variational quantum circuits as the practical backbone of near-term QRL and analyzes how encoding strategies, entanglement structure, and measurement choices shape expressive power and trainability. It also examines quantum-enhanced exploration, showing how stochasticity emerges intrinsically from measurement rather than from externally injected noise. Policy-gradient and value-based approaches are compared, with particular attention to measurement cost, optimization stability, and near-term hardware suitability. This chapter further discusses quantum environments, sample-efficiency considerations, and hands-on implementations such as the quantum multi-armed bandit and quantum grid-world. Together, these topics establish QRL as a rigorous and practically grounded framework for studying sequential decision-making, control, and exploration in the quantum setting.
Quantum kernel methods (QKMs) have emerged as a prominent framework for supervised quantum machine learning. Unlike variational quantum algorithms, which rely on gradient-based optimisation and may suffer from issues such as barren plateaus, non-variational QKMs employ fixed quantum feature maps, with model selection performed classically via convex optimisation and cross-validation. This separation of quantum feature embedding from classical training ensures stable optimisation while leveraging quantum circuits to encode data in high-dimensional Hilbert spaces. In this review, we provide a thorough analysis of non-variational supervised QKMs, covering their foundations in classical kernel theory, constructions of fidelity and projected quantum kernels, and methods for their estimation in practice. We examine frameworks for assessing quantum advantage, including generalisation bounds and necessary conditions for separation from classical models, and analyse key challenges such as exponential concentration, dequantisation via tensor-network methods, and the spectral properties of kernel integral operators. We further discuss structured problem classes that may enable advantage, and synthesise insights from comparative and hardware studies. Overall, this review aims to clarify the regimes in which QKMs may offer genuine advantages, and to delineate the conceptual, methodological, and technical obstacles that must be overcome for practical quantum-enhanced learning.
The quest for successful variational quantum machine learning (QML) relies on the design of suitable parametrized quantum circuits (PQCs), as analogues to neural networks in classical machine learning. Successful QML models must fulfill the properties of trainability and non-dequantization, among others. Recent works have highlighted an intricate interplay between trainability and dequantization of such models, which is still unresolved. In this work we contribute to this debate from the perspective of machine learning, proving a number of results identifying, among others when trainability and non-dequantization are not mutually exclusive. We begin by providing a number of new somewhat broader definitions of the relevant concepts, compared to what is found in other literature, which are operationally motivated, and consistent with prior art. With these precise definitions given and motivated, we then study the relation between trainability and dequantization of variational QML. Next, we also discuss the degrees of "variationalness" of QML models, where we distinguish between models like the hardware efficient ansatz and quantum kernel methods. Finally, we introduce recipes for building PQC-based QML models which are both trainable and nondequantizable, and corresponding to different degrees of variationalness. We do not address the practical utility for such models. Our work however does point toward a way forward for finding more general constructions, for which finding applications may become feasible.