Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

Science

How to benchmark a quantum computer

Ian S. Osborne

Published Jul 24, 2014• 0 citations
Publisher

Abstract

Quantum Computing Quantum machines offer the possibility of performing certain computations much faster than their classical counterparts. However, how to define and measure quantum speedup is a topic of debate. Ronnow et al. describe methods for fairly evaluating the difference in computational power between classical and quantum processors. They define various types of quantum speedup and consider quantum processors that are designed to solve a specific class of problems. Science , this issue p. [420][1] [1]: /lookup/volpage/345/420?iss=6195

DOI
10.1126/science.345.6195.413-l

Related papers

Benchmarking quantum computers: towards a standard performance evaluation approach

Arturo Acuaviva, David Aguirre, Rubén Peña +1 more

Abstract The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a good benchmark such as relevance, reproducibility, fairness, verifiability and usability. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).

arXiv:2407.10941•Quantum Science and Technology•Feb 5, 2026•26 citations

Defining and detecting quantum speedup

Troels F. Rønnow, Zhihui Wang, Joshua Job +6 more

The development of small-scale quantum devices raises the question of how to fairly assess and detect quantum speedup. Here, we show how to define and measure quantum speedup and how to avoid pitfalls that might mask or fake such a speedup. We illustrate our discussion with data from tests run on a D-Wave Two device with up to 503 qubits. By using random spin glass instances as a benchmark, we found no evidence of quantum speedup when the entire data set is considered and obtained inconclusive results when comparing subsets of instances on an instance-by-instance basis. Our results do not rule out the possibility of speedup for other classes of problems and illustrate the subtle nature of the quantum speedup question.

arXiv:1401.2910•Science•Jun 20, 2014•632 citations

Towards a scalable quantum computer

Carmen G. Almudéver, Nader Khammassi, Louis Hutin +5 more

A quantum machine may solve some complex problems that are intractable for even the most powerful classical computers. By exploiting quantum superposition and entanglement phenomena, quantum algorithms can achieve from polynomial to exponential speed up when compared to their best classical counterparts. A quantum computer will be a part of a heterogeneous, multi-core computer in which a classical processor will interact with several accelerators such as FPGAs, GPUs and also a quantum co-processor. Figure 1 shows the different layers of the quantum computer system stack [1]. Building such a quantum system requires contributions from different fields such as physics, electronics, computer science and computer engineering for addressing the following challenges: i) build scalable quantum chips integrating qubits with long coherence times and high-fidelity operations, ii) develop classical control electronics at possibly cryogenic temperatures and iii) create the microarchitecture as well as the software for quantum computation.

International Conference on Design & Technology of Integrated Systems in Nanoscale Era (DTIS)•Apr 1, 2018•8 citations

Quantum outperforms classical

Jelena Stajic

Quantum Computing Quantum computers are expected to be better at solving certain computational problems than classical computers. This expectation is based on (well-founded) conjectures in computational complexity theory, but rigorous comparisons between the capabilities of quantum and classical algorithms are difficult to perform. Bravyi et al. proved theoretically that whereas the number of “steps” needed by parallel quantum circuits to solve certain linear algebra problems was independent of the problem size, this number grew logarithmically with size for analogous classical circuits (see the Perspective by Montanaro). This so-called quantum advantage stems from the quantum correlations present in quantum circuits that cannot be reproduced in analogous classical circuits. Science , this issue p. [308][1]; see also p. [289][2] [1]: /lookup/doi/10.1126/science.aar3106 [2]: /lookup/doi/10.1126/science.aau9555

Science•Oct 18, 2018•0 citations

Quantum Algorithms for Machine Learning and Optimization

P. Sangeetha, Prameela Kumari

Improvising the performance of machine learning for applications in the field of computer science leads to create new algorithms. As these are being optimized, using the algorithms of the classical machine learning in the area of quantum computing are being widely researched. First, we can consider general optimization problems with only functional Evaluations. Quantum Speedups of the problems can be improved compared to the classical algorithms. A frame work for quantum inspired classical algorithms can be introduced. To study the various problems on quantum computers on testing properties and compare the quantum speed ups with the classical counter parts.

PhD Colloquium on Ethically Driven Innovation and Technology for Society (PhD EDITS)•Nov 8, 2020•9 citations

Hybrid benchmarking of quantum algorithms

Andreea-Iulia Lefterovici

Combinatorial optimisation is a main topic for real-world applications, where fast solutions are required. Optimisation algorithms are incredibly powerful, but still face bottlenecks for certain problems. In recent years, quantum algorithms have been proposed as a potential candidate for solving such problems. The question that naturally arises is whether they could provide meaningful speed-ups over classical approaches. The study of asymptotic complexity has become central in the development of quantum algorithms. Nevertheless, having a promising asymptotic improvement does not directly translate into good practical performance. This is already well known for classical algorithms, which can be executed and evaluated directly. When it comes to quantum algorithms, the situation is more nuanced, as solving practically relevant instances would require a fully fault-tolerant quantum computer. In this thesis, we address the challenge of assessing the practical performance of a quantum algorithm without access to quantum hardware. Our strategy is to evaluate how these algorithms would perform under idealised fault-tolerant assumptions and to identify parameter regimes in which quantum approaches could potentially become useful. Our goal is to guide algorithmic development long before implementation becomes possible. We first give a general overview of the quantum (sub)routines that serve as building blocks for more advanced algorithms, as well as of the three hybrid benchmarking techniques used throughout this thesis. Then, we elaborate on each method: we start with the algorithmic descriptions, proceed to the methodology, and conclude with their hybrid evaluation on practically relevant datasets. Within the hybrid benchmarking method 1 (gate count analysis), we analytically derive the minimum number of gates that a quantum version of the classical simplex algorithm would require to solve realistic instances. By executing classical counterparts of the quantum subroutines, we infer the execution speed required per gate for the quantum simplex to outperform its classical counterpart. Within the hybrid benchmarking method 2 (query count analysis), we benchmark functional quantum linear solvers by analytically deriving a precise number of query calls made to the same oracles as a function of problem parameters. This provides insight into which algorithm might be more suitable in a given parameter regime. It allows for a direct comparison only between different quantum algorithms and is independent of classical runtime measurements. Within the hybrid benchmarking method 3 (cycle count analysis), we develop a runtime estimation framework tailored to the quantum tree generator–based search for solving 0-1-knapsack problems. We evaluate its performance on large scale instances beyond the reach of general simulation. We compare it with state-of-the-art classical linear programming solvers in terms of processor cycle counts. Hybrid benchmarking provides a fair, systematic benchmarking methodology, ensuring reproducible and relevant comparisons not only between classical and quantum approaches, but also among different quantum algorithms.

•Jan 1, 2026•0 citations
Quantum Computing
Quantum machines offer the possibility of performing certain computations much faster than their classical counterparts. However, how to define and measure quantum speedup is a topic of debate. Ronnow et al. describe methods for fairly evaluating the difference in computational power between classical and quantum processors. They define various types of quantum speedup and consider quantum processors that are designed to solve a specific class of problems.
Science , this issue p. 420
Abstract The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a good benchmark such as relevance, reproducibility, fairness, verifiability and usability. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).
The development of small-scale quantum devices raises the question of how to fairly assess and detect quantum speedup. Here, we show how to define and measure quantum speedup and how to avoid pitfalls that might mask or fake such a speedup. We illustrate our discussion with data from tests run on a D-Wave Two device with up to 503 qubits. By using random spin glass instances as a benchmark, we found no evidence of quantum speedup when the entire data set is considered and obtained inconclusive results when comparing subsets of instances on an instance-by-instance basis. Our results do not rule out the possibility of speedup for other classes of problems and illustrate the subtle nature of the quantum speedup question.
A quantum machine may solve some complex problems that are intractable for even the most powerful classical computers. By exploiting quantum superposition and entanglement phenomena, quantum algorithms can achieve from polynomial to exponential speed up when compared to their best classical counterparts. A quantum computer will be a part of a heterogeneous, multi-core computer in which a classical processor will interact with several accelerators such as FPGAs, GPUs and also a quantum co-processor. Figure 1 shows the different layers of the quantum computer system stack [1]. Building such a quantum system requires contributions from different fields such as physics, electronics, computer science and computer engineering for addressing the following challenges: i) build scalable quantum chips integrating qubits with long coherence times and high-fidelity operations, ii) develop classical control electronics at possibly cryogenic temperatures and iii) create the microarchitecture as well as the software for quantum computation.
Quantum Computing Quantum computers are expected to be better at solving certain computational problems than classical computers. This expectation is based on (well-founded) conjectures in computational complexity theory, but rigorous comparisons between the capabilities of quantum and classical algorithms are difficult to perform. Bravyi et al. proved theoretically that whereas the number of “steps” needed by parallel quantum circuits to solve certain linear algebra problems was independent of the problem size, this number grew logarithmically with size for analogous classical circuits (see the Perspective by Montanaro). This so-called quantum advantage stems from the quantum correlations present in quantum circuits that cannot be reproduced in analogous classical circuits. Science , this issue p. [308][1]; see also p. [289][2] [1]: /lookup/doi/10.1126/science.aar3106 [2]: /lookup/doi/10.1126/science.aau9555
Improvising the performance of machine learning for applications in the field of computer science leads to create new algorithms. As these are being optimized, using the algorithms of the classical machine learning in the area of quantum computing are being widely researched. First, we can consider general optimization problems with only functional Evaluations. Quantum Speedups of the problems can be improved compared to the classical algorithms. A frame work for quantum inspired classical algorithms can be introduced. To study the various problems on quantum computers on testing properties and compare the quantum speed ups with the classical counter parts.
Combinatorial optimisation is a main topic for real-world applications, where fast solutions are required. Optimisation algorithms are incredibly powerful, but still face bottlenecks for certain problems. In recent years, quantum algorithms have been proposed as a potential candidate for solving such problems. The question that naturally arises is whether they could provide meaningful speed-ups over classical approaches. The study of asymptotic complexity has become central in the development of quantum algorithms. Nevertheless, having a promising asymptotic improvement does not directly translate into good practical performance. This is already well known for classical algorithms, which can be executed and evaluated directly. When it comes to quantum algorithms, the situation is more nuanced, as solving practically relevant instances would require a fully fault-tolerant quantum computer. In this thesis, we address the challenge of assessing the practical performance of a quantum algorithm without access to quantum hardware. Our strategy is to evaluate how these algorithms would perform under idealised fault-tolerant assumptions and to identify parameter regimes in which quantum approaches could potentially become useful. Our goal is to guide algorithmic development long before implementation becomes possible. We first give a general overview of the quantum (sub)routines that serve as building blocks for more advanced algorithms, as well as of the three hybrid benchmarking techniques used throughout this thesis. Then, we elaborate on each method: we start with the algorithmic descriptions, proceed to the methodology, and conclude with their hybrid evaluation on practically relevant datasets. Within the hybrid benchmarking method 1 (gate count analysis), we analytically derive the minimum number of gates that a quantum version of the classical simplex algorithm would require to solve realistic instances. By executing classical counterparts of the quantum subroutines, we infer the execution speed required per gate for the quantum simplex to outperform its classical counterpart. Within the hybrid benchmarking method 2 (query count analysis), we benchmark functional quantum linear solvers by analytically deriving a precise number of query calls made to the same oracles as a function of problem parameters. This provides insight into which algorithm might be more suitable in a given parameter regime. It allows for a direct comparison only between different quantum algorithms and is independent of classical runtime measurements. Within the hybrid benchmarking method 3 (cycle count analysis), we develop a runtime estimation framework tailored to the quantum tree generator–based search for solving 0-1-knapsack problems. We evaluate its performance on large scale instances beyond the reach of general simulation. We compare it with state-of-the-art classical linear programming solvers in terms of processor cycle counts. Hybrid benchmarking provides a fair, systematic benchmarking methodology, ensuring reproducible and relevant comparisons not only between classical and quantum approaches, but also among different quantum algorithms.