Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2605.19736 · IEEE/ACM International Conference on Automated Software Engineering (ASE)

QUTest: A Native Testing Framework for Quantum Programs

José Campos

quant-phcs.SESubmitted May 19, 2026• Announced May 20, 2026• 0 citations
arXivHTMLPDF

Abstract

Quantum programs are often shared as OpenQASM 3 circuits, but tests are still written in host languages such as Python with Qiskit. We present QUTest, a native framework in which both programs and tests are standard .qasm files. Tests follow the Arrange / Act / Assert pattern, while configuration, runtime requirements, and assertions are encoded as pragma comments (//%), preserving compatibility with existing OpenQASM tools. QUTest provides 12 assertion types spanning deterministic, statistical, quantum-state, and structural checks, plus a linter and an environment-aware mode for running the same test across selected runtime versions in isolated environments. Its CLI supports automatic test discovery, runtime compatibility checks, and XML reports for continuous integration. We describe the pragma language, implementation, and a planned evaluation using coverage and mutation testing. QUTest is available at https://github.com/QBugs/qutest. Video demo: https://youtu.be/FvgvsiAXuW0.

Related papers

QuCheck: A Property-based Testing Framework for Quantum Programs in Qiskit

Gabriel Pontolillo, Mohammad Reza Mousavi, Marek Grzesiuk

Property-based testing has been previously proposed for quantum programs in Q# with QSharpCheck; however, this implementation was limited in functionality, lacked extensibility, and was evaluated on a narrow range of programs using a single property. To address these limitations, we propose QuCheck, an improved property-based testing framework for Qiskit. By leveraging Qiskit and the broader Python ecosystem, QuCheck facilitates property construction, introduces flexible input generators and assertions, and supports expressive preconditions. We assessed its effectiveness through mutation analysis on five quantum programs (2-10 qubits), varying the number of properties, inputs, and measurement shots to assess their impact on fault detection and demonstrate the effectiveness of property-based testing across a range of conditions. Results show a strong positive correlation between the mutation score (a measure of fault detection) and number of properties evaluated, with a moderate negative correlation between the false positive rate and number of measurement shots. The most thorough test configurations achieved a mean mutation score of 0.90 averaged across all five algorithms, with a false positive rate between 0 and 0.06. QuCheck identified 47.8% more faults than QSharpCheck, with execution time reduced by 67.4%. These findings highlight the viability of property-based testing for verifying quantum systems.

arXiv:2503.22641•ACM Transactions on Quantum Computing•May 8, 2026•17 citations

QMutPy: a mutation testing tool for Quantum algorithms and applications in Qiskit

Daniel Fortunato, José Campos, Rui Abreu

There is an inherent lack of knowledge and technology to test a quantum program properly. In this paper, building on the definition of syntactically equivalent quantum gates, we describe our efforts in developing a tool, coined QMutPy, leveraging the well-known open-source mutation tool MutPy. We further discuss the design and implementation of QMutPy, and the usage of a novel set of mutation operators that generate mutants for qubit measurements and gates. To evaluate QMutPy’s performance, we conducted a preliminary study on 11 real quantum programs written in the IBM’s Qiskit library. QMutPy has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests. QMutPy is publicly available at https://github.com/danielfobooss/mutpy. Tool demo: https://youtu.be/fC4tOY5trqc.

ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)•Jul 15, 2022•72 citations

Mutation Testing of Quantum Programs: A Case Study With Qiskit

Daniel Fortunato, José Campos, Rui Abreu

As quantum computing is still in its infancy, there is an inherent lack of knowledge and technology to test a quantum program properly. In the classical realm, mutation testing has been successfully used to evaluate how well a program's test suite detects seeded faults (i.e., mutants). In this paper, building on the definition of syntactically equivalent quantum operations, we propose a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further propose QMUTPY, an extension of the well-known and fully automated open-source mutation tool MUTPY. To evaluate QMUTPY's performance, we conducted a case study on 24 real quantum programs written in IBM's QISKIT library. Furthermore, we show how better test suite coverage and improvements to test assertions can increase the test suites' mutation score and quality. QMUTPY has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests and how to improve them.

IEEE Transactions on Quantum Engineering•Jan 1, 2022•83 citations

Muskit: A Mutation Analysis Tool for Quantum Software Testing

Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini +1 more

Given that quantum software testing is a new area of research, there is a lack of benchmark programs and bugs repositories to assess the effectiveness of testing techniques. To this end, quantum mutation analysis focuses on systematically generating faulty versions of Quantum Programs (QPs), called mutants, using mutation operators. Such mutants can be used as benchmarks to assess the quality of test cases in a test suite. Thus, we present Muskit - a quantum mutation analysis tool for QPs coded in IBM's Qiskit language. Muskit defines mutation operators on gates of QPs and selection criteria to reduce the number of mutants to generate. Moreover, it allows for the execution of test cases on mutants and generation of results for test analyses. Muskit is provided as command line interface, GUI, and web application. We validated Muskit by using it to generate and execute mutants for four QPs. Muskit code: https://github.com/Simula-COMPLEX/muskitWeb app: https://qiskitmutantcreatorsrl.pythonanywhere.com/YouTube Video: EbPHJOK_AEA Artifact Available: https://doi.org/10.5281/zenodo.5288917

IEEE/ACM International Conference on Automated Software Engineering (ASE)•Nov 1, 2021•96 citations

A Framework for the Efficient Evaluation of Runtime Assertions on Quantum Computers

Damian Rovara, Lukas Burgholzer, Robert Wille

The continuous growth of quantum computing and the increasingly complex quantum programs resulting from it lead to unprecedented obstacles in ensuring program correctness. Runtime assertions are, therefore, becoming a crucial tool in the development of quantum programs. They assist developers in the debugging process and help to test and verify the program. However, while assertions can be implemented in a straightforward manner on classical computers, physical limitations of quantum computers pose considerable challenges for the evaluation of quantum assertions. Access to the quantum state of a program is limited, execution time is expensive and noise can significantly distort measurement outcomes. To address these problems, this work proposes a framework that assists developers in the evaluation of runtime assertions on real quantum computers. It translates a variety of assertions into sets of measurements, reduces execution overhead where possible and evaluates the measurement results after the execution even in the presence of noise. This approach substantially aids developers in the debugging process, enabling efficient assertion-driven debugging even in large programs. The proposed framework is available as an open-source implementation at https://github.com/munich-quantum-toolkit/debugger

arXiv:2505.03885•IEEE International Conference on Quantum Software (QSW)•Jul 1, 2026•6 citations

Testing Quantum Programs with Multiple Subroutines

Peixun Long, Jianjun Zhao

Errors in quantum programs are challenging to track down due to the uncertainty of quantum programs. Testing is, therefore, an indispensable method for assuring the quality of quantum software. Existing testing methods focus only on testing quantum programs with quantum circuits or single subroutines and, therefore, cannot effectively test quantum programs with multi-subroutines. In this paper, we first discuss several critical issues that must be considered when testing multi-subroutine quantum programs and point out the limitations and problems with existing testing methods. We then present a novel framework for testing multi-subroutine quantum programs that allow for both unit and integration testing. Our framework includes two novel test coverage criteria for the equivalent class partition of quantum variables to guide our testing tasks and techniques to test quantum programs with several common patterns. We also discuss how to generate test cases based on our framework. To evaluate the effectiveness of our testing framework, we implemented a tool called QSharpTester for testing Q\# programs with multiple subroutines. We used it to conduct experiments on hundreds of mutation programs deriving from seven original Q\# programs. The experimental results show that our testing methods can deal with broader types of quantum programs than existing ones and perform well on almost all faulty mutation programs.

arXiv:2208.09206•Aug 19, 2022•15 citations
Quantum programs are often shared as OpenQASM 3 circuits, but tests are still written in host languages such as Python with Qiskit. We present QUTest, a native framework in which both programs and tests are standard .qasm files. Tests follow the Arrange / Act / Assert pattern, while configuration, runtime requirements, and assertions are encoded as pragma comments (//%), preserving compatibility with existing OpenQASM tools. QUTest provides 12 assertion types spanning deterministic, statistical, quantum-state, and structural checks, plus a linter and an environment-aware mode for running the same test across selected runtime versions in isolated environments. Its CLI supports automatic test discovery, runtime compatibility checks, and XML reports for continuous integration. We describe the pragma language, implementation, and a planned evaluation using coverage and mutation testing.
QUTest is available at https://github.com/QBugs/qutest.
Video demo: https://youtu.be/FvgvsiAXuW0.
Property-based testing has been previously proposed for quantum programs in Q# with QSharpCheck; however, this implementation was limited in functionality, lacked extensibility, and was evaluated on a narrow range of programs using a single property. To address these limitations, we propose QuCheck, an improved property-based testing framework for Qiskit. By leveraging Qiskit and the broader Python ecosystem, QuCheck facilitates property construction, introduces flexible input generators and assertions, and supports expressive preconditions. We assessed its effectiveness through mutation analysis on five quantum programs (2-10 qubits), varying the number of properties, inputs, and measurement shots to assess their impact on fault detection and demonstrate the effectiveness of property-based testing across a range of conditions. Results show a strong positive correlation between the mutation score (a measure of fault detection) and number of properties evaluated, with a moderate negative correlation between the false positive rate and number of measurement shots. The most thorough test configurations achieved a mean mutation score of 0.90 averaged across all five algorithms, with a false positive rate between 0 and 0.06. QuCheck identified 47.8% more faults than QSharpCheck, with execution time reduced by 67.4%. These findings highlight the viability of property-based testing for verifying quantum systems.
There is an inherent lack of knowledge and technology to test a quantum program properly. In this paper, building on the definition of syntactically equivalent quantum gates, we describe our efforts in developing a tool, coined QMutPy, leveraging the well-known open-source mutation tool MutPy. We further discuss the design and implementation of QMutPy, and the usage of a novel set of mutation operators that generate mutants for qubit measurements and gates. To evaluate QMutPy’s performance, we conducted a preliminary study on 11 real quantum programs written in the IBM’s Qiskit library. QMutPy has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests. QMutPy is publicly available at https://github.com/danielfobooss/mutpy. Tool demo: https://youtu.be/fC4tOY5trqc.
As quantum computing is still in its infancy, there is an inherent lack of knowledge and technology to test a quantum program properly. In the classical realm, mutation testing has been successfully used to evaluate how well a program's test suite detects seeded faults (i.e., mutants). In this paper, building on the definition of syntactically equivalent quantum operations, we propose a novel set of mutation operators to generate mutants based on qubit measurements and quantum gates. To ease the adoption of quantum mutation testing, we further propose QMUTPY, an extension of the well-known and fully automated open-source mutation tool MUTPY. To evaluate QMUTPY's performance, we conducted a case study on 24 real quantum programs written in IBM's QISKIT library. Furthermore, we show how better test suite coverage and improvements to test assertions can increase the test suites' mutation score and quality. QMUTPY has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests and how to improve them.
Given that quantum software testing is a new area of research, there is a lack of benchmark programs and bugs repositories to assess the effectiveness of testing techniques. To this end, quantum mutation analysis focuses on systematically generating faulty versions of Quantum Programs (QPs), called mutants, using mutation operators. Such mutants can be used as benchmarks to assess the quality of test cases in a test suite. Thus, we present Muskit - a quantum mutation analysis tool for QPs coded in IBM's Qiskit language. Muskit defines mutation operators on gates of QPs and selection criteria to reduce the number of mutants to generate. Moreover, it allows for the execution of test cases on mutants and generation of results for test analyses. Muskit is provided as command line interface, GUI, and web application. We validated Muskit by using it to generate and execute mutants for four QPs. Muskit code: https://github.com/Simula-COMPLEX/muskitWeb app: https://qiskitmutantcreatorsrl.pythonanywhere.com/YouTube Video: EbPHJOK_AEA Artifact Available: https://doi.org/10.5281/zenodo.5288917
The continuous growth of quantum computing and the increasingly complex quantum programs resulting from it lead to unprecedented obstacles in ensuring program correctness. Runtime assertions are, therefore, becoming a crucial tool in the development of quantum programs. They assist developers in the debugging process and help to test and verify the program. However, while assertions can be implemented in a straightforward manner on classical computers, physical limitations of quantum computers pose considerable challenges for the evaluation of quantum assertions. Access to the quantum state of a program is limited, execution time is expensive and noise can significantly distort measurement outcomes. To address these problems, this work proposes a framework that assists developers in the evaluation of runtime assertions on real quantum computers. It translates a variety of assertions into sets of measurements, reduces execution overhead where possible and evaluates the measurement results after the execution even in the presence of noise. This approach substantially aids developers in the debugging process, enabling efficient assertion-driven debugging even in large programs. The proposed framework is available as an open-source implementation at https://github.com/munich-quantum-toolkit/debugger
Errors in quantum programs are challenging to track down due to the uncertainty of quantum programs. Testing is, therefore, an indispensable method for assuring the quality of quantum software. Existing testing methods focus only on testing quantum programs with quantum circuits or single subroutines and, therefore, cannot effectively test quantum programs with multi-subroutines. In this paper, we first discuss several critical issues that must be considered when testing multi-subroutine quantum programs and point out the limitations and problems with existing testing methods. We then present a novel framework for testing multi-subroutine quantum programs that allow for both unit and integration testing. Our framework includes two novel test coverage criteria for the equivalent class partition of quantum variables to guide our testing tasks and techniques to test quantum programs with several common patterns. We also discuss how to generate test cases based on our framework. To evaluate the effectiveness of our testing framework, we implemented a tool called QSharpTester for testing Q\# programs with multiple subroutines. We used it to conduct experiments on hundreds of mutation programs deriving from seven original Q\# programs. The experimental results show that our testing methods can deal with broader types of quantum programs than existing ones and perform well on almost all faulty mutation programs.