Various fields of science face a reproducibility crisis. For quantum software engineering as an emerging field, it is therefore imminent to focus on proper reproducibility engineering from the start. Yet the provision of reproduction packages is almost universally lacking. Actionable advice on how to build such packages is rare, particularly unfortunate in a field with many contributions from researchers with backgrounds outside computer science. In this article, we argue how to rectify this deficiency by proposing a 1-2-3 approach to reproducibility engineering for quantum software experiments: Using a meta-generation mechanism, we generate DOI-safe, long-term functioning and dependency-free reproduction packages. They are designed to satisfy the requirements of professional and learned societies solely on the basis of project-specific research artefacts (source code, measurement and configuration data), and require little temporal investment by researchers. Our scheme ascertains long-term traceability even when the quantum processor itself is no longer accessible. By drastically lowering the technical bar, we foster the proliferation of reproduction packages in quantum software experiments and ease the inclusion of non-CS researchers entering the field.
Various fields of science face a reproducibility crisis. For quantum software engineering as an emerging field, it is therefore imminent to focus on proper reproducibility engineering from the start. Yet the provision of reproduction packages is almost universally lacking. Actionable advice on how to build such packages is rare, particularly unfortunate in a field with many contributions from researchers with backgrounds outside computer science. In this article, we argue how to rectify this deficiency by proposing a 1-2-3 approach to reproducibility engineering for quantum software experiments: Using a meta-generation mechanism, we generate DOI-safe, long-term functioning and dependency-free reproduction packages. They are designed to satisfy the requirements of professional and learned societies solely on the basis of project-specific research artefacts (source code, measurement and configuration data), and require little temporal investment by researchers. Our scheme ascertains long-term traceability even when the quantum processor itself is no longer accessible. By drastically lowering the technical bar, we foster the proliferation of reproduction packages in quantum software experiments and ease the inclusion of non-CS researchers entering the field.
Background. In the realm of software engineering, there are widely accepted guidelines for reporting and creating laboratory packages. Unfortunately, the landscape differs considerably in the emerging field of quantum computing. To the best of our knowledge, no standardized guidelines exist for describing experiments or outlining the necessary structures for quantum software laboratory packages. Aims. This paper endeavors to enhance the replicability and verifiability of quantum software experiments. Method. This objective is pursued through the proposition of guidelines for reporting and the delineation of a structure for laboratory packages tailored to quantum computing experiments. Specifically, we advocate for an extension and adaption of established guidelines in experimental software engineering, integrating novel elements to address the specific requirements of quantum software engineering. Results. In validating the utility and effectiveness of the proposed guidelines, we conducted a review encompassing 11 works (5 focusing on reporting guidelines and 6 on laboratory packages). In particular, this review highlighted the absence of standardized guidelines and structure of laboratory packages for quantum software experiments. Conclusions. Our assessment revealed gaps in information and opportunities for enhancement within the evaluated papers and laboratory packages. Our proposal contributes to the advancement of quantum software engineering research, taking a fundamental step toward fostering rigorous and reliable scientific research in this emerging paradigm.
Quantum computing research increasingly depends on complex software stacks, yet the reproducibility of published results does not receive the priority and longevity mandated by recommendations of large international scientific bodies and best practices in software-centric systems research. In this paper, we present a combined manual and automated large-scale analysis of the reproducibility landscape in quantum computing research, quantify shortcomings, and derive actionable steps forward.
We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability. To place these findings in a broader context, we conduct an automated large-scale screening of nearly 5000 quantum computing papers for the same reproducibility indicators. Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment. This assessment is corroborated by a large-scale automated analysis that yields a consistent code availability rate of 26.8%. Further, it shows that approximately one-third of the papers with accessible code lack machine-readable environment specifications.
The results in this paper indicate that reproducibility is not yet consistently achieved in quantum computing research. In response, we outline a set of practical recommendations that address the observed failure modes and illustrate how reproducibility can be improved in practice.
Replicability and verifiability of empirical results are the cornerstones of scientific endeavor. In the specific area of software engineering, there are widely accepted guidelines for reporting and creating laboratory packages. However, in the realm of quantum computing, the landscape is remarkably different. Current practices in quantum computing involve complex and disparate computing architectures, and, to the best of our knowledge, there exists a notable absence of tailored guidelines for reporting and assembling laboratory packages specific to quantum computing experiments. This gap presents significant challenges to achieving replicability and verifiability in quantum computing research. Addressing this need, our paper introduces a comprehensive framework for reporting software experiments in the quantum computing domain. It adapts Jedlitschka's widely recognized reporting guidelines to the unique requirements of quantum computing. Furthermore, we propose an innovative structure for laboratory packages, synthesizing the methodologies of Guevara et al., and Solari et al., with additional elements tailored to the specificities of quantum computing experiments. This dual approach aims to enhance the replicability and verifiability of experiments in this cutting-edge field, fostering the rigorous and reliable scientific inquiry of quantum software engineering.
<!--HTML--> Abstract In this talk, we explore the critical role that software engineering craftsmanship plays in the success of data science projects and scientific research. We cover practices for ensuring reproducibility, starting with basic concepts such as packaging, version control, testing, and automation. By sharing stories from both academic and industry settings, the talk illustrates how these practices drive reproducibility and project success, benefiting researchers and professionals alike. Bio Francesco is a PhD student at CERN and University of the Basque Country. He studied at KTH and Politecnico di Torino and focuses on the simulation of Quantum Matter with Tensor Network Methods and other techniques related to Quantum Computation. Roman is a Data Scientist at Ergon Informatik, specializing in software engineering for data science and research. He holds a Master’s degree in Physics from ETH Zurich. During his studies, he conducted research at CERN, TU Munich, and the University of Tokyo, focusing on quantum computing and particle physics.
The quantum computing (QC) field is rapidly moving beyond the realm of pure science to become a commercially viable technology that may be able to overcome the drawbacks of traditional computing. Major technology tycoons have spent in building coding frameworks and hardware to create applications specifically designed for quantum computing over the last few years. The development of QC hardware is accelerating, however, the requirement for software-intensive methodology, approaches, procedures, instruments, roles and responsibilities for creating industrial-focused quantum software applications arises from operationalizing the QC. This paper outlines the concept of quantum software engineering (QSE) life cycle, which entails the engineering of quantum requirements, design, implementation, testing and maintenance of quantum software. This paper notably advocates for collaborative efforts between the industrial community and software engineering research to propose practical solutions to support the complete activities for the development of quantum software. The proposed vision makes it easier for researchers and practitioners to suggest new procedures, reference designs, cutting-edge equipment, and methods for utilizing quantum computers and creating the newest and most advanced quantum software.
Reproducibility and comparability of empirical results are at the core tenet of the scientific method in any scientific field. To ease reproducibility of empirical studies, several benchmarks in software engineering research, such as Defects4J, have been developed and widely used. For quantum software engineering research, however, no benchmark has been established yet. In this position paper, we propose a new benchmark-named QBugs-which will provide experimental subjects and an experimental infrastructure to ease the evaluation of new research and the reproducibility of previously published results on quantum software engineering.