Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2512.00766 · ACM Transactions on Software Engineering and Methodology

Code Comments for Quantum Software Development Kits: An Empirical Study on Qiskit

Zenghui Zhou, Yuechen Li, Yi Cai, Jinlong Wen, Xiaohan Yu, Zheng Zheng, Beibei Yin

Published Sep 15, 2026• 1 citation
arXivHTMLPDF

Abstract

Quantum computing is gaining attention from academia and industry. With the quantum Software Development Kits (SDKs), programmers can develop quantum software to explore the power of quantum computing. However, programmers may face challenges in understanding quantum software due to the non-intuitive quantum mechanics. To facilitate software development and maintenance, code comments offered in quantum SDKs serve as a natural language explanation of program functionalities and logical flows. Despite their importance, scarce research systematically reports their value and provides constructive guidelines for programmers. To address this gap, our paper focuses on Qiskit, one of the most popular quantum SDKs, and presents CC4Q, the first dataset of code comments for quantum computing. CC4Q incorporates 12,812 code comment pairs and 28,076 sentence-level code comment units, the latter of which involve heavy human annotation. Regarding the annotation, we validate the applicability of the developer-intent taxonomy used in classical programs, and also propose a new taxonomy considering quantum-specific knowledge. We conduct an empirical study comprehensively interpreting code comments from three perspectives: comment structure and coverage, developers’ intentions, and associated quantum topics. Our findings uncover key differences in code comments between classical and quantum software, and also outline quantum-specific knowledge relevant to quantum software development.

DOI
10.1145/3847113

Related papers

CC4Q Workbench: From Quantum Repositories to Analyzable Code Comments

Zenghui Zhou, Yuechen Li, Zheng Zheng

Quantum software development kits (SDKs) expose high-level APIs whose use requires an understanding of underlying mathematical principles, circuit semantics, and hardware properties. Code comments can provide natural-language or structural explanations that are not explicit in API calls. However, studying comments in a new quantum SDK still requires researchers to coordinate separate steps for comment extraction, quantum-specific classification, and manual verification, resulting in repeated integration effort. To address this problem, we present CC4Q Workbench, an open-source workbench that integrates these steps into a reusable workflow. Given a Python repository at a specified revision, CC4Q Workbench automatically extracts code-comment pairs, segments comments into sentence-level units, and applies two trained classifiers to identify quantum-specific content and assign quantum-knowledge categories. Its web interface supports exploration of bundled datasets, inspection of code snippets, corpus construction from new repositories, manual annotation with model predictions hidden during binary labeling, and classifier fine-tuning using exported annotations. The release provides aggregate statistics for six SDK datasets comprising 39,593 sentence-level code comment units (SCCUs), of which 17,517 are browsable in the web interface. Applied to three additional application-layer Qiskit repositories outside the original CC4Q dataset, the pipeline ran without code changes and produced 11,517 SCCUs. Manual verification of 379 SCCUs showed 85.0% agreement between human annotations and the binary classifier's predictions.

arXiv:2610.32651•Oct 23, 2026•0 citations

Understanding Quantum Software Engineering Challenges An Empirical Study on Stack Exchange Forums and GitHub Issues

Mohamed Raed El aoun, Heng Li, Foutse Khomh +1 more

With the advance of quantum computing, quantum software becomes critical for exploring the full potential of quantum computing systems. Recently, quantum software engineering (QSE) becomes an emerging area attracting more and more attention. However, it is not clear what are the challenges and opportunities of quantum computing facing the software engineering community. This work aims to understand the QSE-related challenges perceived by developers. We perform an empirical study on Stack Exchange forums where developers post-QSE-related questions & answers and Github issue reports where developers raise QSE-related issues in practical quantum computing projects. Based on an existing taxonomy of question types on Stack Overflow, we first perform a qualitative analysis of the types of QSE-related questions asked on Stack Exchange forums. We then use automated topic modeling to uncover the topics in QSE-related Stack Exchange posts and GitHub issue reports. Our study highlights some particularly challenging areas of QSE that are different from that of traditional software engineering, such as explaining the theory behind quantum computing code, interpreting quantum program outputs, and bridging the knowledge gap between quantum computing and classical computing, as well as their associated opportunities.

arXiv:2205.03181•IEEE International Conference on Software Maintenance (ICSM)•Sep 1, 2021•47 citations

Qiskit HumanEval: An Evaluation Benchmark for Quantum Code Generative Models

Sanjay Vishwakarma, Francis Harkins, Siddharth Golecha +7 more

Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit – a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.

arXiv:2406.14712•IEEE International Conference on Quantum Computing and Engineering (QCE)•Sep 15, 2024•40 citations

Large Language Models for Quantum Code Translation: An Exploratory Study

Xiyuan Li, Yuechen Li, Jianjun Zhao

Quantum software development is challenged by quantum-mechanical semantics and fragmented software development kits (SDKs), which complicate cross-SDK code translation and execution. In this paper, we propose an LLM-based approach for translating executable programs between high-level quantum SDKs. We develop a quantum-aware translation workflow whose prompt design and validation reflect quantum semantics and accommodate the version evolution of quantum ecosystems. We conduct a preliminary evaluation of 4 LLM configurations on 43 tasks, translating Qiskit programs to PennyLane and Q#. LLMs achieve a zero-shot Pass@1 of up to 72.33%, while one-shot prompting further improves the translation effectiveness. These results suggest potential for quantum code translation by LLMs.

cs.SECross-listedarXiv:2610.34841•Oct 27, 2026•0 citations

Quantum software development: a survey

Luis Jiménez-Navajas, Fabian Bühler, Frank Leymann +3 more

Over the last few years, quantum computing has been growing at an exponential pace. Every day, new techniques, frameworks, modeling, and programming languages are emerging that aim to facilitate the development of quantum software, which is key to achieving the promising applications of quantum computing. However, which of these are actively used and the degree of satisfaction of researchers and developers regarding these quantum software frameworks and languages is not known. To address this, we conducted a survey to characterize which modeling tools and which quantum programming languages are used during the quantum software lifecycle. Researchers in academia and industry developers were surveyed, and a total of 57 responses were collected. The results indicate that during quantum software development, some models and diagrams are used to guide development. In addition, the survey results show what quantum programming languages are the most used alongside the classical programming languages employed to build hybrid programs, among other important insights. The implications of this survey are: (i) to find out what the current trends are within quantum software development and (ii) to find out what the needs are of quantum software developers with respect to current modeling and programming languages and tools.

Quantum Information and Computation•Jun 1, 2024•6 citations

An Empirical Study on Self-Admitted Technical Debt in Quantum Software

Yuta Ishimoto, Yuto Nakamura, Ryota Katsube +5 more

Quantum computers, which utilize the principles of quantum mechanics, are expected to be applied to a wide range of fields. With the advancement of quantum computer development, a lot of quantum software, which enables the operation of quantum computers, has been developed. It has a distinct nature (e.g., superposition and entanglement of qubits) compared to traditional software, leading to the unique challenges of its development. While prior studies have clarified and defined some unique challenges of quantum software, many remain unclear due to limited research. In this study, we conducted an empirical study of Self-Admitted Technical Debt (SATD) for quantum software. SATD is a type of technical debt, a problem in the code that the developer is aware of. Hence, we conjecture that analyzing SATDs can reveal the unique challenges developers face when developing quantum software. We manually coded 202 comments from the Python® files of the 61 open-source quantum software on GitHub®. The 202 comments correspond to a 95% confidence level with a 5% confidence interval, as in previous studies. The results showed that 88 comments (45.6% of all SATD comments) were quantum-specific SATDs (QSATDs), which require knowledge of quantum computation to repay. Furthermore, we propose a taxonomy for QSATDs. This taxonomy, which consists of four main categories and eight subcategories, classifies QSATDs in terms of quantum-specific aspects such as circuit implementation, backend, and algorithms. Our empirical results are beneficial for quantum software developers, helping them understand implementation areas that require attention. For researchers, our results promote further research, including the exploration of challenges in QSATD repayment.

Asia-Pacific Software Engineering Conference (APSEC)•Dec 3, 2024•6 citations
Quantum computing is gaining attention from academia and industry. With the quantum Software Development Kits (SDKs), programmers can develop quantum software to explore the power of quantum computing. However, programmers may face challenges in understanding quantum software due to the non-intuitive quantum mechanics. To facilitate software development and maintenance, code comments offered in quantum SDKs serve as a natural language explanation of program functionalities and logical flows. Despite their importance, scarce research systematically reports their value and provides constructive guidelines for programmers. To address this gap, our paper focuses on Qiskit, one of the most popular quantum SDKs, and presents CC4Q, the first dataset of code comments for quantum computing. CC4Q incorporates 12,812 code comment pairs and 28,076 sentence-level code comment units, the latter of which involve heavy human annotation. Regarding the annotation, we validate the applicability of the developer-intent taxonomy used in classical programs, and also propose a new taxonomy considering quantum-specific knowledge. We conduct an empirical study comprehensively interpreting code comments from three perspectives: comment structure and coverage, developers’ intentions, and associated quantum topics. Our findings uncover key differences in code comments between classical and quantum software, and also outline quantum-specific knowledge relevant to quantum software development.
Quantum software development kits (SDKs) expose high-level APIs whose use requires an understanding of underlying mathematical principles, circuit semantics, and hardware properties. Code comments can provide natural-language or structural explanations that are not explicit in API calls. However, studying comments in a new quantum SDK still requires researchers to coordinate separate steps for comment extraction, quantum-specific classification, and manual verification, resulting in repeated integration effort. To address this problem, we present CC4Q Workbench, an open-source workbench that integrates these steps into a reusable workflow. Given a Python repository at a specified revision, CC4Q Workbench automatically extracts code-comment pairs, segments comments into sentence-level units, and applies two trained classifiers to identify quantum-specific content and assign quantum-knowledge categories. Its web interface supports exploration of bundled datasets, inspection of code snippets, corpus construction from new repositories, manual annotation with model predictions hidden during binary labeling, and classifier fine-tuning using exported annotations. The release provides aggregate statistics for six SDK datasets comprising 39,593 sentence-level code comment units (SCCUs), of which 17,517 are browsable in the web interface. Applied to three additional application-layer Qiskit repositories outside the original CC4Q dataset, the pipeline ran without code changes and produced 11,517 SCCUs. Manual verification of 379 SCCUs showed 85.0% agreement between human annotations and the binary classifier's predictions.
With the advance of quantum computing, quantum software becomes critical for exploring the full potential of quantum computing systems. Recently, quantum software engineering (QSE) becomes an emerging area attracting more and more attention. However, it is not clear what are the challenges and opportunities of quantum computing facing the software engineering community. This work aims to understand the QSE-related challenges perceived by developers. We perform an empirical study on Stack Exchange forums where developers post-QSE-related questions & answers and Github issue reports where developers raise QSE-related issues in practical quantum computing projects. Based on an existing taxonomy of question types on Stack Overflow, we first perform a qualitative analysis of the types of QSE-related questions asked on Stack Exchange forums. We then use automated topic modeling to uncover the topics in QSE-related Stack Exchange posts and GitHub issue reports. Our study highlights some particularly challenging areas of QSE that are different from that of traditional software engineering, such as explaining the theory behind quantum computing code, interpreting quantum program outputs, and bridging the knowledge gap between quantum computing and classical computing, as well as their associated opportunities.
Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit – a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.
Quantum software development is challenged by quantum-mechanical semantics and fragmented software development kits (SDKs), which complicate cross-SDK code translation and execution. In this paper, we propose an LLM-based approach for translating executable programs between high-level quantum SDKs. We develop a quantum-aware translation workflow whose prompt design and validation reflect quantum semantics and accommodate the version evolution of quantum ecosystems. We conduct a preliminary evaluation of 4 LLM configurations on 43 tasks, translating Qiskit programs to PennyLane and Q#. LLMs achieve a zero-shot Pass@1 of up to 72.33%, while one-shot prompting further improves the translation effectiveness. These results suggest potential for quantum code translation by LLMs.
Over the last few years, quantum computing has been growing at an exponential pace. Every day, new techniques, frameworks, modeling, and programming languages are emerging that aim to facilitate the development of quantum software, which is key to achieving the promising applications of quantum computing. However, which of these are actively used and the degree of satisfaction of researchers and developers regarding these quantum software frameworks and languages is not known. To address this, we conducted a survey to characterize which modeling tools and which quantum programming languages are used during the quantum software lifecycle. Researchers in academia and industry developers were surveyed, and a total of 57 responses were collected. The results indicate that during quantum software development, some models and diagrams are used to guide development. In addition, the survey results show what quantum programming languages are the most used alongside the classical programming languages employed to build hybrid programs, among other important insights. The implications of this survey are: (i) to find out what the current trends are within quantum software development and (ii) to find out what the needs are of quantum software developers with respect to current modeling and programming languages and tools.
Quantum computers, which utilize the principles of quantum mechanics, are expected to be applied to a wide range of fields. With the advancement of quantum computer development, a lot of quantum software, which enables the operation of quantum computers, has been developed. It has a distinct nature (e.g., superposition and entanglement of qubits) compared to traditional software, leading to the unique challenges of its development. While prior studies have clarified and defined some unique challenges of quantum software, many remain unclear due to limited research. In this study, we conducted an empirical study of Self-Admitted Technical Debt (SATD) for quantum software. SATD is a type of technical debt, a problem in the code that the developer is aware of. Hence, we conjecture that analyzing SATDs can reveal the unique challenges developers face when developing quantum software. We manually coded 202 comments from the Python® files of the 61 open-source quantum software on GitHub®. The 202 comments correspond to a 95% confidence level with a 5% confidence interval, as in previous studies. The results showed that 88 comments (45.6% of all SATD comments) were quantum-specific SATDs (QSATDs), which require knowledge of quantum computation to repay. Furthermore, we propose a taxonomy for QSATDs. This taxonomy, which consists of four main categories and eight subcategories, classifies QSATDs in terms of quantum-specific aspects such as circuit implementation, backend, and algorithms. Our empirical results are beneficial for quantum software developers, helping them understand implementation areas that require attention. For researchers, our results promote further research, including the exploration of challenges in QSATD repayment.