Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2603.26718

Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems

Marcin Abram

cs.CYcs.AIcs.MAquant-phSubmitted Mar 18, 2026• Announced Mar 19, 2026
v1 · Mar 18, 2026v2 · Apr 5, 2026
arXivHTMLPDF

Abstract

We analyze the challenges of benchmarking scientific (multi)-agentic systems, including the difficulty of distinguishing reasoning from retrieval, the risks of data/model contamination, the lack of reliable ground truth for novel research problems, the complications introduced by tool use, and the replication challenges due to the continuously changing/updating knowledge base. We discuss strategies for constructing contamination-resistant problems, generating scalable families of tasks, and the need for evaluating systems through multi-turn interactions that better reflect real scientific practice. As an early feasibility test, we demonstrate how to construct a dataset of novel research ideas to test the out-of-sample performance of our system. We also discuss the results of interviews with several researchers and engineers working in quantum science. Through those interviews, we examine how scientists expect to interact with AI systems and how these expectations should shape evaluation methods.

Comments
13 pages, 3 figures

Related papers

Quantum AI-Scientist: A Framework for Quantum Application Discovery in Biomedicine

Min Yang, Sha Cao, Fei Li +1 more

Despite rapid advances in quantum algorithms and hardware, translating these developments into concrete, verifiable real-world applications remains a major challenge. Here, we present Quantum AI-Scientist, a retrieval-augmented, multi-agent framework for discovering potential applications of quantum computing in biomedicine. Unlike conventional AI-scientist systems designed to solve predefined tasks, Quantum AI-Scientist identifies computational problems within a biomedical research domain that may benefit from quantum primitives. Four coordinated agents retrieve and analyze literature data to characterize the capabilities of quantum primitives, identify structurally compatible biomedical problems, translate candidate applications into falsifiable evaluation plans, and synthesize the resulting evidence to rank candidates and recommend further development strategies. The framework maintains a traceable evidence chain of literature-derived evidence, computational formulations, experimental results, and final recommendations. It also explicitly distinguishes structural compatibility from practical executability, empirical improvement, and quantum advantage. Across eight biomedical benchmarks, the staged workflow increased the macro-F1 score for formulation recovery from 82.6% to 88.1% compared with an equal-evidence, single-call baseline, while reducing token consumption at the cost of increased API latency. Case studies involving spatial probe selection and cell lineage inference demonstrate how proposed applications can be translated into executable evaluations of technical feasibility, preliminary empirical improvement, and quantum advantage. Collectively, these results establish a systematic and auditable framework for proposing, testing, and prioritizing biomedical applications of quantum computing.

Proceedings of the AAAI Symposium Series•Nov 6, 2026•0 citations

Reimagining computational quantum research with AI scientists

Yuxuan Du, Hanqing Zhao, Xinbiao Wang +3 more

AI scientists are expanding automated research across scientific domains, yet their ability to turn the scientific record into a foundation for autonomous discovery remains elusive. Here we uncover a fundamental barrier to this transition and demonstrate a route to overcome it in computational quantum science, where advances shape both what we can learn from complex quantum systems and what quantum computing can ultimately deliver. To identify this barrier, we introduce OmniQ, an expert-audited virtual research environment that converts published studies into executable reproduction tasks. Across 59 papers spanning classical simulation, quantum algorithms and quantum-property characterization, we systematically evaluate five agentic systems powered by large language models and six third-party agent harnesses. Our findings reveal that current AI scientists often fail to reproduce published results, exposing a fundamental mismatch between what scientific papers communicate and what autonomous agents require. By contrast, human expert intervention can turn these failures into successful reproductions, showing that the missing information can be identified and converted into reusable research experience. Guided by this observation, we develop Residual Agents (RA), a quantum-tailored self-improving AI scientist that distils externally verified workflow repairs into such experience. RA substantially improves research performance under fixed base models and limited adaptation budgets. For example, RA raises DeepSeek V4.1 Flash performance from 68% to 85%, matching Claude Code with Opus 5 at a third of its cost despite using a weaker base model. We integrate all reproduced results into an extensible platform built on the model context protocol, making verified research workflows directly accessible to AI scientists and allowing new scientific papers to be continuously incorporated.

quant-pharXiv:2611.02833•Nov 3, 2026•0 citations

Towards Verifiable and Self-Correcting AI Physicists for Quantum Many-Body Simulations

Ken Deng, Xiangfei Wang, Guijing Duan +7 more

While large language models (LLMs) promise to revolutionize automated scientific discovery, their application in rigorous real-world physical research is stalled by two critical barriers: a lack of realistic evaluation benchmarks and systemic LLM hallucinations. Here, we address both problems. We introduce QMP-Bench, a pioneering end-to-end research-level benchmark in quantum many-body simulation consisting of $100$ tasks extracted from $21$ high-impact prestigious journals, presenting a challenge even for current frontier LLMs. To establish a paradigm for reliable and transparent AI physicists, we present PhysVEC, a multi-agent framework that enforces self-verifiable and error correction in AI research. PhysVEC seamlessly integrates programming and scientific verifiers to guarantee coding correctness and principle-based physical validity, yielding interpretable evidence and error correction at each step. PhysVEC significantly outperforms existing LLM baselines on various scenarios in QMP-Bench and presents a favorable inference-time scaling, successfully transforming unreliable AI generations into accurate physical reproductions, paving a robust and trustworthy path towards future automated scientific discovery.

arXiv:2604.00149•Mar 31, 2026•8 citations

Explainable Quantum-Logical Multi-Agent Systems for Autonomous Scientific Discovery in High-Dimensional Research Landscapes

Murali Krishna Pasupuleti

Abstract: This study investigates an explainable quantum-logical multi-agent system (QL-MAS) for autonomous scientific discovery in high-dimensional research landscapes. The central methodological problem is that modern discovery settings combine large numbers of variables, heterogeneous evidence types, and partially conflicting reasoning signals, yet conventional single-model pipelines often compress this complexity into opaque rankings. The proposed framework addresses this limitation by treating candidate scientific hypotheses as quantum-logical superpositions of evidence generated by three complementary agents: an evidence agent based on mutual information, a structure agent based on principal-component loadings, and a rule agent based on interpretable decision structures. The resulting architecture does not claim physical quantum computation; rather, it uses quantum-logical amplitude composition as a mathematically disciplined way to represent concurrent support, interference, and consensus across agents. Empirical evaluation is conducted on three real public datasets that approximate heterogeneous scientific discovery landscapes: the Breast Cancer Wisconsin (Diagnostic) dataset (569 observations, 30 variables), the Wine Recognition dataset (178 observations, 13 variables), and the Diabetes dataset used in the least-angle-regression literature (442 observations, 10 baseline variables). These corpora were selected because they are non-synthetic, scientifically grounded, and sufficiently high-dimensional to test feature prioritization, interpretability, and cross-domain transfer. Across the three datasets, the proposed system identifies compact sets of high-salience hypotheses while preserving strong downstream predictive utility. The top-five features selected by QL-MAS attain cross-validated scores of 0.965 on breast cancer classification, 0.955 on wine classification, and 0.465 R² on diabetes progression, remaining close to full-feature baselines while offering substantially more interpretable evidence paths. The results indicate that explainable multi-agent discovery is strongest when high predictive relevance and rule-based consistency converge, and weakest when latent structure is diffuse or poorly aligned with supervised objectives. The analysis therefore supports the conclusion that autonomous discovery systems in research environments should not be assessed solely by predictive accuracy, but by the transparency of their evidential decomposition and by the stability of agreement across reasoning agents. The paper contributes a theoretically coherent framework, a real-data evaluation protocol, and a discussion of limitations, governance, and future directions for autonomous scientific reasoning systems. Keywords: autonomous scientific discovery; explainable AI; multi-agent systems; quantum logic; high-dimensional data; interpretable machine learning; research landscapes

International Journal of Academic and Industrial Research Innovations(IJAIRI)•Apr 6, 2026•9 citations

Introduction to quantum computing for AI agents

Palvi Sharma, Rakesh Kumar, Meenu Gupta

Quantum computing establishes a completely new method for performing calculations, which relies on three fundamental principles called superposition, entanglement, and probabilistic measurement. The application of artificial intelligence (AI) with quantum computing provides the ability to develop quantum AI (QAI) agents, which function as intelligent systems using quantum-based methods for representation, optimization, and probabilistic modeling. The research study analyzed the mathematical foundations of quantum computing, together with its key quantum algorithms used in AI and the architectural designs of hybrid quantum–classical agents, alongside quantum learning models and the complete implementation framework, which includes hardware systems, programming methods, and scalability challenges. The study examined different essential obstacles, which included decoherence, data encoding bottlenecks, gradient instability, benchmarking methods, and the comparison between theoretical and practical quantum advantage. Current quantum computing hardware exists within the noisy intermediate-scale quantum computing stage, which offers scientists various ways to investigate hybrid models and variational methods. Scalable and fault-tolerant quantum computing development will enable businesses to achieve groundbreaking results across various fields, including optimization, probabilistic reasoning, and intricate decision-making. The field of AI needs hardware engineering, algorithm advancement, error correction, and assessment methodology development before it can achieve practical quantum advantage. QAI represents an emerging interdisciplinary research frontier that integrates physics, computer science, mathematics, and ethics into a coherent research agenda.

CRC Press eBooks•Oct 21, 2026•0 citations

ORBIT-Q: Dual-axis benchmarking of autonomous agents in scientific quantum programming

Shi-Xin Zhang, Yu-Qin Chen

Autonomous coding agents perform well on many conventional programming tasks, but scientific computing demands a rigorous validation paradigm that extends beyond simple functional test completion: generated code must preserve physical fidelity, differentiable workflows, framework-native semantics, and scalable representations. We introduce Open Research Benchmark for Integrated Tasks in Quantum Computing (ORBIT-Q) to address this gap. At its core, ORBIT-Q contributes a carefully curated suite of complex, research-level quantum workflows that serves as a challenging testbed for modern scientific programming. ORBIT-Q combines a rigorous multi-tier verification pipeline to support two orthogonal comparisons: different agent harness and model configurations at a fixed quantum software framework, and different quantum software frameworks at a fixed agent. In our systematic evaluations, TensorCircuit-NG (TC) exhibits the highest capability and performance efficiency among the evaluated quantum software frameworks under agent-driven programming, and Codex with GPT-5.5 is the strongest tested agent configuration on TC. However, a significant performance and design gap remains between frontier autonomous agents and human expert reference implementations. We further evaluate two efficiency dimensions: agent-side resource use and artifact-side runtime. Together, these results establish ORBIT-Q as a rigorous benchmark for autonomous scientific programming, framework-agent synergy, and quantum software performance.

quant-pharXiv:2607.03105•Jul 6, 2026•1 citation
We analyze the challenges of benchmarking scientific (multi)-agentic systems, including the difficulty of distinguishing reasoning from retrieval, the risks of data/model contamination, the lack of reliable ground truth for novel research problems, the complications introduced by tool use, and the replication challenges due to the continuously changing/updating knowledge base. We discuss strategies for constructing contamination-resistant problems, generating scalable families of tasks, and the need for evaluating systems through multi-turn interactions that better reflect real scientific practice. As an early feasibility test, we demonstrate how to construct a dataset of novel research ideas to test the out-of-sample performance of our system. We also discuss the results of interviews with several researchers and engineers working in quantum science. Through those interviews, we examine how scientists expect to interact with AI systems and how these expectations should shape evaluation methods.
Despite rapid advances in quantum algorithms and hardware, translating these developments into concrete, verifiable real-world applications remains a major challenge. Here, we present Quantum AI-Scientist, a retrieval-augmented, multi-agent framework for discovering potential applications of quantum computing in biomedicine. Unlike conventional AI-scientist systems designed to solve predefined tasks, Quantum AI-Scientist identifies computational problems within a biomedical research domain that may benefit from quantum primitives. Four coordinated agents retrieve and analyze literature data to characterize the capabilities of quantum primitives, identify structurally compatible biomedical problems, translate candidate applications into falsifiable evaluation plans, and synthesize the resulting evidence to rank candidates and recommend further development strategies. The framework maintains a traceable evidence chain of literature-derived evidence, computational formulations, experimental results, and final recommendations. It also explicitly distinguishes structural compatibility from practical executability, empirical improvement, and quantum advantage. Across eight biomedical benchmarks, the staged workflow increased the macro-F1 score for formulation recovery from 82.6% to 88.1% compared with an equal-evidence, single-call baseline, while reducing token consumption at the cost of increased API latency. Case studies involving spatial probe selection and cell lineage inference demonstrate how proposed applications can be translated into executable evaluations of technical feasibility, preliminary empirical improvement, and quantum advantage. Collectively, these results establish a systematic and auditable framework for proposing, testing, and prioritizing biomedical applications of quantum computing.
AI scientists are expanding automated research across scientific domains, yet their ability to turn the scientific record into a foundation for autonomous discovery remains elusive. Here we uncover a fundamental barrier to this transition and demonstrate a route to overcome it in computational quantum science, where advances shape both what we can learn from complex quantum systems and what quantum computing can ultimately deliver. To identify this barrier, we introduce OmniQ, an expert-audited virtual research environment that converts published studies into executable reproduction tasks. Across 59 papers spanning classical simulation, quantum algorithms and quantum-property characterization, we systematically evaluate five agentic systems powered by large language models and six third-party agent harnesses. Our findings reveal that current AI scientists often fail to reproduce published results, exposing a fundamental mismatch between what scientific papers communicate and what autonomous agents require. By contrast, human expert intervention can turn these failures into successful reproductions, showing that the missing information can be identified and converted into reusable research experience. Guided by this observation, we develop Residual Agents (RA), a quantum-tailored self-improving AI scientist that distils externally verified workflow repairs into such experience. RA substantially improves research performance under fixed base models and limited adaptation budgets. For example, RA raises DeepSeek V4.1 Flash performance from 68% to 85%, matching Claude Code with Opus 5 at a third of its cost despite using a weaker base model. We integrate all reproduced results into an extensible platform built on the model context protocol, making verified research workflows directly accessible to AI scientists and allowing new scientific papers to be continuously incorporated.
While large language models (LLMs) promise to revolutionize automated scientific discovery, their application in rigorous real-world physical research is stalled by two critical barriers: a lack of realistic evaluation benchmarks and systemic LLM hallucinations. Here, we address both problems. We introduce QMP-Bench, a pioneering end-to-end research-level benchmark in quantum many-body simulation consisting of 100100100 tasks extracted from 212121 high-impact prestigious journals, presenting a challenge even for current frontier LLMs. To establish a paradigm for reliable and transparent AI physicists, we present PhysVEC, a multi-agent framework that enforces self-verifiable and error correction in AI research. PhysVEC seamlessly integrates programming and scientific verifiers to guarantee coding correctness and principle-based physical validity, yielding interpretable evidence and error correction at each step. PhysVEC significantly outperforms existing LLM baselines on various scenarios in QMP-Bench and presents a favorable inference-time scaling, successfully transforming unreliable AI generations into accurate physical reproductions, paving a robust and trustworthy path towards future automated scientific discovery.
Abstract: This study investigates an explainable quantum-logical multi-agent system (QL-MAS) for autonomous scientific discovery in high-dimensional research landscapes. The central methodological problem is that modern discovery settings combine large numbers of variables, heterogeneous evidence types, and partially conflicting reasoning signals, yet conventional single-model pipelines often compress this complexity into opaque rankings. The proposed framework addresses this limitation by treating candidate scientific hypotheses as quantum-logical superpositions of evidence generated by three complementary agents: an evidence agent based on mutual information, a structure agent based on principal-component loadings, and a rule agent based on interpretable decision structures. The resulting architecture does not claim physical quantum computation; rather, it uses quantum-logical amplitude composition as a mathematically disciplined way to represent concurrent support, interference, and consensus across agents. Empirical evaluation is conducted on three real public datasets that approximate heterogeneous scientific discovery landscapes: the Breast Cancer Wisconsin (Diagnostic) dataset (569 observations, 30 variables), the Wine Recognition dataset (178 observations, 13 variables), and the Diabetes dataset used in the least-angle-regression literature (442 observations, 10 baseline variables). These corpora were selected because they are non-synthetic, scientifically grounded, and sufficiently high-dimensional to test feature prioritization, interpretability, and cross-domain transfer. Across the three datasets, the proposed system identifies compact sets of high-salience hypotheses while preserving strong downstream predictive utility. The top-five features selected by QL-MAS attain cross-validated scores of 0.965 on breast cancer classification, 0.955 on wine classification, and 0.465 R² on diabetes progression, remaining close to full-feature baselines while offering substantially more interpretable evidence paths. The results indicate that explainable multi-agent discovery is strongest when high predictive relevance and rule-based consistency converge, and weakest when latent structure is diffuse or poorly aligned with supervised objectives. The analysis therefore supports the conclusion that autonomous discovery systems in research environments should not be assessed solely by predictive accuracy, but by the transparency of their evidential decomposition and by the stability of agreement across reasoning agents. The paper contributes a theoretically coherent framework, a real-data evaluation protocol, and a discussion of limitations, governance, and future directions for autonomous scientific reasoning systems. Keywords: autonomous scientific discovery; explainable AI; multi-agent systems; quantum logic; high-dimensional data; interpretable machine learning; research landscapes
Quantum computing establishes a completely new method for performing calculations, which relies on three fundamental principles called superposition, entanglement, and probabilistic measurement. The application of artificial intelligence (AI) with quantum computing provides the ability to develop quantum AI (QAI) agents, which function as intelligent systems using quantum-based methods for representation, optimization, and probabilistic modeling. The research study analyzed the mathematical foundations of quantum computing, together with its key quantum algorithms used in AI and the architectural designs of hybrid quantum–classical agents, alongside quantum learning models and the complete implementation framework, which includes hardware systems, programming methods, and scalability challenges. The study examined different essential obstacles, which included decoherence, data encoding bottlenecks, gradient instability, benchmarking methods, and the comparison between theoretical and practical quantum advantage. Current quantum computing hardware exists within the noisy intermediate-scale quantum computing stage, which offers scientists various ways to investigate hybrid models and variational methods. Scalable and fault-tolerant quantum computing development will enable businesses to achieve groundbreaking results across various fields, including optimization, probabilistic reasoning, and intricate decision-making. The field of AI needs hardware engineering, algorithm advancement, error correction, and assessment methodology development before it can achieve practical quantum advantage. QAI represents an emerging interdisciplinary research frontier that integrates physics, computer science, mathematics, and ethics into a coherent research agenda.
Autonomous coding agents perform well on many conventional programming tasks, but scientific computing demands a rigorous validation paradigm that extends beyond simple functional test completion: generated code must preserve physical fidelity, differentiable workflows, framework-native semantics, and scalable representations. We introduce Open Research Benchmark for Integrated Tasks in Quantum Computing (ORBIT-Q) to address this gap. At its core, ORBIT-Q contributes a carefully curated suite of complex, research-level quantum workflows that serves as a challenging testbed for modern scientific programming. ORBIT-Q combines a rigorous multi-tier verification pipeline to support two orthogonal comparisons: different agent harness and model configurations at a fixed quantum software framework, and different quantum software frameworks at a fixed agent. In our systematic evaluations, TensorCircuit-NG (TC) exhibits the highest capability and performance efficiency among the evaluated quantum software frameworks under agent-driven programming, and Codex with GPT-5.5 is the strongest tested agent configuration on TC. However, a significant performance and design gap remains between frontier autonomous agents and human expert reference implementations. We further evaluate two efficiency dimensions: agent-side resource use and artifact-side runtime. Together, these results establish ORBIT-Q as a rigorous benchmark for autonomous scientific programming, framework-agent synergy, and quantum software performance.