arXiv:2610.17193
Principles and Techniques for Agentic Research in Experimental Physics
Niko R. Reed, Declan M. Daly, Andrew J. Beling, Jordan D. Giandrea, Ronald L. Walsworth, John W. Blanchard, Daniel Ang
Abstract
The use of artificial intelligence (AI) in physics research is increasingly common, yet few guidelines exist for its effective use. Starting with a principles-based approach, we outline a set of ten requirements for collaborative AI agents in experimental physics, spanning scientific norms, effective agent use, and domain expertise. We take a human-centered approach in which agents act as collaborators that amplify rather than replace the expertise of a human researcher (user) who directs the work and is responsible for its results. We then present a framework designed to enforce these requirements, detailing methods for improving off-the-shelf behavior of large language models (LLMs) with custom scaffolding and describing general-purpose scientific workflows and skills. In particular, we highlight the power of closed-loop campaigns where agents work iteratively toward a well-defined goal. We demonstrate the potential of this mode of research through six case studies in experimental physics, covering device design, hardware control, neural network optimization, physical model identification, and data analysis. All six case studies result in useful work products that add utility to our research, including a neural network architecture and solid-state quantum defect photodynamic model that substantially outperform our published work; a microwave waveguide design that extends a published design concept to a fabrication-ready device meeting our experimental constraints; a quantum control pulse sequence candidate that improves simulated performance by more than an order of magnitude; commissioning of an NMR spectrometer to $\sim$1~Hz frequency precision on live hardware; and finally a data-fitting pipeline that achieves, to our knowledge, the first reconstruction of the Ising microstate of an artificial spin-ice array from diffraction-limited magnetic imaging.
AI Summary
How can LLM agents support experimental physics while preserving verification, provenance, safety, and human accountability? This paper presents ten behavioral principles and PhysKit, a publicly available framework of 26 skills, structured records, verification procedures, and closed-loop workflows. Campaigns iterate against frozen evaluators; hardware requests pass deterministic safety gates. Human researchers specify goals, constraints, and verification requirements and assess the results. Six quantum-sensing case studies illustrate the approach. A current-reconstruction network reduces synthetic-test RMSE by about 33% relative to a published baseline, with multiple-seed confirmation. Searching 101 photodynamic model structures produces an eleven-level hBN defect model with fit SSR 0.83 versus 2.13 for the published nine-level model; withheld-power tests and additional datasets assess prediction. Electromagnetic simulations yield a fabrication-ready microwave design predicting a 119 MHz Rabi frequency at 16 W. A pulse-sequence search finds a candidate with roughly $30\times$ lower simulated effective infidelity than a published benchmark across a robustness battery; the highest-scoring candidate fails under modest random disorder. Live NV-NMR commissioning reaches its first scored measurement after 3.2 hours and a best frequency uncertainty of 0.97 Hz in ten minutes, with a 1.47 Hz median at the final settings. A multi-campaign reconstruction recovers a 2,178-island spin-ice microstate with reported misattribution below 2%. The framework lacks controlled quantitative benchmarking. Device and pulse-sequence results await experimental validation; photodynamic models remain physically ambiguous. NMR performance is drift-limited, and complex reconstruction requires substantial human interaction and verification.
All results were generated using commercially available LLMs and agentic harnesses (Claude Code and Codex) and supported by our custom framework, PhysKit.
Error found: The reported 33% neural-network RMSE improvement is measured on the fixed synthetic benchmark repeatedly used to select architectures and training configurations, without a reported untouched final evaluation establishing generalization. The ASI case asserts under 2% misattributed Ising states without explaining its validation, while the reported magnetic-field fit R² of 0.9 alone cannot establish microstate accuracy.
Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.
- Comments
- 25 pages, 7 figures