arXiv:2609.20912
Do Quantum Models Scale Like LLMs?
David S. Berman, Ying-Jer Kao, Roger G. Melko, Alexander G. Stapleton
Abstract
In this work, we study the neural scaling laws of RydbergGPT, an autoregressive transformer model trained on qubit projective measurement data gathered from interacting Rydberg atom arrays. The quantum system is known to exhibit a finite-size remnant of a critical point as the laser detuning parameter is varied. We find that near the critical point the transformer loss as a function of training dataset size is well described by a power-law with a loss floor correction. However, away from criticality the quality of the power-law description is substantially reduced. We then compare the statistical structure of both Rydberg measurements and natural-language corpora using an entropy-normalised, finite sample corrected mutual information "two-point" function. We find that near-critical statistics of the two point functions are closest to those observed in natural-language, whilst other qubit configurations far from the critical point have two-point functions that decay more rapidly. This supports the hypothesis that multi-scale dependence contributes to stable neural scaling, and that scaling behaviour should be viewed as a property of the model-data pair.
AI Summary
The paper asks whether transformers trained on quantum measurement distributions exhibit language-model-like data-scaling laws, and whether their stability relates to multiscale statistical dependence. It studies RydbergGPT, an autoregressive transformer trained on synthetic occupation-basis samples from quantum Monte Carlo simulations of an open $6\times6$ Rydberg array. Model size is fixed, with $R_b/a=1.15$, $\beta\Omega=8$, and six detunings spanning the finite-size critical region near $\delta/\Omega\simeq1.1$. Training-data fractions from 10% to 100% are evaluated over ten seeds and fitted to $\mathcal L(D)=\mathcal L_\infty+BD^{-\gamma}$. At detunings 1.05 and 1.17, aggregate validation fits give $R^2=0.989$ and $0.983$, exponents $\gamma=0.656$ and $0.791$, and loss floors 0.4839 and 0.4683, respectively. Fits become less reproducible and have lower goodness of fit at remote detunings. This tests fixed-capacity data scaling, rather than compute-optimal joint parameter/data scaling. To investigate a possible statistical explanation, the paper compares entropy-normalised, permutation-null-corrected pairwise mutual-information profiles from $10^5$ Rydberg configurations with character-level profiles from AG News and Yahoo Answers. Near-critical distributions show strong local dependence followed by a weaker extended tail, qualitatively resembling the language profiles; remote-detuning profiles approach the null level more rapidly. The proposed connection between multiscale dependence and stable learning curves remains a hypothesis. Stated limitations include the small, finite-temperature array, open boundaries, dependence on the mutual-information reference construction, and symmetry sensitivity. Neither universal exponents nor a shared mechanism with language are established.
Error found: The reported power-law fits use ten dataset fractions spanning one decade at one model size and one lattice size, without alternative-curve comparisons or extrapolation tests, so they establish an in-range description rather than a uniquely supported scaling law. The analysis selects the minimum logged validation loss for each dataset fraction and seed without reporting an independent evaluation of the selected checkpoint, leaving potential selection optimism unquantified.
Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.
- Comments
- 10 pages, 6 figures