Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2610.36113

Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

Min Chen, Junqi Wang, Zeguan Wu, Wenhao Huang, Junyu Liu

cs.LGcs.AIhep-thquant-phSubmitted Oct 26, 2026• Announced Oct 28, 2026
arXivHTMLPDF

Abstract

Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.

AI Summary

The paper asks whether scaling theory and real-space renormalization group (RG) can quantitatively describe grokking—the delayed onset of generalization during neural-network training. It studies modular addition modulo $97$ using a two-layer, quadratic-activation network with $500$ hidden neurons and an ensemble of $30$ training runs. An effective thermodynamic dictionary identifies squared weight norm with an order parameter, weight decay with its conjugate field, epoch with an inverse-temperature-like coordinate, and cross-run norm and loss variances with susceptibility and heat-capacity proxies. Loss-Hessian couplings between neuron norm coordinates define a graph; Fiedler-vector ordering places neurons on a chain. At the selected transition epoch $e_c=145$, nearby pairs have mean absolute coupling approximately $2.1$ times that of distant pairs. Adjacent neurons are then blocked in pairs. A weighted regression over $65$ epoch–weight-decay configurations estimates a two-coordinate coarse-graining map, whose eigenvalue magnitudes yield $y_t=0.6923$ and $y_h=0.6515$. Assuming standard free-energy homogeneity and dimension $d=1$, the resulting exponents are $(\alpha,\beta,\gamma,\nu,\delta)=(0.556,0.503,0.438,1.444,1.870)$. Separate empirical fits differ by $1.2$–$9.7\%$; transition-width scaling across seven task moduli gives $\nu=1.400$, with additional data-collapse tests involving $350$ runs. The stated qualifications are substantive: training seeds are not Gibbs samples and combine initialization variability with quenched dataset disorder; loss variance is only a heat-capacity surrogate; the epoch–temperature correspondence is motivated rather than established; and AdamW decay is not literally an added norm penalty. The RG description uses a two-coupling truncation, while relating task modulus to spatial system size requires additional scaling hypotheses.

Little sign of AI-generated textStrong assumptionsOverclaimsPossible errorMissing comparison

Central claim may not hold: The paper identifies a local blocking-regression Jacobian with critical RG eigenvalues without establishing a critical fixed point corresponding to grokking. The block-sum construction restricts perturbations to block-uniform configurations rather than eliminating internal fluctuations, so it does not establish the free-energy homogeneity used to predict critical exponents. Ordering a fully connected neuron graph onto a chain, with only a modest reported coupling contrast, does not establish the one-dimensional thermodynamic scaling assumed in the exponent formulas. The purported independent correlation-length exponent instead measures transition-time variability versus task modulus, whose connection to neuron-space RG scaling is assumed rather than derived. The direct heat-capacity and susceptibility estimates are unverified seed-variance proxies rather than established thermodynamic responses, making their exponent agreement conditional on the proposed dictionary.

Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.

Comments
22 pages, 10 figures

Related papers

A Deep Dive into the Connections Between the Renormalization Group and Deep Learning in the Ising Model

Taylor, Kelsie

The renormalization group (RG) is an essential technique in statistical physics and quantum field theory, which considers scale-invariant properties of physical theories and how these theories' parameters change with scaling. Deep learning is a powerful computational technique that uses multi-layered neural networks to solve a myriad of complicated problems. Previous research suggests the possibility that unsupervised deep learning may be a form of RG flow, by being a layer-by-layer coarse graining of the original data. We examined this connection on a more rigorous basis for the simple example of Kadanoff block renormalization of the 2D nearest-neighbor Ising model, with our deep learning accomplished via Restricted Boltzmann Machines (RBMs). We developed extensive renormalization techniques for the 1D and 2D Ising model to provide a baseline for comparison. For the 1D Ising model, we successfully used Adam optimization on a correlation length loss function to learn the group flow, yielding results consistent with the analytical model for infinite N. For the 2D Ising model, we successfully generated Ising model samples using the Wolff algorithm, and performed the group flow using a quasi-deterministic method, validating these results by calculating the critical exponent ν. We then examined RBM learning of the Ising model layer by layer, finding a blocking structure in the learning that is qualitatively similar to RG. Lastly, we directly compared the weights of each layer from the learning to Ising spin renormalization, but found quantitative inconsistencies for the simple case of nearest-neighbor Ising models.

arXiv:2308.11075•Jun 26, 2023•0 citations

Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group

Johanna Erdmenger B, Kevin T. Grosvenor, Ro Jefferson

We investigate the analogy between the renormalization group (RG) and deep neural networks, wherein subsequent layers of neurons are analogous to successive steps along the RG. In particular, we quantify the flow of information by explicitly computing the relative entropy or Kullback-Leibler divergence in both the one- and two-dimensional Ising models under decimation RG, as well as in a feedforward neural network as a function of depth. We observe qualitatively identical behavior characterized by the monotonic increase to a parameter-dependent asymptotic value. On the quantum field theory side, the monotonic increase confirms the connection between the relative entropy and the c-theorem. For the neural networks, the asymptotic behavior may have implications for various information maximization methods in machine learning, as well as for disentangling compactness and generalizability. Furthermore, while both the two-dimensional Ising model and the random neural networks we consider exhibit non-trivial critical points, the relative entropy appears insensitive to the phase structure of either system. In this sense, more refined probes are required in order to fully elucidate the flow of information in these models.

arXiv:2107.06898•SciPost Physics•Jan 26, 2022•43 citations

Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis

Donghee Lee, Hye‐Sung Lee, Jaeok Yi

Deep neural network architectures often consist of repetitive structural elements. We introduce an approach that reveals these patterns and can be broadly applied to the study of deep learning. Similarly to how a power strip helps untangle and organize complex cable connections, this approach treats neurons as additional degrees of freedom in interactions, simplifying the structure and enhancing the intuitive understanding of interactions within deep neural networks. Furthermore, it reveals the translational symmetry of deep neural networks, which simplifies the application of the renormalization group transformation, a method that effectively analyzes the scaling behavior of the system. By utilizing translational symmetry and renormalization group transformations, we can analyze critical phenomena. This approach may open new avenues for studying deep neural networks using statistical physics.

arXiv:2410.00396•Physical Review Research•May 30, 2025•6 citations

Renormalization Group connected to Neural Networks

J.J. ter Hoeve

Even though deep learning has proved to be very powerful as the core method of machine learning, theoretical understanding behind its success is still unclear. It is been pointed out in recent years that the behaviour of deep neural networks is reminiscent of a fundamental framework in statistical physics: the renormalization group (RG). Motivated by this analogy, we develop an analytical method of directly obtaining the trained weights W resulting from a training set of 1D-Ising samples with coupling J and temperature T. The found relation W(J) drives a flow that takes 1D-Ising configurations at non-zero temperature to the critical point at T = 0. This behaviour is opposite to what a typical RG-flow dictates.

•Jan 1, 2018•0 citations

Neural Monte Carlo renormalization group

Jui-Hui Chung, Ying-Jer Kao

The key idea behind the renormalization group (RG) transformation is that properties of physical systems with very different microscopic makeups can be characterized by a few universal parameters.However, finding a systematic way to construct RG transformation for particular systems remains difficult due to the many possible choices of the weight factors in the RG procedure.Here we show, by identifying the conditional distribution in the restricted Boltzmann machine and the weight factor distribution in the RG procedure, that a valid real-space RG transformation can be learned without prior knowledge of the physical system.This neural Monte Carlo RG algorithm allows for direct computation of the RG flow and critical exponents.Our results establish a solid connection between the RG transformation in physics and the deep architecture in machine learning, paving the way for further interdisciplinary research.

arXiv:2010.05703•Physical Review Research•Jun 22, 2021•23 citations

Scale-invariant feature extraction of neural network and renormalization group flow

Satoshi Iso, Shotaro Shiba, Sumito Yokoo

Theoretical understanding of how a deep neural network (DNN) extracts features from input images is still unclear, but it is widely believed that the extraction is performed hierarchically through a process of coarse graining. It reminds us of the basic renormalization group (RG) concept in statistical physics. In order to explore possible relations between DNN and RG, we use the restricted Boltzmann machine (RBM) applied to an Ising model and construct a flow of model parameters (in particular, temperature) generated by the RBM. We show that the unsupervised RBM trained by spin configurations at various temperatures from T=0 to T=6 generates a flow along which the temperature approaches the critical value T_{c}=2.27. This behavior is the opposite of the typical RG flow of the Ising model. By analyzing various properties of the weight matrices of the trained RBM, we discuss why it flows towards T_{c} and how the RBM learns to extract features of spin configurations.

arXiv:1801.07172•Physical review. E•May 8, 2018•86 citations
Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.
The paper asks whether scaling theory and real-space renormalization group (RG) can quantitatively describe grokking—the delayed onset of generalization during neural-network training. It studies modular addition modulo 979797 using a two-layer, quadratic-activation network with 500500500 hidden neurons and an ensemble of 303030 training runs.
An effective thermodynamic dictionary identifies squared weight norm with an order parameter, weight decay with its conjugate field, epoch with an inverse-temperature-like coordinate, and cross-run norm and loss variances with susceptibility and heat-capacity proxies. Loss-Hessian couplings between neuron norm coordinates define a graph; Fiedler-vector ordering places neurons on a chain. At the selected transition epoch ec=145e_c=145ec​=145, nearby pairs have mean absolute coupling approximately 2.12.12.1 times that of distant pairs. Adjacent neurons are then blocked in pairs. A weighted regression over 656565 epoch–weight-decay configurations estimates a two-coordinate coarse-graining map, whose eigenvalue magnitudes yield yt=0.6923y_t=0.6923yt​=0.6923 and yh=0.6515y_h=0.6515yh​=0.6515.
Assuming standard free-energy homogeneity and dimension d=1d=1d=1, the resulting exponents are (α,β,γ,ν,δ)=(0.556,0.503,0.438,1.444,1.870)(\alpha,\beta,\gamma,\nu,\delta)=(0.556,0.503,0.438,1.444,1.870)(α,β,γ,ν,δ)=(0.556,0.503,0.438,1.444,1.870). Separate empirical fits differ by 1.21.21.2–9.7%9.7\%9.7%; transition-width scaling across seven task moduli gives ν=1.400\nu=1.400ν=1.400, with additional data-collapse tests involving 350350350 runs.
The stated qualifications are substantive: training seeds are not Gibbs samples and combine initialization variability with quenched dataset disorder; loss variance is only a heat-capacity surrogate; the epoch–temperature correspondence is motivated rather than established; and AdamW decay is not literally an added norm penalty. The RG description uses a two-coupling truncation, while relating task modulus to spatial system size requires additional scaling hypotheses.
The renormalization group (RG) is an essential technique in statistical physics and quantum field theory, which considers scale-invariant properties of physical theories and how these theories' parameters change with scaling. Deep learning is a powerful computational technique that uses multi-layered neural networks to solve a myriad of complicated problems. Previous research suggests the possibility that unsupervised deep learning may be a form of RG flow, by being a layer-by-layer coarse graining of the original data. We examined this connection on a more rigorous basis for the simple example of Kadanoff block renormalization of the 2D nearest-neighbor Ising model, with our deep learning accomplished via Restricted Boltzmann Machines (RBMs). We developed extensive renormalization techniques for the 1D and 2D Ising model to provide a baseline for comparison. For the 1D Ising model, we successfully used Adam optimization on a correlation length loss function to learn the group flow, yielding results consistent with the analytical model for infinite N. For the 2D Ising model, we successfully generated Ising model samples using the Wolff algorithm, and performed the group flow using a quasi-deterministic method, validating these results by calculating the critical exponent ν. We then examined RBM learning of the Ising model layer by layer, finding a blocking structure in the learning that is qualitatively similar to RG. Lastly, we directly compared the weights of each layer from the learning to Ising spin renormalization, but found quantitative inconsistencies for the simple case of nearest-neighbor Ising models.
We investigate the analogy between the renormalization group (RG) and deep neural networks, wherein subsequent layers of neurons are analogous to successive steps along the RG. In particular, we quantify the flow of information by explicitly computing the relative entropy or Kullback-Leibler divergence in both the one- and two-dimensional Ising models under decimation RG, as well as in a feedforward neural network as a function of depth. We observe qualitatively identical behavior characterized by the monotonic increase to a parameter-dependent asymptotic value. On the quantum field theory side, the monotonic increase confirms the connection between the relative entropy and the c-theorem. For the neural networks, the asymptotic behavior may have implications for various information maximization methods in machine learning, as well as for disentangling compactness and generalizability. Furthermore, while both the two-dimensional Ising model and the random neural networks we consider exhibit non-trivial critical points, the relative entropy appears insensitive to the phase structure of either system. In this sense, more refined probes are required in order to fully elucidate the flow of information in these models.
Deep neural network architectures often consist of repetitive structural elements. We introduce an approach that reveals these patterns and can be broadly applied to the study of deep learning. Similarly to how a power strip helps untangle and organize complex cable connections, this approach treats neurons as additional degrees of freedom in interactions, simplifying the structure and enhancing the intuitive understanding of interactions within deep neural networks. Furthermore, it reveals the translational symmetry of deep neural networks, which simplifies the application of the renormalization group transformation, a method that effectively analyzes the scaling behavior of the system. By utilizing translational symmetry and renormalization group transformations, we can analyze critical phenomena. This approach may open new avenues for studying deep neural networks using statistical physics.
Even though deep learning has proved to be very powerful as the core method of machine learning, theoretical understanding behind its success is still unclear. It is been pointed out in recent years that the behaviour of deep neural networks is reminiscent of a fundamental framework in statistical physics: the renormalization group (RG). Motivated by this analogy, we develop an analytical method of directly obtaining the trained weights W resulting from a training set of 1D-Ising samples with coupling J and temperature T. The found relation W(J) drives a flow that takes 1D-Ising configurations at non-zero temperature to the critical point at T = 0. This behaviour is opposite to what a typical RG-flow dictates.
The key idea behind the renormalization group (RG) transformation is that properties of physical systems with very different microscopic makeups can be characterized by a few universal parameters.However, finding a systematic way to construct RG transformation for particular systems remains difficult due to the many possible choices of the weight factors in the RG procedure.Here we show, by identifying the conditional distribution in the restricted Boltzmann machine and the weight factor distribution in the RG procedure, that a valid real-space RG transformation can be learned without prior knowledge of the physical system.This neural Monte Carlo RG algorithm allows for direct computation of the RG flow and critical exponents.Our results establish a solid connection between the RG transformation in physics and the deep architecture in machine learning, paving the way for further interdisciplinary research.
Theoretical understanding of how a deep neural network (DNN) extracts features from input images is still unclear, but it is widely believed that the extraction is performed hierarchically through a process of coarse graining. It reminds us of the basic renormalization group (RG) concept in statistical physics. In order to explore possible relations between DNN and RG, we use the restricted Boltzmann machine (RBM) applied to an Ising model and construct a flow of model parameters (in particular, temperature) generated by the RBM. We show that the unsupervised RBM trained by spin configurations at various temperatures from T=0 to T=6 generates a flow along which the temperature approaches the critical value T_{c}=2.27. This behavior is the opposite of the typical RG flow of the Ising model. By analyzing various properties of the weight matrices of the trained RBM, we discuss why it flows towards T_{c} and how the RBM learns to extract features of spin configurations.