arXiv:2610.36113
Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking
Min Chen, Junqi Wang, Zeguan Wu, Wenhao Huang, Junyu Liu
Abstract
Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.
AI Summary
The paper asks whether scaling theory and real-space renormalization group (RG) can quantitatively describe grokking—the delayed onset of generalization during neural-network training. It studies modular addition modulo $97$ using a two-layer, quadratic-activation network with $500$ hidden neurons and an ensemble of $30$ training runs. An effective thermodynamic dictionary identifies squared weight norm with an order parameter, weight decay with its conjugate field, epoch with an inverse-temperature-like coordinate, and cross-run norm and loss variances with susceptibility and heat-capacity proxies. Loss-Hessian couplings between neuron norm coordinates define a graph; Fiedler-vector ordering places neurons on a chain. At the selected transition epoch $e_c=145$, nearby pairs have mean absolute coupling approximately $2.1$ times that of distant pairs. Adjacent neurons are then blocked in pairs. A weighted regression over $65$ epoch–weight-decay configurations estimates a two-coordinate coarse-graining map, whose eigenvalue magnitudes yield $y_t=0.6923$ and $y_h=0.6515$. Assuming standard free-energy homogeneity and dimension $d=1$, the resulting exponents are $(\alpha,\beta,\gamma,\nu,\delta)=(0.556,0.503,0.438,1.444,1.870)$. Separate empirical fits differ by $1.2$–$9.7\%$; transition-width scaling across seven task moduli gives $\nu=1.400$, with additional data-collapse tests involving $350$ runs. The stated qualifications are substantive: training seeds are not Gibbs samples and combine initialization variability with quenched dataset disorder; loss variance is only a heat-capacity surrogate; the epoch–temperature correspondence is motivated rather than established; and AdamW decay is not literally an added norm penalty. The RG description uses a two-coupling truncation, while relating task modulus to spatial system size requires additional scaling hypotheses.
Central claim may not hold: The paper identifies a local blocking-regression Jacobian with critical RG eigenvalues without establishing a critical fixed point corresponding to grokking. The block-sum construction restricts perturbations to block-uniform configurations rather than eliminating internal fluctuations, so it does not establish the free-energy homogeneity used to predict critical exponents. Ordering a fully connected neuron graph onto a chain, with only a modest reported coupling contrast, does not establish the one-dimensional thermodynamic scaling assumed in the exponent formulas. The purported independent correlation-length exponent instead measures transition-time variability versus task modulus, whose connection to neuron-space RG scaling is assumed rather than derived. The direct heat-capacity and susceptibility estimates are unverified seed-variance proxies rather than established thermodynamic responses, making their exponent agreement conditional on the proposed dictionary.
Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.
- Comments
- 22 pages, 10 figures