arXiv:2609.22965
Exact Realization of Deep ReLU Computation in a Microscopic Statistical-Mechanical System
Junxu Li
Abstract
Understanding the physical origin of the piecewise-linear computation in deep rectified linear unit (ReLU) networks remains a fundamental challenge. Here we establish an exact statistical-mechanical realization of arbitrary ReLU networks. Starting from a microscopic configuration space and its state multiplicities, we construct a partition function without prescribing a neural-network activation function. The resulting system admits equivalent descriptions in terms of cascaded quantum operations with post selection, or fermionic transport model. We rigorously prove that in $β\to+\infty$ limit, the thermodynamic observables of this system exactly reproduce the hidden states, outputs, and loss function of an arbitrary deep ReLU network. Crucially, we demonstrate that the piecewise-linear behavior of the ReLU network emerges from a first-order phase transition in the microscopic system, where the discontinuities precisely coincide with the boundaries of linear regions of ReLU network. This work establishes an exact physical foundation for deep neural computation and provides a novel statistical-mechanical perspective on neural architecture design.
AI Summary
The paper asks whether arbitrary deep ReLU networks can be represented by a microscopic statistical-mechanical construction whose equilibrium observables reproduce the network computation, rather than taking ReLU as a prescribed activation rule. The construction augments affine maps to linear maps, represents signed weights through two nonnegative branches, and proposes implementing these branches using controlled quantum operations, measurement, and post-selection. Measurement trajectories also receive a directed fermionic-transport interpretation. Binary variables $\boldsymbol{\alpha}$ specify which layer sites transmit or block trajectories. Their Gibbs distribution is $q(\boldsymbol{\alpha})\propto\exp[\beta\Pi(\boldsymbol{\alpha})]$, where $\Pi=\sum_{t=1}^{T}2^{T-t}\pi^{(t)}$ combines trajectory-weighted population imbalances. The main theorem claims that, as $\beta\to\infty$, this ensemble selects the ReLU activation pattern for arbitrary inputs, weights, depths, and widths. Layerwise observables then satisfy $\langle\widetilde{\mathbf z}^{(t)}\rangle\to\widetilde{\mathbf h}^{(t)}/\prod_{\tau=0}^{t}C^{(\tau)}$, giving corresponding output and squared-loss identities. The proof uses bounds on backward-propagated coefficients to establish layerwise sign selection. Numerical illustrations use a $1\times6\times6\times1$ network approximating $\sin(\pi x)+1$, with $\beta$ between 0 and 100; reported losses coincide with the ReLU loss for $\beta\geq10$. Exactness is restricted to the zero-temperature limit. The finite-temperature examples show nearly degenerate configurations near activation boundaries. Jumps in the limiting response $\langle\partial\Pi/\partial x\rangle$ are interpreted as first-order transitions corresponding to changes between ReLU linear regions.
Central claim may not hold: Theorem 1's arbitrary-network hidden-state identity is false because a hidden neuron with no effective continuation can have an unconstrained gate whose ensemble average is half its nonzero ReLU activation. The specified Hadamard–controlled-unitary circuit produces an imbalance depending on unsigned input populations, rather than the signed populations required for the claimed affine transformation. The unitary-embedding derivation confuses amplitudes with measurement probabilities and contains mutually inconsistent normalization equations. The microscopic entropy derivation treats signed input weights as probabilities, making its joint probability and logarithm invalid for negative inputs. Finite-beta discontinuities are incompatible with the finite Gibbs model under analytic parameter dependence, and the claimed universal zero-temperature boundary correspondence is not established.
Written by gpt-6.1-sol from the paper's full text. Automated: it can be wrong.
- Comments
- 7 pages, 3 figures