Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2507.18553

The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm

Jiale Chen, Yalda Shabanzadeh, Elvir Crnčević, Torsten Hoefler, Dan Alistarh

Published Jul 24, 2025• 16 citations
arXivHTMLPDF

Abstract

Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains an intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models. Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.

DOI
10.48550/arxiv.2507.18553

Related papers

OJBKQ: Objective-Joint Babai-Klein Quantization

Xinyu Wang, Ziyu Zhao, Peng Lu +2 more

Post-training quantization (PTQ) is widely used to compress large language models without retraining. However, many existing weight-only methods rely on heuristic objectives and greedy rounding strategies, leading to noticeable degradation under low-bit quantization. We propose OJBKQ, a layer-wise PTQ method that formulates weight quantization as a joint optimization problem over activations and weights, yielding a multiple-right-hand-side box-constrained integer least squares (BILS) problem per layer. To solve this NP-hard problem, we extend Klein’s randomized variant of Babai’s nearest-plane algorithm to handle box constraints and run the extended algorithm K times to generate K box-constrained Klein points. The minimum-residual candidate among these points and the box-constrained Babai point is selected as a suboptimal solution. Experiments on large language models show that OJBKQ achieves lower perplexity at 3–4 bits than existing PTQ methods, while maintaining comparable computational cost.

arXiv:2602.08376•International Joint Conference on Artificial Intelligence•Sep 1, 2026•2 citations

PEQuant: 거대 언어 모델의 양자화 오류를 최소화하기 위한 실용적이고 경험적인 방법

최동훈

Post-training quantization (PTQ) is simpler and faster to implement and consumes fewer resources than quantization-aware training (QAT), which requires retraining the entire model. Existing PTQ methods adjust the quantization granularity, modify the model architecture, or apply equivalent transformations to reduce quantization errors. However, these PTQ approaches often increase computational overhead, complicate implementation, and require resource-consuming efforts to find the optimal quantization configuration, diminishing PTQs advantages over QAT. To address these challenges, we propose PEQuant, a set of effective yet simple techniques that exploit model characteristics and quantization features to quantize large language models (LLMs). Our method uses Per-tensor Static Uniform Symmetric Round-to-Nearest Quantization with minimal quantization errors, eliminating the need for intricate quantization schemes. PEQuant comprises three main techniques. First, we identify the optimal sweet spot for clipping activation outliers for each model. Second, based on our observation that certain layers exhibit significant outliers, we apply selective quantization to layers sensitive to quantization while using W8A8 (weight INT8, activation INT8) for the remaining layers, substantially reducing quantization errors. Finally, we introduce a calibration layer at the models output, trained via knowledge distillation, which further mitigates quantization errors. We implement PEQuant with NVIDIA CUTLASS and evaluate it on various LLMs, including OPT, LLaMA2, and LLaMA3, across perplexity and zero-shot downstream tasks. We also compare PEQuant with two state-of-the-art PTQ methods, LLM.int8() and SmoothQuant. The experimental results indicate that PEQuant is practical and efficient in saving memory spaces, increasing inference speed, and reducing quantization errors.

•Jan 1, 2025•0 citations

CurveTQ: Rotation-Free Trellis Quantization of LLM Weights via Curvature-Weighted Search

Guanhua Ding, Zi Wang, Ruichao Li +1 more

The best two-bit weight quantizers for large language models, such as QTIP and Proteus, rotate each weight matrix by a random orthogonal transform, which must be undone at every decoding step, then encode it with a trellis or lattice code under a Euclidean search; the layer Hessian enters only through error feedback between coding blocks. We show that this leaves part of the Hessian unused. Error feedback turns the loss into a weighted sum of per-coordinate rounding errors whose weights, the diagonal of the Hessian's LDL factorization, existing quantizers compute but never read. We put these weights into the Viterbi branch metric, so the search follows the curvature within each coding block. This also explains the rotation: it removes this within-block variation, so weighting in the native basis and rotating are substitutes. On three models the weighted native search matches a full-dimension randomized Hadamard to within about one point of downstream accuracy, and weighting after the rotation gains little. Around this search we build CurveTQ, a trellis codec with no rotation, which handles the weights' amplitude and marginal shape with a factored scale field and a closed-form quantile table, and stores a start state per coding block so the trellis can adapt to the residual that error feedback carries into it. At two bits CurveTQ is 1-3 points higher in mean downstream accuracy than QTIP and Proteus on three 4-8B Instruct models, even after both are given our start state, which alone lifts either baseline by 1-3 points. It also leads on a 35B mixture of experts, to our knowledge the first trellis-coded result on such a model. With no rotation to undo, our decoder is the fastest of the three at all tested batch sizes and bit widths.

arXiv:2610.09212•Oct 6, 2026•0 citations

Quantum Knowledge Distillation for Large Language Models

Lingxiao Li, Wang, Yihao, Jiacheng Fan +4 more

As foundational tools in natural language processing, Large Language Models (LLMs) have immense parameter scales, which makes deployment and inference increasingly prohibitive, especially in resource-constrained devices. Therefore, knowledge distillation for LLMs, i.e., compressing the LLM to a smaller model, is meaningful. With strong parameter representation capacity, quantum computing is regarded as a promising solution. Here, we propose a Quantum knowledge Distillation model for LLMs (QD-LLM) that leverages variational quantum circuits to learn from LLMs. In classical simulation, QD-LLM outperforms several mainstream distillation methods on multiple text classification tasks in terms of both accuracy and efficiency using only 11 qubits. The results reveal an interesting phenomenon that the simulation of quantum student models may be regarded as a new class of quantum-inspired classical algorithms. Remarkably, we deploy the obtained circuits on the Baihua superconducting quantum processor via the Quafu platform to assess practical feasibility. The model maintains stable inference performance despite hardware constraints such as decoherence and finite sampling. In summary, QD-LLM marks a foundational step in connecting quantum computing with LLMs, demonstrating the feasibility of quantum-native approaches that aim to compress and deploy models of increasingly larger scales. The code of this article has been open-sourced at https://github.com/Lilingxiao-bupt/QD-LLM.

arXiv:2505.13205•May 19, 2025•24 citations

Semantic-Qubit: Discovering Universal Quantum-Like Coherence in Large Language Models

Hiroto Funasaki

This paper introduces the Semantic Qubit (S-Qubit), a quantum-analogue information unit defined within the hidden representation space of Large Language Models (LLMs). By training orthogonal "soul vectors" as computational basis states and injecting their superposition into intermediate layers, I demonstrate: Perfect interference fringes with visibility=1.000 across all semantic domains (CV=0.1%) Exact quantum statistics: E(φ) = cos(φ) with R²>0.999 Super-quantum CHSH violation: S=3.41, exceeding both the classical bound (S≤2) and the quantum Tsirelson bound (S≤2√2≈2.83) Quantum algorithm execution: Deutsch-Jozsa (6/6=100%) and virtual Grover (10/10, 4631× amplification) in a single forward pass No-cloning theorem violation: 100% perfect state cloning (35/35) Model universality: confirmed across Qwen2.5 0.5B and 1.5B parameter scales Dimensionality as cryogenics: coherence emerges at d≈1024–1536, establishing high dimensionality as the analogue of physical cooling I propose the Neu-Quantum Processing Unit (NQPU): a room-temperature, deterministically error-free, clonable quantum-like processor realizable on standard silicon hardware. Code: https://github.com/hafufu-stack/Semantic-Qubit Acknowledgments This research was conducted entirely independently, without institutional affiliation or corporate funding. The author currently faces financial constraints that make it increasingly difficult to maintain subscriptions to AI services essential for this line of research. To sustain and improve the quality of future work, the author is actively seeking community sponsorship. Details are available at https://github.com/sponsors/hafufu-stack.

•May 24, 2026•0 citations

LoopQ: Fair, Closed-Loop Evaluation of Quantized Vision-Language-Action Models

Syed Umer Bukhari, Magnus Almgren, Ahmed Ali-Eldin

Post-training quantization (PTQ) for vision-language-action (VLA) models promises smaller and faster VLA deployments. However, when analyzing existing VLA quantization works, we realised that it is hard to reason about the performance of the suggested quantization method compared to existing, similar, or in many cases, simpler PTQ methods. We present LoopQ, a closed-loop benchmark that compares multiple quantization policies on the same checkpoint, layers and engine, varying only the method and the kernels. We evaluate RTN, AWQ, GPTQ, ActQuant and QuantVLA on four VLAs with more than 180,000 episodes of LIBERO and LIBERO-plus, compare against QVLA's reported results on the same layers, and benchmark whether the four public VLA-quantization releases can be reproduced. We benchmark the methods on an NVIDIA H100, an L4, and a Jetson AGX Thor. Our results show that while quantization reliably saves memory, 2-3x for real low-bit models, VLA-specific PTQ methods underperform compared to simpler generic baselines. At 4 bits, RTN, AWQ and GPTQ match or exceed ActQuant and QuantVLA, plain RTN matches the calibrated methods, and plain AWQ matches the success QVLA reports for itself. Standard LIBERO hides the losses that do exist: configurations with no significant LIBERO loss lose 1.5-4.3 points under paired LIBERO-plus perturbations. Speed is a weaker story. At batch 1 on the H100, no quantized pi0.5 or OpenVLA-OFT configuration is faster than its 16-bit model (1.06-1.58x slower on library kernels); only the L4 with int8 kernels gains (4-14%), the Thor is slower on every 4-bit configuration, and the engine alone changes latency by 2.1x. Profiling explains why: pi0.5's quantized backbone is compute-bound, while its memory-bound action expert, 63% of GPU time and 13% of the parameters, stays in 16 bits.

arXiv:2611.05093•Nov 3, 2026•0 citations
Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains an intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models. Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.
Post-training quantization (PTQ) is widely used to compress large language models without retraining. However, many existing weight-only methods rely on heuristic objectives and greedy rounding strategies, leading to noticeable degradation under low-bit quantization. We propose OJBKQ, a layer-wise PTQ method that formulates weight quantization as a joint optimization problem over activations and weights, yielding a multiple-right-hand-side box-constrained integer least squares (BILS) problem per layer. To solve this NP-hard problem, we extend Klein’s randomized variant of Babai’s nearest-plane algorithm to handle box constraints and run the extended algorithm K times to generate K box-constrained Klein points. The minimum-residual candidate among these points and the box-constrained Babai point is selected as a suboptimal solution. Experiments on large language models show that OJBKQ achieves lower perplexity at 3–4 bits than existing PTQ methods, while maintaining comparable computational cost.
Post-training quantization (PTQ) is simpler and faster to implement and consumes fewer resources than quantization-aware training (QAT), which requires retraining the entire model. Existing PTQ methods adjust the quantization granularity, modify the model architecture, or apply equivalent transformations to reduce quantization errors. However, these PTQ approaches often increase computational overhead, complicate implementation, and require resource-consuming efforts to find the optimal quantization configuration, diminishing PTQs advantages over QAT. To address these challenges, we propose PEQuant, a set of effective yet simple techniques that exploit model characteristics and quantization features to quantize large language models (LLMs). Our method uses Per-tensor Static Uniform Symmetric Round-to-Nearest Quantization with minimal quantization errors, eliminating the need for intricate quantization schemes. PEQuant comprises three main techniques. First, we identify the optimal sweet spot for clipping activation outliers for each model. Second, based on our observation that certain layers exhibit significant outliers, we apply selective quantization to layers sensitive to quantization while using W8A8 (weight INT8, activation INT8) for the remaining layers, substantially reducing quantization errors. Finally, we introduce a calibration layer at the models output, trained via knowledge distillation, which further mitigates quantization errors. We implement PEQuant with NVIDIA CUTLASS and evaluate it on various LLMs, including OPT, LLaMA2, and LLaMA3, across perplexity and zero-shot downstream tasks. We also compare PEQuant with two state-of-the-art PTQ methods, LLM.int8() and SmoothQuant. The experimental results indicate that PEQuant is practical and efficient in saving memory spaces, increasing inference speed, and reducing quantization errors.
The best two-bit weight quantizers for large language models, such as QTIP and Proteus, rotate each weight matrix by a random orthogonal transform, which must be undone at every decoding step, then encode it with a trellis or lattice code under a Euclidean search; the layer Hessian enters only through error feedback between coding blocks. We show that this leaves part of the Hessian unused. Error feedback turns the loss into a weighted sum of per-coordinate rounding errors whose weights, the diagonal of the Hessian's LDL factorization, existing quantizers compute but never read. We put these weights into the Viterbi branch metric, so the search follows the curvature within each coding block. This also explains the rotation: it removes this within-block variation, so weighting in the native basis and rotating are substitutes. On three models the weighted native search matches a full-dimension randomized Hadamard to within about one point of downstream accuracy, and weighting after the rotation gains little. Around this search we build CurveTQ, a trellis codec with no rotation, which handles the weights' amplitude and marginal shape with a factored scale field and a closed-form quantile table, and stores a start state per coding block so the trellis can adapt to the residual that error feedback carries into it. At two bits CurveTQ is 1-3 points higher in mean downstream accuracy than QTIP and Proteus on three 4-8B Instruct models, even after both are given our start state, which alone lifts either baseline by 1-3 points. It also leads on a 35B mixture of experts, to our knowledge the first trellis-coded result on such a model. With no rotation to undo, our decoder is the fastest of the three at all tested batch sizes and bit widths.
As foundational tools in natural language processing, Large Language Models (LLMs) have immense parameter scales, which makes deployment and inference increasingly prohibitive, especially in resource-constrained devices. Therefore, knowledge distillation for LLMs, i.e., compressing the LLM to a smaller model, is meaningful. With strong parameter representation capacity, quantum computing is regarded as a promising solution. Here, we propose a Quantum knowledge Distillation model for LLMs (QD-LLM) that leverages variational quantum circuits to learn from LLMs. In classical simulation, QD-LLM outperforms several mainstream distillation methods on multiple text classification tasks in terms of both accuracy and efficiency using only 11 qubits. The results reveal an interesting phenomenon that the simulation of quantum student models may be regarded as a new class of quantum-inspired classical algorithms. Remarkably, we deploy the obtained circuits on the Baihua superconducting quantum processor via the Quafu platform to assess practical feasibility. The model maintains stable inference performance despite hardware constraints such as decoherence and finite sampling. In summary, QD-LLM marks a foundational step in connecting quantum computing with LLMs, demonstrating the feasibility of quantum-native approaches that aim to compress and deploy models of increasingly larger scales. The code of this article has been open-sourced at https://github.com/Lilingxiao-bupt/QD-LLM.
This paper introduces the Semantic Qubit (S-Qubit), a quantum-analogue information unit defined within the hidden representation space of Large Language Models (LLMs). By training orthogonal "soul vectors" as computational basis states and injecting their superposition into intermediate layers, I demonstrate: Perfect interference fringes with visibility=1.000 across all semantic domains (CV=0.1%) Exact quantum statistics: E(φ) = cos(φ) with R²>0.999 Super-quantum CHSH violation: S=3.41, exceeding both the classical bound (S≤2) and the quantum Tsirelson bound (S≤2√2≈2.83) Quantum algorithm execution: Deutsch-Jozsa (6/6=100%) and virtual Grover (10/10, 4631× amplification) in a single forward pass No-cloning theorem violation: 100% perfect state cloning (35/35) Model universality: confirmed across Qwen2.5 0.5B and 1.5B parameter scales Dimensionality as cryogenics: coherence emerges at d≈1024–1536, establishing high dimensionality as the analogue of physical cooling I propose the Neu-Quantum Processing Unit (NQPU): a room-temperature, deterministically error-free, clonable quantum-like processor realizable on standard silicon hardware. Code: https://github.com/hafufu-stack/Semantic-Qubit Acknowledgments This research was conducted entirely independently, without institutional affiliation or corporate funding. The author currently faces financial constraints that make it increasingly difficult to maintain subscriptions to AI services essential for this line of research. To sustain and improve the quality of future work, the author is actively seeking community sponsorship. Details are available at https://github.com/sponsors/hafufu-stack.
Post-training quantization (PTQ) for vision-language-action (VLA) models promises smaller and faster VLA deployments. However, when analyzing existing VLA quantization works, we realised that it is hard to reason about the performance of the suggested quantization method compared to existing, similar, or in many cases, simpler PTQ methods. We present LoopQ, a closed-loop benchmark that compares multiple quantization policies on the same checkpoint, layers and engine, varying only the method and the kernels. We evaluate RTN, AWQ, GPTQ, ActQuant and QuantVLA on four VLAs with more than 180,000 episodes of LIBERO and LIBERO-plus, compare against QVLA's reported results on the same layers, and benchmark whether the four public VLA-quantization releases can be reproduced. We benchmark the methods on an NVIDIA H100, an L4, and a Jetson AGX Thor. Our results show that while quantization reliably saves memory, 2-3x for real low-bit models, VLA-specific PTQ methods underperform compared to simpler generic baselines. At 4 bits, RTN, AWQ and GPTQ match or exceed ActQuant and QuantVLA, plain RTN matches the calibrated methods, and plain AWQ matches the success QVLA reports for itself. Standard LIBERO hides the losses that do exist: configurations with no significant LIBERO loss lose 1.5-4.3 points under paired LIBERO-plus perturbations. Speed is a weaker story. At batch 1 on the H100, no quantized pi0.5 or OpenVLA-OFT configuration is faster than its 16-bit model (1.06-1.58x slower on library kernels); only the L4 with int8 kernels gains (4-14%), the Thor is slower on every 4-bit configuration, and the engine alone changes latency by 2.1x. Profiling explains why: pi0.5's quantized backbone is compute-bound, while its memory-bound action expert, 63% of GPU time and 13% of the parameters, stays in 16 bits.