Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

GO-ViT: 이상치 채널 그룹화를 통한 비전 트랜스포머 완전 양자화

기준하

Published Jan 1, 2025• 0 citations
OpenAlex

Abstract

In this study, we focus on two important branches in quantization. The first is to enable lower bit operations to take advantage of lower memory usage, increased computation speed, and reduced model size. And the second is to fully quantize to enable integer-only operations and utilize On-device utilities that enables integer-accelerated operations. In particular, the layer norm input activation required for full quantization is large in value and the presence of outliers is the main cause of performance degradation in low bit settings. In particular, since the distribution of outliers in actviations is consistent in specific channels, more efficient quantization is possible when this information is utilized in the quantization process. The GO-ViT method uses a groupwise scaling method that distinguishes inliers and outliers and determines scaling factors for each corresponding channel to improve performance even in low bits. To verify the versatile applicability of the GO-ViT methodology, which can be used in both Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT), we conducted experiments on various datasets such as ImageNet, CIFAR-100, and Something-Something v2 and several transformer architectures such as Swin, ViT, DeiT, and Video Swin. The experimental results show that GO-ViT can improve the Top-1 accuracy at low bit compared to the PTQ baseline model. In addition, it shows higher performance than the QAT baseline model tested on the Cifar100 dataset by preserving high values of activations while starting with high accuracy from the beginning.

Related papers

PEQuant: 거대 언어 모델의 양자화 오류를 최소화하기 위한 실용적이고 경험적인 방법

최동훈

Post-training quantization (PTQ) is simpler and faster to implement and consumes fewer resources than quantization-aware training (QAT), which requires retraining the entire model. Existing PTQ methods adjust the quantization granularity, modify the model architecture, or apply equivalent transformations to reduce quantization errors. However, these PTQ approaches often increase computational overhead, complicate implementation, and require resource-consuming efforts to find the optimal quantization configuration, diminishing PTQs advantages over QAT. To address these challenges, we propose PEQuant, a set of effective yet simple techniques that exploit model characteristics and quantization features to quantize large language models (LLMs). Our method uses Per-tensor Static Uniform Symmetric Round-to-Nearest Quantization with minimal quantization errors, eliminating the need for intricate quantization schemes. PEQuant comprises three main techniques. First, we identify the optimal sweet spot for clipping activation outliers for each model. Second, based on our observation that certain layers exhibit significant outliers, we apply selective quantization to layers sensitive to quantization while using W8A8 (weight INT8, activation INT8) for the remaining layers, substantially reducing quantization errors. Finally, we introduce a calibration layer at the models output, trained via knowledge distillation, which further mitigates quantization errors. We implement PEQuant with NVIDIA CUTLASS and evaluate it on various LLMs, including OPT, LLaMA2, and LLaMA3, across perplexity and zero-shot downstream tasks. We also compare PEQuant with two state-of-the-art PTQ methods, LLM.int8() and SmoothQuant. The experimental results indicate that PEQuant is practical and efficient in saving memory spaces, increasing inference speed, and reducing quantization errors.

•Jan 1, 2025•0 citations

LoopQ: Fair, Closed-Loop Evaluation of Quantized Vision-Language-Action Models

Syed Umer Bukhari, Magnus Almgren, Ahmed Ali-Eldin

Post-training quantization (PTQ) for vision-language-action (VLA) models promises smaller and faster VLA deployments. However, when analyzing existing VLA quantization works, we realised that it is hard to reason about the performance of the suggested quantization method compared to existing, similar, or in many cases, simpler PTQ methods. We present LoopQ, a closed-loop benchmark that compares multiple quantization policies on the same checkpoint, layers and engine, varying only the method and the kernels. We evaluate RTN, AWQ, GPTQ, ActQuant and QuantVLA on four VLAs with more than 180,000 episodes of LIBERO and LIBERO-plus, compare against QVLA's reported results on the same layers, and benchmark whether the four public VLA-quantization releases can be reproduced. We benchmark the methods on an NVIDIA H100, an L4, and a Jetson AGX Thor. Our results show that while quantization reliably saves memory, 2-3x for real low-bit models, VLA-specific PTQ methods underperform compared to simpler generic baselines. At 4 bits, RTN, AWQ and GPTQ match or exceed ActQuant and QuantVLA, plain RTN matches the calibrated methods, and plain AWQ matches the success QVLA reports for itself. Standard LIBERO hides the losses that do exist: configurations with no significant LIBERO loss lose 1.5-4.3 points under paired LIBERO-plus perturbations. Speed is a weaker story. At batch 1 on the H100, no quantized pi0.5 or OpenVLA-OFT configuration is faster than its 16-bit model (1.06-1.58x slower on library kernels); only the L4 with int8 kernels gains (4-14%), the Thor is slower on every 4-bit configuration, and the engine alone changes latency by 2.1x. Profiling explains why: pi0.5's quantized backbone is compute-bound, while its memory-bound action expert, 63% of GPU time and 13% of the parameters, stays in 16 bits.

arXiv:2611.05093•Nov 3, 2026•0 citations

Tail-Aware Post-Training Quantization for 3D Geometry Models

Sicheng Pan, Chen Tang, Shuzhao Xie +6 more

The burgeoning complexity and scale of 3D geometry models pose significant challenges for deployment on resource-constrained platforms. While Post-Training Quantization (PTQ) enables efficient inference without retraining, conventional methods, primarily optimized for 2D Vision Transformers, fail to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead. To address these challenges, we propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline specifically engineered for 3D geometric learning. Our contribution is threefold: (1) To overcome the data-scale bottleneck in 3D datasets, we develop a progressive coarse-to-fine calibration construction strategy that constructs a highly compact subset to achieve both statistical purity and geometric representativeness. (2) We reformulate the quantization interval search as an optimization problem and introduce a ternary-search-based solver, reducing the computational complexity from $\mathcal{O}(N)$ to $\mathcal{O}(\log N)$ for accelerated deployment. (3) To mitigate quantization error accumulation, we propose TRE-Guided Module-wise Compensation, which utilizes a Tail Relative Error (TRE) metric to adaptively identify and rectify distortions in modules sensitive to long-tailed activation outliers. Extensive experiments on the VGGT and Pi3 benchmarks demonstrate that TAPTQ consistently outperforms state-of-the-art PTQ methods in accuracy while significantly reducing calibration time. The code will be released soon.

arXiv:2602.01741•Feb 2, 2026•2 citations

Enhancing Low-bit Quantization of Vision Models via Rotation Transformation

Jaewoo Park, Yunjeong Yong, Jihae Lee

최신 비전 모델들은 활성화 이상치로 인해 저비트 양자화 시 성능 저하가 크게 발생한다. 이를 완화하기 위해 본 논문에서는 LLM에서 검증된 직교 행렬 기반 회전 변환을 RF-DETR, RT-DETR, YOLOv12에 적용하였다. 실험 결과 회전 변환은 레이어별 첨도를 낮춰 긴 꼬리 분포와 극단적 이상치의 영향을 효과적으로 완화하였고, 저비트 설정에서 탐지 성능 저하를 유의미하게 줄였다. 특히 RF-DETR은 W4A8에서 Baseline에 근접한 성능을 회복하였으며, RT-DETR은 QRT-DETR 대비 추가적인 성능 향상을 통해 저비트 환경에서의 민감도를 낮추었다. YOLOv12 또한 저비트 설정에서 일관된 성능 개선을 보여 양자화 오류 완화에 기여함을 확인하였다.더불어 회전 변환 적용 후에도 추론 시간은 baseline(43.75ms)과 유사한 수준(41.31ms)으로 유지되어, 제안 기법이 추가적인 런타임 오버헤드를 유의미하게 증가시키지 않음을 시사한다.

Journal of the Institute of Electronics and Information Engineers•Mar 30, 2026•0 citations

UQ-ViT: Harmonizing Extreme Activations with Hardware-Friendly Uniform Quantization in Vision Transformers

Tao Jiang, Yucheng Jiang, Xiwen Yao +2 more

Post-Training Quantization enables efficient Vision Transformer (ViTs) deployment with a small calibration data, and its prevalent use of uniform quantization harnesses AI accelerator matrix cores for high-speed inference. However, the application of uniform quantization is fundamentally challenged by the extreme non-uniformity of activation distributions.Specifically, the power-law nature of post-Softmax attention scores and the significant inter-channel variance in post-GELU activations create a dilemma for conventional quantization, as it struggles to preserve critical high-magnitude values without sacrificing overall precision. To resolve this core conflict, we introduce UQ-ViT (Uniform Quantization for Vision Transformers), a novel uniform quantization framework designed to reconcile high precision with hardware efficiency. Central to UQ-ViT are two operators: Dynamic Elimination of Maximum (DeMax) and Normalization Quantization (NormQuant). DeMax is a quantization operator for post-Softmax attention scores that utilizes uniform quantization. It dynamically eliminates and preserves dominant values, effectively mitigating quantization loss from the extreme values in the power-law distribution. NormQuant utilizes a per-channel quantization strategy during quantization and reverts to a per-tensor format for dequantization, achieving both high accuracy and computational efficiency. Crucially, it is applicable to any linear layer, enabling effective quantization of post-GELU activations in ViTs. Through extensive experiments on various ViTs and vision tasks, including image classification, object detection, and instance segmentation, we demonstrate that our proposed approach outperforms existing methods, achieving superior accuracy while ensuring hardware friendliness.

Proceedings of the AAAI Conference on Artificial Intelligence•Mar 14, 2026•1 citation

When Does Quantum Machine Learning Outperform Classical Machine Learning? Identifying Dataset Characteristics That Enable Quantum Advantage

Sungmin Kim, Juntaek Oh, Zichen Luo +2 more

양자머신러닝(Quantum Machine Learning, QML)에 대한 기업의 관심이 높아지고 있음에도 불구하고, 조직들은 언제 QML 투자가 고전 머신러닝 대안보다 정당화되는지를 판단할 데이터 기반 프레임워크를 갖추지 못하고 있다. 본 연구는 PRISMA 2020 체계적 문헌고찰을 통해 확인된 35편의 동료심사 연구로부터 추출한 92개 벤치마크 데이터셋을 대상으로, 양자 우위 발현 조건에 대한 최초의 메타분석적 ․ 통계적 검증 연구를 수행함으로써 이러한 공백을 다룬다. 각 데이터셋에 대해 표본 규모, 차원성, 노이즈 수준, 비선형 구조, 특성 간 상관관계를 아우르는 18개의 측정 가능한 데이터 특성을 추출하고, 최고 성능 QML모델과 고전 최신기법(SOTA) 간의 성능 격차(Gap_from_SOTA)를 산출하였다. 비모수 집단비교, HC3 이분산성-강건표준오차를 적용한 다중회귀분석, 그리고 5개 모델(XGBoost, LightGBM, CatBoost, RandomForest, GradientBoosting)에 대한 SHAP 설명가능 인공지능을 활용한 앙상블 머신러닝이라는 세 가지 독립적 분석 방법이 적용되었다. 분석 결과, 노이즈 수준, 비선형성 격차, 최대 특성 간 상관관계가 QML 우위를 결정하는 세 가지 지배적 예측변수로 일관되게 확인되었으며, 5개 앙상블 모델 전반에서 특성 중요도 순위가 동일하게 나타났다. 이들 예측변수로부터 QML이 고전 방법을 능가할 가능성이 높은 조건, 즉 낮은 노이즈, 우호적인 비선형 구조, 약한 특성 간 상관관계를 함께 특징짓는 정량적 의사결정 임계값을 도출하였다. 도메인 분석 결과 에너지/전력 및 기상/기후 분야에서 유의하게 높은 양자 우위가 나타난 반면, 약물/화학 및 금융 분야는 고전 방법에 유리한 것으로 나타났다. 시계열 과제는 QML의 이점이 가장 큰 반면, 다중분류는 고전ML에 유리한 경향을 보였다. 아키텍처 중에서는 QLSTM 계열 모델이 가장 높고 일관된 우위를 달성하였다.

Journal of Intelligence and Information Systems•Sep 30, 2026•0 citations
In this study, we focus on two important branches in quantization. The first is to enable lower bit operations to take advantage of lower memory usage, increased computation speed, and reduced model size. And the second is to fully quantize to enable integer-only operations and utilize On-device utilities that enables integer-accelerated operations. In particular, the layer norm input activation required for full quantization is large in value and the presence of outliers is the main cause of performance degradation in low bit settings. In particular, since the distribution of outliers in actviations is consistent in specific channels, more efficient quantization is possible when this information is utilized in the quantization process. The GO-ViT method uses a groupwise scaling method that distinguishes inliers and outliers and determines scaling factors for each corresponding channel to improve performance even in low bits. To verify the versatile applicability of the GO-ViT methodology, which can be used in both Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT), we conducted experiments on various datasets such as ImageNet, CIFAR-100, and Something-Something v2 and several transformer architectures such as Swin, ViT, DeiT, and Video Swin. The experimental results show that GO-ViT can improve the Top-1 accuracy at low bit compared to the PTQ baseline model. In addition, it shows higher performance than the QAT baseline model tested on the Cifar100 dataset by preserving high values of activations while starting with high accuracy from the beginning.
Post-training quantization (PTQ) is simpler and faster to implement and consumes fewer resources than quantization-aware training (QAT), which requires retraining the entire model. Existing PTQ methods adjust the quantization granularity, modify the model architecture, or apply equivalent transformations to reduce quantization errors. However, these PTQ approaches often increase computational overhead, complicate implementation, and require resource-consuming efforts to find the optimal quantization configuration, diminishing PTQs advantages over QAT. To address these challenges, we propose PEQuant, a set of effective yet simple techniques that exploit model characteristics and quantization features to quantize large language models (LLMs). Our method uses Per-tensor Static Uniform Symmetric Round-to-Nearest Quantization with minimal quantization errors, eliminating the need for intricate quantization schemes. PEQuant comprises three main techniques. First, we identify the optimal sweet spot for clipping activation outliers for each model. Second, based on our observation that certain layers exhibit significant outliers, we apply selective quantization to layers sensitive to quantization while using W8A8 (weight INT8, activation INT8) for the remaining layers, substantially reducing quantization errors. Finally, we introduce a calibration layer at the models output, trained via knowledge distillation, which further mitigates quantization errors. We implement PEQuant with NVIDIA CUTLASS and evaluate it on various LLMs, including OPT, LLaMA2, and LLaMA3, across perplexity and zero-shot downstream tasks. We also compare PEQuant with two state-of-the-art PTQ methods, LLM.int8() and SmoothQuant. The experimental results indicate that PEQuant is practical and efficient in saving memory spaces, increasing inference speed, and reducing quantization errors.
Post-training quantization (PTQ) for vision-language-action (VLA) models promises smaller and faster VLA deployments. However, when analyzing existing VLA quantization works, we realised that it is hard to reason about the performance of the suggested quantization method compared to existing, similar, or in many cases, simpler PTQ methods. We present LoopQ, a closed-loop benchmark that compares multiple quantization policies on the same checkpoint, layers and engine, varying only the method and the kernels. We evaluate RTN, AWQ, GPTQ, ActQuant and QuantVLA on four VLAs with more than 180,000 episodes of LIBERO and LIBERO-plus, compare against QVLA's reported results on the same layers, and benchmark whether the four public VLA-quantization releases can be reproduced. We benchmark the methods on an NVIDIA H100, an L4, and a Jetson AGX Thor. Our results show that while quantization reliably saves memory, 2-3x for real low-bit models, VLA-specific PTQ methods underperform compared to simpler generic baselines. At 4 bits, RTN, AWQ and GPTQ match or exceed ActQuant and QuantVLA, plain RTN matches the calibrated methods, and plain AWQ matches the success QVLA reports for itself. Standard LIBERO hides the losses that do exist: configurations with no significant LIBERO loss lose 1.5-4.3 points under paired LIBERO-plus perturbations. Speed is a weaker story. At batch 1 on the H100, no quantized pi0.5 or OpenVLA-OFT configuration is faster than its 16-bit model (1.06-1.58x slower on library kernels); only the L4 with int8 kernels gains (4-14%), the Thor is slower on every 4-bit configuration, and the engine alone changes latency by 2.1x. Profiling explains why: pi0.5's quantized backbone is compute-bound, while its memory-bound action expert, 63% of GPU time and 13% of the parameters, stays in 16 bits.
The burgeoning complexity and scale of 3D geometry models pose significant challenges for deployment on resource-constrained platforms. While Post-Training Quantization (PTQ) enables efficient inference without retraining, conventional methods, primarily optimized for 2D Vision Transformers, fail to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead. To address these challenges, we propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline specifically engineered for 3D geometric learning. Our contribution is threefold: (1) To overcome the data-scale bottleneck in 3D datasets, we develop a progressive coarse-to-fine calibration construction strategy that constructs a highly compact subset to achieve both statistical purity and geometric representativeness. (2) We reformulate the quantization interval search as an optimization problem and introduce a ternary-search-based solver, reducing the computational complexity from O(N)\mathcal{O}(N)O(N) to O(log⁡N)\mathcal{O}(\log N)O(logN) for accelerated deployment. (3) To mitigate quantization error accumulation, we propose TRE-Guided Module-wise Compensation, which utilizes a Tail Relative Error (TRE) metric to adaptively identify and rectify distortions in modules sensitive to long-tailed activation outliers. Extensive experiments on the VGGT and Pi3 benchmarks demonstrate that TAPTQ consistently outperforms state-of-the-art PTQ methods in accuracy while significantly reducing calibration time. The code will be released soon.
최신 비전 모델들은 활성화 이상치로 인해 저비트 양자화 시 성능 저하가 크게 발생한다. 이를 완화하기 위해 본 논문에서는 LLM에서 검증된 직교 행렬 기반 회전 변환을 RF-DETR, RT-DETR, YOLOv12에 적용하였다. 실험 결과 회전 변환은 레이어별 첨도를 낮춰 긴 꼬리 분포와 극단적 이상치의 영향을 효과적으로 완화하였고, 저비트 설정에서 탐지 성능 저하를 유의미하게 줄였다. 특히 RF-DETR은 W4A8에서 Baseline에 근접한 성능을 회복하였으며, RT-DETR은 QRT-DETR 대비 추가적인 성능 향상을 통해 저비트 환경에서의 민감도를 낮추었다. YOLOv12 또한 저비트 설정에서 일관된 성능 개선을 보여 양자화 오류 완화에 기여함을 확인하였다.더불어 회전 변환 적용 후에도 추론 시간은 baseline(43.75ms)과 유사한 수준(41.31ms)으로 유지되어, 제안 기법이 추가적인 런타임 오버헤드를 유의미하게 증가시키지 않음을 시사한다.
Post-Training Quantization enables efficient Vision Transformer (ViTs) deployment with a small calibration data, and its prevalent use of uniform quantization harnesses AI accelerator matrix cores for high-speed inference. However, the application of uniform quantization is fundamentally challenged by the extreme non-uniformity of activation distributions.Specifically, the power-law nature of post-Softmax attention scores and the significant inter-channel variance in post-GELU activations create a dilemma for conventional quantization, as it struggles to preserve critical high-magnitude values without sacrificing overall precision. To resolve this core conflict, we introduce UQ-ViT (Uniform Quantization for Vision Transformers), a novel uniform quantization framework designed to reconcile high precision with hardware efficiency. Central to UQ-ViT are two operators: Dynamic Elimination of Maximum (DeMax) and Normalization Quantization (NormQuant). DeMax is a quantization operator for post-Softmax attention scores that utilizes uniform quantization. It dynamically eliminates and preserves dominant values, effectively mitigating quantization loss from the extreme values in the power-law distribution. NormQuant utilizes a per-channel quantization strategy during quantization and reverts to a per-tensor format for dequantization, achieving both high accuracy and computational efficiency. Crucially, it is applicable to any linear layer, enabling effective quantization of post-GELU activations in ViTs. Through extensive experiments on various ViTs and vision tasks, including image classification, object detection, and instance segmentation, we demonstrate that our proposed approach outperforms existing methods, achieving superior accuracy while ensuring hardware friendliness.
양자머신러닝(Quantum Machine Learning, QML)에 대한 기업의 관심이 높아지고 있음에도 불구하고, 조직들은 언제 QML 투자가 고전 머신러닝 대안보다 정당화되는지를 판단할 데이터 기반 프레임워크를 갖추지 못하고 있다. 본 연구는 PRISMA 2020 체계적 문헌고찰을 통해 확인된 35편의 동료심사 연구로부터 추출한 92개 벤치마크 데이터셋을 대상으로, 양자 우위 발현 조건에 대한 최초의 메타분석적 ․ 통계적 검증 연구를 수행함으로써 이러한 공백을 다룬다. 각 데이터셋에 대해 표본 규모, 차원성, 노이즈 수준, 비선형 구조, 특성 간 상관관계를 아우르는 18개의 측정 가능한 데이터 특성을 추출하고, 최고 성능 QML모델과 고전 최신기법(SOTA) 간의 성능 격차(Gap_from_SOTA)를 산출하였다. 비모수 집단비교, HC3 이분산성-강건표준오차를 적용한 다중회귀분석, 그리고 5개 모델(XGBoost, LightGBM, CatBoost, RandomForest, GradientBoosting)에 대한 SHAP 설명가능 인공지능을 활용한 앙상블 머신러닝이라는 세 가지 독립적 분석 방법이 적용되었다. 분석 결과, 노이즈 수준, 비선형성 격차, 최대 특성 간 상관관계가 QML 우위를 결정하는 세 가지 지배적 예측변수로 일관되게 확인되었으며, 5개 앙상블 모델 전반에서 특성 중요도 순위가 동일하게 나타났다. 이들 예측변수로부터 QML이 고전 방법을 능가할 가능성이 높은 조건, 즉 낮은 노이즈, 우호적인 비선형 구조, 약한 특성 간 상관관계를 함께 특징짓는 정량적 의사결정 임계값을 도출하였다. 도메인 분석 결과 에너지/전력 및 기상/기후 분야에서 유의하게 높은 양자 우위가 나타난 반면, 약물/화학 및 금융 분야는 고전 방법에 유리한 것으로 나타났다. 시계열 과제는 QML의 이점이 가장 큰 반면, 다중분류는 고전ML에 유리한 경향을 보였다. 아키텍처 중에서는 QLSTM 계열 모델이 가장 높고 일관된 우위를 달성하였다.