Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

IEEE Transactions on Machine Learning in Communications and Networking

Spectral-Guided Mask Generation for Error Correction Code Transformers

Linh Nguyen, Quoc Bao Phan, Tuy Tan Nguyen

Published Jan 1, 2026• 0 citations
Publisher

Abstract

Neural decoders have recently advanced error correction performance, with transformer-based architectures in the error correction code transformer (ECCT) framework achieving state-of-the-art results among neural decoders. However, the multiple-masks ECCT suffers from suboptimal mask selection, which impacts decoding performance. In this work, we propose spectral-guided ECCT, leveraging the Laplacian matrix and its eigenvectors from the variable-variable graph to identify weakly connected node clusters for optimized mask generation. By embedding spectral properties into the decoding process, our method constructs more effective masks that improve information flow across variable and check nodes. Experimental results demonstrate consistent gains over existing ECCT-based models, achieving superior decoding accuracy and reliability while significantly improving training efficiency. This work highlights the potential of spectral graph theory to address structural challenges in neural decoders and provides a principled direction for future error correction design.

DOI
10.1109/tmlcn.2026.3698041

Related papers

Multiple-Masks Error Correction Code Transformer for Short Block Codes

Seong‐Joon Park, Hee-Youl Kwak, Sang‐Hyo Kim +3 more

With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.

IEEE Journal on Selected Areas in Communications•Apr 10, 2025•13 citations

How to Mask in Error Correction Code Transformer: Systematic and Double Masking

Seong‐Joon Park, Kwak, Hee-Youl, Sang‐Hyo Kim +3 more

In communication and storage systems, error correction codes (ECCs) are pivotal in ensuring data reliability. As deep learning's applicability has broadened across diverse domains, there is a growing research focus on neural network-based decoders that outperform traditional decoding algorithms. Among these neural decoders, Error Correction Code Transformer (ECCT) has achieved the state-of-the-art performance, outperforming other methods by large margins. To further enhance the performance of ECCT, we propose two novel methods. First, leveraging the systematic encoding technique of ECCs, we introduce a new masking matrix for ECCT, aiming to improve the performance and reduce the computational complexity. Second, we propose a novel transformer architecture of ECCT called a double-masked ECCT. This architecture employs two different mask matrices in a parallel manner to learn more diverse features of the relationship between codeword bits in the masked self-attention blocks. Extensive simulation results show that the proposed double-masked ECCT outperforms the conventional ECCT, achieving the state-of-the-art decoding performance with significant margins.

arXiv:2308.08128•Aug 16, 2023•9 citations

Accelerating Error Correction Code Transformers

Matan Levy, Yoni Choukroun, Lior Wolf

Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.

arXiv:2410.05911•Oct 8, 2024•7 citations

Error Correction Code Residual Transformer

Elaheh Bassak, Elvino Silveira Sousa

The Error Correction Code Transformer (ECCT) is a seminal state-of-the-art model-free neural decoder for linear block codes, leveraging parity-check-matrix-aware masked self-attention to exploit algebraic code structure. A well-known limitation of deep Transformer architectures trained with Pre-Layer Normalization (PreNorm), however, is the uncontrolled growth of hidden-state magnitudes with depth, which progressively dilutes the contribution of individual layers, a phenomenon known as PreNorm dilution, thus effectively limiting the benefit of increased architectural depth. In this paper, we propose EC-CRT, a novel Transformer-based neural decoder that integrates Attention Residuals (AttnRes) into the ECCT framework to address this limitation. AttnRes replaces fixed, additive residual connections with learned, input-dependent softmax attention over all preceding layer outputs, enabling each decoding layer to selectively aggregate the most informative earlier representations and directly mitigating PreNorm dilution. Extensive experiments on LDPC(49,24), Polar(64,32), BCH(31,16), and BCH(63,51) codes under a deep configuration and a shallow configuration demonstrate that ECCRT consistently and substantially outperforms ECCT in the deep setting across all codes and SNR values, with gains that widen at higher SNRs, while delivering comparable performance in the shallow setting. The improvement is achieved with negligible additional parameters, only one learned pseudo-query vector per Transformer layer. To the best of our knowledge, this is the first work to demonstrate the benefit of Attention Residuals outside the large language model regime, confirming that this technique generalizes across model scales and application domains.

IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)•Sep 1, 2026•0 citations

Lowering the Error Floor of Error Correction Code Transformer

Park, Taewoo, Seong‐Joon Park, Kwak, Hee-Youl +2 more

With the success of transformer architectures across diverse applications, the error correction code transformer (ECCT) has gained significant attention for its superior decoding performance. In spite of its advantages, the error floor problem in ECCT decoding remains unexplored. We present the first investigation into this issue, revealing that ECCT encounters error floors, limiting its effectiveness in practical settings. To address this error floor problem, we adopt a hybrid decoding framework that integrates ECCT with conventional hard decision decoders. Unlike prior hybrid decoding schemes, our key contribution lies in proposing a novel loss function that explicitly takes into account the interaction between ECCT and hard decision decoders during training. The proposed loss function guides ECCT to focus on residual errors that are not corrected by the hard decision stages, effectively lowering the error floor. Simulation results confirm that the hybrid decoder trained with the proposed loss function achieves substantial performance gains over standard ECCT in both the waterfall and the error floor regions.

arXiv:2502.09065•Feb 13, 2025•3 citations

Spectral-Aligned Pruning for Universal Error-Correcting Code Transformers

Sanghyeon Cho, Taewoo Park, Seong‐Joon Park +4 more

Universal channel decoders based on transformers-such as the Foundation Error Correction Code Transformer (FECCT)-achieve competitive decoding performance across diverse code families with a single shared backbone, optionally followed by code-specific finetuning. However, the high computational complexity and large parameter footprint of FECCT present substantial obstacles to practical deployment. To address these challenges, we investigate structured pruning for FECCT and propose Spectral-Aligned Pruning (SAP), a structure-aware framework that enables cross-code reuse of structured pruning masks by leveraging the spectrum of the corresponding bipartite graph. SAP is grounded in classical graph analysis of codes: the two algebraically largest adjacency eigenvalues provide compact spectral proxies for degree scale, expansion ratio, and minimum-distance lower bounds. These quantities are directly relevant to decoding performance: degree scale reflects how densely codeword bits and parity checks are connected; expansion ratio influences how information propagates across the bipartite graph; and minimum distance characterizes codeword separation. Based on this connection, SAP uses these two leading eigenvalues as a lightweight code signature for pruning-mask retrieval. Empirically, this two-dimensional signature yields stable library selection equivalent to higher-dimensional spectral signatures in our evaluation. After pruning, SAP uses parameter-efficient low-rank adaptation (LoRA) to recover decoding performance while storing only a small number of additional adapter parameters per code. Experiments across diverse codes show that SAP achieves decoding performance comparable to dedicated per-code pruning, while enabling substantial reductions in computational cost and model memory footprint through kernel-level structured pruning. Code is available at https://github.com/iil-postech/sap.

arXiv:2602.01602•Feb 2, 2026•0 citations
Neural decoders have recently advanced error correction performance, with transformer-based architectures in the error correction code transformer (ECCT) framework achieving state-of-the-art results among neural decoders. However, the multiple-masks ECCT suffers from suboptimal mask selection, which impacts decoding performance. In this work, we propose spectral-guided ECCT, leveraging the Laplacian matrix and its eigenvectors from the variable-variable graph to identify weakly connected node clusters for optimized mask generation. By embedding spectral properties into the decoding process, our method constructs more effective masks that improve information flow across variable and check nodes. Experimental results demonstrate consistent gains over existing ECCT-based models, achieving superior decoding accuracy and reliability while significantly improving training efficiency. This work highlights the potential of spectral graph theory to address structural challenges in neural decoders and provides a principled direction for future error correction design.
With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.
In communication and storage systems, error correction codes (ECCs) are pivotal in ensuring data reliability. As deep learning's applicability has broadened across diverse domains, there is a growing research focus on neural network-based decoders that outperform traditional decoding algorithms. Among these neural decoders, Error Correction Code Transformer (ECCT) has achieved the state-of-the-art performance, outperforming other methods by large margins. To further enhance the performance of ECCT, we propose two novel methods. First, leveraging the systematic encoding technique of ECCs, we introduce a new masking matrix for ECCT, aiming to improve the performance and reduce the computational complexity. Second, we propose a novel transformer architecture of ECCT called a double-masked ECCT. This architecture employs two different mask matrices in a parallel manner to learn more diverse features of the relationship between codeword bits in the masked self-attention blocks. Extensive simulation results show that the proposed double-masked ECCT outperforms the conventional ECCT, achieving the state-of-the-art decoding performance with significant margins.
Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.
The Error Correction Code Transformer (ECCT) is a seminal state-of-the-art model-free neural decoder for linear block codes, leveraging parity-check-matrix-aware masked self-attention to exploit algebraic code structure. A well-known limitation of deep Transformer architectures trained with Pre-Layer Normalization (PreNorm), however, is the uncontrolled growth of hidden-state magnitudes with depth, which progressively dilutes the contribution of individual layers, a phenomenon known as PreNorm dilution, thus effectively limiting the benefit of increased architectural depth. In this paper, we propose EC-CRT, a novel Transformer-based neural decoder that integrates Attention Residuals (AttnRes) into the ECCT framework to address this limitation. AttnRes replaces fixed, additive residual connections with learned, input-dependent softmax attention over all preceding layer outputs, enabling each decoding layer to selectively aggregate the most informative earlier representations and directly mitigating PreNorm dilution. Extensive experiments on LDPC(49,24), Polar(64,32), BCH(31,16), and BCH(63,51) codes under a deep configuration and a shallow configuration demonstrate that ECCRT consistently and substantially outperforms ECCT in the deep setting across all codes and SNR values, with gains that widen at higher SNRs, while delivering comparable performance in the shallow setting. The improvement is achieved with negligible additional parameters, only one learned pseudo-query vector per Transformer layer. To the best of our knowledge, this is the first work to demonstrate the benefit of Attention Residuals outside the large language model regime, confirming that this technique generalizes across model scales and application domains.
With the success of transformer architectures across diverse applications, the error correction code transformer (ECCT) has gained significant attention for its superior decoding performance. In spite of its advantages, the error floor problem in ECCT decoding remains unexplored. We present the first investigation into this issue, revealing that ECCT encounters error floors, limiting its effectiveness in practical settings. To address this error floor problem, we adopt a hybrid decoding framework that integrates ECCT with conventional hard decision decoders. Unlike prior hybrid decoding schemes, our key contribution lies in proposing a novel loss function that explicitly takes into account the interaction between ECCT and hard decision decoders during training. The proposed loss function guides ECCT to focus on residual errors that are not corrected by the hard decision stages, effectively lowering the error floor. Simulation results confirm that the hybrid decoder trained with the proposed loss function achieves substantial performance gains over standard ECCT in both the waterfall and the error floor regions.
Universal channel decoders based on transformers-such as the Foundation Error Correction Code Transformer (FECCT)-achieve competitive decoding performance across diverse code families with a single shared backbone, optionally followed by code-specific finetuning. However, the high computational complexity and large parameter footprint of FECCT present substantial obstacles to practical deployment. To address these challenges, we investigate structured pruning for FECCT and propose Spectral-Aligned Pruning (SAP), a structure-aware framework that enables cross-code reuse of structured pruning masks by leveraging the spectrum of the corresponding bipartite graph. SAP is grounded in classical graph analysis of codes: the two algebraically largest adjacency eigenvalues provide compact spectral proxies for degree scale, expansion ratio, and minimum-distance lower bounds. These quantities are directly relevant to decoding performance: degree scale reflects how densely codeword bits and parity checks are connected; expansion ratio influences how information propagates across the bipartite graph; and minimum distance characterizes codeword separation. Based on this connection, SAP uses these two leading eigenvalues as a lightweight code signature for pruning-mask retrieval. Empirically, this two-dimensional signature yields stable library selection equivalent to higher-dimensional spectral signatures in our evaluation. After pruning, SAP uses parameter-efficient low-rank adaptation (LoRA) to recover decoding performance while storing only a small number of additional adapter parameters per code. Experiments across diverse codes show that SAP achieves decoding performance comparable to dedicated per-code pruning, while enabling substantial reductions in computational cost and model memory footprint through kernel-level structured pruning. Code is available at https://github.com/iil-postech/sap.