Transformer architectures have emerged as promising deep learning (DL) tools for modeling complex sequence-to-sequence interactions in channel decoding. However, existing transformer-based decoders for error correction codes (ECCs) demonstrate inferior decoding performance and generalization capabilities compared to conventional algebraic decoders, especially in short-code regimes. In this work, we propose a novel latent-attention based transformer (LAT) decoder for polar codes that addresses these limitations through three pivotal innovations. First, we develop a latent-attention mechanism that supersedes the conventional self-attention mechanism. This architectural modification enables independent learning of the Query and Key matrices for code-aware attention computation, decoupling them from the Value matrix. As a result, the LAT decoder emphasizes position-wise decoding interactions while mitigating interference from context correlations that are misaligned with the inherent code structure. Second, we devise an advanced training framework incorporating three synergistic components: entropy-aware importance sampling that emphasizes low-probability regions in the signal constellation space, experience reflow that introduces empirical labels to improve characterization of decoding boundaries, and dynamic label smoothing for likelihood-based regularization. Third, we propose a code-aware masking scheme which allows dynamic adaptation to varying code configurations without modifying the network architecture. Numerical evaluations demonstrate that the proposed LAT decoder achieves near maximum-likelihood (ML) performance in terms of both bit error rate (BER) and block error rate (BLER) for short-length polar codes. Moreover, the architecture exhibits robust generalization capabilities across a wide range of code rates and code lengths.
Transformer architectures have emerged as promising deep learning (DL) tools for modeling complex sequence-to-sequence interactions in channel decoding. However, existing transformer-based decoders for error correction codes (ECCs) demonstrate inferior decoding performance and generalization capabilities compared to conventional algebraic decoders, especially in short-code regimes. In this work, we propose a novel latent-attention based transformer (LAT) decoder for polar codes that addresses these limitations through three pivotal innovations. First, we develop a latent-attention mechanism that supersedes the conventional self-attention mechanism. This architectural modification enables independent learning of the Query and Key matrices for code-aware attention computation, decoupling them from the Value matrix. As a result, the LAT decoder emphasizes position-wise decoding interactions while mitigating interference from context correlations that are misaligned with the inherent code structure. Second, we devise an advanced training framework incorporating three synergistic components: entropy-aware importance sampling that emphasizes low-probability regions in the signal constellation space, experience reflow that introduces empirical labels to improve characterization of decoding boundaries, and dynamic label smoothing for likelihood-based regularization. Third, we propose a code-aware masking scheme which allows dynamic adaptation to varying code configurations without modifying the network architecture. Numerical evaluations demonstrate that the proposed LAT decoder achieves near maximum-likelihood (ML) performance in terms of both bit error rate (BER) and block error rate (BLER) for short-length polar codes. Moreover, the architecture exhibits robust generalization capabilities across a wide range of code rates and code lengths.
The Error Correction Code Transformer (ECCT) is a seminal state-of-the-art model-free neural decoder for linear block codes, leveraging parity-check-matrix-aware masked self-attention to exploit algebraic code structure. A well-known limitation of deep Transformer architectures trained with Pre-Layer Normalization (PreNorm), however, is the uncontrolled growth of hidden-state magnitudes with depth, which progressively dilutes the contribution of individual layers, a phenomenon known as PreNorm dilution, thus effectively limiting the benefit of increased architectural depth. In this paper, we propose EC-CRT, a novel Transformer-based neural decoder that integrates Attention Residuals (AttnRes) into the ECCT framework to address this limitation. AttnRes replaces fixed, additive residual connections with learned, input-dependent softmax attention over all preceding layer outputs, enabling each decoding layer to selectively aggregate the most informative earlier representations and directly mitigating PreNorm dilution. Extensive experiments on LDPC(49,24), Polar(64,32), BCH(31,16), and BCH(63,51) codes under a deep configuration and a shallow configuration demonstrate that ECCRT consistently and substantially outperforms ECCT in the deep setting across all codes and SNR values, with gains that widen at higher SNRs, while delivering comparable performance in the shallow setting. The improvement is achieved with negligible additional parameters, only one learned pseudo-query vector per Transformer layer. To the best of our knowledge, this is the first work to demonstrate the benefit of Attention Residuals outside the large language model regime, confirming that this technique generalizes across model scales and application domains.
With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.
Polar codes have been chosen by 3GPP as the official error-correcting codes in the control plane of 5G NR enhanced Mobile Broadband (eMBB) due to their low complexity decoding and near-DMC capacity achievement. Recently, deep learning methods have produced promising results in many fields, including linear and polar code decoding. Using end-to-end DL approach provides flexibility, thus empowering us to integrate powerful DL algorithms into channel decoders. In this paper, we propose a novel one-shot decoding framework for polar codes using Attention-based Recurrent Neural Network (ATRNN). Furthermore, we evaluate the decoding performance of ATRNN using bit error rate (BER) and compare it with the state-of-the-art techniques such as successive cancellation (SC).
We propose a self-supervised deep learning-based decoding scheme that enables one-shot decoding of polar codes. In the proposed scheme, rather than using the information bit vectors as labels for training the neural network (NN) through supervised learning as the conventional scheme did, the NN is trained to function as a bounded distance decoder by leveraging the generator matrix of polar codes through self-supervised learning. This approach eliminates the reliance on predefined labels, empowering the potential to train directly on the actual data within communication systems and thereby enhancing the applicability. Furthermore, computer simulations demonstrate that (i) the bit error rate (BER) and block error rate (BLER) performances of the proposed scheme can approach those of the maximum a posteriori (MAP) decoder for very short packets and (ii) the proposed NN decoder (NND) exhibits much superior generalization ability compared to the conventional one.
Error correcting codes (ECCs) are indispensable for reliable transmission in communication systems. The recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance. In this paper, we propose a novel Cross-attention Message-Passing Transformer (CrossMPT), which shares key operational principles with conventional message-passing decoders. While conventional transformer-based decoders employ self-attention mechanism without distinguishing between the types of input vectors (i.e., magnitude and syndrome vectors), CrossMPT updates the two types of input vectors separately and iteratively using two masked cross-attention blocks. The mask matrices are determined by the code's parity-check matrix, which explicitly captures the irrelevant relationship between two input vectors. Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders for various code classes. Notably, CrossMPT achieves this decoding performance improvement, while significantly reducing the memory usage, complexity, inference time, and training time.
In this paper, we propose a novel deep learning-based framework for the joint design and decoding of linear block codes, the end-to-end deep coding cross-attention message-passing Transformer (E2E DC-CrossMPT). To improve linear block code design and decoding, we redesign the conventional error correction code (ECC) decoder, CrossMPT, to fit within an end-to-end framework. The redesigned decoder separately utilizes magnitude and syndrome vectors obtained from the received signals as inputs. It further employs one-hot encoding based syndrome embedding and incorporates a parity-check matrix into the output layers. Experimental results demonstrate that, across various code lengths and code rates, E2E DC-CrossMPT consistently outperforms both traditional decoding algorithms and a conventional end-to-end deep coding model in terms of decoding performance. Moreover, the codes designed by E2E DC-CrossMPT achieve superior error-correction capability compared with both traditional linear block codes and those designed by the conventional end-to-end deep coding model, while requiring lower computational complexity.