Research map / Architectures and efficiency / Attention and transformers
CrossMPT: Cross-attention Message-passing Transformer for Error Correcting Codes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CrossMPT, a transformer ECC decoder that separately updates magnitude and syndrome embeddings via two PCM-masked cross-attention blocks, outperforming ECCT with lower FLOPs and training/inference time.
Problem
Transformer-based neural ECC decoders concatenate magnitude and syndrome inputs and use self-attention without distinguishing them, leading to high memory/computation cost and suboptimal decoding, especially for long codes.
Approach
Treat magnitude and syndrome as distinct modalities and update them alternately with two masked cross-attention blocks, using H^T and H as attention masks so the attention maps directly encode the Tanner-graph magnitude-syndrome relations, mimicking message passing.
Key results
- CrossMPT achieves higher negative-log-BER than ECCT on all listed codes at 4, 5, and 6 dB; e.g., (121,80) LDPC at 6 dB: 18.15 vs 16.44 for ECCT.
- For (63,45) BCH, CrossMPT uses 11.8M FLOPs vs 14.0M for ECCT and mask density 32.45% vs 53.09%.
- For (384,320) WRAN LDPC, training time per epoch drops from 305 s (ECCT) to 104 s (CrossMPT) with FLOPs 10.0M vs 29.3M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ECCT self-attention with separate masked cross-attention updates for magnitude and syndrome | ECCT · BCH, Polar, LDPC, and Turbo codes | negative natural logarithm of BER: 6.39 -> 6.98; 5.92 -> 7.52; 6.40 -> 7.06; 4.74 -> 5.55 (4 dB) | Helped |
| replace ECCT self-attention with separate masked cross-attention updates for magnitude and syndrome | ECCT · LDPC codes | negative natural logarithm of BER | Helped |
| add masking of magnitude-magnitude and syndrome-syndrome relationships (ablation) | ECCT · BCH (31,16) | negative natural logarithm of BER: 6.39 -> 6.52; 8.29 -> 8.55; 10.66 -> 11.42 | No clear effect |
| add masking of magnitude-magnitude and syndrome-syndrome relationships (ablation) | ECCT · BCH (63,45) | negative natural logarithm of BER: 5.60 -> 5.53; 7.79 -> 7.74; 10.93 -> 10.88 | No clear effect |
| add masking of magnitude-magnitude and syndrome-syndrome relationships (ablation) | ECCT · Polar (64,48) | negative natural logarithm of BER: 6.36 -> 6.25 | No clear effect |
| add masking of magnitude-magnitude and syndrome-syndrome relationships (ablation) | ECCT · LDPC (121,60) | negative natural logarithm of BER: 5.17 -> 4.98 | No clear effect |
| share cross-attention block parameters instead of training them separately (ablation) | CrossMPT · ECC decoding | decoding performance | No clear effect |
| apply denoising diffusion training to CrossMPT (ablation) | CrossMPT · BCH, Polar, and MacKay codes | negative natural logarithm of BER: 5.03 -> 5.23; 7.52 -> 10.21; 7.97 -> 8.85 (4 dB) | Helped |
| use CrossMPT rather than ECCT | ECCT · BCH (63,45) | FLOPs: 14.0~M -> 11.8~M | Helped |
| use CrossMPT rather than ECCT | ECCT · WRAN (384,320) | training time per epoch: 305~s -> 104~s | Helped |
Surprising findings
- Separate query/key-value roles for magnitude and syndrome perform nearly identically when weights are trained separately, so weights are shared.
- Masking magnitude-magnitude and syndrome-syndrome relations in ECCT yields no significant performance change.
Limitations
- Gains over ECCT are smaller on some codes (e.g., polar (64,48) at 4 dB) and CrossMPT does not beat SCL (L=4) on several polar codes.
- Evaluated only on AWGN with all-zero codeword training and simulated channels; Rayleigh results are only in appendix.
Experimental setup
Datasets: simulated AWGN channel codewords (BCH, polar, LDPC, MacKay, CCSDS, turbo PCMs)
Benchmarks: BCH (31,16) BCH (63,36/45/51) Polar (64,32)-(128,96) LDPC (49,24)-(121,80) MacKay (96,48) CCSDS (128,64) Turbo (132,40)
Metrics: negative natural log of BER FLOPs inference time training time per epoch mask density
Baselines: BP Hyper BP AR BP ECCT SCL decoder DDECCT ML decoder
Builds on: ECCT belief propagation min-sum denoising diffusion ECCT
Closest papers by meaning
- Efficient Message-Passing Transformer for Error Correcting Codes (ICLR 2026 · similarity 0.90)
- Denoising Diffusion Error Correction Codes (ICLR 2023 · similarity 0.84)
- Learning Linear Block Error Correction Codes (ICML 2024 · similarity 0.83)
- DMax: Aggressive Parallel Decoding for dLLMs (NeurIPS 2026 · similarity 0.80)
- Structured Masked Diffusion for Joint Multiuser Decoding (NeurIPS 2026 · similarity 0.80)
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (NeurIPS 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.