Research map / Architectures and efficiency / Attention and transformers

Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PHSA, a natively trainable block-sparse attention that builds block representatives by fusing mean-pooled global keys with punctuation-token keys, plus extreme-sparsity training, evaluated on Qwen3-0.6B/8B against dense attention and InfLLM v2.

Problem

Sparse attention methods use coarse average-pooled block representatives that blur semantic boundaries and degrade sharply under extremely low token activation ratios, losing critical information in long contexts.

Approach

Augment block-level mean-pooled key representatives with a punctuation-only pooled representative (punctuation tokens as semantic boundary anchors), fused via a scalar gate, and train natively under very low Top-K selection.

Key results

• On NIAH at 4k with training Top-K=2, PHSA scores 81.8 at inference Top-K=1 versus 78.6 for InfLLM v2 (training-free comparison also reported).
• At 20B training tokens PHSA averages 48.28 on general benchmarks vs 47.38 for InfLLM v2 and 48.29 for dense attention; at 100B tokens PHSA at Top-K=4 averages 49.24 vs dense 48.96.
• On LongBench, PHSA averages 28.13 vs dense 27.86 and InfLLM v2 27.40 (Qwen3-0.6B-scale training).

Surprising findings

• Training with a very low Top-K=1 or excessively small Top-K degrades performance, and Top-K=0 yields complete failure (0 scores), indicating a collapse at extreme sparsity.
• Adding Chinese punctuation to training (PHSA_en+zh) helped Chinese MQZ, suggesting English-only punctuation set hurts non-English performance.

Limitations

• Wholesale use of all punctuation marks introduces noise; a screened subset is not explored.
• Experiments are limited to small Qwen3 models (0.6B training, 8B training-free) and English punctuation during training.
• Efficiency/latency gains are asserted rather than measured in the presented text.

Experimental setup

Datasets: dclm map-cc ultrachat tuluv3 finemath megamath proprietary educational data

Benchmarks: GSM8K MathQA MATH ARC-C ARC-C-ZH ARC-E MMLU CMMLU BBH LAMBADA XStoryCloze HumanEval

Metrics: accuracy average score NIAH score

Baselines: Dense attention InfLLM v2 NSA

Builds on: InfLLM v2 NSA SepLLM

Closest papers by meaning

• Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.91)
• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.90)
• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.90)
• Training-Free Long-Context Scaling of Large Language Models (ICML 2024 · similarity 0.89)
• SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.89)
• Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning (NeurIPS 2025 · similarity 0.89)