Research map / Large language models / Reasoning and chain of thought

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

ICML 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TriAttention, a KV cache compression method that scores keys using pre-RoPE Q/K concentration centers via a trigonometric series over Q-K distance, plus norm-based terms, to retain top keys during long chain-of-thought reasoning.

Problem

KV cache memory bottleneck in long reasoning decoding; existing post-RoPE attention-based compression methods estimate key importance from few usable recent queries (rotated by RoPE), leading to poor top-key selection and unstable reasoning.

Approach

Observes that pre-RoPE Q/K vectors concentrate around fixed non-zero centers stable across positions; substituting these centers into the RoPE logit yields a trigonometric series in Q-K distance that predicts which distances receive attention, so keys are scored from centers and positions without observing attention, with norms added as a complement weighted by (1 - mean resultant length).

Key results

• On AIME25 with 32K generation, TriAttention matches Full Attention accuracy (40.8%) at 2.5x higher throughput or 10.7x KV memory reduction, while R-KV reaches about half the accuracy at the same efficiency.
• At a 2048-token budget on AIME25 with Qwen3-8B, TriAttention scores 32.9% vs R-KV's 17.5%; on AIME24 it scores 42.1% vs 25.4%.
• On MATH 500 with a 1024-token budget, TriAttention reaches 68.4% vs Full Attention's 69.6%, with 6.3x throughput (1405 vs 223 tokens/s).

Surprising findings

• Calibrating Q/K statistics on coding data gives comparable reasoning accuracy to reasoning-domain calibration, and calibration data quality (e.g., HTML vs chat) shows no clear accuracy correlation.
• On AIME25 with a 4096 budget, TriAttention (43.3%) exceeds Full Attention (40.8%).

Limitations

• Current implementation lacks a dedicated inference kernel, so further latency reductions are possible.
• Evaluation is limited to mathematical reasoning and general benchmarks; coding and agentic tasks are left to future work.
• Uses a fixed per-model budget across heads; head-specific budgets are not explored.

Experimental setup

Datasets: AIME 2024 AIME 2025 MATH 500 LongBench RULER coding calibration data ShareGPT chat data

Benchmarks: AIME24 AIME25 MATH 500 LongBench RULER Recursive State Query

Metrics: accuracy pass rate throughput (tokens/s) KV memory reduction stack exact match

Baselines: Full Attention SnapKV R-KV H2O LazyEviction StreamingLLM PyramidKV KnormPress Ada-KV+SnapKV

Builds on: RoPE R-KV SnapKV H2O StreamingLLM

Closest papers by meaning

• Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression (ICML 2026 · similarity 0.90)
• DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads (ICLR 2025 · similarity 0.90)
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.89)
• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference (ICML 2026 · similarity 0.89)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.89)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.89)