Research map / Large language models / Retrieval and knowledge
BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BTR, which precomputes 1-bit calibrated binary token representations of passages in the FiD reader, plus distillation and recovery losses and token merging, cutting storage ~100x and speeding reader inference 2-4x while keeping over 95% of Atlas performance.
Problem
Retrieval-augmented LM readers are slow because they cross-attend over many retrieved passages at inference; prior precomputation methods (DeFormer, LUMEN) need terabytes of storage for Wikipedia-scale corpora.
Approach
Binarize passage token representations at an intermediate encoder layer after layernorm (calibrated binarization with saved variance to recover scale), train with straight-through estimation, and recover accuracy via passage representation recovery and query-aware passage token distillation; add offline and runtime token merging to cut storage and compute.
Key results
- BTR-Atlas base reaches 49.5 EM on NQ vs 52.1 for Atlas base (95%) with 3.1x speedup and 127 GB storage for 3.2B tokens.
- Binary passage representations reduce storage from 12,804 GB to 127 GB on NQ (over 100x) per the ablation.
- BTR-Atlas large achieves 4.0x speedup on NQ while retaining 96% of Atlas large EM (56.1 vs 58.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add binary passage representations and BTR compression to Atlas base | Atlas base · NQ, TriviaQA, WebQuestions, FEVER, MMLU | inference speedup: 3.1; 2.5; 2.6; 3.1; 2.6 | Helped |
| Add binary passage representations and BTR compression to Atlas large | Atlas large · NQ, TriviaQA, WebQuestions, FEVER, MMLU | inference speedup: 4.0; 3.9; 3.6; 4.1; 2.5 | Helped |
| Add binary passage representations and BTR compression to Atlas base | Atlas base · NQ, TriviaQA, WebQuestions, FEVER, MMLU | task score: 52.1 -> 49.5; 69.3 -> 66.7; 46.4 -> 43.8; 72.9 -> 70.2; 38.6 -> 35.4 | Hurt |
| Add binary passage representations to BTR-Atlas base (ablation) | BTR-Atlas base · NQ | storage: 12,804 -> 127 | Helped |
| Add binary passage representations to BTR-Atlas base (ablation) | BTR-Atlas base · NQ | accuracy: 50.8 -> 49.5 | Hurt |
| Add offline token compression to BTR-Atlas base (ablation) | BTR-Atlas base · NQ | storage: 159 -> 127 | Helped |
| Add offline token compression to BTR-Atlas base (ablation) | BTR-Atlas base · NQ | accuracy: 49.8 -> 49.5 | Hurt |
| Add runtime intra-passage compression to BTR-Atlas base (ablation) | BTR-Atlas base · NQ | throughput: 28.1 -> 31.5 | Helped |
| Add runtime cross-passage compression to BTR-Atlas base (ablation) | BTR-Atlas base · NQ | throughput: 24.6 -> 31.5 | Helped |
| Add passage representation recovery loss (ablation) | BTR-Atlas base · NQ | accuracy: 47.4 -> 49.5 | Helped |
| Add query-aware passage token distillation (ablation) | BTR-Atlas base · NQ | accuracy: 48.2 -> 49.5 | Helped |
| Increase offline token compression ratio (ablation) | BTR-Atlas base · NQ | storage: 151; 143; 135; 127; 119; 111; 103; 95; 87; 80 | Helped |
Surprising findings
- Directly annealing tanh binarization degrades representation quality because large-scale pre-layernorm activations saturate, and a naive layernorm before binarization performs worse.
Limitations
- Extending to decoder-only readers is non-trivial because passage representations depend on the query sequentially.
- Speedup shrinks for longer queries and the approach uses representation sizes that may not scale to larger models without compression.
- Accuracy still drops by about 4-5% relative to the original models, and the encoder-only BERT variant cannot use runtime compression.
Experimental setup
Datasets: NaturalQuestions TriviaQA WebQuestions FEVER MMLU Wikipedia dump (Dec 2018)
Benchmarks: NaturalQuestions TriviaQA WebQuestions FEVER MMLU
Metrics: exact match accuracy queries per second speedup storage (GB)
Baselines: Atlas Atlas-Q DeFormer DensePhrase LLaMA2-7B
Builds on: Atlas Fusion-in-Decoder DeFormer DensePhrase DPR
Closest papers by meaning
- One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models (AAAI 2025 · similarity 0.88)
- Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT (ICML 2024 · similarity 0.88)
- zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.87)
- ARC-Encoder: learning compressed text representations for large language models (NeurIPS 2026 · similarity 0.87)
- Rep2Text: Decoding Full Text from a Single LLM Token Representation (NeurIPS 2026 · similarity 0.87)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.