Research map / Large language models / Reasoning and chain of thought

zip2zip: Inference-Time Adaptive Tokenization via Online Compression

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes zip2zip, which applies online LZW compression at inference time to create hypertokens, with a hyper-encoder for their embeddings, and uptrains Phi-3 models via LoRA to cut token counts by 15-40%.

Problem

Static tokenizers with fixed vocabularies fragment domain- or language-specific text into long token sequences, raising compute, latency and cost; enlarging vocabularies yields diminishing returns and static vocabularies cannot adapt at inference time.

Approach

Dynamically expand the active vocabulary during decoding by running online LZW compression over base tokens to form hypertokens, embed hypertokens with a learned hyper-encoder over constituent base-token embeddings, and train the LM on LZW-compressed sequences with an auxiliary reconstruction loss.

Key results

• Token efficiency gains of 13-41% sequence length reduction; e.g., Llama-3 tokenizer code bytes/token 4.1 to 6.3 (+54%).
• On Phi-3.5-4B, zip2zip GSM8K accuracy drops to 0.15 from 0.82 base and 0.40 for continued pretraining, while most other tasks stay near baseline.
• Throughput on H100 for Phi-3.5-4B with 512+256 setting rises from 1347.2 to 2722.1 prefill tokens/s and decode from 54.4 to 79.8 tokens/s.

Measured effects reported in the paper

Surprising findings

• Only about 25% of created hypertokens are reused during generation, indicating codebook waste.
• Merge size M=2 performs worse than M=3 and larger merge sizes, a non-monotonic trend in perplexity and loss.

Limitations

• Performance drop on GSM8K suggests adaptive tokenization harms numerical reasoning.
• Perplexity increases under compression, indicating the model does not reach the theoretically equivalent transported model.
• Codebook grows linearly with context and only about 25% of hypertokens are reused.

Experimental setup

Datasets: HuggingFaceFW/fineweb-edu devngho/the-stack-llm-annotations-v2 AI-MO/NuminaMath-1.5 HuggingFaceH4/ultrachat_200k HuggingFaceFW/fineweb-2 Wikitext The Pile Paloma (mC4, dC4) WMT14 WMT16

Benchmarks: ARC-Challenge ARC-Easy HellaSwag OpenbookQA PIQA Winogrande GSM8K LAMBADA WMT14 En-Fr WMT16 En-De WMT16 En-Ro Wikitext

Metrics: accuracy byte-level perplexity BLEU CHRF TER bytes per token (token efficiency) throughput (tokens/sec)

Baselines: Base model (Phi-3.5-4B, Phi-3-14B) Continued pretraining with LoRA HuggingFace fast BPE tokenizer Dynamic Vocab

Builds on: byte pair encoding LoRA Lempel-Ziv-Welch compression Dynamic Vocab

Closest papers by meaning

• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.90)
• Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.89)
• Rep2Text: Decoding Full Text from a Single LLM Token Representation (NeurIPS 2026 · similarity 0.89)
• MrT5: Dynamic Token Merging for Efficient Byte-level Language Models (ICLR 2025 · similarity 0.89)
• Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.88)
• Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference (AAAI 2025 · similarity 0.88)