Research map / Large language models / Reasoning and chain of thought
Gram2Token: Enabling Run-time GPU-Native Grammar-Constrained Decoding for LLMs
Hantao Hua, Jiming Su, Hao Tang, Yiping Yao, Feng Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gram2Token, a GPU-native grammar-constrained decoding approach for LLMs that avoids CPU-side parser control flow and CPU-GPU synchronization in the token decoding loop.
Problem
Grammar-constrained decoding for structured LLM outputs (JSON, SQL, domain-specific programs) often runs byte-level parser logic inside the token-level decoding loop, causing CPU-side control flow and CPU-GPU synchronization bottlenecks under continuous batching.
Approach
Moves grammar constraint enforcement onto the GPU so that run-time masking runs natively within the decoding loop, removing per-step CPU parser execution and synchronization. (Abstract truncated; details beyond this are not stated.)
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token pre-alignment (precomputed token-level transitions) (ablation) | Gram2Token · Qwen3-8B, B=64 | throughput (TPS): 1567 -> 1873 | Improved |
| add token categorization (ablation) | Gram2Token · Qwen3-8B, B=64 | throughput (TPS): 419 -> 1873 | Improved |
| add token categorization (ablation) | Gram2Token · Qwen3-8B, B=64 | resident table memory (MB): 754 -> 92 | Improved |
| add token pre-alignment (ablation) | Gram2Token · Qwen3-8B, B=64 | resident table memory (MB): 862 -> 92 | Improved |
| add token pre-alignment (ablation) | Gram2Token · Qwen3-8B, B=64 | time-to-first-token (ms): 787 -> 1060 | Hurt |
| use Gram2Token instead of Pre3 | Pre3 · Schema-diverse batching, 4 models | throughput speedup: 1.38x geometric mean, up to 1.85x | Improved |
| use Gram2Token instead of Pre3 | Pre3 · LLaMA3-8B, B=64 | time-to-first-token (ms): 2898 -> 5076 | Hurt |
| increase batch size from 4 to 512 (ablation) | Gram2Token · break-even vs Pre3 | tokens until break-even: 539 -> 46 | Improved |
| use Gram2Token instead of Pre3 on Mistral-7B at small batch | Pre3 · Mistral-7B, B=4 | throughput (TPS): 32 -> 44 | Improved |
Closest papers by meaning
- Flexible and Efficient Grammar-Constrained Decoding (ICML 2025 · similarity 0.89)
- Grammar-Aligned Decoding (NeurIPS 2024 · similarity 0.89)
- TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation (ICML 2025 · similarity 0.89)
- Earley-Driven Dynamic Pruning for Efficient Structured Decoding (ICML 2025 · similarity 0.89)
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.88)
- AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
