AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Gram2Token: Enabling Run-time GPU-Native Grammar-Constrained Decoding for LLMs

Hantao Hua, Jiming Su, Hao Tang, Yiping Yao, Feng Zhu

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Gram2Token, a GPU-native grammar-constrained decoding approach for LLMs that avoids CPU-side parser control flow and CPU-GPU synchronization in the token decoding loop.

Problem

Grammar-constrained decoding for structured LLM outputs (JSON, SQL, domain-specific programs) often runs byte-level parser logic inside the token-level decoding loop, causing CPU-side control flow and CPU-GPU synchronization bottlenecks under continuous batching.

Approach

Moves grammar constraint enforcement onto the GPU so that run-time masking runs natively within the decoding loop, removing per-step CPU parser execution and synchronization. (Abstract truncated; details beyond this are not stated.)

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add token pre-alignment (precomputed token-level transitions) (ablation)Gram2Token · Qwen3-8B, B=64throughput (TPS): 1567 -> 1873Improved
add token categorization (ablation)Gram2Token · Qwen3-8B, B=64throughput (TPS): 419 -> 1873Improved
add token categorization (ablation)Gram2Token · Qwen3-8B, B=64resident table memory (MB): 754 -> 92Improved
add token pre-alignment (ablation)Gram2Token · Qwen3-8B, B=64resident table memory (MB): 862 -> 92Improved
add token pre-alignment (ablation)Gram2Token · Qwen3-8B, B=64time-to-first-token (ms): 787 -> 1060Hurt
use Gram2Token instead of Pre3Pre3 · Schema-diverse batching, 4 modelsthroughput speedup: 1.38x geometric mean, up to 1.85xImproved
use Gram2Token instead of Pre3Pre3 · LLaMA3-8B, B=64time-to-first-token (ms): 2898 -> 5076Hurt
increase batch size from 4 to 512 (ablation)Gram2Token · break-even vs Pre3tokens until break-even: 539 -> 46Improved
use Gram2Token instead of Pre3 on Mistral-7B at small batchPre3 · Mistral-7B, B=4throughput (TPS): 32 -> 44Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy