Research map / Large language models / Reasoning and chain of thought
PinTok: Tokenizers Deserve Dedicated Pinned CPU-Compute and Memory
Sean Choi, Myungheon Chin, Ernest K. Ryu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PinTok, a system that runs LLM tokenizers on dedicated pinned CPU compute and memory to cut inference overhead from scheduling, core selection, and data copying.
Problem
Tokenization in LLM inference runs as a standard CPU process, incurring overhead from scheduling delays, core selection, data copying, and other system-level costs.
Approach
Treats the tokenizer as a component worth accelerating by giving it dedicated pinned CPU cores and memory rather than running it as a generic OS process.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PinTok pinned tokenizer instead of Python tokenizer | tokenizer system · ChromaDB retrieval | average latency (ms): 20.06 -> 11.35 | Improved |
| use PinTok instead of Python tokenizer | tokenizer system · ChromaDB insertion | average latency (ms): 56.26 -> 47.55 | Improved |
| use PinTok instead of Python tokenizer | tokenizer system · Qwen3 4B query (TTFT) | average latency (ms): 422.26 -> 413.55 | Improved |
| use PinTok instead of Python tokenizer | tokenizer system · GPT-OSS 120b query (TTFT) | average latency (ms): 1357.26 -> 1348.55 | No clear effect |
| use PinTok instead of Rust tokenizer | tokenizer system · ChromaDB retrieval | average latency (ms): 16.73 -> 11.35 | Improved |
| use PinTok in tokenizer benchmark | tokenizer system · Tokenizer benchmark | latency and throughput: up to 95% avg latency reduction, 2,084% throughput | Improved |
Closest papers by meaning
- Getting the most out of your tokenizer for pre-training and domain adaptation (ICML 2024 · similarity 0.87)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.87)
- D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models (NeurIPS 2024 · similarity 0.86)
- From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · similarity 0.86)
- From Language Models over Tokens to Language Models over Characters (ICML 2025 · similarity 0.86)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
