Research map / Large language models / Instruction and fine-tuning

Getting the most out of your tokenizer for pre-training and domain adaptation

ICML 2024 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies how BPE tokenizer size, pre-tokenization regex, and training data affect code LLM compression and downstream code generation, and tests switching tokenizers during fine-tuning of pre-trained LLMs.

Problem

Tokenizer design (vocabulary size, pre-tokenization regex, training data) is usually fixed by borrowing and rarely optimized, and pre-trained LLMs keep their original tokenizer during domain fine-tuning even when it is sub-optimal for the domain, hurting inference efficiency and effective context le

Approach

Systematically ablates BPE tokenizer hyper-parameters for code, derives inference- and memory-optimal vocabulary size formulas, and shows that a pre-trained code LLM can switch to a more compressive tokenizer during fine-tuning (with FVT embedding initialization) given enough tokens (>=50B) without losing downstream performance.

Key results

• Changing the tokenizer of a pre-trained 7B Llama 2 model to GPT-4 80k tokenizer for 500B tokens gives HumanEval Pass@1 of 30.8 vs 32.1 for Code Llama, with comparable MBPP performance.
• GPT-4 style regex tokenizer at 32k compresses code by about 19% relative to the Llama tokenizer (NSL 0.81 on code), with Identity tokenizer achieving up to 0.59 NSL on code at 100k.
• Fine-tuning with a switched tokenizer recovers performance after about 50B tokens; with only 5B tokens a large gap to the Llama tokenizer remains.

Measured effects reported in the paper

Surprising findings

• Vocabulary size from 32k to 256k has no significant effect on HumanEval Pass@1 (Pearson correlation -0.13, p=0.87).
• Identity tokenizer (no pre-tokenization) gives the best compression but loses significant downstream performance unless token healing is used.

Limitations

• Gains from tokenizer switching require long fine-tuning (>=50B tokens); short regimes degrade performance.
• Evaluation limited to HumanEval and MBPP code generation and models of 1.5B and 7B size.
• Several differences between tokenizers (e.g., Punct vs GPT-4) are not statistically significant.

Experimental setup

Datasets: CCnet Wikipedia The Stack HumanEval MBPP

Benchmarks: HumanEval MBPP

Metrics: Pass@1 Pass@100 Compile@1 Normalized Sequence Length Bytes per Token Renyi entropy

Baselines: Llama tokenizer Code Llama 7B GPT-2 tokenizer Identity tokenizer InCoder tokenizer

Builds on: Byte-Pair Encoding Fast Vocabulary Transfer Code Llama GPT-4 pre-tokenization regex

Closest papers by meaning

• Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.89)
• Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.89)
• Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.89)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations (NeurIPS 2025 · similarity 0.88)
• zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.88)
• Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.88)