Research map / Large language models / Language, safety and interpretability
Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes dual-part token embeddings (shared learnable part plus per-token random part, with normalization and AdaCos cosine loss) so Transformers can handle unseen interchangeable tokens such as atomic propositions, and introduces alpha-covariance to measure robustness to alpha-renaming.
Problem
Language models cannot handle interchangeable tokens (e.g., bound variables or atomic propositions) beyond the vocabulary seen in training, and are not robust to alpha-renaming, which preserves meaning.
Approach
Construct embeddings for interchangeable tokens on the fly as a concatenation of a shared learnable vector and a freshly randomized vector per token (resampled each training forward pass), tied with the output projection and combined with normalization and AdaCos; this lets the model accept arbitrarily many interchangeable tokens and encourages alpha-equivalence.
Key results
- On LTL solving, the proposed model trained on at most 5 APs reaches 90.76% accuracy in the generalization experiment, close to the full 10-AP vocabulary baseline and above the alpha-renaming and vanilla baselines.
- Llama 3.2 3B achieves 24.33% correct on LTLRandom35 versus 95.94% for the proposed method (perturbed dataset).
- Alpha-covariance on perturbed LTL data: baseline 64.93% for 3 AP and 40.91% for 5 AP, versus 97.66% and 98.29% for the proposed method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dual-part token embeddings instead of fixed learned embeddings | LTL solver · LTL generalization to larger AP counts | accuracy | Helped |
| Use dual-part token embeddings instead of fixed learned embeddings | Propositional logic assignment predictor · Propositional logic generalization to larger AP counts | accuracy | Helped |
| Use dual-part token embeddings instead of fixed learned embeddings | LTL solver · LTLRandom35 perturbed | correct: 34.13% -> 95.94% | Helped |
| Use dual-part token embeddings instead of fixed learned embeddings | Propositional logic assignment predictor · PropRandom35 perturbed | correct: 41.57% -> 93.25% | Helped |
| Remove f_bn normalization (ablation) | Proposed method · LTL | correct: 90.76% to 29.53% | Hurt |
| Remove f_bn normalization (ablation) | Proposed method · Propositional logic | correct: 77.70% to 14.12% | Hurt |
| Remove AdaCos (ablation) | Proposed method · LTL | correct: 90.76% vs. 81.45% | Hurt |
| Remove AdaCos and f_fn (ablation) | Proposed method · Propositional logic | correct | No clear effect |
| Add f_fn normalization (ablation) | Copying model with dual-part embeddings · Copying hyperparameter search | edit distance: -0.29 correlation | Hurt |
| Add AdaCos loss (ablation) | Copying model with dual-part embeddings · Copying hyperparameter search | edit distance: -0.41 correlation | Hurt |
| Use Neighboring Points random vectors instead of other random-vector methods (ablation) | Copying model with dual-part embeddings · Copying hyperparameter search | edit distance: -0.14 correlation | Helped |
| Use alpha-renaming augmentation instead of ordinary baseline training | LTL solver · LTLRandom35 perturbed | alpha-covariance, 5 AP: 40.91% -> 98.86% | Helped |
Surprising findings
- Removing the per-vector normalization f_bn drops LTL accuracy from 90.76% to 29.53%, showing the balance between shared and random parts is critical.
- The alpha-renaming baseline achieves higher in-distribution correctness and alpha-covariance than the proposed method on perturbed LTL data.
Limitations
- Not directly applicable to natural language, where tokens carry semantic information not preserved by renaming.
- Requires manually defining the set of interchangeable tokens and training from scratch, hindering use with pretrained models.
- Slight decrease in in-distribution performance on LTL solving compared with the baseline.
Experimental setup
Datasets: LTLRandom35 PropRandom35 synthetic copying strings
Benchmarks: LTLRandom35 test set PropRandom35 test set copying with extendable vocabulary test set
Metrics: correct ratio exact match alpha-covariance edit distance
Baselines: Full vocabulary baseline Alpha-renaming baseline Llama 3.2 3B DeepLTL baseline
Builds on: DeepLTL AdaCos RoPE tree-positional encoding weight tying
Closest papers by meaning
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.87)
- Soft Token Alignment for Cross-Lingual Reasoning (NeurIPS 2026 · similarity 0.87)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.86)
- Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs (ICML 2026 · similarity 0.86)
- When the Same Coefficients Reach Different Places: Asymmetric Realizability in Transplanting Tokenizers across Large Language Models (NeurIPS 2026 · similarity 0.85)
- Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.