Research map / Large language models / Language, safety and interpretability
Unnatural Languages Are Not Bugs but Features for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that GCG-searched "unnatural" strings, unreadable to humans, carry semantic content LLMs can use across tasks and models, and that fine-tuning on unnatural LIMA instructions matches natural fine-tuning on AlpacaEval 2.0.
Problem
Non-human-readable strings (e.g., jailbreak suffixes, filler tokens) are usually viewed as bugs for aligned LLMs; it is unclear whether they carry usable latent meaning.
Approach
Search, via a GCG-style gradient-guided discrete optimization across multiple models and reconstruction tasks, for a semantically equivalent but syntactically unreadable version of a natural string, then test whether such unnatural text can be understood in QA, used for instruction tuning, and how models process it.
Key results
- Unnatural contexts reach 80.4% average accuracy on SynContextQA versus 93% natural (zero-shot, across models), while Shuf-InJ reaches 48%.
- Models fine-tuned on unnatural LIMA achieve about 49.7% average winrate in LC AlpacaEval 2.0 against natural-LIMA-tuned counterparts (49.78%, 47.13%, 52.22% for Llama-3-8B, Gemma-2-9B, Llama-3-70B).
- On SimGSM8K, unnatural contexts reach 54% average accuracy versus 88% natural and 27% Shuf-InJ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use searched unnatural contexts instead of natural contexts | LLMs answering context-based questions · SynContextQA (Transfer models average) | accuracy: 0.98 -> 0.80 | Hurt |
| Use searched unnatural contexts instead of shuffled text with injected tokens | LLMs answering context-based questions · SynContextQA (Transfer models average) | accuracy: 0.54 -> 0.80 | Helped |
| Use searched unnatural contexts instead of natural contexts | LLMs answering questions · SimGSM8K (Transfer models average) | accuracy: 0.88 -> 0.54 | Hurt |
| Use searched unnatural contexts instead of shuffled text with injected tokens | LLMs answering questions · SimGSM8K (Transfer models average) | accuracy: 0.27 -> 0.54 | Helped |
| Fine-tune on unnatural rather than natural LIMA instructions | Instruction-tuned LLMs · MixEval overall average | score: 0.557 -> 0.547 | No clear effect |
| Fine-tune on unnatural rather than random instructions | Instruction-tuned LLMs · MixEval overall average | score: 0.499 -> 0.547 | Helped |
| Fine-tune on unnatural rather than empty instructions | Instruction-tuned LLMs · MixEval overall average | score: 0.516 -> 0.547 | Helped |
| Use eight natural-language in-context examples instead of one (ablation) | Mistral-7B base model · SimGSM8K unnatural contexts | accuracy: 0.23 -> 0.38 | Helped |
| Use eight natural-language in-context examples instead of one (ablation) | Llama-3-8B base model · SimGSM8K unnatural contexts | accuracy: 0.33 -> 0.42 | Helped |
| Remove original natural tokens from search candidates and initialize without natural tokens (ablation) | Unnatural-language search · Unnatural-language performance | performance | Hurt |
| Use searched unnatural examples rather than salient-token-only examples | LLMs answering SimGSM8K · SimGSM8K | accuracy: 0.18 -> 0.42 | Helped |
| Use searched unnatural examples rather than random-injection salient-token examples | LLMs answering SimGSM8K · SimGSM8K | accuracy: 0.17 -> 0.42 | Helped |
Surprising findings
- Unnatural strings searched on Mistral and Vicuna transfer to GPT-4o and GPT-3.5 comprehension, suggesting a shared rather than model-specific property.
- Unnatural SimGSM8K comprehension remains far below natural accuracy, and fine-tuning on unnatural GSM8K does not match natural fine-tuning, which the authors attribute to long, complex contexts.
Limitations
- Searched strings still contain tokens from the natural text, so they are not fully unnatural.
- GCG-style search is computationally expensive, limiting scale of experiments.
- Unnatural language does not transfer uniformly; complex tasks such as GSM8K fine-tuning underperform natural versions.
Experimental setup
Datasets: LIMA GSM8K SynContextQA (generated with GPT-3.5) SimGSM8K EN_EWT Universal Dependencies
Benchmarks: SynContextQA SimGSM8K LC AlpacaEval 2.0 MixEval
Metrics: accuracy exact match winrate F1 NLI accuracy
Baselines: Shuffled language with injected special tokens (Shuf-InJ) Random tokens instruction Empty instruction Saliency-based token dropping (pure_top)
Builds on: GCG LIMA structural probing
Closest papers by meaning
- When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.86)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.86)
- From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data (ICLR 2025 · similarity 0.86)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.