Research map / Large language models / Language, safety and interpretability
Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LLMs learn spurious syntactic-template–domain correlations that degrade entity-knowledge performance cross-domain and can be exploited to bypass refusals, with a benchmark for detecting it.
Problem
LLMs may rely on syntactic templates (PoS-tag sequences) learned as associated with a domain rather than on semantics and domain knowledge, which can degrade performance and create security vulnerabilities.
Approach
Formalizes template–domain spurious correlation, builds a synthetic TRex-based dataset where each domain has a unique PoS template, and defines in-domain vs cross-domain evaluation under semantic-preserving, semantic-breaking, and paraphrase perturbations to detect syntactic-domain reliance in trained models.
Key results
- On the synthetic dataset, OLMo-2 Instruct models show cross-domain drops of roughly 0.40-0.60 on Exact, Synonym, and Antonym settings after fine-tuning (e.g., 1B Exact 0.93 in-domain to 0.42 cross-domain).
- On OLMo-2-7B-Instruct over 1000 WildJailbreak requests, prefixing a chain-of-thought template drops refusal rate from 40% to 2.5% in the Exact setting.
- On Sentiment140, GPT-4o-mini Synonym accuracy falls from 1.00 in-domain to 0.44 cross-domain (delta -0.56).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply another domain's template instead of the in-domain template | OLMo-2-Instruct 1B · Synthetic entity-pair dataset, Exact | accuracy: ↓ 0.51 | Hurt |
| Apply another domain's template instead of the in-domain template | OLMo-2-Instruct 7B · Synthetic entity-pair dataset, Synonym | accuracy: ↓ 0.48 | Hurt |
| Apply another domain's template instead of the in-domain template | OLMo-2-Instruct 13B · Synthetic entity-pair dataset, Antonym | accuracy: ↓ 0.37 | Hurt |
| Apply another domain's template instead of the in-domain template | OLMo-2-Instruct 13B · Synthetic entity-pair dataset, Disfluent | accuracy: ↑ 0.11 | Helped |
| Apply another domain's template instead of the in-domain template | OLMo-2-Instruct 13B · Synthetic entity-pair dataset, Paraphrase | accuracy: ↓ 0.34 | Hurt |
| Apply another domain's template instead of the in-domain template | OLMo-2-7B · Sentiment140, Synonym | accuracy: ↓0.37 | Hurt |
| Apply another domain's template instead of the in-domain template | GPT-4o-mini · Sentiment140, Synonym | accuracy: ↓0.56 | Hurt |
| Apply another domain's template instead of the in-domain template | GPT-4o · E-SNLI, Antonym | accuracy: ↓0.17 | Hurt |
| Add chain-of-thought template as a prefix | OLMo-2-7B-Instruct · WildJailbreak, Exact | refusal rate: 0.400 -> 0.025 | Hurt |
| Add chain-of-thought template as a suffix | OLMo-2-7B-Instruct · WildJailbreak, Disfluent | refusal rate: 0.400 -> 0.549 | Helped |
| Add math template as a prefix | OLMo-2-7B-Instruct · WildJailbreak, Antonym | refusal rate: 0.399 -> 0.189 | Hurt |
| Add another domain's template | Llama-4-Maverick · Sentiment140, Synonym | accuracy: 0.00 | No clear effect |
Surprising findings
- Syntax can override semantics: OLMo-2-13B Instruct scores similarly on Antonym (0.92) and Synonym (0.92) in-domain despite opposite meanings.
- Llama-4-Maverick cross-domain performance barely changes, which the authors attribute to entity memorization rather than syntactic-domain correlation.
Limitations
- Results on GPT-4o, GPT-4o-mini, and Llama-4 are speculative because training on FlanV2 cannot be confirmed.
- Reasoning models and chain-of-thought-trained models are not investigated.
- Synthetic dataset relies on single templates per domain and fine-tuning on a narrow set of PID-based domains.
Experimental setup
Datasets: TRex (LAMA) FlanV2 subset (Sentiment140, E-SNLI, Newsroom, CoT) Tulu-V3 WildJailbreak
Benchmarks: TRex synthetic test set FlanV2 Sentiment140 FlanV2 E-SNLI FlanV2 Newsroom WildJailbreak
Metrics: accuracy ROUGE-2 refusal rate in-domain vs cross-domain performance delta
Baselines: OLMo-2-7B GPT-4o-mini GPT-4o Llama-4-Maverick
Builds on: TRex FlanV2 Tulu-V3 OLMo-2
Closest papers by meaning
- Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.88)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.88)
- Correlated Errors in Large Language Models (ICML 2025 · similarity 0.87)
- Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.87)
- A Polar coordinate system represents syntax in large language models (NeurIPS 2024 · similarity 0.87)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.