Research map / Large language models / Language, safety and interpretability
Idiosyncrasies in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs have distinguishable output "idiosyncrasies" by fine-tuning LLM2vec text embedding classifiers to identify the source model of a text, achieving 97.1% five-way accuracy among chat APIs.
Problem
Whether different LLMs produce outputs that are distinguishable from one another, and what features make them distinct, is not well understood.
Approach
Frame source-LLM identification as a synthetic N-way text classification task, fine-tune a decoder-based embedding model (LLM2vec) with LoRA on outputs from each model, and use the resulting accuracy as a measure of idiosyncrasy; then probe the sources via shuffling, rewriting, markdown-only and TF-IDF analyses.
Key results
- Five-way classification of chat APIs (ChatGPT, Claude, Grok, Gemini, DeepSeek) reaches 97.1% held-out accuracy versus 20% chance.
- Within the Qwen2.5 family (7B, 14B, 32B, 72B), the four-way instruct classifier reaches 59.8%, and the best pairwise accuracy is 85.5%.
- Shuffling words drops chat accuracy only to 88.9% while shuffling letters drops it to 39.1%; summarizing rewritten outputs drops accuracy to 58.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLM2vec on outputs from different chat APIs | LLM2vec classifier · Chat APIs | accuracy: 97.1 | Helped |
| Constrain responses to one paragraph of at most 100 words (ablation) | LLM2vec classifier on instruct LLMs · Instruct LLMs | accuracy: 96.3 -> 93.0 | Hurt |
| Constrain responses to plain text without formatting (ablation) | LLM2vec classifier on instruct LLMs · Instruct LLMs | accuracy: 96.3 -> 91.4 | Hurt |
| Increase the number of input tokens (ablation) | LLM2vec classifier · Chat APIs, instruct LLMs, and base LLMs | accuracy | Helped |
| Use LLM2vec embeddings instead of GPT-2 embeddings (ablation) | Sequence classifier · Chat APIs | accuracy: 92.1 -> 97.1 | Helped |
| Increase the number of training samples (ablation) | Sequence classifier · Chat APIs, instruct LLMs, and base LLMs | accuracy | Helped |
| Remove special characters from responses (ablation) | LLM2vec classifier · Chat APIs | accuracy: 97.1 -> 95.1 | Hurt |
| Shuffle words in responses (ablation) | LLM2vec classifier · Chat APIs | accuracy: 97.1 -> 88.9 | Hurt |
| Shuffle letters in responses (ablation) | LLM2vec classifier · Chat APIs | accuracy: 97.1 -> 39.1 | Hurt |
| Keep markdown elements and replace other text with placeholders (ablation) | LLM2vec classifier · Chat APIs | accuracy: 97.1 -> 73.1 | Hurt |
| Paraphrase responses while preserving semantic meaning (ablation) | LLM2vec classifier · Chat APIs | accuracy: 97.8 -> 91.4 | Hurt |
| Translate responses into Chinese (ablation) | LLM2vec classifier · Chat APIs | accuracy: 97.8 -> 91.8 | Hurt |
Surprising findings
- Classifying outputs from the same LLM under different decoding strategies is only marginally above chance (about 38% for 5-way temperature classification vs 20% chance).
- A single first token already gives around 50% accuracy for chat APIs and instruct LLMs.
Limitations
- Evaluated on a closed set of source LLMs; open-set or large unknown source lists are not addressed beyond a 10-way initial test.
- Chat API outputs were generated at specific API versions and time windows, so results may not persist as models change.
- Open-ended analyses rely on LLM judges whose own biases may affect the descriptions.
Experimental setup
Datasets: UltraChat FineWeb Cosmopedia LmsysChat WildChat
Benchmarks: UltraChat Cosmopedia LmsysChat WildChat
Metrics: classification accuracy ROUGE-1 ROUGE-L BERTScore
Baselines: ELMo BERT T5 GPT-2 TF-IDF logistic regression
Builds on: LLM2vec LoRA Name That Dataset
Closest papers by meaning
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.88)
- Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification (ICLR 2025 · similarity 0.88)
- LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (ICLR 2025 · similarity 0.87)
- Language Model Cascades: Token-Level Uncertainty And Beyond (ICLR 2024 · similarity 0.87)
- Synthetic Text Generation for Training Large Language Models via Gradient Matching (ICML 2025 · similarity 0.87)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.