Research map / Large language models / Agents, code and math

PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

ICLR 2025 · Large language models › Agents, code and math · approach: LLMs for optimization and heuristics · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PhyloLM, which adapts Nei genetic distance and neighbor-joining to LLM next-token outputs on prompt "genes", reconstructing model family trees and predicting leaderboard benchmark scores from the resulting distances.

Problem

Understanding how LLMs are related (fine-tuning lineage, shared training data) and predicting their benchmark capabilities is hard when training details are opaque and many models are never benchmarked.

Approach

Treats LLMs as populations, prompt contexts as genes, and sampled next tokens as alleles; computes Nei genetic similarity from output token frequencies across many probes, converts it to a distance, and builds dendrograms with neighbor joining. The distance is then used with a logistic regression to predict benchmark scores.

Key results

• Across 111 open-source and 45 closed models, PhyloLM dendrograms cluster models into their known families (e.g., Llama, Mistral, Qwen, Bloom) and recover the ground-truth topology for 5 Mistral-family models.
• Phylogenetic distance predicts Open LLM Leaderboard benchmark scores with mean Pearson r = 0.68±0.04 across benchmarks and gene sets, explaining about 48% of between-model variance.
• Within-family prediction still yields mean r = 0.64±0.05, explaining about 52% of variance within families.

Measured effects reported in the paper

Surprising findings

• Some GPT-3 models (ada, babbage, curie) cluster with OPT, Pythia and Falcon-RW, suggesting shared CommonCrawl-based training data.

Limitations

• Common ancestors are present among the studied LLMs, violating the assumption of phylogenetic methods that ancestors are not sampled.
• Chat interfaces add template tokens that distort the gene-allele comparison, so only 14 of 45 proprietary models were used as completion models.
• Benchmark prediction explains only about half of the variance; temperature effects were not explored.

Experimental setup

Datasets: open-web-math MBXP Hugging Face Open LLM Leaderboard (MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K)

Benchmarks: MMLU ARC HellaSwag TruthfulQA Winogrande GSM8K

Metrics: Pearson correlation variance explained mean squared error

Baselines: MMLU

Builds on: Nei genetic distance neighbor joining

Closest papers by meaning

• PhyloGen: Language Model-Enhanced Phylogenetic Inference via Graph Structure Generation (NeurIPS 2024 · similarity 0.90)
• Language Modeling by Language Models (NeurIPS 2025 · similarity 0.87)
• Bias Amplification in Language Model Evolution: An Iterated Learning Perspective (NeurIPS 2024 · similarity 0.86)
• PhyloGFN: Phylogenetic inference with generative flow networks (ICLR 2024 · similarity 0.86)
• Deep Generative Models for Phylogenetic Inference with Complex Evolutionary Processes (NeurIPS 2026 · similarity 0.86)
• PhyloVAE: Unsupervised Learning of Phylogenetic Trees via Variational Autoencoders (ICLR 2025 · similarity 0.85)