Research map / Large language models / Agents, code and math

Law of the Weakest Link: Cross Capabilities of Large Language Models

ICLR 2025 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CrossEval, a 1,400-prompt benchmark with human-rated responses for seven individual and seven cross LLM capabilities, uses GPT-4o as a multi-reference judge, and reports that cross-capability scores track the weakest constituent capability ("Law of the Weakest Link").

Problem

Current LLM evaluation measures capabilities in isolation, while real-world tasks require combining several abilities; it is unclear how individual capability levels relate to performance on tasks that demand multiple capabilities at once.

Approach

Define seven individual and seven cross capabilities with taxonomies, build an open-ended benchmark with multiple model responses rated by human experts and explained, use an LLM judge with point-deduction prompting that references human-annotated examples, and test whether cross-capability scores follow the weakest individual capability.

Key results

• GPT-4o judge reaches overall Pearson 0.697 with human ratings; Claude 3.5 Sonnet 0.696.
• Of 58 cross-capability scenarios with a >3 point gap, 38 fall below both individual capabilities and 20 lie between, closer to the weaker one.
• Principle-based system prompting improved Image Recognition by 3.71 (Claude 3 Haiku) and 6.19 (Gemini 1.5 Flash) points, with larger cross-capability gains when the weaker capability was targeted.

Measured effects reported in the paper

Surprising findings

• No cross-capability score came close to or exceeded the stronger individual capability; average individual score 65.72 vs cross 58.67.
• Llama 3.1 405B scored below 50 on Tool Use, the weakest capability tested.

Limitations

• Judge scoring distributions differ across evaluators, so a mixture of judges was not used.
• Only pairwise cross capabilities and 100 prompts per capability; the Law of the Weakest Link analysis uses only 58 scenarios with a gap.
• Tool use evaluation is limited by evaluator APIs lacking web browsing and Gemini code-interpreter non-text outputs being excluded.

Experimental setup

Datasets: CrossEval

Benchmarks: CrossEval

Metrics: Likert score (1-5, linearly mapped to 1-100) Pearson correlation Spearman correlation Kendall tau inter-rater agreement Krippendorff's alpha

Baselines: GPT-4o mini Llama 3.1 405B Claude 3.5 Sonnet GPT-4o

Builds on: Llama 3 BiGGen Bench Amdahl's Law LLM-as-a-Judge

Closest papers by meaning

• Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.90)
• The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model (ICLR 2025 · similarity 0.88)
• EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.88)
• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.88)
• Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond (ICLR 2025 · similarity 0.88)
• Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.88)