Research map / Large language models / Language, safety and interpretability
Uncovering Competency Gaps in Large Language Models and Their Benchmarks
Maty Bohacek, Nino Scherrer, Nicholas Dufour, Thomas Leung, Christoph Bregler, Stephanie C.Y. Chan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| name the protagonists with foreign names (Omar, Mahmoud) (ablation) | Qwen3-4B · LogicBench-style prompt | accuracy (%): 80% -> 30% | Hurt |
| remove datapoints tied to best-performing concepts (ablation) | Competency Gaps (Llama3.1-8B) · Benchmark suite | median X_model: lowered by 0.6% | Hurt |
| remove datapoints tied to worst-performing concepts (ablation) | Competency Gaps (Llama3.1-8B) · Benchmark suite | median X_model: increased by 1.3% | Improved |
| randomly drop 20% of examples per benchmark (100 reruns) (ablation) | Competency Gaps (Llama3.1-8B) · Benchmark suite | std of X_model: 0.014 | No clear effect |
| swap Llama SAE for Gemma 2 2B activations and SAE (ablation) | Competency Gaps (Llama3.1-8B) · Benchmark suite | score distributions | No clear effect |
| remove datapoints tied to best-performing concepts (Gemma) (ablation) | Competency Gaps (Gemma2-2B) · Benchmark suite | median X_model: lowered by 0.8% | Hurt |
| remove datapoints tied to worst-performing concepts (Gemma) (ablation) | Competency Gaps (Gemma2-2B) · Benchmark suite | median X_model: increased by 0.5% | Improved |
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.89)
- Paloma: A Benchmark for Evaluating Language Model Fit (NeurIPS 2024 · similarity 0.89)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.89)
- Aligning Language Model Benchmarks with Pairwise Preferences (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
