Research map / Learning theory and algorithms / Prediction and decision losses
What are the best Systems? New Perspectives on NLP Benchmarking
Pierre Colombo, Nathan Noiry, Ekhine Irurozki, Stephan CLEMENCON
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use two-level Borda ranking aggregation instead of mean aggregation | system ranking · synthetic corrupted tasks | corrupted tasks needed to reach error 0.75: 2 -> 10 | Improved |
| use two-level instead of one-level Borda aggregation of instance-level rankings (ablation) | Kemeny/Borda ranking · synthetic corrupted tasks | corrupted tasks needed to reach error 0.75: 5 -> 10 | Improved |
| use ranking-based aggregation instead of mean when one task is rescaled | system ranking · synthetic rescaling | ranking error: error >90% at factor 2 for mean; unaffected for ranking | Improved |
| use Borda rank aggregation instead of mean on task-level benchmarks | system ranking · GLUE/SGLUE/XTREME | ranking agreement with mean (Kendall tau): tau 0.82 to 0.92 | Mixed |
| use Borda rank aggregation instead of mean when randomly dropping tasks | system ranking · GLUE task subsampling | Kendall correlation to full-task ranking | Improved |
| use two-level or one-level Borda instead of mean aggregation on NLG instance-level data when removing metrics | system ranking · NLG datasets (TAC, FLICKR, etc.) | Kendall correlation to full ranking | Improved |
Closest papers by meaning
- Majority of the Bests: Improving Best-of-N via Bootstrapping (NeurIPS 2025 · similarity 0.84)
- How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.84)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.83)
- Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation (AAAI 2025 · similarity 0.82)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.82)
- Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
