Research map / Large language models / AI and society
IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Research
Yuanhao Shen, Daniel Xavier de Sousa, Ricardo Marçal de Andrade Nascimento, Hongyu Guo, Xiaodan Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 5 few-shot examples to the prompt | LLMs on IPI · IDRBench IPI | Macro-F1: gpt-4o-mini 0.587 -> 0.510; avg 0.556 -> 0.483 | Mixed |
| use reasoning models instead of non-reasoning models | I3 idea integration · IDRBench I3 Subset 1 | Macro-F1: around 0.6-0.8 -> around 0.2-0.6 | Hurt |
| replace cross-discipline negatives with same-discipline negatives | all LLMs on I3 · IDRBench I3 Subset 2 vs Subset 1 | Macro-F1: e.g. gpt-4o-mini 0-shot 0.750 -> 0.588 | Hurt |
| add 3 few-shot examples to the prompt | gpt-4o-mini on I3 Subset 2 · IDRBench I3 Subset 2 | Macro-F1: 0.588 -> 0.672 | Improved |
| add 3 few-shot examples to the prompt | Gemini 2.0 on I3 Subset 1 · IDRBench I3 Subset 1 | Macro-F1: 0.666 -> 0.434 | Hurt |
| use full paper content instead of title and abstract (ablation) | IDRBench tasks · IDRBench | performance | No clear effect |
| compare models on I2R ranking | 10 LLMs · IDRBench I2R | MRR: approximately 0.670 MRR | No clear effect |
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.87)
- HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models (ICLR 2026 · similarity 0.87)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.87)
- ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code (NeurIPS 2025 · similarity 0.87)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
