Research map / Large language models / Retrieval and knowledge
MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MIR-Bench, a many-shot (up to 2048 shots) in-context pattern recognition benchmark built from introductory coding functions with LLM-written data generators, and evaluates ~21 LLMs on it.
Problem
Existing pattern recognition benchmarks for LLMs use few examples (<10), while many-shot in-context reasoning, aggregating information from hundreds to thousands of input-output examples, is largely unevaluated beyond classification.
Approach
Generate many-shot inductive/transductive tasks automatically from existing coding-benchmark solution functions by having GPT-4o write data generators (20000 shots and 10 test cases per function), then select a hard MIR-Core subset whose difficulty grows most from few to many shots.
Key results
- The best model, o1-mini-0912, reaches under 0.7 accuracy on MIR-Extended, while most models such as GPT-4o-0806 stay under 0.4.
- Performance saturates at about 256 shots for most models; duplicated-shot experiments show the saturation stems from distraction rather than retrieval failure.
- Transductive (no CoT) answers have higher accuracy than CoT answers across all 21 models tested, e.g. o1-preview 0.797 without CoT vs 0.588 with CoT.
- Embedding-based RAG with 64 selected shots gives no significant gain over random selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Omit CoT rather than produce CoT (ablation) | MIR-Core evaluated with Claude-3.5-Sonnet, GPT-4o-0806, and o1-mini-0912 · MIR-Core, averaged over 4–2048 shots | accuracy: Claude-3.5-Sonnet: 0.585 -> 0.775; GPT-4o-0806: 0.488 -> 0.540; o1-mini-0912: 0.334 -> 0.696 | Helped |
| Use SolverLearner (write code, then execute) (ablation) | MIR-Core; effect differs across models · MIR-Core, 16–1024 shots | accuracy: DeepSeek-R1: +0.022, +0.007, +0.018, +0.003; GPT4o-0806: +0.012, -0.033, -0.029, +0.004; Gemini-1.5 Pro-002: -0.029, -0. | Mixed |
| Select 64 shots by embedding similarity rather than random selection (ablation) | MIR-Core with GPT-4o-0806 and Gemini-1.5 Pro · MIR-Core, 128–2048 candidate shots | accuracy: GPT-4o: 51.77 vs 50.40; 49.83 vs 50.17; 49.33 vs 50.90; 49.60 vs 51.03; 50.53 vs 51.63; Gemini-1.5-Pro: 51.53 vs 51.20; | No clear effect |
| Increase the number of shots beyond 512 | Most evaluated LLMs on MIR-Extended · MIR-Extended, many-shot ICL | accuracy | Hurt |
| Add incorrect examples to the in-context shots (ablation) | LLMs on MIR-Core · MIR-Core, 64 shots and error ratios through 3/4 | accuracy | Hurt |
| Provide out-of-domain meta-shot reasoning demonstrations (ablation) | Gemini-1.5 Pro-002 and Gemini-1.5 Flash-002 · MIR-Core, forced-CoT accuracy | accuracy: Gemini-1.5 Pro-002, 4 shots: 38.67 -> 42.97 (0 to 32 meta-shots); Gemini-1.5 Flash-002, 4 shots: 37.93 -> 40.74 | Mixed |
| Force CoT instead of no CoT (ablation) | Most non-thinking LLMs on MIR-Core · MIR-Core, forced CoT vs no CoT | accuracy | Hurt |
| Use forced CoT instead of forced nonsense (ablation) | GPT-4o, Gemini, and Mistral models on MIR-Core · MIR-Core, forced CoT vs forced nonsense | accuracy | Helped |
Surprising findings
- Answers without chain-of-thought are unanimously more accurate than those with CoT across all 21 models, and forced CoT often underperforms no CoT.
- LLMs tend to underestimate inductive reasoning difficulty of simple-looking functions, which required a multi-round self-reflection difficulty labeling scheme.
Limitations
- Difficulty factors used to select MIR-Core are not decisive; a more explainable selection rule is needed.
- Evaluation is limited to text; multimodal pattern recognition is not covered.
- Some underlying functions may be non-unique given the input-output pairs.
Experimental setup
Datasets: HumanEval+ MBPP+ APPS
Benchmarks: MIR-Extended MIR-Core
Metrics: exact match accuracy accuracy gap D between few-shot and many-shot
Baselines: o1-preview o1-mini GPT-4o Claude-3.5-Sonnet Gemini-1.5 Pro Mistral-Large-2 SolverLearner
Builds on: SolverLearner many-shot in-context learning HumanEval+ MBPP+ APPS
Closest papers by meaning
- From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation (ICLR 2025 · similarity 0.91)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.91)
- MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning (ICML 2025 · similarity 0.89)
- What Makes In-context Learning Effective for Mathematical Reasoning (ICML 2025 · similarity 0.88)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.88)
- LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.