Research map / Vision-language and multimodal / Multimodal LLMs
Task Me Anything
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Task-Me-Anything, a benchmark generation engine that programmatically builds user-tailored perception VQA benchmarks from a scene-graph taxonomy and uses budgeted active query approximation to evaluate MLMs.
Problem
General-purpose MLM benchmarks are too broad and uninformative for developers who need to know which model performs best on their specific visual use case or where models fail at fine-grained perceptual skills.
Approach
A taxonomy-driven, programmatic task generator (over real scene graphs, 2D sticker composites, and 3D Blender tabletop scenes) lets users pose queries (top-K, threshold, model comparison, debugging) answered by evaluating MLMs on a query-relevant subset, with fitting/active approximation (Gaussian Process) to stay within an evaluation budget.
Key results
- Active query approximation achieves Top-K mean rank 10.79 and hit rate 70.55% vs 46.81 / 42.30% for random sampling.
- Best open-source model (LLaVA-Next-34B / InternVL-Chat-1.5-24B) outperforms best proprietary model by ~8% on spatial reasoning and ~7% on 3D attribute understanding.
- GPT-4o drops 5% to 50% on Top-5 hard objects, relations, and attributes found by the system; model rankings correlate ≥0.8 with MMMU and MMBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use active task selection instead of random sampling (ablation) | Task-Me-Anything query approximation · Top-K query | Mean Rank: 46.81 -> 10.79 | Helped |
| Use active task selection instead of random sampling (ablation) | Task-Me-Anything query approximation · Top-K query | Hit Rate: 42.30 -> 70.55 | Helped |
| Fit a function approximator from random samples instead of using the random subset as proxy (ablation) | Task-Me-Anything query approximation · Top-K query | Mean Rank: 46.81 -> 34.43 | Helped |
| Use active task selection instead of random sampling (ablation) | Task-Me-Anything query approximation · Threshold query | Recall: 42.48 -> 46.83 | Helped |
| Use active task selection instead of fitting on random samples (ablation) | Task-Me-Anything query approximation · Threshold query | Precision: 47.45 -> 47.39 | No clear effect |
| Use active task selection instead of random sampling (ablation) | Task-Me-Anything query approximation · Model comparison query | Recall: 24.58 -> 54.88 | Helped |
| Use active task selection instead of fitting on random samples (ablation) | Task-Me-Anything query approximation · Model comparison query | F1-score: 52.59 -> 61.87 | Helped |
| Use active task selection instead of random sampling (ablation) | Task-Me-Anything query approximation · Model debugging query | Recall: 23.27 -> 43.84 | Helped |
| Use active task selection instead of fitting on random samples (ablation) | Task-Me-Anything query approximation · Model debugging query | F1-score: 43.86 -> 56.44 | Helped |
| Evaluate challenging tasks selected by active Top-K approximation rather than random tasks | MLMs evaluated on Task-Me-Anything-2024 · Task-Me-Anything-2024 vs Task-Me-Anything-Random | accuracy: 10-30% | Hurt |
Surprising findings
- Smaller InstructBLIP-7B significantly outperforms InstructBLIP-13B on relation understanding, and small video models often beat larger ones.
- GPT-4o struggles more with recognizing rotating/moving furniture, food, and plants and with distinguishing colors.
Limitations
- Programmatically generated tasks may be unrealistic and biased toward synthetic distributions.
- Adding new task generators requires coding skills.
- Query approximation can be inaccurate under small budgets; current scope is perception rather than complex reasoning.
Experimental setup
Datasets: Visual Genome GQA AGQA Action Genome Objaverse-LVIS
Benchmarks: Task-Me-Anything-Random Task-Me-Anything-2024 Task-Me-Anything-DB
Metrics: accuracy precision recall F1 mean rank hit rate Spearman correlation
Baselines: MME MMBench BLINK MMMU SEED-Bench GQA AGQA
Builds on: CLEVR GQA SHAPES Objaverse Action Genome
Closest papers by meaning
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.90)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.89)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.89)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.89)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.