Research map / Large language models / Language, safety and interpretability
ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the ACT pipeline in which MLLMs annotate data and a second MLLM criticizes likely errors so a limited human budget reviews only suspicious samples, plus a modified ACT loss from active M-estimation for downstream training.
Problem
LLM/MLLM-generated labels fall short of human quality, and it is unclear how to combine limited human review with machine annotation so downstream models match fully human-labeled training.
Approach
Use one MLLM as annotator and another MLLM (black-box or white-box) as criticizer to estimate per-sample error probability, sample high-risk items for human review under a budget (normalization, exponential weighting, or thresholding), and train with a debiased ACT loss adapted from active M-estimation.
Key results
- ACT with the exponential/thresholding loss reduces the human-machine downstream gap to under 2% on most datasets (e.g., 0.71% CIFAR10, 0.08% VQA-RAD) while saving roughly 70%-90% of human cost.
- Machine-only training yields up to 10.15% lower accuracy than human-only training (Fashion).
- On Stanford Cars with a 9.56% budget, normalization leaves a 76.34% gap while exponential weighting and thresholding reduce it to 1.69% and 1.88%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACT exponential-weighting loss instead of normalization loss (ablation) | ACT data training · Cars test | accuracy (%): 11.54 $\pm$ 0.96 -> 86.19 $\pm$ 0.14 | Helped |
| Use ACT exponential-weighting loss instead of normalization loss (ablation) | ACT data training · CIFAR10 test | accuracy (%): 64.70 $\pm$ 5.46 -> 87.73 $\pm$ 0.36 | Helped |
| Use ACT exponential-weighting loss instead of normalization loss (ablation) | ACT data training · Fashion test | accuracy (%): 69.27 $\pm$ 7.25 -> 89.73 $\pm$ 0.35 | Helped |
| Use ACT exponential-weighting loss instead of normalization loss (ablation) | ACT data training · VQA-RAD test | accuracy (%): 62.55 $\pm$ 3.01 -> 67.73 $\pm$ 1.33 | Helped |
| Train on ACT data rather than machine-only labels | Cross-entropy training · CIFAR10 test | accuracy (%): 81.55 $\pm$ 1.93 -> 85.59 $\pm$ 0.52 | Helped |
| Train on ACT data with exponential-weighting loss rather than machine-only labels | CIFAR10 test · CIFAR10 test | accuracy (%): 81.55 $\pm$ 1.93 -> 87.73 $\pm$ 0.36 | Helped |
| Use ACT exponential-weighting loss rather than ACT Cross-entropy loss (ablation) | ACT data training · VQA-RAD test | accuracy (%): 61.83 $\pm$ 3.27 -> 67.73 $\pm$ 1.33 | Helped |
| Use ACT exponential-weighting loss rather than ACT Cross-entropy loss (ablation) | ACT data training · Cars test | accuracy (%): 85.88 $\pm$ 0.26 -> 86.19 $\pm$ 0.14 | Helped |
| Use thresholding rather than exponential-weighting ACT loss (ablation) | ACT data training · CIFAR10 test | accuracy (%): 87.73 $\pm$ 0.36 -> 87.95 $\pm$ 0.35 | No clear effect |
| Use thresholding rather than exponential-weighting ACT loss (ablation) | ACT data training · Fashion test | accuracy (%): 89.73 $\pm$ 0.35 -> 89.16 $\pm$ 0.89 | No clear effect |
| Use thresholding rather than exponential-weighting ACT loss (ablation) | ACT data training · Cars test | accuracy (%): 86.19 $\pm$ 0.14 -> 86.00 $\pm$ 0.26 | No clear effect |
| Use ACT Cross-entropy rather than ACT exponential-weighting loss (ablation) | ACT data training · Fashion test | accuracy (%): 89.73 $\pm$ 0.35 -> 87.50 $\pm$ 0.86 | Hurt |
Surprising findings
- CoT does not consistently help annotation, though it helps criticism (up to 22.46% ABS gain over naive).
- Cross-criticism (a different model as criticizer) typically outperforms self-criticism, attributed to self-preference bias.
Limitations
- Focuses primarily on classification tasks; does not address summarization or open-ended QA.
- Efficiency is bounded by current MLLM capabilities; white-box criticizers lag black-box ones.
- Human budget in the main experiments is set from GPT-4o's annotation accuracy (an 'ideal' budget), not an independent practical setting.
Experimental setup
Datasets: CIFAR10 Fashion-MNIST Stanford Cars TweetEval-Emotion TweetEval-Irony VQA-RAD ImageNet-1k
Benchmarks: CIFAR10 Fashion-MNIST Stanford Cars TweetEval-Emotion TweetEval-Irony VQA-RAD
Metrics: accuracy Area under Budget Sensitivity (ABS) Annotation Quality Gain (AQG) test accuracy
Baselines: CDI naive annotation normalization ACT loss Human only Machine only
Builds on: active M-estimation chain-of-thought LLM-as-a-judge power tuning
Closest papers by meaning
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.87)
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.87)
- Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.87)
- L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.87)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.