Research map / Vision-language and multimodal / Human and embodied modeling
Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Modeling Collaborator uses an LLM (PaLM 2) with PaLI-X VQA/captioning tools to auto-annotate mined images for subjective visual concepts, then distills them into a lightweight MLP classifier on CLIP/ALIGN features, needing only ~100 human labels.
Problem
Building classifiers for nuanced, subjective visual concepts requires thousands of manually labeled images (Agile Modeling needs ~2,000 labels, ~30 minutes), which is costly and tedious and suffers from crowd-rater inconsistency.
Approach
Replace human labeling with an LLM-driven annotator that decomposes a concept into atomic yes/no questions answered by a VQA VLM, plus a caption, and then uses chain-of-thought reasoning to label images; the labels train a lightweight student classifier refined by active learning, with users only validating on about 100 images.
Key results
- On the Agile Modeling dataset, the Annotator's average F1 on hard concepts is 0.68 versus 0.65 for PaLI-X and 0.56 for CuPL; hard-concept F1 gains are large (e.g., hateful memes +15%).
- The distilled student reaches 0.66 auPR overall, +1% over crowd annotators (0.65) and within about 2% of the user-labeled model, using 100 human labels versus 2,000 in Agile Modeling.
- On Hateful Memes, the student model (F1 0.56) and teacher (F1 0.66) outperform a CLIP+MLP trained on human labels (F1 0.48) without using the training labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Modeling Collaborator annotator instead of PaLI-X | Subjective visual classification · Agile Modeling hard concepts | F1: 0.65 -> 0.68 | Helped |
| Use the Modeling Collaborator annotator instead of PaLI-X | Subjective visual classification · Agile Modeling easy concepts | F1: 0.80 -> 0.82 | Helped |
| Use the Modeling Collaborator annotator instead of PaLI-X | Subjective visual classification · Agile Modeling overall average | F1: 0.72 -> 0.74 | Helped |
| Use the Modeling Collaborator annotator instead of CLIP | Hateful Memes classification · Hateful Memes | F1: 0.66 -> 0.66 | No clear effect |
| Train a student on Modeling Collaborator annotations instead of human annotations | Hateful Memes classification · Hateful Memes | F1: 0.48 -> 0.56 | Helped |
| Use Modeling Collaborator annotations instead of crowd annotations | Distilled classifiers · Agile Modeling hard concepts | auPR: 0.57 -> 0.60 | Helped |
| Use Modeling Collaborator annotations instead of crowd annotations | Distilled classifiers · Agile Modeling easy concepts | auPR: 0.72 -> 0.73 | Helped |
| Use Modeling Collaborator annotations instead of crowd annotations | Distilled classifiers · Agile Modeling overall average | auPR: 0.65 -> 0.66 | Helped |
Surprising findings
- Verbose, overly complex concept descriptions can degrade performance on simple concepts relative to plain PaLI-X.
- PaLI-X answers change with prompt phrasing on subjective concepts such as gourmet tuna.
Limitations
- Performance inherits errors from its components (PaLM 2, PaLI-X, CLIP), including VQA errors and LLM hallucinations.
- Verbose descriptions can hurt simple concepts; the Annotator needs per-concept strategy selection with a 100-image validation set.
- CLIP features struggle to distinguish real stop signs from depictions, and Hateful Memes performance remains low.
Experimental setup
Datasets: LAION Agile Modeling dataset Hateful Memes
Benchmarks: Agile Modeling dataset (14 concepts) Hateful Memes
Metrics: precision recall F1 accuracy auPR
Baselines: CLIP CuPL PaLI-X ALIGN Agile Modeling crowd annotators
Builds on: Agile Modeling Toolformer CuPL CLIP PaLI-X
Closest papers by meaning
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.87)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.87)
- Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR 2024 · similarity 0.87)
- Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · similarity 0.87)
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.86)
- The Neglected Tails in Vision-Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.