Research map / Vision-language and multimodal / Human and embodied modeling
LEMoN: Label Error Detection using Multimodal Neighbors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LEMoN, a training-free label error detector for image-caption datasets that combines CLIP image-text distance with multimodal nearest-neighbor discordance scores, evaluated on eight classification and captioning datasets.
Problem
Identifying mislabeled image-caption pairs in large noisy multimodal datasets, where prior label error detection is mostly unimodal, classification-only, and often requires a downstream trained model.
Approach
Score each image-caption pair by its CLIP image-text distance plus two neighbor-based terms: the caption distances of image-space neighbors' pairs and the image distances of caption-space neighbors' pairs, with exponential weights that downweight distant or likely-mislabeled neighbors.
Key results
- LEMoN (oracle-tuned) reaches 98.1 AUROC on cifar10 and 90.8 on cifar100 for label error detection, beating training-free baselines on three of four classification datasets.
- On captioning, LEMoN-opt achieves 95.6 AUROC on mscoco and 86.0 on mmimdb, outperforming CLIP Similarity and other neighborhood baselines.
- Fine-tuning GIT on data filtered by LEMoN (removing 40%) improves mscoco B@4 from 35.0 (no filtering) to 38.0 vs 37.8 for CLIP Similarity filtering.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LEMoN opt rather than the strongest baseline | Label error detection · Stanford Cars real noise | AUROC: 72.2 (1.7) -> 73.1 (0.5) | Helped |
| Use LEMoN opt rather than the strongest baseline | Label error detection · CIFAR-10 human noise | AUROC: 98.3 (0.1) -> 98.1 (0.0) | Hurt |
| Use LEMoN opt rather than the strongest baseline | Label error detection · CIFAR-100 human noise | AUROC: 92.3 (0.3) -> 90.8 (0.0) | Hurt |
| Use LEMoN opt rather than the strongest baseline | Label error detection · MSCOCO category noise | AUROC: 99.3 (0.0) -> 95.6 (0.2) | Hurt |
| Use LEMoN opt rather than the strongest baseline | Label error detection · MMIMDb category noise | AUROC: 82.7 (0.7) -> 86.0 (0.1) | Helped |
| Filter the 40% highest mislabel scores with LEMoN | GIT captioner with LoRA · Flickr30k | BLEU-4: 27.9 (0.7) -> 29.5 (0.6) | Helped |
| Filter the 40% highest mislabel scores with LEMoN | GIT captioner with LoRA · MSCOCO | BLEU-4: 35.0 (0.1) -> 38.0 (0.2) | Helped |
| Pretrain CLIP on noisy MIMIC-CXR data rather than use BiomedCLIP | LEMoN opt · MIMIC-CXR random noise | AUROC: 73.7 (1.7) -> 80.0 (0.9) | Helped |
| Pretrain CLIP on noisy MIMIC-CXR data rather than use CheXzero | LEMoN opt · MIMIC-CXR category noise | AUROC: 88.5 (0.4) -> 77.2 (0.8) | Hurt |
| Remove the tau weighting terms (ablation) | LEMoN · Label error detection datasets | Label error detection performance: about 1% | Hurt |
| Remove components from the full score (ablation) | LEMoN · Label error detection datasets | Label error detection performance | Hurt |
| Add the image-text distance term d_mm (ablation) | LEMoN · Label error detection datasets | Label error detection performance | Helped |
Surprising findings
- CLIP pretrained from scratch only on the noisy MIMIC-CXR training set outperforms BiomedCLIP for label error detection.
- Manual annotation of top-200 flagged real-world samples shows only 5.5% to 41% actual mislabels, with LEMoN higher than CLIP Similarity on every dataset.
Limitations
- Assumes a binary oracle for mislabels, whereas real mislabels are often ambiguous.
- Some false positives may flag legitimate atypical examples (rare concepts, minority groups), so removal may introduce bias.
- Several hyperparameters; the best configuration requires a labeled validation set, and the fixed setting drops about 1.6 AUROC on average.
Experimental setup
Datasets: CIFAR-10N CIFAR-100N miniImageNet (red annotations) Stanford Cars (red annotations) MSCOCO Flickr30k MMIMDB MIMIC-CXR CC3M VTAB Datacomp
Benchmarks: CIFAR-10N CIFAR-100N miniImageNet Stanford Cars MSCOCO Flickr30k MMIMDB MIMIC-CXR VTAB
Metrics: AUROC AUPRC F1 BLEU-4 CIDEr ROUGE classification accuracy
Baselines: AUM Datamap Confident Learning CLIP Logits CLIP Similarity SimiFeat-V SimiFeat-R Discrepancy Deep k-NN LLaVA prompting CapFilt VDC
Builds on: CLIP Deep k-NN SimiFeat Discrepancy (semantic neighborhood scores)
Closest papers by meaning
- Noise-Aware Image Captioning with Progressively Exploring Mismatched Words (AAAI 2024 · similarity 0.89)
- Sieve: Multimodal Dataset Pruning using Image Captioning Models (CVPR 2024 · similarity 0.87)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.86)
- Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models (ICLR 2025 · similarity 0.86)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.86)
- BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.