Research map / Detection and segmentation / Open-set and few-shot detection
Generating Enhanced Negatives for Training Language-Based Object Detectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates LLM-written negative text descriptions and GLIGEN-inpainted negative images (with filtering) to train language-based object detectors GLIP-T and FIBER-B, improving OmniLabel and D3 AP.
Problem
Language-based object detectors are trained with discriminative objectives but existing negatives (other descriptions in the image or random descriptions) are weak; the space of free-form text negatives is huge and human-curated negatives are costly.
Approach
Use instruction-tuned LLMs (LLM-based foils, re-combination, in-context summaries) to generate semantically related contradicting text negatives, distill them into a finetuned LLaMA-7B, and use GLIGEN text-to-image inpainting to create negative images, filtered by box-size rules and CLIP similarity, as extra training signal.
Key results
- Adding generated negatives improves GLIP-T by +2.9 AP on OmniLabel and FIBER-B by +2.4 AP on OmniLabel.
- FIBER-B on D3 full-description AP rises from 22.7 to 26.0 (+3.3) with generated negative texts and images.
- Box and CLIP filters raise the fraction of non-noisy generated images from 47% to 84% in a human study of 100 samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add generated negative data | FIBER-B · OmniLabel | AP: 25.7 -> 28.1 | Helped |
| Add generated negative data | FIBER-B · D3 | AP: 22.7 -> 26.0 | Helped |
| Add generated negative data | GLIP-T · OmniLabel | AP: 19.3 -> 22.2 | Helped |
| Add generated negative data | GLIP-T · D3 | AP: 19.1 -> 21.4 | Helped |
| Train with rule-based foil negatives (ablation) | FIBER-B · OmniLabel | AP: 25.7 -> 26.4 | Helped |
| Train with LLM-based foil negatives (ablation) | FIBER-B · OmniLabel | AP: 25.7 -> 26.5 | Helped |
| Train with LLM recombination negatives (ablation) | FIBER-B · OmniLabel | AP: 25.7 -> 26.9 | Helped |
| Train with in-context-summary negatives (ablation) | FIBER-B · D3 | AP: 22.7 -> 25.7 | Helped |
| Add raw generated negative images without filtering (ablation) | FIBER with negative texts · D3 | AP: 25.9 -> 23.0 | Hurt |
| Apply box and CLIP filters to generated negative images (ablation) | FIBER with negative texts · OmniLabel | APc: 30.7 -> 31.1 | Helped |
| Concatenate filtered positive and negative images (ablation) | FIBER with negative texts · OmniLabel | APc: 31.1 -> 31.7 | Helped |
| Ensemble weights of text-only and text-plus-image models (ablation) | FIBER with negative texts · OmniLabel | APc: 31.7 -> 32.1 | Helped |
Surprising findings
- Raw generated negative images without filtering hurt performance (FIBER AP_d on OmniLabel drops from 22.3 to 22.4 and D3 full drops only slightly, and OmniLabel APc drops), showing noise sensitivity.
- Negative images help OmniLabel but give little gain on D3, attributed to a domain gap in D3 image regions clustered by category.
Limitations
- Raw generated images are noisy (only 47% non-noisy before filtering), requiring filters and manual checks.
- Negative images give limited gains on D3 due to domain gap between generated regions and D3 data.
- Relies on Winoground-style in-context examples and a human-checked seed set for generation.
Experimental setup
Datasets: Flickr30k Objects365 Winoground OmniLabel D3
Benchmarks: OmniLabel OmniLabel-Negative D3
Metrics: AP APc APd APd-P Full AP Pres AP Abs AP
Baselines: GLIP-T FIBER-B Detic MDETR OFA-L OWL-ViT-L G-DINO-B OFA-DOD Rule-based foils LLM-based foils
Builds on: GLIP FIBER GLIGEN Winoground LLaMA
Closest papers by meaning
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.89)
- Language-conditioned Detection Transformer (CVPR 2024 · similarity 0.88)
- Open-Vocabulary Object Detection via Language Hierarchy (NeurIPS 2024 · similarity 0.87)
- Expanding the Scope of Negatives: Boosting Image-Text Matching with Negatives Distribution Guided Learning (AAAI 2025 · similarity 0.87)
- The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding (CVPR 2024 · similarity 0.87)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.