Research map / Detection and segmentation / Open-set and few-shot detection
Hyperbolic Learning with Synthetic Captions for Open-World Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Bootstraps dense synthetic region captions with BLIP-2 and trains an open-world detector (HyperLearner) with hyperbolic contrastive and entailment losses to align visual and caption embeddings in a "caption entails object" hierarchy.
Problem
Open-world detection (detecting objects by class labels or free-form text) needs large manually annotated caption data, which is expensive; synthetic captions are cheap but hallucinate, adding noise to alignment.
Approach
Generate dense region-level synthetic captions with a pre-trained VLM captioner and align visual and caption embeddings in hyperbolic (Lorentz) space with a contrastive loss plus an entailment-cone loss, so captions entail their object embeddings and hallucinated content is not directly aligned to visual features.
Key results
- HyperLearner (O365, GoldG) reaches 48.4 zero-shot and 57.4 fine-tuned mAP on COCO, versus 48.1 and 57.1 for Grounding DINO-T with the same pre-training.
- On LVIS minival with O365+GoldG, HyperLearner gets 31.3 AP and 30.7 APr, outperforming GLIP-T (C) and Grounding DINO-T on rare classes by +13.0 and +16.3 APr.
- On ODinW zero-shot, HyperLearner (O365, GoldG) scores 45.2 APavg, versus 44.4 for GLIP-T(C) and 44.9 for Grounding DINO-T (with Cap4M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HyperLearner instead of Grounding-DINO-T2 with O365 and GoldG pre-training | COCO2017 val · COCO2017 val zero-shot | mAP: 48.1 -> 48.4 | Helped |
| use HyperLearner instead of Grounding-DINO-T2 with O365 and GoldG pre-training | COCO2017 val · COCO2017 val fine-tuning | mAP: 57.1 -> 57.4 | Helped |
| use HyperLearner instead of Grounding-DINO-T with O365, GoldG, and Cap4M | LVIS minival · LVIS minival | AP: 27.4 -> 31.3 | Helped |
| use HyperLearner instead of Grounding-DINO-T with O365, GoldG, and Cap4M | LVIS minival · LVIS minival | APr: 20.8 -> 30.7 | Helped |
| use HyperLearner instead of Grounding-DINO-T with O365, GoldG, and Cap4M | ODinW · ODinW zero-shot | Test APavg: 44.9 -> 45.2 | Helped |
| replace the standard detection objective with hyperbolic learning (ablation) | HyperLearner · COCO | mAP: 42.3 -> 48.4 | Helped |
| replace vanilla contrastive caption learning with hyperbolic learning (ablation) | HyperLearner · COCO | mAP: 46.8 -> 48.4 | Helped |
| add synthetic caption learning to the standard detection objective (ablation) | HyperLearner · LVIS | AP: 13.3 -> 23.6 | Helped |
| replace vanilla contrastive caption learning with hyperbolic learning (ablation) | HyperLearner · LVIS | AP: 23.6 -> 31.3 | Helped |
| add synthetic caption learning to the standard detection objective (ablation) | HyperLearner · ODinW | mAP: 26.3 -> 38.6 | Helped |
| replace vanilla contrastive caption learning with hyperbolic learning (ablation) | HyperLearner · ODinW | mAP: 38.6 -> 45.2 | Helped |
| add captions from ground-truth boxes, grid regions, and proposal regions (ablation) | HyperLearner · COCO | mAP: 42.3 -> 48.4 | Helped |
Surprising findings
- The ablation shows the hyperbolic objective converges slower than the vanilla contrastive baseline but keeps improving, which saturates early.
- Replacing the hyperbolic objective with a vanilla contrastive loss (Lbaseline) gives 46.8 COCO vs 48.4 for the hyperbolic objective, while dropping captions entirely gives 42.3.
Limitations
- Relies on a single caption generator (BLIP-2) and synthetic captions can still hallucinate.
- Cross-modal attention module is needed for referring expressions, and its gains come from RefCOCO-specific localization rather than COCO.
- Comparisons to GLIP and Grounding DINO use the same backbone but differing pre-training data; full-scale VLM pretraining is not matched.
Experimental setup
Datasets: Objects365 GoldG COCO LVIS ODinW RefCOCO RefCOCO+ RefCOCOg
Benchmarks: COCO LVIS minival ODinW RefCOCO RefCOCO+ RefCOCOg
Metrics: mAP AP APr APavg top-1 accuracy
Baselines: GLIP-T Grounding DINO-T Detic DETCLIP-T MDETR Faster R-CNN Deformable DETR DQ-DETR
Builds on: GLIP CenterNetV2 Lorentz hyperbolic embeddings entailment cones BLIP-2
Closest papers by meaning
- DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.88)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.87)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.86)
- Language-conditioned Detection Transformer (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.