Research map / Vision-language and multimodal / Human and embodied modeling
Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces pixel-level visual entity linking (PL-VEL) and builds Mask-OVEN-Wiki (~5M mask-entity annotations) via reverse region-to-entity annotation, then trains an Osprey-based MLLM with visual semantic tokenization.
Problem
Visual entity linking usually relies on text queries, which are hard to write for complex scenes; pixel-mask references are a more convenient alternative but no large-scale dataset exists for linking masks to knowledge-base entities.
Approach
Reverse the annotation direction: start from known entity labels in OVEN-Wiki and segment their regions with open-vocabulary segmentation (Grounded-SAM and SEEM), enhanced by Wikidata hypernym expansion, GPT-3.5 relational expressions, model ensembling and heuristic filtering; the model uses region-interacted attention via FastSAM-based visual semantic tokens.
Key results
- Manual evaluation of 2,000 sampled annotations gives 94.8% annotation accuracy after knowledge augmentation, model ensemble and filtering (vs. 81.4% for Grounded-SAM with entity labels).
- Osprey-7B fine-tuned on Mask-OVEN-Wiki reaches 20.0% test accuracy vs 1.3% zero-shot.
- Visual semantic tokenization (Osprey-Seg-7B) reaches 25.2% overall test accuracy, about 5 points above the fine-tuned Osprey baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on MaskOven-Wiki | Osprey-7B · MaskOven-Wiki test | accuracy: 1.3 -> 20.0 | Helped |
| Add visual semantic tokenization to the fine-tuned model (ablation) | Osprey-7B · MaskOven-Wiki test | accuracy: 20.0 -> 25.2 | Helped |
| Add visual semantic tokenization to the fine-tuned model (ablation) | Osprey-7B · MaskOven-Wiki validation | accuracy (entity split): 19.0 -> 22.7 | Helped |
| Add visual semantic tokenization to the fine-tuned model (ablation) | Osprey-7B · MaskOven-Wiki validation | accuracy (query split): 6.2 -> 11.7 | Helped |
| Add pre-training before fine-tuning (ablation) | Osprey-7B · MaskOven-Wiki validation | accuracy (Overall): 18.7 -> 19.0 | Helped |
| Add pre-training before fine-tuning (ablation) | Osprey-7B · MaskOven-Wiki validation | accuracy (Overall): 22.4 -> 24.0 | Helped |
| Add intensional knowledge augmentation to entity-label references (ablation) | Grounded-SAM · MaskOven-Wiki annotation evaluation | annotation accuracy: 81.4 -> 86.1 | Helped |
| Add extensional reference augmentation (ablation) | Grounded-SAM · MaskOven-Wiki annotation evaluation | annotation accuracy: 81.4 -> 83.0 | Helped |
| Use model ensemble and heuristic filtering (ablation) | Reverse annotation framework · MaskOven-Wiki annotation evaluation | annotation accuracy: 81% to 95% | Helped |
| Use reverse annotation rather than direct PL-VEL annotation | MaskOven-Wiki annotation · MaskOven-Wiki annotation evaluation | accuracy: 25.5% -> 94.8% | Helped |
| Add retrieval augmentation | AutoVER-7B peer model · Oven-Wiki validation | accuracy: 23.8 vs 42.3 | Helped |
Surprising findings
- Pre-training on the wiki split contributes only 0.1 to 1.6 points, contrasting with GER-ALD's findings.
- A GPT-4V direct segmentation-and-naming approach achieves only about 25% accuracy, far below the reverse annotation accuracy.
Limitations
- Query split questions with extra intents (e.g., 'made of', 'produced by') fall outside the VEL scope and hurt mask methods.
- Fine-tuning used only about 7% of the samples (capped at 50 per entity) due to compute constraints.
- Final linking accuracy remains around 25%, indicating the task is still far from solved.
Experimental setup
Datasets: OVEN-Wiki Mask-OVEN-Wiki Wikidata COCO ImageNet-21k
Benchmarks: Mask-OVEN-Wiki validation Mask-OVEN-Wiki test OVEN-Wiki
Metrics: accuracy annotation accuracy (manual)
Baselines: CLIP CLIP Fusion CLIP2CLIP PaLI-3B PaLI-17B BLIP-2 GPT-4V GlaMM Osprey-7B AutoVER
Builds on: OVEN-Wiki GER-ALD Osprey Grounded-SAM SEEM
Closest papers by meaning
- Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach (NeurIPS 2024 · similarity 0.88)
- A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.87)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.86)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.86)
- MOFI: Learning Image Representations from Noisy Entity Annotated Images (ICLR 2024 · similarity 0.85)
- Visual Data-Type Understanding does not emerge from scaling Vision-Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.