Research map / Detection and segmentation / Open-set and few-shot detection
OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triad
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OCRT, an object-concept-relation framework that adds slot-based object decomposition, importance-weighted concept filtering, and a concept graph of relation tokens to boost SAM and CLIP robustness and generalization in the open world.
Problem
Foundation models such as SAM and CLIP lose generalizability under distribution shift, weak supervision (box/point/poly prompts), and adversarial attacks, and existing fixes are task- or model-specific.
Approach
Decompose visual features into slot-based object-centric representations, weight and keep the most informative concepts via a cosine-similarity importance score, then build a flexible-degree concept graph whose aggregated relation tokens are fed into the FM during fine-tuning, without changing the FM architecture.
Key results
- On kvasir-SEG under poly prompts, OCRT outperforms SAM by 35.91 mIoU points (per Table 1).
- On CAMO, OCRT improves over SAM by 13.60 to 20.18 points across box, point, and poly prompts (per Table 3).
- On zero-shot classification with eps=4/255 attack, OCRT4 averages 33.4 vs FARE4 32.4 and TeCoA4 31.9 across datasets (Table 2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OCRT to SAM | SAM · COCO 2017, box prompt | mIoU: 74.29 -> 78.74 | Helped |
| Add OCRT to SAM | SAM · kvasir-SEG, poly prompt | mIoU: +35.91 | Helped |
| Add OCRT to SAM | SAM · CAMO, point prompt | mIoU: +8.26 | Helped |
| Add OCRT to CLIP | CLIP · ImageNet, clean zero-shot classification | accuracy: 70.4 -> 73.6 | Mixed |
| Add OCRT to CLIP | CLIP · ImageNet, ε = 4/255 zero-shot classification | accuracy: 33.3 -> 35.6 | Helped |
| Add OCRT to CLIP | CLIP in LLaVA · COCO image captioning, ℓ∞ attack | CIDEr: 57.1 -> 59.4 | Helped |
| Add OCRT to CLIP | CLIP in LLaVA · COCO image captioning, clean | CIDEr: 102.4 -> 105.0 | Mixed |
| Remove object-centric representations (ablation) | OCRT · SAM segmentation benchmarks | mIoU | Hurt |
| Remove informative concepts extraction (ablation) | OCRT · SAM segmentation benchmarks | mIoU | Hurt |
| Remove relational reasoning (ablation) | OCRT · SAM segmentation benchmarks | mIoU | Hurt |
| Vary the number of object-centric representations K (ablation) | OCRT · VOC | mIoU | No clear effect |
| Vary the number of relation tokens Nr (ablation) | OCRT · VOC | mIoU | No clear effect |
Surprising findings
- Clean CLIP remains best on POPE hallucination (84.5 avg) while OCRT2 (82.4) is closest to it, though OCRT does not surpass the original clean model on that benchmark.
Limitations
- Gains on clean zero-shot accuracy under adversarial fine-tuning are modest (about 1-2 points over FARE)
- Method is limited to VFM and MMFM per the authors' scope statement
- Hyperparameters K and K-tilde chosen empirically; sensitivity shown only in ablations
Experimental setup
Datasets: SA-1B VOC COCO 2017 OCID CAMO COD10K ISIC kvasir-SEG ImageNet COCO Flickr30k TextVQA
Benchmarks: COCO 2017 Pascal VOC kvasir-SEG ISIC CAMO COD10K OCID ImageNet CIFAR10 CIFAR100 COCO captioning Flickr30k
Metrics: mIoU accuracy CIDEr VQA accuracy
Baselines: SAM TENT SHOT TRIBE DePT WDASS WESAM TeCoA FARE CLIP
Builds on: Slot Attention WESAM FARE TeCoA SAM
Closest papers by meaning
- Large-Scale Visual Relationship Understanding (AAAI 2019 · similarity 0.87)
- Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation (AAAI 2025 · similarity 0.87)
- Interpreting Object-level Foundation Models via Visual Precision Search (CVPR 2025 · similarity 0.86)
- Explaining Domain Shifts in Language: Concept Erasing for Interpretable Image Classification (CVPR 2025 · similarity 0.85)
- Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.85)
- Unknown Prompt the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.