Research map / Detection and segmentation / Open-set and few-shot detection
Object-Centric Concept-Bottlenecks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Object-Centric Concept Bottlenecks (OCB), which combine object proposals (Mask R-CNN or SAM) with SpLiCE concept activations on the whole image and object crops, aggregated before a linear predictor, and introduces the COCOLogic benchmark.
Problem
Concept bottleneck models rely on holistic image encodings, limiting their expressiveness for object-centric, complex visual tasks such as multi-label and logic-based classification on natural images.
Approach
Extend CBMs with object-level concept activations: pretrained object proposals are cropped, each crop is encoded by a pretrained concept discovery module (SpLiCE), and image-level and object-level concept activations are aggregated (concat, sum, max, count, sum+count) and fed to a linear predictor, with no concept or object supervision.
Key results
- OCB (RCNN) reaches 85.75 mAP on PASCAL-VOC vs 82.42 for the SpLiCE CBM and 89.60 for CLIP+MLP.
- On COCOLogic, OCB (RCNN) reaches 68.84 balanced accuracy vs 58.84 for the CBM and 65.95 for CLIP+MLP.
- Max and sum aggregation perform best overall; no single aggregation is best across all datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add object-level concepts from Mask-RCNN proposals to whole-image concepts | SpLiCE CBM · PASCAL-VOC, COCO(h), COCO(l), SUN397, COCOLogic | mAP / accuracy / balanced accuracy: 82.42 ± 0.01 -> 85.75 ± 0.01; 84.73 ± 0.00 -> 87.33 ± 0.00; 59.19 ± 0.00 -> 64.12 ± 0.00; 74.79 ± 0.01 -> 75.28 ± 0.04; | Helped |
| Add object-level concepts using SAM proposals | SpLiCE CBM · PASCAL-VOC, COCO(h), COCO(l), SUN397 | mAP / accuracy: 82.42 ± 0.01 -> 84.37 ± 0.01; 84.73 ± 0.00 -> 86.91 ± 0.01; 59.19 ± 0.00 -> 63.33 ± 0.00; 74.79 ± 0.01 -> 75.13 ± 0.02 | Helped |
| Use Mask-RCNN instead of SAM for object proposals | OCB · COCOLogic | balanced accuracy: 62.42 ± 0.10 -> 68.84 ± 0.11 | Helped |
| Use Mask-RCNN instead of SAM for object proposals | OCB · SUN397 | accuracy: 75.13 ± 0.02 -> 75.28 ± 0.04 | No clear effect |
| Increase concept capacity by reducing SpLiCE sparsity regularization to match OCB (ablation) | CBM · PASCAL-VOC, COCO(h), COCO(l), SUN397, COCOLogic | mAP / accuracy / balanced accuracy: 82.42 ± 0.01 -> 82.90 ± 0.01; 84.73 ± 0.00 -> 86.31 ± 0.00; 59.19 ± 0.00 -> 61.70 ± 0.00; 74.79 ± 0.01 -> 74.85 ± 0.01; | Helped |
| Remove object-level encodings (ablation) | OCB · PASCAL-VOC | mAP: 85.75 ± 0.01 -> 82.42 ± 0.01 | Hurt |
| Remove image-level encoding (ablation) | OCB · PASCAL-VOC | mAP: 85.75 ± 0.01 -> 77.48 ± 0.03 | Hurt |
| Use count or sum + count aggregation rather than max or sum (ablation) | OCB · PASCAL-VOC | mAP: 82.98 ± 0.01 -> 85.67 ± 0.01; 83.30 ± 0.01 -> 85.67 ± 0.01 | Helped |
| Use count or sum + count aggregation rather than max or sum (ablation) | OCB · SUN397 | accuracy: 74.02 ± 0.02 -> 69.32 ± 0.02; 73.65 ± 0.01 -> 68.88 ± 0.03 | Hurt |
| Use sum instead of count-based aggregation (ablation) | OCB · COCOLogic | balanced accuracy: 59.82 ± 0.02 -> 65.44 ± 0.03; 60.16 ± 0.55 -> 65.44 ± 0.03 | Helped |
| Use max instead of concat aggregation (ablation) | OCB · COCOLogic | balanced accuracy: 62.16 ± 0.14 -> 65.33 ± 0.13 | Helped |
| Increase maximum object proposals from 1 to 10 (ablation) | OCB with max aggregation · COCO(l) | mAP: 60.90 ± 0.00 -> 64.12 ± 0.00 | Helped |
Surprising findings
- Removing image-level encodings from OCB, leaving only object encodings, drops PASCAL-VOC performance from 85.75 to 77.48.
- Adding more object proposals or lower minimum object size degrades COCOLogic and SUN397 performance beyond a point, unlike the object-based tasks.
Limitations
- Performance is bounded by the pretrained CLIP-based concept discovery (SpLiCE), which yields noisy concepts for small object crops.
- Aggregation methods other than concatenation lose object-to-concept traceability.
- Only linear predictors were evaluated on COCOLogic, which is not linearly separable.
Experimental setup
Datasets: PASCAL-VOC MSCOCO SUN397 COCOLogic
Benchmarks: PASCAL-VOC COCO(h) COCO(l) SUN397 COCOLogic
Metrics: mAP accuracy balanced accuracy
Baselines: CBM (SpLiCE) CBM (equal capacity) CLIP+MLP C2F-CBM
Builds on: Concept Bottleneck Models SpLiCE Mask R-CNN Segment Anything
Closest papers by meaning
- VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance (NeurIPS 2024 · similarity 0.89)
- Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations (NeurIPS 2026 · similarity 0.89)
- Hybrid Concept Bottleneck Models (CVPR 2025 · similarity 0.88)
- V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer (AAAI 2025 · similarity 0.88)
- Coarse-to-Fine Concept Bottleneck Models (NeurIPS 2024 · similarity 0.88)
- Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.