Research map / Detection and segmentation / Open-set and few-shot detection
COCONut: Modernizing COCO Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-annotates COCO segmentation with an assisted-manual pipeline and data engine to produce the COCONut dataset (383K images, 5.18M panoptic masks) with human-verified labels and a new validation benchmark.
Problem
COCO's segmentation annotations are coarse, inconsistent across semantic/instance/panoptic tasks, incomplete, and the benchmark has plateaued, so it is no longer a reliable benchmark for modern segmentation models.
Approach
Combine machine proposals (DETA boxes, kMaX-DeepLab stuff masks) with human raters editing via box/point prompts (box-kMaX Box2Mask, CFR Point2Mask), then iteratively retrain proposal models on human-verified data in a data engine to scale a universal, consistently labeled segmentation dataset.
Key results
- Training kMaX-DeepLab with ConvNeXt-L on COCONut-L raises COCONut-val PQ from 58.3 (COCO-trained) to 62.7 and relabeled COCO-val PQ from 60.4 to 65.1.
- Expert-vs-expert agreement reaches 98.1% (thing) and 97.3% (stuff); assisted-manual annotation takes 10 sec per thing mask versus 10 min purely manual.
- Pseudo-labeled COCO unlabeled data gives only ~0.1 PQ gain, whereas adding fully human-labeled COCONut-B yields larger improvements.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale training data from COCO to COCONut-L | kMaX-DeepLab with ConvNeXt-L · relabeled COCO-val | PQ: 60.4 -> 65.1 | Helped |
| Scale training data from COCO to COCONut-L | kMaX-DeepLab with ConvNeXt-L · COCONut-val | PQ: 58.3 -> 62.7 | Helped |
| Scale training data from COCONut-S to COCONut-L (ablation) | kMaX-DeepLab with ConvNeXt-L · COCONut-val | PQ: 59.4 -> 62.7 | Helped |
| Add machine pseudo-labels for the COCO unlabeled set (ablation) | COCO training data · COCONut-val | PQ: 58.3 -> 58.5 | No clear effect |
| Add machine pseudo-labels for the COCO unlabeled set (ablation) | COCONut-S training data · COCONut-val | PQ: 59.4 -> 59.5 | No clear effect |
| Use human labels for added training images instead of machine pseudo-labels (ablation) | COCONut-S training data · COCONut-val | PQ: 59.5 -> 61.3 | Helped |
| Evaluate on relabeled COCO-val instead of original COCO-val | kMaX-DeepLab trained on COCO · validation set | PQ: 57.9 -> 60.4 | Helped |
| Evaluate on COCONut-val instead of COCO-val | kMaX-DeepLab trained on COCONut-L · validation set | PQ: 58.1 -> 62.7 | Helped |
Surprising findings
- Pseudo-label augmentation from machine predictions gives only marginal PQ gains (about 0.1), much less than human-labeled data.
- COCO-val performance saturates as training data grows from COCONut-B to COCONut-L, so the original 5K-image validation set is inadequate for evaluating modern segmenters.
Limitations
- Pseudo-labels from machine models provide limited benefit, so scaling still depends on costly human verification.
- Stuff annotation remains slower in the assisted pipeline due to revising superpixel annotations.
- Only 133 COCO-style classes and daily images are covered; the dataset is restricted to COCO-related class definitions.
Experimental setup
Datasets: COCO Objects365 COCO panoptic Sama-COCO
Benchmarks: COCO-val relabeled COCO-val COCONut-val
Metrics: PQ AP mask mIoU
Baselines: COCO ADE20K LVIS EntitySeg Sama-COCO
Builds on: COCO kMaX-DeepLab DETA CFR Objects365
Closest papers by meaning
- COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation (NeurIPS 2025 · similarity 0.89)
- K-Net: Towards Unified Image Segmentation (NeurIPS 2021 · similarity 0.88)
- Large-Scale Interactive Object Segmentation With Human Annotators (CVPR 2019 · similarity 0.87)
- The Devil is in the Object Boundary: Towards Annotation-free Instance Segmentation using Foundation Models (ICLR 2024 · similarity 0.86)
- COCOTree: A Dataset and Benchmark for Open Tree-Structured Visual Decomposition (NeurIPS 2026 · similarity 0.85)
- MaSS13K: A Matting-level Semantic Segmentation Benchmark (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.