Research map / Vision-language and multimodal / CLIP and zero-shot
OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OVFoodSeg, an open-vocabulary food ingredient segmentation framework that enriches frozen CLIP text embeddings with image-specific information via a BLIP-2-style FoodLearner pre-trained on Recipe-1M+ and an Image-Informed Text Encoder, trained on top of SAN.
Problem
Food ingredient segmentation from images suffers from large intra-class visual variance, emergence of new ingredients, and high annotation cost; existing open-vocabulary segmenters use static CLIP text embeddings and closed vocabularies.
Approach
Augment static CLIP class text embeddings with image-conditioned query tokens from a BERT-style FoodLearner (pre-trained with ITC, ITM, and LM losses on food image-recipe pairs), summing the averaged visual tokens with text embeddings before SAN-based mask classification.
Key results
- On FoodSeg103, OVFoodSeg improves novel-class mIoU over SAN by about 4.9% on average across three random class splits (e.g., 30.0 vs 25.6 on split 1).
- On FoodSeg195, OVFoodSeg gains about 3.5% novel mIoU over SAN on average across splits (e.g., 24.0 vs 20.0 on split 1).
- SimSeg reaches higher overall mIoU on FoodSeg195 but runs about 30 times slower (0.3 fps vs 9.8 fps on V100).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use image-informed text representations with FoodLearner | OVFoodSeg · FoodSeg103 novel classes | mIoU: 4.9% | Helped |
| Use image-informed text representations with FoodLearner | OVFoodSeg · FoodSeg195 novel classes | mIoU: 3.5% | Helped |
| Add Recipe-1M+ pre-training to FoodLearner (ablation) | OVFoodSeg · FoodSeg103 novel classes | mIoU | Helped |
| Add Recipe-1M+ pre-training to FoodLearner (ablation) | OVFoodSeg · FoodSeg195 novel classes | mIoU | Helped |
| Add LM loss during FoodLearner pre-training (ablation) | OVFoodSeg · FoodSeg103 | mIoUn: 26.5 -> 31.1 | Helped |
| Add ITM loss during FoodLearner pre-training (ablation) | OVFoodSeg · FoodSeg103 | mIoUn: 27.3 -> 31.1 | Helped |
| Add ITC loss during FoodLearner pre-training (ablation) | OVFoodSeg · FoodSeg103 | mIoUn: 30.8 -> 31.1 | Helped |
| Use no prompt template instead of the default single prompt (ablation) | OVFoodSeg · FoodSeg103 | mIoUn: 30.2; 31.1 | Hurt |
| Use ViLD prompt templates instead of the default single prompt (ablation) | OVFoodSeg · FoodSeg103 | mIoUn: 29.4; 31.1 | Hurt |
| Use ImageNet prompt templates instead of the default single prompt (ablation) | OVFoodSeg · FoodSeg103 | mIoUn: 29.3; 31.1 | Hurt |
Surprising findings
- Using no prompt template or the default 'a photo of' template gives better novel-class mIoU than prompt-ensembling methods such as ViLD and ImageNet templates.
- Without Recipe-1M+ pre-training, the OVFoodSeg variant still beats SAN on novel classes by 4.0% and 2.4%.
Limitations
- FoodSeg195 is not publicly available due to confidentiality, limiting reproducibility.
- SimSeg achieves better overall mIoU on FoodSeg195, so the method does not dominate on all metrics.
- Prompt-averaging approaches do not fully exploit text embeddings, as acknowledged by the authors.
Experimental setup
Datasets: Recipe-1M+ FoodSeg103 FoodSeg195
Benchmarks: FoodSeg103 FoodSeg195
Metrics: mIoU mIoU novel mIoU base mIoU overall
Baselines: MaskCLIP OVSeg SimSeg FreeSeg SAN HIPIE
Builds on: SAN BLIP-2 CLIP Recipe-1M+
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.85)
- Pic2Prep: A Multimodal Conversational Agent for Cooking Assistance (AAAI 2025 · similarity 0.85)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.85)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.85)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.85)
- LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.