Research map / Vision-language and multimodal / Cross-modal learning
Enhancing Recipe Retrieval with Foundation Models: A Data Augmentation Perspective
Fangzhou Song, Bin Zhu, Yanbin Hao, Shuo Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adapter layers to both CLIP image and recipe encoders (ablation) | zero-shot CLIP · Recipe1M 10k image-to-recipe | R@1: 14.9 -> 39.2 | Improved |
| add recipe self-alignment loss L_rec (ablation) | adapter CLIP baseline · Recipe1M 10k image-to-recipe | R@1: 39.2 -> 42.3 | Improved |
| add LLM visual imagination description with adapter text encoder (ablation) | DAR baseline · Recipe1M 10k image-to-recipe | R@1: 42.3 -> 43.7 | Improved |
| add SAM image segments with adapter layers in the segment encoder instead of frozen encoder (ablation) | DAR baseline · Recipe1M 10k image-to-recipe | R@1: 43.0 -> 42.1 | Hurt |
| add SAM segments with frozen encoder (ablation) | DAR baseline · Recipe1M 10k image-to-recipe | R@1: 42.3 -> 43.0 | Improved |
| replace triplet loss with circle loss (ablation) | DAR baseline · Recipe1M 10k image-to-recipe | R@1: 36.9 -> 42.3 | Improved |
| add visual imagination description at test time (DAR+) | DAR · Recipe1M 10k image-to-recipe | R@1: 44.2 -> 47.4 | Improved |
| also add image segments at test time (DAR++ vs DAR+) | DAR+ · Recipe1M 10k recipe-to-image | R@1: image-to-recipe 47.4 -> 47.8; recipe-to-image 48.3 -> 47.4 | Mixed |
| use number of SAM segments n=4 instead of fewer or more (ablation) | DAR++ · Recipe1M 10k | recipe-to-image R@1: n=1: 46.2, n=6: 46.9, n=4: 47.4 | Improved |
| use visual imagination description instead of LLM summary (ablation) | DAR · Recipe1M 10k | R@1: +0.7% image-to-recipe, +0.9% recipe-to-image | Improved |
| apply test-time description and segment augmentation to H-T(ViT) | H-T(ViT) · Recipe1M 10k image-to-recipe | R@1: 33.5 -> 42.4 | Improved |
| loss weights alpha=beta=1 instead of 0.1 (ablation) | multi-level circle loss · Recipe1M 10k image-to-recipe | R@1: 43.3 -> 44.2 | Improved |
Closest papers by meaning
- CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval (AAAI 2021 · similarity 0.92)
- Learning Cross-Modal Embeddings for Cooking Recipes and Food Images (CVPR 2017 · similarity 0.92)
- Revamping Cross-Modal Recipe Retrieval With Hierarchical Transformers and Self-Supervised Learning (CVPR 2021 · similarity 0.92)
- OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation (CVPR 2024 · similarity 0.86)
- Pic2Prep: A Multimodal Conversational Agent for Cooking Assistance (AAAI 2025 · similarity 0.85)
- Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
