AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Enhancing Recipe Retrieval with Foundation Models: A Data Augmentation Perspective

Fangzhou Song, Bin Zhu, Yanbin Hao, Shuo Wang

ECCV 2024 · Vision-language and multimodal › Cross-modal learning · approach: Probabilistic embeddings for cross-modal retrieval · 14 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add adapter layers to both CLIP image and recipe encoders (ablation)zero-shot CLIP · Recipe1M 10k image-to-recipeR@1: 14.9 -> 39.2Improved
add recipe self-alignment loss L_rec (ablation)adapter CLIP baseline · Recipe1M 10k image-to-recipeR@1: 39.2 -> 42.3Improved
add LLM visual imagination description with adapter text encoder (ablation)DAR baseline · Recipe1M 10k image-to-recipeR@1: 42.3 -> 43.7Improved
add SAM image segments with adapter layers in the segment encoder instead of frozen encoder (ablation)DAR baseline · Recipe1M 10k image-to-recipeR@1: 43.0 -> 42.1Hurt
add SAM segments with frozen encoder (ablation)DAR baseline · Recipe1M 10k image-to-recipeR@1: 42.3 -> 43.0Improved
replace triplet loss with circle loss (ablation)DAR baseline · Recipe1M 10k image-to-recipeR@1: 36.9 -> 42.3Improved
add visual imagination description at test time (DAR+)DAR · Recipe1M 10k image-to-recipeR@1: 44.2 -> 47.4Improved
also add image segments at test time (DAR++ vs DAR+)DAR+ · Recipe1M 10k recipe-to-imageR@1: image-to-recipe 47.4 -> 47.8; recipe-to-image 48.3 -> 47.4Mixed
use number of SAM segments n=4 instead of fewer or more (ablation)DAR++ · Recipe1M 10krecipe-to-image R@1: n=1: 46.2, n=6: 46.9, n=4: 47.4Improved
use visual imagination description instead of LLM summary (ablation)DAR · Recipe1M 10kR@1: +0.7% image-to-recipe, +0.9% recipe-to-imageImproved
apply test-time description and segment augmentation to H-T(ViT)H-T(ViT) · Recipe1M 10k image-to-recipeR@1: 33.5 -> 42.4Improved
loss weights alpha=beta=1 instead of 0.1 (ablation)multi-level circle loss · Recipe1M 10k image-to-recipeR@1: 43.3 -> 44.2Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy