Research map / Vision-language and multimodal / Cross-modal learning
Revamping Cross-Modal Recipe Retrieval With Hierarchical Transformers and Self-Supervised Learning
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical Transformer recipe encoding instead of Transformer sentence encodings with average pooling (ablation) | Recipe encoder · Recipe1M validation, image-to-recipe | R1: 20.2 -> 24.4 | Helped |
| Replace LSTM sentence encoders with Transformer encoders in the hierarchical recipe encoder (ablation) | Recipe encoder · Recipe1M validation, image-to-recipe | R1: 19.8 -> 24.4 | Helped |
| Add ingredients and instructions to a title-only recipe representation (ablation) | Recipe encoder · Recipe1M validation, image-to-recipe | R1: 6.0 -> 24.4 | Helped |
| Add instructions to an ingredients-only recipe representation (ablation) | Recipe encoder · Recipe1M validation, image-to-recipe | R1: 19.1 -> 22.4 | Helped |
| Add the self-supervised recipe-component loss (ablation) | Image-recipe retrieval model · Recipe1M test, 10k, image-to-recipe | R1: 26.8 -> 27.3 | Helped |
| Train with recipe-only samples in addition to paired samples (ablation) | Image-recipe retrieval model · Recipe1M test, 10k, image-to-recipe | R1: 26.8 -> 27.9 | Helped |
| Use learned projection functions rather than identity functions in component-alignment losses (ablation) | Self-supervised recipe loss · Recipe1M test, 10k, image-to-recipe | R1: 0.5 point decrease | Helped |
| Replace ResNet-50 with ViT as the image encoder (ablation) | Image-recipe retrieval model · Recipe1M test, 10k, image-to-recipe | R1: 27.9 -> 33.5 | Helped |
| Hallucinate a missing title embedding from available recipe components (ablation) | Image-recipe retrieval model · Recipe1M test, 10k, image-to-recipe, title missing | R1: 22.7 -> 24.2 | Helped |
Closest papers by meaning
- CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval (AAAI 2021 · similarity 0.95)
- Learning Cross-Modal Embeddings for Cooking Recipes and Food Images (CVPR 2017 · similarity 0.94)
- Pic2Prep: A Multimodal Conversational Agent for Cooking Assistance (AAAI 2025 · similarity 0.86)
- OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation (CVPR 2024 · similarity 0.83)
- Robust Self-Paced Hashing for Cross-Modal Retrieval with Noisy Labels (AAAI 2025 · similarity 0.82)
- Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.