Research map / Vision-language and multimodal / Human and embodied modeling
Pic2Prep: A Multimodal Conversational Agent for Cooking Assistance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Demonstrates Pic2Prep, a multimodal chatbot that extracts ingredients and cooking instructions from food images (BLIP fine-tuned on Stable Diffusion-generated food images) and maps cooking actions to ingredients via CookGen and a Mistral-based mapper.
Problem
Existing food computation models give only basic nutritional insights and cannot link ingredients to cooking actions or provide step-by-step instructions needed for substitution and recipe modification from food images.
Approach
Builds a pipeline that generates a synthetic food-image dataset from Recipe1M titles and ingredients via Stable Diffusion, fine-tunes two BLIP models for instructions and ingredients, and links predicted cooking actions to ingredients with a Mistral-trained mapper, exposed through a conversational interface.
Key results
- No quantitative results are reported; the paper presents a system demonstration with a user interface and example queries.
Limitations
- No quantitative evaluation of BLIP extraction, CookGen, or mapper accuracy is reported
- Ingredient substitution and dietary recommendation are proposed but not implemented
- Fallback to 'I don't know' for out-of-scope queries; limited scope to complex cooking processes
Experimental setup
Datasets: Recipe1M
Builds on: BLIP CookGen Recipe1M Stable Diffusion
Closest papers by meaning
- OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation (CVPR 2024 · similarity 0.85)
- NutriBench: A Dataset for Evaluating Large Language Models in Nutrition Estimation from Meal Descriptions (ICLR 2025 · similarity 0.84)
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (ICLR 2024 · similarity 0.82)
- What can large language models do for sustainable food? (ICML 2025 · similarity 0.82)
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.82)
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.