Research map / Large language models / AI and society
Leveraging Computer Vision and Visual LLMs for Cost-Effective and Consistent Street Food Safety Assessment in Kolkata India
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Combines YOLOv10 detection, MobileNetV3 feature clustering, and fine-tuned lightweight VLLMs (PaliGemma, Florence-2) to automate street food safety infrastructure surveys in Kolkata, compared against ChatGPT-4o and human surveyors.
Problem
Human street food safety surveys are costly, logistically hard, and subject to surveyor bias; automated assessment of food safety infrastructure from field video is needed for developing-country settings.
Approach
A three-stage edge-friendly pipeline (sharp-frame selection with optical flow, YOLOv10 detection, MobileNetV3 feature clustering to 5 representatives per class) feeds representative crops to lightweight VLLMs fine-tuned on ChatGPT-4o-generated answers to survey yes/no questions, with processing largely on smartphones.
Key results
- On disagreement cases (250 of 780), fine-tuned Florence-2 generally outperformed the human survey team on accuracy, precision, recall, and F1.
- ChatGPT-4o slightly outperformed fine-tuned PaliGemma and Florence-2 on the full test set.
- PaliGemma inference took 154.24 seconds at batch size 16 and Florence-2 took 178.42 seconds at batch size 6 on an RTX 3090; ChatGPT took 30 minutes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ChatGPT-4o instead of the fine-tuned VLLMs | Street food safety assessment · Full test set | overall performance | Helped |
| Use Florence-2 instead of the human survey team | Street food safety assessment · Disagreements only | accuracy | Helped |
| Use Florence-2 instead of the human survey team | Street food safety assessment · Disagreements only | precision | Helped |
| Use Florence-2 instead of the human survey team | Street food safety assessment · Disagreements only | recall | Helped |
| Use Florence-2 instead of the human survey team | Street food safety assessment · Disagreements only | F1 score | Helped |
Surprising findings
- On disagreement cases, the human survey team had the lowest scores across all metrics, below the VLLMs.
- ChatGPT-4o was slower than the fine-tuned local models, despite the paper's main cost argument being framed around cloud APIs.
Limitations
- Ground truth on disagreement cases relies solely on a single expert's opinion, and expert review was applied only where ChatGPT and surveyors disagreed.
- Object detection favours precision over recall, so some objects are missed (underclassified as background).
- Misclassification between water tanks and hand washing stations, and errors in height interpretation, occur.
Experimental setup
Datasets: Kolkata field survey images and videos (FAO-funded) COCO
Benchmarks: Kolkata street food survey test set (258 images, 864 questions)
Metrics: accuracy precision recall F1 score mAP50 mAP50-90 inference time
Baselines: ChatGPT-4o human survey team
Builds on: YOLOv10 PaliGemma Florence-2 MobileNetV3 K-Means++
Closest papers by meaning
- Combining Deep Learning and Street View Imagery to Map Smallholder Crop Types (AAAI 2024 · similarity 0.80)
- AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark (NeurIPS 2025 · similarity 0.80)
- LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models (ICML 2025 · similarity 0.80)
- OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation (CVPR 2024 · similarity 0.80)
- HarvestNet: A Dataset for Detecting Smallholder Farming Activity Using Harvest Piles and Remote Sensing (AAAI 2024 · similarity 0.79)
- Giving Cars Eyes: Using Computer Vision to Prevent Road Traffic Accidents in Nigeria (AAAI 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.