Research map / Label-efficient and robust learning / Recognition and long-tail
Combining Deep Learning and Street View Imagery to Map Smallholder Crop Types
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an automated pipeline that curates Google Street View field images, classifies crop types with CNNs trained on web, expert, or GPT-4V labels, and uses the labels to train a Sentinel-2 random forest producing a 10m crop type map of Thailand.
Problem
Crop type maps are hard to create in low- and middle-income smallholder countries because ground truth labels are costly to collect via field surveys.
Approach
Combine land-cover-filtered, heading-corrected Google Street View imagery with weakly labeled web images or zero-shot GPT-4V labels to auto-generate tens of thousands of crop type ground references, then train a Sentinel-2 harmonic-regression random forest to map crops wall-to-wall.
Key results
- Random forest trained on 81,000 CNN-labeled ground references achieved 93% overall accuracy on the country map, versus 69% for expert-only GSV references (984 samples).
- GPT-4V zero-shot classification reached 95% accuracy on the expert-labeled GSV test set, and a CNN trained on GPT-4V labels reached 92%.
- Field/not-field filtering raised precision from 0.42 to 0.98 at recall 0.95.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine WebCC and iNaturalist training images instead of using either source alone | ResNet-50 street-view crop classifier · Thailand street-view test set | Overall Acc: 0.82 $\pm$ 0.04 (WebCC), 0.81 $\pm$ 0.04 (iNaturalist) -> 0.85 $\pm$ 0.02 | Helped |
| Train the street-view CNN on expert-labeled GSV images rather than weak web labels | ResNet-50 street-view crop classifier · Thailand street-view test set | Overall Acc: 0.85 $\pm$ 0.02 -> 0.93 $\pm$ 0.01 | Helped |
| Use GPT-4V zero-shot crop labels instead of a trained classifier | Crop classification in street-view images · Thailand street-view test set | Overall Acc: 0.92 $\pm$ 0.01 -> 0.95 $\pm$ 0.00 | Helped |
| Use 81,000 CNN-generated ground references rather than 984 expert-labeled references | Random forest crop-type mapper · Thailand remote sensing test set | Overall Acc: 0.69 $\pm$ 0.04 -> 0.93 $\pm$ 0.01 | Helped |
| Generate ground-reference labels with the expert-trained CNN rather than weakly supervised CNNs | Random forest crop-type mapper · Thailand remote sensing test set | Overall Acc: 0.80 $\pm$ 0.01 (WebCC), 0.78 $\pm$ 0.02 (iNaturalist) -> 0.93 $\pm$ 0.01 | Helped |
| Add camera heading toward fields to the baseline image-collection approach (ablation) | GSV field-image filtering · Thailand field-image filtering | Precision: 0.07 -> 0.14 | Helped |
| Add cropland filtering (ablation) | GSV field-image filtering · Thailand field-image filtering | Precision: 0.14 -> 0.31 | Helped |
| Add tree-cover filtering to cropland filtering (ablation) | GSV field-image filtering · Thailand field-image filtering | Precision: 0.31 -> 0.42 | Helped |
| Add a field/not-field CNN after the land-cover filters (ablation) | GSV field-image filtering · Thailand field-image filtering | Precision: 0.42 -> 0.98 | Helped |
| Use sliding windows instead of whole-image classification (ablation) | Expert-labeled street-view crop classifier · Thailand street-view test set | Overall Acc: 82% -> 90% | Helped |
| Add MHP probability thresholding at 0.90 to sliding-window classification (ablation) | Expert-labeled street-view crop classifier · Thailand street-view test set | Overall Acc: 90% -> 93% | Helped |
| Increase the MHP threshold (ablation) | Street-view crop classifiers · Thailand street-view test set | Overall F1 | Helped |
Surprising findings
- Zero-shot GPT-4V outperformed every CNN trained on weakly or expert-labeled street-view data (95% vs 93% best CNN).
- Automated 81,000-sample labels from a CNN outperformed training on about 1,000 expert labels for the country-wide map, despite label noise.
Limitations
- GPT-4V query limits make labeling more than a few thousand images per month difficult.
- Street-view coverage is uneven globally and updated only about every 3 years.
- Only validated GPT-4V for the four main crops; 'Other' class performance weaker for weakly supervised models.
Experimental setup
Datasets: Google Street View Thailand images OpenStreetMap ESA WorldCover 10m Sentinel-2 L2A Google Images Creative Commons (WebCC) iNaturalist FAO planted area statistics
Benchmarks: Expert-labeled GSV test set (Thailand) Expert-labeled field points for remote sensing map (1,600 fields)
Metrics: overall accuracy F1 precision recall
Baselines: Baseline (Midwest GSV method) WebCC-trained CNN iNaturalist-trained CNN Expert GSV-only ground references
Builds on: ResNet harmonic regression random forest Google Street View crop mapping approach
Closest papers by meaning
- Scalable Vision-Guided Crop Yield Estimation (AAAI 2026 · similarity 0.89)
- HarvestNet: A Dataset for Detecting Smallholder Farming Activity Using Harvest Piles and Remote Sensing (AAAI 2024 · similarity 0.88)
- Fields of The World: A Machine Learning Benchmark Dataset for Global Agricultural Field Boundary Segmentation (AAAI 2025 · similarity 0.87)
- Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind (NeurIPS 2025 · similarity 0.87)
- Exact: Exploring Space-Time Perceptive Clues for Weakly Supervised Satellite Image Time Series Semantic Segmentation (CVPR 2025 · similarity 0.86)
- GTPBD: A Fine-Grained Global Terraced Parcel and Boundary Dataset (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.