Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes CLIP and GPT-3 (davinci-003) for country-specific object knowledge and proposes geography knowledge regularization for soft prompts trained on Europe (or USA) that generalize to other geographies.
Problem
Object recognition models degrade under geographical domain shift, and collecting target-geography training data is expensive, so the paper asks how to make class representations geographically robust without target data.
Approach
Augment CLIP prompts with country names and LLM-generated country-specific visual descriptors, then regularize learned soft prompts (CoOp-style) toward the ensembled geographic knowledge embeddings of unseen target countries, so the prompts trained on a source geography generalize to target geographies.
Key results
- CountryInPrompt+LLM Reg improves ViT-B/16 target accuracy on DollarStreet by +2.8 in Africa and +1.6 total over the best soft prompt baseline (KgCoOp).
- On hardest classes (<40% CoOp recall), CountryInPrompt+LLM Reg gains +8.7 over CoOp, versus +4.1 for KgCoOp.
- A 16-shot-per-class regularized model trained only on Europe outperforms a 12-shot-per-class target-trained CoOp on DollarStreet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add geography knowledge regularization using both country prompts and LLM descriptors | Soft prompts trained on Europe data · DollarStreet Africa | balanced accuracy: +2.8 | Helped |
| Add geography knowledge regularization using both country prompts and LLM descriptors | Soft prompts trained on Europe data · DollarStreet Asia | balanced accuracy: +1.2 | Helped |
| Add geography knowledge regularization using both country prompts and LLM descriptors | Soft prompts trained on Europe data · DollarStreet Americas | balanced accuracy: +1.6 / -0.1 | Mixed |
| Add geography knowledge regularization using both country prompts and LLM descriptors | Soft prompts trained on USA data · GeoNet Asia | balanced accuracy: +1.3; +1.4 | Helped |
| Add country names and geo-specific LLM descriptors to zero-shot prompts | CLIP · DollarStreet Total | top-1 balanced accuracy: +3.4 | Helped |
| Use country-specific LLM descriptions instead of general object descriptions | Zero-shot CLIP · DollarStreet Total | top-1 balanced accuracy: 52.6% vs. 51.4% | Helped |
| Combine country-name prompts with country-specific LLM descriptors rather than either knowledge source alone (ablation) | Zero-shot CLIP · DollarStreet | top-1 balanced accuracy | Mixed |
| Use CountryInPrompt knowledge rather than the combined CountryInPrompt and LLM knowledge (ablation) | Geography knowledge regularization · DollarStreet Africa | balanced accuracy: 56.8 vs. 57.2 | Hurt |
| Use CountryLLM knowledge rather than the combined CountryInPrompt and LLM knowledge (ablation) | Geography knowledge regularization · DollarStreet Africa | balanced accuracy: 55.6 vs. 57.2 | Hurt |
| Use combined CountryInPrompt and LLM knowledge for regularization instead of CountryInPrompt alone (ablation) | Soft prompts · DollarStreet classes with CoOp recall <40% | balanced accuracy: 37.5 -> 39.9 | Helped |
| Use combined CountryInPrompt and LLM knowledge for regularization instead of CountryLLM alone (ablation) | Soft prompts · DollarStreet classes with CoOp recall <40% | balanced accuracy: 36.8 -> 39.9 | Helped |
Surprising findings
- Adding country names to CLIP prompts yields gains mainly in Africa and Asia but slight drops in Americas/Europe for ViT encoders.
- Regularized source-only prompts can outperform few-shot target-trained prompts, despite using no target data.
Limitations
- LLM knowledge may be hallucinated, imprecise, or biased, and CLIP representations may be unrepresentative for some countries.
- Gains are modest (typically 1-3 points) and continent-level disparities remain.
- GeoNet experiments use only the top 10 most frequent countries for target knowledge due to LLM query cost.
Experimental setup
Datasets: DollarStreet GeoNet (GeoImNet)
Benchmarks: DollarStreet GeoNet
Metrics: balanced accuracy top-3 accuracy
Baselines: Zero-Shot CLIP GeneralLLM CoOp CoCoOp KgCoOp
Builds on: CoOp KgCoOp CLIP classification by description
Closest papers by meaning
- Textual Supervision Enhances Geospatial Representations in Vision-Language Models (ICML 2026 · similarity 0.86)
- Enhancing Target-unspecific Tasks through a Features Matrix (ICML 2025 · similarity 0.85)
- Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models (NeurIPS 2025 · similarity 0.85)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.85)
- SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.85)
- No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.