Research map / Vision-language and multimodal / Human and embodied modeling
Low-Resource Vision Challenges for Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the LITE benchmark of truly low-resource image tasks (circuit diagrams, historic maps, mechanical drawings) and three adaptation baselines on frozen vision foundation models: generative-data contrastive augmentation, sub-kernel tokenization, and specialized-domain attention.
Problem
Vision foundation models and transfer methods fail on low-resource image tasks that combine scarce data, fine-grained differences, and specialized domains far from natural images.
Approach
Three simple, combinable baselines, one per challenge: generative (Stable Diffusion) label-preserving and label-breaking augmentations with a contrastive loss; splitting the patch projection kernel into learned-weighted sub-kernels for finer tokens; and learned global attention maps shared across tokens, added to a frozen foundation model adapted with AdaptFormer.
Key results
- Zero-shot ImageBind reaches only 19.3% Top-1 on circuit diagram classification; combining all three baselines with ImageBind+AdaptFormer reaches 24.1% Top-1 and 49.3% Top-5.
- On historic map retrieval, ImageBind with the baselines reaches 36.4 R@1 versus 30.3 for AdaptFormer alone.
- Adding AdaptFormer to CLIP raises mechanical drawing retrieval R@1 by +33.7%, and the proposed baselines add a further +4.7% R@1 for CLIP and DINOv2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine generated data, fine-grained tokenization, and specialized-domain attention | ImageBind · Circuit Diagram Classification | Top-1 accuracy (%): 19.8 -> 24.1 | Helped |
| Combine generated data, fine-grained tokenization, and specialized-domain attention | ImageBind · Historic Map Retrieval | Recall@1: 30.3 -> 36.4 | Helped |
| Combine generated data, fine-grained tokenization, and specialized-domain attention | ImageBind · Mechanical Drawing Retrieval | Recall@1: 54.3 -> 60.0 | Helped |
| Add label-preserving and label-breaking generated images (ablation) | ImageBind · Circuit Diagram Classification | Top-1 accuracy (%): 19.3 -> 21.3 | Helped |
| Use selected sub-kernels for smaller image regions (ablation) | ImageBind · Circuit Diagram Classification | Top-1 accuracy (%): 19.3 -> 20.9 | Helped |
| Add learned global attention maps for specialized domains (ablation) | ImageBind · Circuit Diagram Classification | Top-1 accuracy (%): 19.3 -> 20.6 | Helped |
| Combine all three proposed baselines | ImageBind · Circuit Diagram Classification | Top-1 accuracy (%): 19.3 -> 24.1 | Helped |
Surprising findings
- Current vision foundation models such as CLIP, SAM, and DINOv2 perform poorly zero-shot on low-resource tasks, with some near-chance retrieval R@1 (e.g., SAM 0.7 on historic maps).
- Full-parameter finetuning degrades circuit classification accuracy (13.2% Top-1) below zero-shot transfer (19.3%).
Limitations
- Proposed baselines remain far from solving the tasks (24.1% Top-1 on circuit classification).
- Baselines are overconfident and struggle with relations between multiple image regions and rare image styles.
- Does not explicitly address intra-class variation, imbalanced representation, or input-to-ground-truth domain shift.
Experimental setup
Datasets: LITE benchmark (circuit diagrams, historic maps from Old Maps Online, satellite images from Google Maps, mechanical draw
Benchmarks: LITE circuit diagram classification LITE historic map retrieval LITE mechanical drawing retrieval
Metrics: Top-1 accuracy Top-5 accuracy Recall@1 Recall@5 mean rank
Baselines: Zero-shot CLIP BLIP SAM AIM DINOv2 ImageBind AdaptFormer Mixup CutMix Random Erasing DA-Fusion PLEor
Builds on: AdaptFormer Stable Diffusion DA-Fusion Adaptive-FGSBIR CLIP
Closest papers by meaning
- LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models (ICLR 2025 · similarity 0.85)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.84)
- VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models (NeurIPS 2025 · similarity 0.84)
- Learning from Offline Foundation Features with Tensor Augmentations (NeurIPS 2024 · similarity 0.84)
- LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models (NeurIPS 2026 · similarity 0.84)
- Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurement (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.