Research map / Vision-language and multimodal / CLIP and zero-shot
VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VT-FSL, which uses an LLM conditioned on class names and support images (Cross-modal Iterative Prompting) to write class descriptions and a text-to-image model to synthesize extra shots, then aligns text, support and synthetic features via a kernelized Gram-determinant volume loss (Cross-modal Geometric Alignment).
Problem
Few-shot image classification where support features are semantically thin and LLM-generated text from class names alone hallucinates attributes that contradict the visual evidence.
Approach
Generate visually grounded class descriptions by prompting an LLM with both class names and support images in a single structured four-stage pass, use those descriptions to synthesize additional images, and align the fused textual, support, and synthetic visual embeddings by minimizing a kernelized parallelotope volume contrastive loss.
Key results
- On mini-ImageNet 1-shot, VT-FSL reaches 83.66% vs. 81.14% for ECER and 78.94% for SemFew, with the full model achieving the best results on all ten benchmarks.
- On fine-grained benchmarks, VT-FSL outperforms the second-best SUITED by 3.0%-10.3% in 1-shot.
- Ablations show textual and visual prompts together plus the alignment loss give the best accuracy; the kernelized volume loss beats InfoNCE and linear volume loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add textual prompts (ablation) | VT-FSL · miniImageNet 1-shot | accuracy: 68.47 ± 0.43 -> 78.82 ± 0.36 | Helped |
| Add textual prompts (ablation) | VT-FSL · CIFAR-FS 1-shot | accuracy: 76.43 ± 0.45 -> 84.76 ± 0.36 | Helped |
| Add visual prompts (ablation) | VT-FSL · miniImageNet 1-shot | accuracy: 68.47 ± 0.43 -> 79.17 ± 0.35 | Helped |
| Add visual prompts (ablation) | VT-FSL · CIFAR-FS 1-shot | accuracy: 76.43 ± 0.45 -> 86.01 ± 0.34 | Helped |
| Add kernelized alignment loss to textual prompts (ablation) | VT-FSL · miniImageNet 1-shot | accuracy: 78.82 ± 0.36 -> 78.96 ± 0.36 | Helped |
| Add kernelized alignment loss to visual prompts (ablation) | VT-FSL · CIFAR-FS 5-shot | accuracy: 89.75 ± 0.28 -> 90.13 ± 0.28 | Helped |
| Use textual and visual prompts with alignment loss (ablation) | VT-FSL · miniImageNet 1-shot | accuracy: 82.08 ± 0.31 -> 83.66 ± 0.31 | Helped |
| Replace the proposed kernelized volume loss with InfoNCE (ablation) | VT-FSL · miniImageNet 1-shot | accuracy: 83.66 -> 79.96 | Hurt |
| Replace kernelized volume loss with linear volume loss (ablation) | VT-FSL · miniImageNet 1-shot | accuracy: 83.66 -> 82.33 | Hurt |
| Use support-only inference prototype instead of combined text and visual prototype (ablation) | VT-FSL · miniImageNet 1-shot | accuracy: 83.66 -> 71.63 | Hurt |
| Use textual-only inference prototype instead of combined text and visual prototype (ablation) | VT-FSL · CIFAR-FS 1-shot | accuracy: 88.67 -> 87.43 | Hurt |
| Use visual-only inference prototype instead of combined text and visual prototype (ablation) | VT-FSL · miniImageNet 1-shot | accuracy: 83.66 -> 82.83 | Hurt |
Surprising findings
- InfoNCE contrastive loss degrades accuracy below the no-contrastive baseline on mini-ImageNet 1-shot (79.96% vs 82.49%).
Limitations
- Relies on external LLM and text-to-image generators, whose outputs may still be inaccurate; the paper notes the fusion factor u is tuned on validation data.
- Limitations are only discussed in the supplemental material, per the checklist.
Experimental setup
Datasets: mini-ImageNet tiered-ImageNet CIFAR-FS FC100 CUB-200-2011 Stanford-Cars Stanford-Dogs Places Plantae
Benchmarks: mini-ImageNet tiered-ImageNet CIFAR-FS FC100 CUB-200-2011 Stanford-Dogs Stanford-Cars Places Plantae
Metrics: accuracy (%) 95% confidence interval
Baselines: ProtoNet AM3 DeepEMD SUN FewTURE CPEA SP SemFew ECER SIFT UAP SUITED
Builds on: CLIP SemFew ECER Visformer Chain-of-Thought prompting
Closest papers by meaning
- Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning (AAAI 2025 · similarity 0.88)
- DSV-LFS: Unifying LLM-Driven Semantic Cues with Visual Features for Robust Few-Shot Segmentation (CVPR 2025 · similarity 0.88)
- Text Augmented Correlation Transformer For Few-shot Classification & Segmentation (CVPR 2025 · similarity 0.88)
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.88)
- Simple Semantic-Aided Few-Shot Learning (CVPR 2024 · similarity 0.88)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.