Research map / Large language models / Language, safety and interpretability
LLaFS: When Large Language Models Meet Few-Shot Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLaFS, which fine-tunes CodeLlama-7B with LoRA to directly output 16-point polygon segmentations for few-shot segmentation, guided by support images and a ChatGPT-derived region-attribute table, with pseudo-sample curriculum pretraining.
Problem
Few-shot segmentation relies on limited, biased information from a handful of annotated support images; the paper asks whether an LLM's prior knowledge can supplement this and directly produce segmentation for few-shot tasks.
Approach
Uses a text LLM end-to-end: a task instruction asks for 16-point polygon vertices, and a fine-grained in-context instruction built from ChatGPT-generated class attributes, CLIP region-attribute matching on superpixels of support images, and an expert-guided iterative refinement supplies multimodal guidance. Pseudo support-query pairs with curriculum pretraining (decreasing provided polygon points,
Key results
- LLaFS reaches 74.2 mIoU on PASCAL-5i 1-shot (Fold mean), improving over the second-best method by 4.1 points.
- On COCO-20i, LLaFS gets 47.5 (fold-0 1-shot) and a 60.0 mean 5-shot, 3.9 and 4.0 point gains over the second-best method in 1-shot and 5-shot.
- Removing pseudo-sample curriculum pretraining drops mIoU from 74.2 to 63.5 (-10.7) in the 1-shot PASCAL-5i ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaFS instead of HDMNet | few-shot segmentation · PASCAL-5i 1-shot | mIoU: +4.1% | Helped |
| Use LLaFS instead of HDMNet | few-shot segmentation · PASCAL-5i 5-shot | mIoU: +3.8% | Helped |
| Use LLaFS instead of HDMNet | few-shot segmentation · COCO-20i 1-shot | mIoU: +3.9% | Helped |
| Use LLaFS instead of HDMNet | few-shot segmentation · COCO-20i 5-shot | mIoU: +4.0% | Helped |
| Add support images as a demonstration example (ablation) | LLaFS · PASCAL-5i 1-shot | mIoU: +17.3% | Helped |
| Add class attributes to the fine-grained in-context instruction (ablation) | LLaFS · PASCAL-5i 1-shot | mIoU: +3.5% | Helped |
| Add the region-attribute corresponding table (ablation) | LLaFS · PASCAL-5i 1-shot | mIoU: +4.5% | Helped |
| Add expert-guide instruction refinement (ablation) | LLaFS · PASCAL-5i 1-shot | mIoU: +3.6% | Helped |
| Add iterative refinement (ablation) | LLaFS · PASCAL-5i 1-shot | mIoU: +2.4% | Helped |
| Use detailed segmentation task instructions instead of an abstract summary (ablation) | LLaFS · PASCAL-5i 1-shot | mIoU: 6.5% | Helped |
| Add the fine-grained in-context instruction (ablation) | LLaFS · PASCAL-5i 1-shot | mIoU: 7.2% | Helped |
| Add the refinement network (ablation) | LLaFS · PASCAL-5i 1-shot | mIoU: 5.1% | Helped |
Surprising findings
- Without any support images the LLM still reaches 56.9 mIoU, showing substantial prior knowledge from the LLM alone.
- Increasing the polygon side count beyond 16 slightly decreased performance, suggesting task complexity limits gains.
Limitations
- Relies on ChatGPT API for attribute and ambiguity generation, adding dependence on a closed model.
- Polygon representation with a fixed 16 points limits edge accuracy, requiring a refinement network to fix boundaries.
- Evaluated only on PASCAL-5i and COCO-20i, the two standard FSS benchmarks.
Experimental setup
Datasets: PASCAL-5i COCO-20i synthetic pseudo support-query pairs (Bezier contours + Gaussian noise)
Benchmarks: PASCAL-5i COCO-20i
Metrics: mIoU
Baselines: NTRENet BAM AAFormer SSP IPMT ABCNet HDMNet MIANet MSI SCCAN MM-Former
Builds on: BLIP-2 instruction tuning in-context learning CLIP LoRA polygon-based instance segmentation (Polygon-RNN-style polar representation)
Closest papers by meaning
- DSV-LFS: Unifying LLM-Driven Semantic Cues with Visual Features for Robust Few-Shot Segmentation (CVPR 2025 · similarity 0.89)
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.88)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.88)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.87)
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.87)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.