Research map / Vision-language and multimodal / CLIP and zero-shot

Homology Consistency Constrained Efficient Tuning for Vision-Language Models

NeurIPS 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Test-time adaptation of VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Homology Consistency constraint that aligns persistent-homology track structure of image and text latent manifolds (plus a deviating perturbation) to improve efficient adapter tuning of CLIP for few-shot and domain generalization.

Problem

Efficient transfer of vision-language models with few samples risks disrupting general image-text correspondence, so learned alignment generalizes poorly beyond observed samples.

Approach

Builds Vietoris-Rips complexes and filtrations over image and text embeddings, tracks 0-dimensional persistence tracks, and adds a loss forcing text-manifold tracks to coincide with image-manifold tracks, plus a deviating perturbation that spreads related images around text embeddings; tailored to TaskRes (residual blending) and Tip-Adapter-F (key-value cache).

Key results

• HC-TAF improves Tip-Adapter-F by 1.04% average at 16 shots and HC-TR improves TaskRes by 0.40% at 16 shots.
• HC*-TAF reaches 66.40% on ImageNet at 16 shots, above APE-T's 66.07%, the best prior baseline listed.
• On domain generalization with ViT-B/16, HC-TAF reaches 61.06% average target accuracy vs 60.95% for Tip-Adapter-F.

Measured effects reported in the paper

Surprising findings

• Deviating perturbation alone (without track coincidence) degrades baseline performance, and HC-TR underperforms TaskRes on several target domains with ResNet and ViT-B/32 backbones.

Limitations

• Higher-dimensional homology classes are not explored; only 0-dimensional classes are used.
• Evaluated only on few-shot recognition tasks and domain generalization, not other VLM downstream tasks.
• Gains are modest (often under 1%) and no error bars or statistical significance tests are reported.

Experimental setup

Datasets: ImageNet Caltech101 DTD EuroSAT FGVCAircraft Flowers102 Food101 OxfordPets StanfordCars SUN397 UCF101 ImageNetV2

Benchmarks: ImageNet Caltech101 DTD EuroSAT FGVCAircraft Flowers102 Food101 OxfordPets StanfordCars SUN397 UCF101 ImageNetV2

Metrics: accuracy

Baselines: Zero-shot CLIP Linear probe CLIP CLIP-Adapter TaskRes Tip-Adapter-F GraphAdapter APE-T

Builds on: TaskRes Tip-Adapter-F CLIP CLIP-Adapter

Closest papers by meaning

• Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning (NeurIPS 2026 · similarity 0.91)
• DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.87)
• Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.87)
• Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.86)
• Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models (NeurIPS 2025 · similarity 0.86)
• Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.86)