Research map / Vision-language and multimodal / Cross-modal learning

Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ToMA, which aligns persistent-homology-selected edge directions (H0 death edges and H1 birth edges on a 1-skeleton filtration) across image and text embeddings in semi-supervised CLIP adaptation, improving over SemiCLIP on remote sensing and fashion benchmarks.

Problem

Semi-supervised vision-language adaptation to specialized domains uses pairwise objectives that ignore global manifold structure, while existing topology-based alignment relies on persistence diagram matching that is non-injective and ignores image-text pairing.

Approach

Use persistent homology on a 1-skeleton (vertex and edge) filtration to select H0-death edges (MST) and H1-birth edges (cycle-closing non-MST edges), then align their directions across modalities via cross-modal pairing maps using cosine-based direction consistency, added to the SemiCLIP objectives.

Key results

• On remote sensing zero-shot classification (L=U), ToMA reaches 97.9% on WHU-RS19 vs 95.6% for retrained SemiCLIP and 84.8% on RSICD-CLS vs 81.4%.
• Under L!=U, ToMA improves SemiCLIP on all remote sensing zero-shot datasets, with ToMA-domain reaching 87.5% on UCM-CLS, a 4.7% gain over SemiCLIP.
• On fashion zero-shot classification, ToMA-domain obtains 49.1% super-class and 46.5% sub-class on FashionGen, vs 43.4% and 43.0% for SemiCLIP; gains on fashion retrieval are modest (e.g., 23.5 vs 23.3 I2T R@5 on Fashion200k).

Surprising findings

• Persistence-image and distance-based topology objectives often degrade or give less stable gains than the proposed edge-direction alignment.
• Birth-only H1 edges alone (without H0) already improve over SemiCLIP, and the H1-only variant is informative despite not constructing 2-simplices.

Limitations

• Evaluation limited to specialized domains (remote sensing, fashion, plus limited SciCap/Simpsons retrieval) and two tasks.
• Gains on retrieval are less consistent than on zero-shot classification; H0+H1 advantage varies across tasks.
• Restricted to a 1-skeleton lightweight topological construction without higher-order simplices.

Experimental setup

Datasets: RSICD UCM Sydney RESISC45 WHU-RS19 RSSCN7 AID Fashion200k FashionGen Polyvore Outfits SciCap Simpsons

Benchmarks: RSICD-CLS UCM-CLS WHU-RS19 RSSCN7 AID RSICD retrieval UCM retrieval Sydney retrieval Fashion200k FashionGen Polyvore Outfits SciCap

Metrics: Top-1 accuracy Recall@5

Baselines: CLIP (original) CLIP (fine-tuned) CLIP (oracle) Hard-PL Soft-PL S-CLIP SemiCLIP HC ToMCLIP TopKD Topological Autoencoders

Builds on: SemiCLIP S-CLIP CLIP ToMCLIP Topological Autoencoders

Closest papers by meaning

• Homology Consistency Constrained Efficient Tuning for Vision-Language Models (NeurIPS 2024 · similarity 0.91)
• Mind the Heads: Topological Representation Alignment for Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.89)
• DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.88)
• Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models (NeurIPS 2026 · similarity 0.88)