Research map / Architectures and efficiency / Attention and transformers
Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UFC, a Transformer that jointly performs feature and cost aggregation via interleaved integrative self- and cross-attention in a coarse-to-fine design, with dense zoom-in inference, for semantic and geometric dense matching.
Problem
Dense visual correspondence suffers because feature aggregation and cost aggregation are used separately, and cost volumes lack semantic context while features lack matching-distribution coherence.
Approach
Jointly aggregate concatenated features and cost volume with self-attention (separate value embeddings for features and cost), then apply cross-attention whose attention map is the convolved, sharpened cost volume, interleaved coarse-to-fine, and select per-pixel flows across multi-scale zoom-in predictions by confidence.
Key results
- On SPair-71k, UFC reaches 8.40 PCK@0.01 and 64.4 PCK@0.10 ... reported as outperforming all prior methods at all PCK thresholds, with 88.0 at 0.05 on PF-PASCAL-side results.
- On HPatches, UFC + (C) achieves 1.73 average AEPE versus 2.52 for ECO-TR, and UFC achieves 7.88 average AEPE in the dense setting.
- Component ablation on HPatches AEPE drops from 36.8 (baseline) to 7.88 with integrative self- and cross-attention, matching distribution, hierarchical processing and dense zoom-in.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate feature and cost self-attention instead of using feature self-attention alone (ablation) | UFC aggregation · SPair-71k | PCK @ α_bbox 0.1: 36.1 -> 51.7 | Helped |
| Integrate feature and cost self-attention instead of using feature self-attention alone (ablation) | UFC aggregation · HPatches | AEPE: 78.2 -> 36.8 | Helped |
| Add feature cross-attention to feature self-attention (ablation) | Feature aggregation · SPair-71k | PCK @ α_bbox 0.1: 36.1 -> 38.5 | Helped |
| Add feature cross-attention to feature self-attention (ablation) | Feature aggregation · HPatches | AEPE: 78.2 -> 59.9 | Helped |
| Add feature cross-attention to joint feature and cost self-attention (ablation) | UFC integrative aggregation · SPair-71k | PCK @ α_bbox 0.1: 51.7 -> 56.5 | Helped |
| Add feature cross-attention to joint feature and cost self-attention (ablation) | UFC integrative aggregation · HPatches | AEPE: 36.8 -> 26.2 | Helped |
| Add integrative self-attention to the feature self- and cross-attention baseline (ablation) | UFC · SPair-71k | PCK @ α_bbox 0.1: 51.7 -> 54.7 | Helped |
| Add integrative self-attention to the feature self- and cross-attention baseline (ablation) | UFC · HPatches | AEPE: 36.8 -> 32.8 | Helped |
| Add integrative cross-attention (ablation) | UFC · SPair-71k | PCK @ α_bbox 0.1: 54.7 -> 58.4 | Helped |
| Add the aggregated cost volume as a matching-distribution cross-attention map (ablation) | UFC · SPair-71k | PCK @ α_bbox 0.1: 58.4 -> 59.9 | Helped |
| Add hierarchical processing (ablation) | UFC · SPair-71k | PCK @ α_bbox 0.1: 59.9 -> 64.4 | Helped |
| Add dense zoom-in at inference (ablation) | UFC · HPatches | AEPE: 10.9 -> 7.88 | Helped |
Surprising findings
- Naive sequential feature-then-cost aggregation performs worse than integrative aggregation, though both beat either alone.
- Existing COTR-style sparse or quasi-dense evaluation inflates reported results compared with conventional dense evaluation, per the authors' argument.
Limitations
- Does not explicitly model matchability or uncertainty, so occlusions are handled poorly.
- Global correlation cost volumes are costly at high resolution; dense zoom-in increases memory and run time with larger k.
Experimental setup
Datasets: SPair-71k PF-PASCAL PF-WILLOW HPatches ETH3D DPED-CityScape-ADE MegaDepth COCO Sintel
Benchmarks: SPair-71k PF-PASCAL PF-WILLOW HPatches ETH3D
Metrics: PCK AEPE
Baselines: CATs CATs++ VAT NeMF SCorrSAN TransforMatcher COTR ECO-TR PUMP GLU-Net-GOCor PDC-Net+ GMFlow
Builds on: CATs++ GMFlow COTR GOCor RANSAC-Flow
Closest papers by meaning
- CATs: Cost Aggregation Transformers for Visual Correspondence (NeurIPS 2021 · similarity 0.94)
- Deep Stereo Matching With Explicit Cost Aggregation Sub-Architecture (AAAI 2018 · similarity 0.88)
- Deep Semantic Feature Matching (CVPR 2017 · similarity 0.87)
- MATCHA: Towards Matching Anything (CVPR 2025 · similarity 0.87)
- OmniGlue: Generalizable Feature Matching with Foundation Model Guidance (CVPR 2024 · similarity 0.86)
- ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.