Research map / Detection and segmentation / CNN features
Pixel-level Semantic Correspondence through Layout-aware Representation Learning and Multi-scale Matching Integration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LPMFlow, an end-to-end semantic correspondence pipeline combining layout-aware representation learning (with a replace-recover RPTC task), progressive feature super-resolution, and multi-scale matching flow integration, reaching SOTA on SPair-71K and PF-PASCAL/PF-WILLOW.
Problem
Establishing precise pixel-level semantic correspondence across object instances is hard due to confusing regions with similar appearance, inconsistent object scales, and indistinguishable nearby pixels.
Approach
A three-module pipeline: a transformer with region-based position embedding and a referenced patch token correction self-supervised task learns layout-consistent features; progressive feature super-resolution builds cross-scale 4D similarity tensors; soft-argmax flows from these tensors are fused and refined coarse-to-fine with swin attention to produce 2x2-pixel-unit flow.
Key results
- Reaches 65.6% PCK@0.1 (bbox) on SPair-71K, about 3.5 points above the second-best method (ACTR at 62.1%).
- Ablation: removing the PFSR module drops SPair-71K PCK@0.1 by 3.6 points (65.6 to 62.0).
- Generalization from PF-PASCAL-trained model to PF-WILLOW shows improvements, with a 1.1% gain at the stricter bbkp 0.1 metric.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LPMFlow instead of ACTR* | semantic correspondence · SPair-71K, αbbox=0.1 | PCK: 62.1 -> 65.6 | Helped |
| Use LPMFlow instead of ACTR* | semantic correspondence · PF-PASCAL, αimg=0.05 | PCK: 81.2 -> 82.4 | Helped |
| Use LPMFlow instead of ACTR* | semantic correspondence · PF-WILLOW, αbkp=0.1 | PCK: 79.9 -> 81.0 | Helped |
| Add LARL (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 63.2 -> 65.6 | Helped |
| Add PFSR (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 62.0 -> 65.6 | Helped |
| Add MMFI (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 63.9 -> 65.6 | Helped |
| Add gradual RPTC guidance (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 64.5 -> 65.6 | Helped |
| Add self-contrastive loss (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 63.9 -> 65.6 | Helped |
| Add region-based position embedding (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 64.8 -> 65.6 | Helped |
| Add interactive super-resolution (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 64.1 -> 65.6 | Helped |
| Add internal super-resolution (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 63.8 -> 65.6 | Helped |
| Use a feature super-resolution block instead of bilinear interpolation (ablation) | LPMFlow · SPair-71K, αbbox=0.1 | PCK: 63.4 -> 65.6 | Helped |
Limitations
- Heavy reliance on the [CLS] token
- Sparse keypoint annotations for supervision
- Not suited to multi-instance correspondence
Experimental setup
Datasets: SPair-71K PF-PASCAL ImageNet-1K
Benchmarks: SPair-71K PF-PASCAL PF-WILLOW
Metrics: PCK@alpha
Baselines: NC-Net SCOT DHPF CHM CATs MMNet TransMatcher ACTR
Builds on: iBOT ACTR CATs Swin Transformer InfoNCE
Closest papers by meaning
- Correspondence Transformers With Asymmetric Feature Learning and Matching Flow Super-Resolution (CVPR 2023 · similarity 0.89)
- SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representations (CVPR 2025 · similarity 0.89)
- Deep Semantic Feature Matching (CVPR 2017 · similarity 0.89)
- Bridging Viewpoint Gaps: Geometric Reasoning Boosts Semantic Correspondence (CVPR 2025 · similarity 0.85)
- Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps (CVPR 2024 · similarity 0.85)
- SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matching (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.