Research map / Label-efficient and robust learning / Recognition and long-tail
KeyPoint Relative Position Encoding for Face Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KP-RPE, a keypoint-conditioned relative position encoding for ViTs that improves robustness to affine misalignment in face recognition (and gait recognition).
Problem
ViT-based recognition degrades sharply under unseen affine transformations (scale, translation, pose), which occur when face alignment fails on low-quality images; standard RPE cannot adapt its query-key offsets to image content.
Approach
Make the relative position bias table a function of image-specific keypoints: compute differences between a patch-location mesh grid and keypoints, linearly project to a keypoint-dependent bias table f(P), and index it by query location and quantized product distance, so pixel significance is anchored to keypoints rather than only to proximity.
Key results
- On unaligned CFP-FP, ViT+KP-RPE reaches 93.56% verification vs 72.81% for ViT and 77.91% for ViT+iRPE (ViT-Small ablation).
- On TinyFace Rank-1, ViT+KP-RPE (WebFace4M, ViT-Base) reaches 75.80% vs 74.81% for AdaFace ViT baseline; with WebFace12M it reaches 76.18%.
- On Gait3D, SwinGait-2D + KP-RPE improves Rank-1 from 67.1 to 68.2 and mAP from 58.76 to 60.81.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add KP-RPE (ablation) | ViT-small · IJB-C unaligned | Verification TAR@0.01%: 21.62 -> 91.85 | Helped |
| Add KP-RPE (ablation) | ViT-small · CFPFP unaligned | Verification TAR@0.01%: 72.81 -> 93.56 | Helped |
| Add KP-RPE (ablation) | ViT-small · IJB-S | Rank-1: 59.60 -> 63.44 | Helped |
| Add KP-RPE | AdaFace ViT · TinyFace | Rank-1: 74.81 -> 75.80 | Helped |
| Add KP-RPE | AdaFace ViT · IJB-S | Rank-1: 71.90 -> 72.78 | Helped |
| Add KP-RPE | AdaFace ViT · AgeDB | Verification Accuracy: 97.48 -> 97.67 | Helped |
| Add KP-RPE | AdaFace ViT · IJB-C | TAR@FAR=0.01%: 97.14 -> 97.13 | No clear effect |
| Add KP-RPE | SwinGait-2D · Gait3D | mAP: 58.76 -> 60.81 | Helped |
| Replace absolute f(P) with relative keypoint-dependent f(P) (ablation) | KP-RPE · IJB-S | Rank-1: 9.97 -> 62.51 | Helped |
| Use MultiHead f(P) instead of Relative f(P) (ablation) | KP-RPE · CFPFP unaligned | Verification TAR@0.01%: 89.70 -> 93.56 | Helped |
Surprising findings
- KP-RPE trained only on aligned data retains high performance on heavily unaligned inputs (IJB-C unaligned TAR jumps from 21.62 to 91.85 for ViT-Small).
Limitations
- Requires keypoint supervision/detection, limiting use on datasets without consistent topology.
- Keypoint detector (MobileNet RetinaFace) differs from the ResNet50 RetinaFace used to align test sets; performance depends on landmark quality.
Experimental setup
Datasets: MS1MV2 MS1MV3 WebFace4M WebFace12M Gait3D MNIST AffNIST
Benchmarks: TinyFace IJB-S CFP-FP AgeDB IJB-C Gait3D AffNIST
Metrics: Rank-1 Rank-5 verification accuracy TAR@FAR=0.01% rank-n mAP mINP
Baselines: ViT ViT + iRPE AdaFace ArcFace CurricularFace URL PFE SwinGait-2D GaitBase DyGait HSTL
Builds on: iRPE AdaFace Relative Position Encoding (Shaw et al.) Swin Transformer
Closest papers by meaning
- Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General (NeurIPS 2026 · similarity 0.85)
- KPNet: Towards Minimal Face Detector (AAAI 2020 · similarity 0.84)
- FAR: Flexible Accurate and Robust 6DoF Relative Camera Pose Estimation (CVPR 2024 · similarity 0.83)
- From Correspondences to Pose: Non-minimal Certifiably Optimal Relative Pose without Disambiguation (CVPR 2024 · similarity 0.82)
- Cameras as Relative Positional Encoding (NeurIPS 2025 · similarity 0.82)
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.