Research map / 3D vision / Face and shape
Looking 3D: Anomaly Detection with 2D-3D Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces conditional anomaly detection that checks a query chair image against a reference 3D shape, releases the BrokenChairs-180K synthetic dataset, and proposes CMT, a transformer with top-k sparse cross-attention and view-agnostic feature alignment.
Problem
Detecting and localizing geometric anomalies in an object photo when the normal reference is given as a 3D shape, where the query viewpoint is unknown and there is a domain gap between textured images and untextured shapes.
Approach
Render the reference shape from multiple views, add 3D positional encodings, and use a correspondence-guided transformer whose cross-attention is restricted to the top-k most similar multi-view patches, with a self-supervised pseudo-correspondence contrastive loss to learn view-agnostic local features.
Key results
- CMT reaches 84.7 AUC and 75.4% accuracy on BrokenChairs-180K, versus 74.6 AUC and 64.7% accuracy for the query-only ResNet18-FPN baseline.
- Ablation shows view-view alignment loss raises AUC from 78.5 to 81.6, and the full combination of both alignment losses reaches 84.7.
- Human participants scored 70.6% accuracy on the task, while CMT scored 74.8% on the same subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the reference 3D shape and correspondence matching | CMT on BrokenChairs-180K test · BrokenChairs-180K test | accuracy: 64.7 -> 75.4 | Helped |
| Add view-agnostic local feature alignment (VLFA) (ablation) | CMT on BrokenChairs-180K test · BrokenChairs-180K test | accuracy: 67.1 -> 72.3 | Helped |
| Add correspondence-guided attention (CGA) (ablation) | CMT on BrokenChairs-180K test · BrokenChairs-180K test | accuracy: 72.3 -> 73.7 | Helped |
| Add 3D positional encoding (3DPE) (ablation) | CMT on BrokenChairs-180K test · BrokenChairs-180K test | accuracy: 73.7 -> 75.4 | Helped |
| Use query-view alignment loss only instead of no alignment loss (ablation) | CMT on BrokenChairs-180K test · BrokenChairs-180K test | AUC: 78.5 -> 78.6 | No clear effect |
| Add view-view alignment loss (ablation) | CMT on BrokenChairs-180K test · BrokenChairs-180K test | AUC: 78.5 -> 81.6 | Helped |
| Combine query-view and view-view alignment losses (ablation) | CMT on BrokenChairs-180K test · BrokenChairs-180K test | AUC: 81.6 -> 84.7 | Helped |
| Use top-k sparse attention with k=100 rather than using all available tokens (ablation) | CMT on BrokenChairs-180K test · BrokenChairs-180K test | AUC: 84.7 | Helped |
| Jointly train classification and bounding-box localization (ablation) | CMT on BrokenChairs-180K test · BrokenChairs-180K test | AUC: 84.7 -> 85.9 (+1.2) | Helped |
| Use CMT rather than human participants for anomaly classification | Perceptual study subset · Perceptual study subset | accuracy: 70.6% -> 74.8% | Helped |
Surprising findings
- Standard cross-attention to multi-view images barely improves over the query-only baseline (about 1.6% accuracy gain), indicating the model fails to exploit the reference shape without correspondence guidance.
Limitations
- Dataset is synthetic and limited to the chair category with only one anomaly per query image.
- Model assumes rigid objects and needs an accurate reference 3D shape.
- Only shape anomalies are covered; appearance anomalies such as fading or discoloration are excluded.
Experimental setup
Datasets: BrokenChairs-180K PartNet PhotoShape SVBRDF materials
Benchmarks: BrokenChairs-180K
Metrics: AUC accuracy average precision viewpoint prediction accuracy
Baselines: ResNet18-FPN ResNet18-FPN w/ SA blocks Vision Transformer (ViT) LFD Lin et al. ICCV'21 standard cross-attention CMT variants
Builds on: Vision Transformer Feature Pyramid Network generalized IoU loss PhotoShape
Closest papers by meaning
- Towards Generalizable 3D Anomaly Detection via Relational Inconsistency Modeling (NeurIPS 2026 · similarity 0.88)
- Odd-One-Out: Anomaly Detection by Comparing with Neighbors (CVPR 2025 · similarity 0.87)
- Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection (NeurIPS 2026 · similarity 0.87)
- Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network (CVPR 2024 · similarity 0.87)
- PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection (NeurIPS 2024 · similarity 0.87)
- Registration is a Powerful Rotation-Invariance Learner for 3D Anomaly Detection (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.