Research map / Vision-language and multimodal / Human and embodied modeling
Distilling CLIP with Dual Guidance for Learning Discriminative Human Body Shape Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a frozen CLIP ViT-L/14 into a ResNet-50 person ReID student using optimal-transport alignment of local features to CLIP body-shape text tokens, MSE global feature distillation, and SMPL-based 3D shape regularization on synthetic meshes.
Problem
Appearance-based person re-identification fails in long-term, cloth-changing scenarios, and 3D-shape-based ReID is limited by the scarcity of real images with ground-truth 3D body shapes.
Approach
Use CLIP to automatically label images with paired linguistic body-shape descriptors, then distill CLIP both via optimal-transport alignment of local patch features to shape-aware text tokens and via MSE alignment of global features to the CLIP image encoder, plus a 3D SMPL regularizer trained on synthetic body meshes without clothing or texture.
Key results
- On Celeb-reID-light, CLIP3DReID reaches 26.3 mAP versus 21.8 for the best baseline (3DInvarReID).
- On LTCC, mAP improves to 21.7 from 19.1 (AIM) and Rank-1 to 42.1 from 40.9 (3DInvarReID).
- Ablation: removing the OT loss drops Celeb-reID mAP from 19.2 to 17.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add synthetic mesh images during training (ablation) | CLIP3DReID · Celeb-reID | mAP: 17.9 -> 19.2 | Helped |
| add local visual-text alignment loss (LOT) (ablation) | CLIP3DReID · Celeb-reID | mAP: 17.0 -> 19.2 | Helped |
| add global feature alignment loss (Lglobal) (ablation) | CLIP3DReID · Celeb-reID | mAP: 16.5 -> 19.2 | Helped |
| add 3D reconstruction regularization (L3D-Regu) (ablation) | CLIP3DReID · Celeb-reID | mAP: 18.5 -> 19.2 | Helped |
| add LOT, Lglobal, and L3D-Regu (ablation) | CLIP3DReID · Celeb-reID | mAP: 14.2 -> 19.2 | Helped |
| replace CLIP ViT-L/14 teacher with ViT-B/32 (ablation) | CLIP3DReID · Celeb-reID | mAP: 19.2 -> 15.2 | Hurt |
| replace CLIP ViT-L/14 teacher with ViT-B/16 (ablation) | CLIP3DReID · Celeb-reID | mAP: 19.2 -> 17.7 | Hurt |
| use CLIP3DReID instead of 3DInvarReID | person ReID · Celeb-reID-light | mAP: 21.8 -> 26.3 | Helped |
| use CLIP3DReID instead of 3DInvarReID | person ReID · CCDA | mAP: 21.7 -> 25.7 | Helped |
Surprising findings
- Synthetic-only 3D regularization pipeline achieves CD-L2 comparable to baselines on HBW without any real 3D body shape ground truth.
Limitations
- Limited gains on LTCC and PRCC attributed to low image resolution, which weakens CLIP's semantic labeling.
- Relies on the fixed set of 16 hand-designed phrase pairs and CLIP-based labeling of images.
- Gains on short-term benchmarks are modest, as the method is tailored to long-term scenarios.
Experimental setup
Datasets: Celeb-reID Celeb-reID-light PRCC LTCC CCVID DeepChange CCDA Market-1501 MSMT17 HBW SMPL synthetic meshes
Benchmarks: Celeb-reID Celeb-reID-light PRCC LTCC CCVID DeepChange CCDA Market-1501 MSMT17 HBW
Metrics: mAP Rank-1 CD-L2
Baselines: ReIDCaps RCSAnet CASE-Net CAL 3DInvarReID 3DSL FSAM AIM SHAPY
Builds on: CLIP 3DInvarReID BodyTalk Sinkhorn distance
Closest papers by meaning
- Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructions (CVPR 2024 · similarity 0.88)
- CLIMB-ReID: A Hybrid CLIP-Mamba Framework for Person Re-Identification (AAAI 2025 · similarity 0.87)
- SEAS: ShapE-Aligned Supervision for Person Re-Identification (CVPR 2024 · similarity 0.86)
- TF-CLIP: Learning Text-Free CLIP for Video-Based Person Re-identification (AAAI 2024 · similarity 0.86)
- Exploring the Camera Bias of Person Re-identification (ICLR 2025 · similarity 0.86)
- From Poses to Identity: Training-Free Person Re-Identification via Feature Centralization (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.