Research map / Vision-language and multimodal / Cross-modal learning
Flexible Biometrics Recognition: Bridging the Multimodality Gap through Attention Alignment and Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Flexible Biometric Recognition (FBR) using a Multimodal Fusion Attention ViT (MFA-ViT) with Multimodal Prompt Tuning (MPT) to fuse face, periocular and soft-biometric attributes for intra- and cross-modality face-periocular recognition.
Problem
Face and periocular recognition each suffer under occlusion, and existing multimodal or cross-modality methods trade off intra-modality against cross-modality matching while requiring all modalities at deployment.
Approach
A shared-parameter ViT with depth-wise convolutional multimodal fusion attention layers and a multimodal prompt tuning mechanism that jointly embeds face, periocular and soft-biometric tokens, producing a modality-invariant embedding usable from either face or periocular input alone.
Key results
- MFA-ViT/MPT reaches 94.82% f-f and 89.98% p-p rank-1 on Ethnic, outperforming the best competitor ViT/VPT with soft-biometrics (92.95% and 85.68%).
- On Cross-Modal DB, MFA-ViT/MPT achieves 72.01% f-p and 75.96% p-f, versus 65.99% and 69.11% for ViT/VPT with soft-biometrics.
- Using the joint multimodal prompt embeddings (PRM) instead of the CLS token as classification head input improves rank-1 accuracy across all four datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MFA-ViT/MPT instead of ViT/VPT with soft-biometric attributes | FBR · Ethnic, FaceScrub, IMDB, and Cross-Modal DB; intra- and cross-modality recognition | rank-1 accuracy: 92.95 -> 94.82; 85.68 -> 89.98; 83.37 -> 86.70; 86.10 -> 89.07 | Helped |
| Remove the soft-biometric attributes (ablation) | MFA-ViT · Ethnic, FaceScrub, IMDB, and Cross-Modal DB; intra- and cross-modality recognition | rank-1 accuracy | Hurt |
| Add multimodal-prompt tuning (MPT) (ablation) | MFA-ViT · Ethnic, FaceScrub, IMDB, and Cross-Modal DB; intra- and cross-modality recognition | rank-1 accuracy | Helped |
| Replace MPT with visual prompt tuning (VPT) (ablation) | MFA-ViT · Ethnic, FaceScrub, IMDB, and Cross-Modal DB; intra- and cross-modality recognition | rank-1 accuracy | Hurt |
| Use joint prompt embeddings (PRM) rather than the class token (CLS) as the classification-head input (ablation) | MFA-ViT/MPT · Ethnic, FaceScrub, IMDB, and Cross-Modal DB; intra- and cross-modality recognition | rank-1 accuracy: 94.57 -> 94.82; 89.18 -> 89.98; 86.24 -> 86.70; 88.34 -> 89.07 | Helped |
Surprising findings
- Baseline models trained independently on face or periocular score below 1% in cross-modality matching, showing the distinct nature of the two modalities despite periocular being a face subset.
Limitations
- Evaluated only on face and periocular modalities; extension to other biometrics is left to future work.
- Cross-modality accuracy (around 72-76% on Cross-Modal DB) remains well below intra-modality accuracy.
Experimental setup
Datasets: VGGFace2 MAAD-Face Ethnic FaceScrub IMDB Cross-Modal DB
Benchmarks: Ethnic FaceScrub IMDB Cross-Modal DB
Metrics: rank-1 recognition accuracy
Baselines: Baseline (unimodal face/periocular) Tiong et al. [32] George et al. [5] Ng et al. [22] ViT/VPT HA-ViT
Builds on: Visual Prompt Tuning HA-ViT Large-margin softmax loss Feature tokenizer
Closest papers by meaning
- mmFAS: Multimodal Face Anti-Spoofing Using Multi-Level Alignment and Switch-Attention Fusion (AAAI 2025 · similarity 0.83)
- SapiensID 2.0: Aligning Human Recognition Foundation Models with Human Perception (NeurIPS 2026 · similarity 0.82)
- TokenMatcher: Diverse Tokens Matching for Unsupervised Visible-Infrared Person Re-Identification (AAAI 2025 · similarity 0.81)
- FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification (ICML 2025 · similarity 0.81)
- SapiensID: Foundation for Human Recognition (CVPR 2025 · similarity 0.81)
- X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing (ICLR 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.