Research map / Label-efficient and robust learning / Test-time adaptation
Test-Time Personalization with Meta Prompt for Gaze Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TPGaze, which personalizes a gaze estimator at test time by tuning a small padding-based prompt (under 1% of ResNet-18 parameters) with a symmetry loss, after meta-learning the prompt initialization on source data.
Problem
Personalizing appearance-based gaze estimation to a new person at test time without gaze labels or source data, efficiently enough for edge devices, while ensuring the unsupervised loss actually reduces gaze error.
Approach
Replace convolutional zero/reflect padding with tunable prompt parameters, freeze the rest of the network, and meta-initialize the prompt (MAML-style meta-auxiliary learning) so that adapting it with a left-right symmetry loss on a few unlabeled images reduces the supervised gaze error.
Key results
- On four cross-dataset tasks (ETH-XGaze/Gaze360 to MPIIGaze/EyeDiap), TPGaze achieves 6.30, 5.89, 6.62 and 5.04 degrees gaze error, the best among compared methods including RUDA (6.86, 6.84, 6.96, 5.32).
- Adapts with only 0.125M tunable parameters versus 12.20M for RUDA and 116.9M for PnP-GA.
- Per-iteration adaptation time including loss is 0.029s versus 0.287s for RUDA, about 10x faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TPGaze instead of source-only baseline | ResNet-18 gaze estimation · four cross-dataset validations | gaze error (degree): 8.02 -> 6.30; 7.30 -> 5.89; 7.79 -> 6.62; 8.19 -> 5.04 | Helped |
| use TPGaze instead of RUDA* | ResNet-18 gaze estimation · four cross-dataset validations | gaze error (degree): 6.86 -> 6.30; 6.84 -> 5.89; 6.96 -> 6.62; 5.32 -> 5.04 | Helped |
| use prompt tuning rather than updating all network parameters (ablation) | TPGaze · four cross-dataset validations | gaze error (degree): 6.47 -> 6.30; 6.10 -> 5.89; 6.71 -> 6.62; 5.58 -> 5.04 | Helped |
| add meta-initialization to the prompt (ablation) | TPGaze · four cross-dataset validations | gaze error (degree): 6.59 -> 6.30; 6.18 -> 5.89; 6.68 -> 6.62; 5.76 -> 5.04 | Helped |
| increase prompted layers from 0 to 9 convolutional layers (ablation) | TPGaze · four cross-dataset validations | gaze error (degree): 8.02 -> 6.30; 7.30 -> 5.89; 7.79 -> 6.62; 8.19 -> 5.04 | Helped |
| increase prompted layers from 9 to 13 convolutional layers (ablation) | TPGaze · four cross-dataset validations | gaze error (degree): 6.30 -> 6.05; 5.89 -> 5.63; 6.62 -> 6.96; 5.04 -> 5.45 | Mixed |
| increase prompted layers from 9 to 17 convolutional layers (ablation) | TPGaze · four cross-dataset validations | gaze error (degree): 6.30 -> 12.45; 5.89 -> 14.23; 6.62 -> 22.67; 5.04 -> 27.34 | Hurt |
| increase personalization images from 1 to 15 (ablation) | TPGaze · four cross-dataset validations | average gaze error (degree): 6.06 -> 5.74 | Helped |
Surprising findings
- Prompt tuning outperforms updating all network parameters for personalization (e.g., 5.04 vs 5.58 on G->D).
- Adding prompts to all 17 convolutional layers causes a catastrophic performance drop (gaze error above 12 degrees).
Limitations
- Compared SF-UDA methods (PnP-GA, RUDA) were re-implemented under the personalization setting, and PnP-GA's pretrained ensemble was unavailable, so 10 pre-training epochs were used instead.
- Performance degrades sharply when prompts are placed in the deepest layers, so the prompt placement must be tuned by hand (9 layers chosen).
Experimental setup
Datasets: ETH-XGaze Gaze360 MPIIGaze EyeDiap
Benchmarks: MPIIGaze EyeDiap
Metrics: gaze error (degrees) tunable parameters adaptation time per iteration
Baselines: DAGEN GazeAdv Gaze360 PnP-GA RUDA Update All No Meta
Builds on: MAML meta-auxiliary learning visual prompt tuning RUDA PnP-GA
Closest papers by meaning
- CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model (AAAI 2024 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.86)
- Gaze from Origin: Learning for Generalized Gaze Estimation by Embedding the Gaze Frontalization Process (AAAI 2024 · similarity 0.86)
- A Pedestrian is Worth One Prompt: Towards Language Guidance Person Re-Identification (CVPR 2024 · similarity 0.85)
- Bootstrap AutoEncoders With Contrastive Paradigm for Self-supervised Gaze Estimation (ICML 2024 · similarity 0.85)
- Prompt Learning via Meta-Regularization (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.