Research map / Vision-language and multimodal / CLIP and zero-shot

Prompt-based Visual Alignment for Zero-shot Policy Transfer

ICML 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PVA, which uses CLIP-based prompt tuning (global, domain-specific, instance-conditional tokens) to supervise a UNet visual aligner that maps all CARLA weather domains to one unified domain, enabling zero-shot PPO policy transfer from limited multi-domain data.

Problem

Zero-shot policy transfer in visual RL: agents overfit to training domains and degrade on unseen visual domains, and existing representation and translation methods need abundant multi-domain data and lack explicit semantic constraints.

Approach

Learn prompt tokens (global, domain-specific, instance-conditional) with a frozen CLIP so that the prompt describes each image's semantics, then train a UNet visual aligner whose output image matches the prompt for a target domain (global CLIP match, patch CLIP match with threshold rejection, and VGG feature loss), so the RL policy sees a unified domain.

Key results

• On CARLA, PVA reaches 2178.4 on unseen ClearSunset and 1775.65 on unseen WetCloudySunset, versus 2079.0 for LUSR on ClearSunset and 582.8 for LUSR on WetCloudySunset.
• With the same 100 images, PVA scores 2004.6 on ClearNoon and 1825.9 on ClearSunset, versus CURL 1227.4 / 199.5 and AdaIN 1420.8 / 313.9.
• Removing the feature (VGG) loss drops HardRainNoon return from 1825.9 to 147.7, and removing the global loss drops ClearSunset return from 2178.4 to 993.8.

Measured effects reported in the paper

Surprising findings

• The paper reports that the domain-specific prompt has the largest impact among prompt components, and LUSR scores higher than PVA on ClearSunset in the 10000-image comparison? No, PVA is higher; but in the prompt-length study, lengthening the domain-specific prompt beyond 5 degrades performance.
• Fixed-template prompts yield very low returns on some seen domains (e.g., 118.63 on HardRainNoon), showing sensitivity of the aligner to the prompt design.

Limitations

• Evaluation is limited to one simulated driving task (CARLA) with a few weather domains, plus a small MuJoCo appendix experiment.
• Single-seed-style tables with no variance or confidence intervals are reported, and several tables contain inconsistent or repeated numbers.
• Method depends on hand-chosen hyperparameters (prompt lengths, temperatures, learning rates) and on a pretrained CLIP whose domain coverage is not analyzed.

Experimental setup

Datasets: CARLA simulator weather domains (ClearNoon, HardRainNoon; unseen ClearSunset, WetCloudySunset, SoftRainSunset) MuJoCo background-altered tasks (cartpole-balance, cheetah-run, point-mass)

Benchmarks: CARLA autonomous driving weather-domain generalization MuJoCo cartpole-balance, cheetah-run, point-mass with altered backgrounds

Metrics: episode return

Baselines: LUSR DARLA CURL UNIT4RL CycleGAN pix2pix AdaIN

Builds on: CoOp CoCoOp PPO LUSR DARLA

Closest papers by meaning

• Prompt-Based Distribution Alignment for Unsupervised Domain Adaptation (AAAI 2024 · similarity 0.88)
• VLMs-Guided Representation Distillation for Efficient Vision-Based Reinforcement Learning (CVPR 2025 · similarity 0.87)
• Fourier Guided Adaptive Adversarial Augmentation for Generalization in Visual Reinforcement Learning (AAAI 2025 · similarity 0.87)
• Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.87)
• Enhancing Domain Adaptation through Prompt Gradient Alignment (NeurIPS 2024 · similarity 0.86)
• SiT: Symmetry-invariant Transformers for Generalisation in Reinforcement Learning (ICML 2024 · similarity 0.86)