Research map / Vision-language and multimodal / CLIP and zero-shot
Perception Encoder: The best visual embeddings are not at the output of the network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Perception Encoder (PE), a contrastively pretrained image-video encoder family whose intermediate layers carry general features, and aligns them to the output via language and spatial alignment tuning.
Problem
Vision encoders are usually specialized by pretraining objective (contrastive, captioning, or self-supervised); the paper asks whether one scalable contrastive recipe can yield state-of-the-art features for zero-shot tasks and downstream perception tasks.
Approach
A robust image contrastive recipe (progressive resolution, large batch, LAMB, RoPE, heavy augmentation, mask regularization) plus a synthetic video data engine yields a unified image-video encoder; its intermediate layers hold general features, surfaced at the output by language alignment (LLM tuning) and spatial alignment (self-distillation plus SAM 2 correspondence distillation).
Key results
- PE Core G reaches 86.6 average zero-shot image classification and 83.7 average zero-shot video classification, outperforming SigLIP2-g-opt and InternVideo2 on most benchmarks.
- Intermediate layers of PE Core G match AIMv2-3B on language tasks and DINOv2-g on spatial tasks; language-aligned PE Lang G reaches 82.2 average across MLLM OCR, captioning, VQA and video tasks.
- Spatially aligned PE Spatial G reportedly sets a new absolute state of the art on COCO detection with a simpler decoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add human-refined training data to PLM (ablation) | PLM video captioner · AuroraCap | Score: 2.2 -> 3.4 | Helped |
| finetune on recaptioned video data with 22M videos | PEcore G · Zero-shot video classification average | accuracy: +3.9% | Helped |
| finetune on recaptioned video data with 22M videos | PEcore G · Zero-shot video retrieval average | retrieval score: +11.1% | Helped |
| use video captions and frame captions in addition to metadata (ablation) | PE video data engine · Video data engine ablation, average image | score: 75.4 -> 78.2 | Helped |
| use video captions and frame captions in addition to metadata (ablation) | PE video data engine · Video data engine ablation, average video | score: 56.0 -> 61.6 | Helped |
| use human-refined PLM video captions instead of captions without human-refined data (ablation) | PE video data engine · Video data engine ablation, average video | score: 60.9 -> 61.6 | Mixed |
| increase LLM scale from 1B to 3B (ablation) | PE language alignment · Language alignment average | score: 76.5 -> 78.1 | Helped |
| unfreeze the LLM (ablation) | PE language alignment · Language alignment average | score: 78.1 -> 78.4 | Helped |
| replace a linear projector with a 2-layer MLP (ablation) | PE language alignment · Language alignment average | score: 77.2 -> 78.1 | Helped |
| project from layer 47 rather than the last layer (layer 50) (ablation) | PE language alignment · Language alignment average | score: 75.9 -> 78.1 | Helped |
| add LayerScale and DropPath encoder regularization (ablation) | PE language alignment · Language alignment average | score: 78.1 -> 79.9 | Helped |
| increase training from the 20M-sample ablation to 70M samples | PE language alignment · Language alignment average | score: +2.1 points to 82.2 | Helped |
Surprising findings
- Intermediate layers of a pure CLIP-trained encoder rival captioning and self-supervised models on language and spatial tasks, while the final layer degrades sharply.
- Horizontal flip augmentation, usually thought to hurt OCR, appears to help: PE Core G drops only 0.1% on flipped TextCaps retrieval.
Limitations
- Video encoding relies on simple average pooling over 8 frames rather than temporal modeling
- Final-layer features of the contrastive encoder remain weak for some tasks, requiring alignment tuning
- Video data engine depends on proprietary or large LLM components (Llama models) for caption synthesis
Experimental setup
Datasets: MetaCLIP curated image-text data PE Video Dataset (PVD) ImageNet-1k Kinetics MSR-VTT COCO Visual Genome Flickr-Entities The Cauldron Densely Captioned Images
Benchmarks: ImageNet val ImageNet v2 ObjectNet ImageNet Adversarial ImageNet Rendition ImageNet Sketch MS-COCO Flickr-30k Kinetics 400 Kinetics 600 Kinetics 700 MSR-VTT
Metrics: top-1 accuracy recall@1 CIDEr box mAP average accuracy
Baselines: SigLIP2 InternVL-C EVA 18B AIMv2-3B DINOv2-g InternVideo2 MetaCLIP CLIP
Builds on: CLIP OpenCLIP MetaCLIP Perception Language Model DINOv2
Closest papers by meaning
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.91)
- Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.91)
- PerceptionGPT: Effectively Fusing Visual Perception into LLM (CVPR 2024 · similarity 0.90)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Unifying Specialized Visual Encoders for Video Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.