Research map / Detection and segmentation / CNN features
Object Recognition as Next Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames object recognition as next-token prediction with a LLaMA-based language decoder using a non-causal mask, one-shot parallel sampling, and a truncated 6-block decoder.
Problem
Object recognition with fixed label galleries (linear classifiers, CLIP) limits the set of recognizable objects and generative VLMs need predefined prompts or exemplars; the paper asks whether labels can be decoded directly from image embeddings without predefined vocabularies.
Approach
A language decoder auto-regressively decodes label tokens from image embeddings, with a non-causal attention mask that makes tokens of different labels independent and treats image tokens as a prefix; this enables one-shot parallel sampling of multiple labels ranked by probability, and a compact decoder built by keeping only the first 6 LLaMA blocks plus the output layer.
Key results
- On CC3M, COCO, and OpenImages with top-10 predictions, the 70M-trained 1.78B model reaches F1 0.603 on OpenImages and 0.758 on COCO, outperforming the listed baselines on recall across datasets.
- The truncated 1.78B decoder (6 of 32 LLaMA blocks) matches the full 7B-decoder performance (F1 0.570 vs 0.569 on OpenImages/CC3M top-5 range) while being 4.5x faster in inference.
- Combining one-shot sampling with the truncated decoder gives an 18.1x speed-up over the full model with greedy sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on G70M instead of G3M | Object Recognition as Next Token Prediction · COCO Captions | F1: 0.702 -> 0.758 | Helped |
| truncate LLaMA 7B to 6 decoder blocks instead of retaining 32 (ablation) | Object Recognition as Next Token Prediction · CC3M | F1: 0.632 -> 0.627 | No clear effect |
| truncate LLaMA 7B to 1 decoder block instead of retaining 32 (ablation) | Object Recognition as Next Token Prediction · CC3M | F1: 0.632 -> 0.620 | Hurt |
| use truncated LLaMA 7B instead of LLaMA 3B at similar model scale (ablation) | Object Recognition as Next Token Prediction · CC3M | F1: 0.599 -> 0.615 | Helped |
| use one-shot sampling instead of greedy search (ablation) | Object Recognition as Next Token Prediction · CC3M | F1: 0.624 -> 0.627 | Helped |
| use one-shot sampling instead of beam search (ablation) | Object Recognition as Next Token Prediction · CC3M | F1: 0.608 -> 0.627 | Helped |
| add image-prefix masking (ablation) | Object Recognition as Next Token Prediction with greedy search · CC3M | F1: 0.611 -> 0.613 | Helped |
| add image-prefix masking (ablation) | Object Recognition as Next Token Prediction with one-shot sampling · COCO Captions | F1: 0.687 -> 0.689 | Helped |
| model labels independently instead of conditionally across labels (ablation) | Object Recognition as Next Token Prediction with greedy search · CC3M | F1: 0.613 -> 0.625 | Helped |
| model labels independently instead of conditionally across labels (ablation) | Object Recognition as Next Token Prediction with one-shot sampling · COCO Captions | F1: 0.689 -> 0.698 | Helped |
| use LLaMA 2 instead of LLaMA 1 for the truncated decoder (ablation) | Object Recognition as Next Token Prediction · CC3M | F1: 0.627 -> 0.627 | No clear effect |
| rank labels by full-label probabilities instead of initial-token probabilities (ablation) | Object Recognition as Next Token Prediction · CC3M | F1: 0.627 -> 0.627 | No clear effect |
Surprising findings
- Truncating LLaMA 7B to its first 11 blocks (3B) or 6 blocks (1.78B) keeps performance nearly unchanged, and the truncated decoder outperforms a natively smaller LLaMA 3B at similar scale.
- Greedy and beam search suffer from repetition issues in label generation, which the non-causal one-shot method avoids.
Limitations
- One-shot sampling can suffer from a competition issue where labels sharing an initial token are omitted.
- CLIP and VQA-style baselines are evaluated with prompts and galleries that may disadvantage them; results rely on weakly supervised labels extracted from noisy captions.
- Evaluation uses a semantic similarity metric rather than standard label accuracy, and the metric is sensitive to the number of predictions.
Experimental setup
Datasets: CC3M COCO Captions SBU LAION-Synthetic-115M OpenImages V7
Benchmarks: CC3M validation COCO Captions validation OpenImages V7 validation
Metrics: semantic recall precision F1 (BERTScore-style with sentence-BERT) precision-recall curves
Baselines: CLIP CaSED Open Flamingo LLaVA 1.0 LLaVA 1.5 BLIP-2 InstructBLIP
Builds on: CLIP LLaMA BERTScore column mask
Closest papers by meaning
- Language-conditioned Detection Transformer (CVPR 2024 · similarity 0.87)
- Dual Pose-invariant Embeddings: Learning Category and Object-specific Discriminative Representations for Recognition and Retrieval (CVPR 2024 · similarity 0.86)
- Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · similarity 0.85)
- Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models (ICLR 2026 · similarity 0.85)
- Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers (ICML 2025 · similarity 0.85)
- Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.