Research map / Vision-language and multimodal / Multimodal LLMs

On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows discriminative classifiers outperform generative MLLM classifiers for closed-set temporal action understanding due to label-subword semantic overlap, and proposes GAD, a discriminative classifier with an auxiliary generative objective used only during training.

Problem

MLLMs used as generative classifiers for closed-set action understanding are inefficient (autoregressive decoding) and suffer from semantic overlap across action labels sharing subwords, causing confusion between similar actions.

Approach

Use a learnable [CLS] token on top of an MLLM for one-step discriminative classification, and add a generative (previous-action / task-label) objective only during fine-tuning as an auxiliary regularizer; inference uses only the discriminative head.

Key results

• GAD (Llama3-8B) reaches 67.3 step and 51.6 next-action top-1 on COIN, versus 66.4/51.0 for the discriminative baseline.
• On EPIC-Kitchens-100 the discriminative Llama3.2-1B classifier gains about 6 segment-F1 points over the generative one (23.2 vs 16.7), with roughly 1.8x faster inference.
• On Ego4DGoalStep the discriminative classifier runs about 4x faster than the generative one (53.6 vs 17.8 FPS for Llama3.2-1B).

Surprising findings

• Randomly remapping label subwords while keeping overlap unchanged does not hurt generative performance, suggesting subword semantics matter little for closed-set labels.
• Two-stage generative-then-classifier training drops performance sharply (e.g., COIN step 35.9 vs 64.1 for discriminative).

Limitations

• Discriminative models remain limited to the closed set and cannot handle unseen actions.
• Task-specific fine-tuning reduces general MLLM question-answering ability.
• Reported FPS excludes vision encoder computation.

Experimental setup

Datasets: COIN THUMOS'14 CrossTask EPIC-Kitchens-100 Ego4D GoalStep Breakfast

Benchmarks: COIN THUMOS'14 CrossTask EPIC-Kitchens-100 Ego4DGoalStep

Metrics: top-1 accuracy segment-wise F1 point-wise F1 frame accuracy FPS

Baselines: Testra MAT CMeRT VideoTaskGraph Videollm-online-8B Videollm-MoD-8B StreamMind-8B Qwen2.5-VL-7B

Builds on: LLaVA LoRA

Closest papers by meaning

• LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.88)
• Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.87)
• SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.86)
• Grounding Multimodal Large Language Models in Actions (NeurIPS 2024 · similarity 0.86)
• From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding (CVPR 2024 · similarity 0.86)
• Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.85)