Research map / Vision-language and multimodal / Human and embodied modeling
Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses Joint Energy-Based Models with a single mixing coefficient alpha to interpolate between discriminative and generative training at fixed architecture, and finds human alignment across six benchmarks peaks at intermediate alpha.
Problem
Whether human-like visual representations are better explained by discriminative or generative learning is unresolved because prior comparisons confound the objective with architecture, scale, and training data.
Approach
Vary only the generative-discriminative mixing coefficient alpha in a Joint Energy-Based Model with fixed Wide ResNet architecture and data, then evaluate on six human-alignment benchmarks; the best-aligned models lie at intermediate alpha rather than at either endpoint.
Key results
- On BAPPS, the hybrid JEM at alpha=0.6 nearly matches the best discriminative baseline on JND (57.5% vs 58.4%) while achieving the highest 2AFC score.
- On shape-texture cue conflict, shape bias increases monotonically with alpha, with the best hybrid JEMs reaching shape bias close to 0.7, above AlexNet and ResNet-50.
- On ClickMe diagnostic feature alignment, the best alignment is at alpha=0.3, with both endpoints performing worse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Interpolate JEM objective from discriminative (alpha=0) to generative (alpha=1) with architecture fixed | JEM (WRN) · six human-alignment benchmarks | human alignment | Mixed |
| Add a small generative term (hybrid alpha=0.6 vs pure discriminative) | JEM · BAPPS JND | JND mAP: 57.5% vs 58.4% for best discriminative baseline | Helped |
| Hybrid objective (alpha=0.6) on BAPPS 2AFC | JEM · BAPPS 2AFC | 2AFC agreement | Helped |
| Add a small generative contribution (alpha=0.5) to discriminative JEM | JEM · Gloss perception | Pearson correlation with human judgments | Helped |
| Shift JEM toward generative objective (alpha up to 0.8) for CIFAR-10H response uncertainty | JEM · CIFAR-10H | cross-entropy with human distributions | Helped |
| Add generative term with alpha=0.3 (diagnostic feature attribution) | JEM · ClickMe | normalized Spearman correlation | Helped |
| Increase alpha (generative weight) for shape bias | JEM · Shape-texture cue conflict | shape bias: best hybrid JEMs close to 0.7 | Helped |
| Apply SGLD refinement at test time (ablation) | JEM with alpha>=0.3 · Shape-texture cue conflict | shape bias | Helped |
| Use frozen pretrained VAE features without JEM training (ablation) | VAE baseline · ImageNet-based benchmarks | alignment | Hurt |
| Increase generative fraction beyond hybrid regime (alpha near 1) | JEM · ClickMe | normalized Spearman correlation | Hurt |
| Gradient-norm rescaling of generative term | JEM training · all benchmarks | training stability | No clear effect |
Surprising findings
- The purely discriminative JEM (alpha=0) already outperforms the purely generative JEM on gloss, and the best hybrid surpasses the original PixelVAE result, suggesting the earlier generative advantage was due to architecture confounds.
- Optimal alpha varies by benchmark (about 0.3 for ClickMe to about 0.8 for CIFAR-10H uncertainty).
Limitations
- Energy-based training is computationally demanding and needs careful stabilization.
- Results use small JEM models (WRN) rather than modern architectures or large-scale generative classifiers.
- Optimal alpha varies across benchmarks, so alignment is not a single scalar property; post-hoc interpretations of the shift are untested.
Experimental setup
Datasets: CIFAR-10 CIFAR-10H ImageNet BAPPS gloss perception dataset (Storrs et al.) ClickMe texture-shape cue-conflict benchmark Model-vs-Human benchmark suite
Benchmarks: BAPPS 2AFC BAPPS JND gloss perception CIFAR-10H Model-vs-Human OOD generalization shape-texture cue conflict ClickMe
Metrics: 2AFC accuracy (%) JND mAP Pearson correlation cross-entropy to human distributions Cohen's kappa error consistency shape bias normalized Spearman correlation
Baselines: AlexNet SqueezeNet PixelVAE ResNet-18 BiGAN Imagen Stable Diffusion
Builds on: Joint Energy-Based Models (JEM) Wide ResNet contrastive divergence Stochastic Gradient Langevin Dynamics LPIPS
Closest papers by meaning
- When does perceptual alignment benefit vision representations? (NeurIPS 2024 · similarity 0.90)
- Evaluating alignment between humans and neural network representations in image-based learning tasks (NeurIPS 2024 · similarity 0.89)
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.88)
- LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.88)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.88)
- Attention Alignment Between Humans and Vision-Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.