Research map / Large language models / AI and society
The Generative AI Paradox: “What It Can Create, It May Not Understand”
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests the "Generative AI Paradox" hypothesis that generative models' generation ability exceeds their understanding relative to humans, via selective (discriminative) and interrogative (QA about own generations) evaluations in language and vision.
Problem
Generative models produce expert-level outputs yet make basic understanding errors that non-expert humans would not; the paper asks whether generation capability is contingent on understanding in models versus humans.
Approach
Operationalizes understanding relative to generation via selective evaluation (choosing correct generations among discriminative candidates) and interrogative evaluation (answering questions about the model's own generated content), comparing models against humans with matched generation performance.
Key results
- On XSUM, GPT-4 reaches 100% accuracy choosing the correct summary among easy negatives but only 19% against hard negatives, while humans stay consistently high.
- OpenCLIP accuracy drops from 100% on easy negatives to about 69% on hard negatives in vision discrimination, versus near-perfect human accuracy.
- In interrogative vision evaluation, human QA accuracy (~95-98%) exceeds all tested models, with the best BLIP-2 (flan-t5-xxl) at about 85-91%, and Bard lowest at 56-74%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hard negatives instead of easy negatives | GPT-4 on XSUM · XSUM | discriminative accuracy: 100% -> 19% | Hurt |
| Use hard negatives instead of easy negatives | OpenCLIP · Vision tasks | discriminative accuracy: 100% to ∼69% | Hurt |
| Evaluate models on discriminative versions of generation tasks rather than generation alone | GPT-3.5 and GPT-4 versus humans · Language benchmarks | performance relative to humans | Mixed |
| Evaluate image understanding with discriminative selection | CLIP and OpenCLIP versus humans · Vision benchmarks | discriminative accuracy | Hurt |
| Ask models questions about their generated content | GPT-3.5 and GPT-4 · Language interrogative evaluation | question-answering accuracy relative to humans | Hurt |
| Ask image-understanding models questions about generated images | BLIP-2, BingChat, and Bard versus humans · TIFAv1.0 | question-answering accuracy | Hurt |
| Increase constraint complexity with arbitrary positions and strict counting requirements | GPT-4 generation · Collie-v1 | constraint satisfaction rate | Hurt |
| Ask questions about constrained generations rather than assess generation itself | GPT-4 · Collie-v1 | performance on constraints | Helped |
Surprising findings
- Workers often preferred GPT-4 generated responses over human-written ground-truth responses, citing length and writing style.
- GPT-4 made errors answering simple questions about stories it had just generated (e.g., how many times Bob sketched the house).
Limitations
- Only a small set of popular models (GPT-3.5/GPT-4, Midjourney, CLIP variants) were studied; smaller models not evaluated.
- Human baseline uses non-expert crowdworkers on AMT; human experts not evaluated, and human self-generation understanding is assumed rather than measured.
- Possible dataset contamination for proprietary models; only one divergence between humans and models is tested.
Experimental setup
Datasets: Mutual+ DREAM FaithDial Topioca RACE XSUM CommonSenseQA SocialIQA HellaSwag PIQA AlphaNLI WaNLI
Benchmarks: Mutual+ DREAM FaithDial Topioca RACE XSUM CommonSenseQA SocialIQA HellaSwag PIQA AlphaNLI WaNLI
Metrics: human-judged generation success rate discriminative accuracy question answering accuracy constraint satisfaction rate
Baselines: CLIP OpenCLIP BLIP-2 InstructBLIP Bard BingChat GPT-3.5-turbo
Builds on: CLIP TIFA COLLIE
Closest papers by meaning
- Position: Stop Making Unscientific AGI Performance Claims (ICML 2024 · similarity 0.87)
- When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration (NeurIPS 2025 · similarity 0.86)
- Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space (NeurIPS 2024 · similarity 0.86)
- Transcendence: Generative Models Can Outperform The Experts That Train Them (NeurIPS 2024 · similarity 0.85)
- Does GPT Really Get It? A Hierarchical Scale to Quantify Human and AI’s Understanding of Algorithms (AAAI 2025 · similarity 0.85)
- A Mathematical Framework for AI-Human Integration in Work (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.