Research map / Large language models / AI and society

Position: Stop Making Unscientific AGI Performance Claims

ICML 2024 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing AGI performance claims based on latent-space patterns are unscientific, illustrated with linear probes on random projections, PCA of yield curves, and FOMC-RoBERTa probes tested on made-up sentences.

Problem

Finding meaningful, externally correlated patterns in model latent spaces is being cited as evidence of AGI or 'understanding', and academic and commercial incentives plus human anthropomorphism encourage such over-interpretation.

Approach

Shows that simple models (random projections, PCA, autoencoders, and a fine-tuned RoBERTa) also yield latent representations predictive of external variables, and that a probe on FOMC-RoBERTa responds to nonsense bird-related sentences, so latent-pattern findings alone should not count as evidence of AGI; proposes a 'parrot test' with null and alternative hypotheses and calls for stronger hypothes

Key results

• Linear probe on an untrained random-projection network predicts world coordinates out-of-sample, showing useful representations arise without training.
• Probe on FOMC-RoBERTa final-layer activations yields predictions for bird-inflation sentences that differ from random-noise baseline, though less than for economic sentences (10 sentences per category, not statistically significant).
• In an autoencoder example, adding a latent embedding to a GDP growth regression lowered BIC from -860.391 to -864.499.

Measured effects reported in the paper

Surprising findings

• Probes on an untrained neural network already recover geographic coordinates out-of-sample.
• FOMC-RoBERTa probe predictions for nonsense bird-inflation sentences deviate from random noise, in a way similar to economically relevant sentences.

Limitations

• Adversarial sentence test uses only 10 sentences per category, so results are not statistically significant.
• Probe-based experiments are small-scale examples rather than rigorous benchmarks; authors admit regularization parameter was not carefully chosen.
• Argument relies on conceptual and social-science review rather than systematic empirical evidence.

Experimental setup

Datasets: world_place.csv (Li et al. world models) US Treasury par yield curve rates Trillion Dollar Words (FOMC communications) FRED real GDP (GDPC1) CPI and PPI indicators

Metrics: RMSE correlation mean directional accuracy MSE BIC

Baselines: autoregressive models (BIC-selected lag)

Builds on: Li et al. world models linear probing FOMC-RoBERTa Trillion Dollar Words

Closest papers by meaning

• The Generative AI Paradox: “What It Can Create, It May Not Understand” (ICLR 2024 · similarity 0.87)
• The AI Race: Why Current Neural Network-based Architectures are a Poor Basis for Artificial General Intelligence (AAAI 2025 · similarity 0.86)
• AGI-Elo: How Far Are We From Mastering A Task? (NeurIPS 2025 · similarity 0.86)
• Does GPT Really Get It? A Hierarchical Scale to Quantify Human and AI’s Understanding of Algorithms (AAAI 2025 · similarity 0.86)
• Position: Stop treating `AGI' as the north-star goal of AI research (ICML 2025 · similarity 0.85)
• Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.85)