Research map / Large language models / AI and society

Position: We Can’t Understand AI Using our Existing Vocabulary

ICML 2025 · Large language models › AI and society · approach: Human-centered trust and agency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing that AI interpretability should develop neologisms (new learned words for human or machine concepts), with a proof-of-concept that learns soft-token embeddings for length, diversity, and quality words on a frozen Gemma model.

Problem

Humans and machines conceptualize the world differently, and existing human vocabulary may be insufficient to reference and control machine concepts or communicate human concepts to machines.

Approach

Frame interpretability as a communication problem and introduce neologisms: new word embeddings trained with preference losses while the base model stays frozen, so the word means a human or machine concept and is usable compositionally in natural language prompts.

Key results

• Base Gemma never meets the specified length constraints on the tested instructions, while the length neologism yields many more compliant responses.
• The diversity neologism substantially raises the probability of finding the correct number in the guessing game; baseline Gemma reaches only about 20% after 100 guesses.
• Responses for 'extremely good' neologism average 3.8 Gemma self-score versus 3.2 for 'extremely not good' (50 samples each).

Measured effects reported in the paper

Surprising findings

• Gemma's self-evaluation is biased toward score 4 out of 5, so the score gap is modest even though it is in the expected direction.

Limitations

• Proof of concept is limited to simple concepts (length, diversity, quality) and a toy guessing game.
• Evaluation of the quality neologism relies on Gemma scoring its own outputs, which is a biased judge.

Experimental setup

Datasets: LIMA

Benchmarks: length-constrained generation on held-out LIMA instructions number guessing game for diversity

Metrics: fraction of responses meeting length constraint probability of finding correct number vs number of samples average Gemma self-score

Baselines: Gemma base model without neologism

Builds on: DPO soft prompting representation engineering APO

Closest papers by meaning

• When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration (NeurIPS 2025 · similarity 0.85)
• Does GPT Really Get It? A Hierarchical Scale to Quantify Human and AI’s Understanding of Algorithms (AAAI 2025 · similarity 0.85)
• The Generative AI Paradox: “What It Can Create, It May Not Understand” (ICLR 2024 · similarity 0.84)
• Towards Trustworthy Autonomous Systems via Conversations and Explanations (AAAI 2024 · similarity 0.83)
• Position: Theory of Mind Benchmarks are Broken for Large Language Models (ICML 2025 · similarity 0.83)
• Position: Explainable AI Cannot Advance Without Better User Studies (ICML 2025 · similarity 0.83)