Research map / Probabilistic and scientific modeling / Causal discovery
From Causal to Concept-Based Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes geometric concept definitions (affine subspaces of latent space) and proves identifiability of concepts from n+2 conditional datasets, validated on synthetic data, CLIP embeddings, and LLM truthfulness steering.
Problem
Causal representation learning needs on the order of latent dimension d_z interventions to identify latent factors, which is infeasible for high-dimensional data and for non-manipulable concepts. The paper asks whether a small set of human-interpretable concepts can be identified from conditional ra
Approach
Defines concepts as affine subspaces {Z: AZ=b} of the latent space, with concept-conditional distributions obtained by soft conditioning (Gaussian noisy filtering) rather than interventions. Shows that n atomic concepts are identifiable (up to linear reparametrization) from roughly n+2 environments, independent of the latent dimension d_z.
Key results
- On synthetic data with n=4 concepts and d_z=8, linear mixing reaches R2 0.97 and MCC 0.87 using m=n+1 conditional environments.
- Concept valuations on 3d-Shapes from two CLIP models (ViT-B/32 and RN101) concentrate around concept hyperplanes and are approximately linearly related across models.
- Replacing steering vectors with steering matrices in Inference-Time Intervention improves TruthfulQA accuracy on LLaMA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace steering vectors with steering matrices | Inference-Time Intervention for LLaMA · TruthfulQA | accuracy | Helped |
Limitations
- Identifiability only up to linear reparametrization and only for the span of the chosen atomic concepts; the full latent map is not recovered.
- Main theorem assumes Gaussian filtering noise and linearly independent atoms; the paper notes other noise families would need modified proofs.
- CLIP experiments do not directly match the theory's training objective, so they only show approximate consistency.
Experimental setup
Datasets: 3d-Shapes TruthfulQA synthetic Gaussian mixture data
Benchmarks: 3d-Shapes TruthfulQA
Metrics: R2 mean correlation coefficient (MCC) accuracy
Baselines: Inference-Time Intervention
Builds on: iVAE causal representation learning inference-time intervention linear representation hypothesis
Closest papers by meaning
- Learning Discrete Concepts in Latent Hierarchical Models (NeurIPS 2024 · similarity 0.91)
- Toward Causal Generative Modeling: From Representation to Generation (AAAI 2025 · similarity 0.88)
- LLM Interpretability with Identifiable Temporal-Instantaneous Representation (NeurIPS 2025 · similarity 0.88)
- Restyling Unsupervised Concept Based Interpretable Networks with Generative Models (ICLR 2025 · similarity 0.88)
- Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity (NeurIPS 2024 · similarity 0.87)
- Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventions (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.