Research map / Large language models / AI and society
The Foundation Model Transparency Index
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Third edition of the Foundation Model Transparency Index scoring 13 AI developers on 100 indicators, finding average transparency fell from 58 to 40.69 and testing an AI agent for information gathering.
Problem
Foundation model developers' transparency practices are opaque and declining, and there is no systematic, longitudinal measurement of company-level transparency across the AI supply chain.
Approach
Extends the FMTI with a revised 100-indicator rubric spanning upstream, model, and downstream domains, combining developer-submitted reports with manual and agent-assisted public-information gathering, followed by company engagement before finalizing scores.
Key results
- Average transparency score fell from 58 in 2024 to 40.69 in 2025 across 13 developers.
- IBM scored 95/100, the highest in the Index's history, while xAI and Midjourney scored 14.
- Claude 4-based agent found additional relevant information for 13 indicators on average versus 8 missed by the human team across six companies.
Surprising findings
- Average transparency declined from 2024 to 2025 despite the prior edition's improvement, with Meta and Mistral dropping sharply.
- No developer disclosed train-test overlap, and no company disclosed reproducible model-level mitigations.
Limitations
- Only 7 of 23 contacted companies submitted reports; six scored from public information only.
- Agent produces false positives and false negatives, so expert verification remains necessary.
Experimental setup
Datasets: company transparency reports public company disclosures (model cards, websites, GitHub, arXiv)
Benchmarks: FMTI 2025 100-indicator rubric
Metrics: transparency score out of 100 indicator agreement rate agent miss and additional-find counts
Baselines: AI21 Labs Alibaba Amazon Anthropic DeepSeek Google IBM Meta Midjourney Mistral OpenAI Writer
Builds on: Foundation Model Transparency Index (2023) FMTI 2024
Closest papers by meaning
- Position: On the Societal Impact of Open Foundation Models (ICML 2024 · similarity 0.87)
- Position: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them? (ICML 2024 · similarity 0.86)
- Making Transparency Advocates: An Educational Approach Towards Better Algorithmic Transparency in Practice (AAAI 2025 · similarity 0.83)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.82)
- Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transition (NeurIPS 2025 · similarity 0.82)
- Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.