Research map / Large language models / AI and society

CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets

NeurIPS 2026 · Large language models › AI and society · approach: Benchmarking and reproducibility practice

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CroissantMiner, a 602-paper benchmark (102 human-validated gold, 500 LLM silver) for end-to-end extraction of the 30-field Croissant 1.1 metadata from ML dataset papers, and evaluates 24 systems including single-pass LLMs and four agentic architectures.

Problem

Populating Croissant machine-readable dataset metadata (core and Responsible AI fields) from dataset papers is manual and labor-intensive, and no benchmark existed for schema-complete automated extraction on real dataset papers.

Approach

A benchmark pairing dataset papers with gold and silver annotations against the full 30-field Croissant 1.1 schema, plus a systematic comparison showing single-pass full-context extraction beats four agentic decomposition architectures, with a human-audited LLM judge for long-form RAI fields.

Key results

• Claude Sonnet 4.6 single-pass achieves the highest composite score of 0.709 (95% CI 0.688-0.729) among ranked systems.
• All four agentic architectures underperform single-pass with the same backbone; with Sonnet 4.6, ReAct scores 0.652 and Locator-Extractor 0.566 versus 0.709 for single-pass.
• Single-pass extraction returns empty values for only 2.4% of documented RAI cells, versus 10.9-14.8% for the agentic variants.

Surprising findings

• Agentic decomposition consistently underperforms single-pass extraction with the same backbone, despite added steps and tools.
• Agentic configurations cost 1.2x to 7.8x more per paper than single-pass without improving scores.

Limitations

• Benchmark conflates extraction capability with incompleteness of dataset documentation (27.7% of gold cells have no value).
• LLM judge agreement with humans is imperfect on difficult fields, so judge-based scores do not replace full human evaluation.
• Gold labels began as Claude Sonnet 4.5 pre-fills, introducing possible anchoring to the seed model and family.

Experimental setup

Datasets: Hugging Face top-downloaded dataset papers

Benchmarks: CroissantMiner

Metrics: composite score exact match token-F1 LLM judge 1-3 ordinal score Cohen's quadratic-weighted kappa Gwet's AC1 Krippendorff's alpha

Baselines: Parallel Specialists Triage + Critique Locator-Extractor ReAct MOLE MeXtract

Builds on: Croissant ReAct

Closest papers by meaning

• Croissant Baker: Metadata Generation for Discoverable, Governable, and Reusable ML Datasets (NeurIPS 2026 · similarity 0.94)
• Croissant: A Metadata Format for ML-Ready Datasets (NeurIPS 2024 · similarity 0.93)
• Bridging the Data Provenance Gap Across Text, Speech, and Video (ICLR 2025 · similarity 0.84)
• Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on HuggingFace (ICLR 2024 · similarity 0.83)
• Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs (ICLR 2026 · similarity 0.83)
• The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.83)