Research map / Large language models / Agents, code and math
The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Alchemist, which prompts LLMs to generate labeling programs instead of labeling each data point, aggregates their outputs with weak supervision (Snorkel), and trains a local distilled model, cutting annotation API cost by ~500x.
Problem
Using large pretrained models as data annotators is expensive (per-point API calls), produces static labels that are hard to extend or audit, and struggles with non-text modalities.
Approach
Task LLMs to generate small Python labeling programs (executed locally) rather than querying labels per data point; aggregate noisy program outputs with weak supervision label models, and extend to images by using LLM-identified concepts scored by a local CLIP feature extractor.
Key results
- Alchemist with GPT-3.5 improves distilled-model accuracy on 5 of 8 text datasets (e.g., MedAbs 0.346 vs 0.311, Cancer 0.968 vs 0.716) while reducing estimated labeling cost roughly 500x overall.
- On Waterbirds, Alchemist with GPT4o and CLIP ViT-L/14 reaches worst-group accuracy 0.467 vs 0.335 for CLIP zero-shot, with a reduced gap (0.335 vs 0.569).
- Combining RoboShot with Alchemist (CLIP ViT-L/14) yields average accuracy 0.803 and worst-group accuracy 0.569.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace per-example zero-shot annotation with generated labeling programs and Snorkel aggregation | Alchemist with GPT-3.5 · Text datasets | accuracy or F1-score: YouTube 0.871 -> 0.891; SMS 0.907 -> 0.900; Yelp 0.845 -> 0.575; IMDb 0.737 -> 0.662; MedAbs 0.311 -> 0.346; Cancer 0.71 | Mixed |
| Extract task-concept features with CLIP before generating labeling programs | Alchemist for Waterbirds · Waterbirds | worst group accuracy: GPT4o, ViT-B/32: 0.367 -> 0.283; Claude 3, ViT-B/32: 0.022 -> 0.463; GPT4o, ViT-L/14: 0.367 -> 0.467; Claude 3, ViT-L/14 | Mixed |
| Add dataset descriptions to program-generation prompts (ablation) | Alchemist prompting · Text datasets | label model performance | Mixed |
| Add 5 data exemplars to program-generation prompts (ablation) | Alchemist prompting · Text datasets | label model performance | Mixed |
| Add keywords to program-generation prompts (ablation) | Alchemist prompting · Text datasets | label model performance | Mixed |
| Add specialized labeling rules to program-generation prompts (ablation) | Alchemist prompting · Text datasets | label model performance | Mixed |
| Collect a more diverse set of programs from different prompt categories (ablation) | Alchemist · SMS, Yelp, IMDb | label model performance | Helped |
| Replace Snorkel aggregation with alternative weak-supervision label models (ablation) | Alchemist · Text datasets | accuracy or F1-score | Mixed |
| Add RoboShot spurious-feature rejection before Alchemist program labeling (ablation) | Alchemist with GPT4o · Waterbirds | worst group accuracy: 0.283 -> 0.375 (CLIP ViT-B/32); 0.467 -> 0.569 (CLIP ViT-L/14) | Helped |
| Vary GPT-4 API temperature among 0.0, 0.5, and 1.0 (ablation) | Alchemist · YouTube, SMS, French, Cancer | label model performance: YouTube end model: 0.795, 0.870, 0.803; SMS: 0.866, 0.854, 0.930; French: 0.690, 0.690, 0.690; Cancer: 0.935, 0.940, 0.9 | No clear effect |
| Generate programs from labeled examples rather than manually crafted functions | Alchemist compared with human-crafted labeling functions · SMS | coverage: 0.41 -> 1.00 | Helped |
| Generate programs with GPT4o instead of using human-crafted labeling functions | Alchemist · Census Income Classification | label model accuracy: 0.681 -> 0.725 | Helped |
Surprising findings
- Vanilla Alchemist with GPT4o on raw image pixels achieves only 0.395 average accuracy on Waterbirds, showing direct program generation fails for images.
- On Yelp, Alchemist with GPT-3.5 (0.575) underperforms zero-shot prompting (0.845) on accuracy, and on SMS and IMDb zero-shot prompting is also competitive.
Limitations
- Performance depends on the capability of the underlying language model to generate effective programs.
- For very complex tasks, generated programs can become long and hard to validate.
- Image handling relies on a two-step concept pipeline with a local CLIP extractor, and direct program generation on pixels performs poorly.
Experimental setup
Datasets: YouTube spam SMS spam Yelp IMDb Finance French book reviews MedAbs Cancer Waterbirds ECG heartbeat Census income
Benchmarks: YouTube SMS Yelp IMDb MedAbs Cancer Finance French Waterbirds WRENCH datasets
Metrics: accuracy F1-score worst group accuracy estimated API cost coverage
Baselines: Zero-shot Prompting Group Prompting WRENCH human-crafted labeling functions Vanilla Alchemist CLIP zero-shot prompting
Builds on: Snorkel WRENCH DataSculpt ScriptoriumWS RoboShot
Closest papers by meaning
- Self Iterative Label Refinement via Robust Unlabeled Learning (NeurIPS 2025 · similarity 0.84)
- Collaborative Refining for Learning from Inaccurate Labels (NeurIPS 2024 · similarity 0.84)
- Pearls from Pebbles: Improved Confidence Functions for Auto-labeling (NeurIPS 2024 · similarity 0.83)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.83)
- Optimal and Efficient Binary Questioning for Accelerated Annotation (AAAI 2025 · similarity 0.83)
- Cost-aware LLM-based Online Dataset Annotation (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.