Research map / Large language models / Language, safety and interpretability
BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BeanCounter, a 159B-token dataset of business disclosures from SEC EDGAR, shows it is less toxic than C4 around demographic terms, and finds continued pretraining on it lowers toxic generation and improves finance tasks for Pythia-1.4B and Phi-1.5.
Problem
Web-scraped LLM pretraining corpora raise concerns about toxicity, bias, PII, and factual accuracy, and large public domain-specific sources of business text are scarce.
Approach
Builds a large public corpus from all EDGAR filings including attachments, with timestamps and a fraud-labeled split, and measures toxicity conditionally on demographic descriptors using Perspective API on sentences rather than random spans.
Key results
- Continued pre-training on BeanCounter reduces RealToxicityPrompts toxicity score by 18-33% (e.g., Pythia-1.4B 0.0579 to 0.0385; Phi-1.5 0.0224 to 0.0184).
- Phi-1.5-BeanCounter improves Fin NER F-1 from 0.71 to 0.74 and FPB F-1 from 0.93 to 0.95.
- Sentences with demographic descriptors in BeanCounter have 59-89% lower average Perspective toxicity than matched C4-en sentences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue pre-training on Pythia-1.4B with BeanCounter | Pythia-1.4B · Fin NER test | F-1 score: 0.72 -> 0.73 | Helped |
| Continue pre-training on Pythia-1.4B with BeanCounter | Pythia-1.4B · FPB | F-1 score: 0.92 -> 0.93 | Helped |
| Continue pre-training on Pythia-1.4B with BeanCounter | Pythia-1.4B · RealToxicityPrompts | Toxicity score: 0.0579 -> 0.0385 | Helped |
| Continue pre-training on Pythia-1.4B with BeanCounter | Pythia-1.4B · RealToxicityPrompts | Perspective score: 0.1297 -> 0.1118 | Helped |
| Continue pre-training on Phi-1.5 with BeanCounter | Phi-1.5 · Fin NER test | F-1 score: 0.71 -> 0.74 | Helped |
| Continue pre-training on Phi-1.5 with BeanCounter | Phi-1.5 · FPB | F-1 score: 0.93 -> 0.95 | Helped |
| Continue pre-training on Phi-1.5 with BeanCounter | Phi-1.5 · RealToxicityPrompts | Toxicity score: 0.0224 -> 0.0184 | Helped |
| Continue pre-training on Phi-1.5 with BeanCounter | Phi-1.5 · RealToxicityPrompts | Perspective score: 0.0901 -> 0.0887 | Helped |
| Continue pre-training on Pythia-1.4B with BeanCounter | Pythia-1.4B · MuSR | Normalized accuracy: 0.3498 -> 0.3645 | Helped |
| Continue pre-training on Pythia-1.4B with BeanCounter | Pythia-1.4B · BBH | Normalized accuracy: 0.3174 -> 0.3135 | Hurt |
| Continue pre-training on Phi-1.5 with BeanCounter | Phi-1.5 · MMLU-Pro | Normalized accuracy: 0.17 -> 0.1252 | Hurt |
| Continue pre-training on Phi-1.5 with BeanCounter | Phi-1.5 · MuSR | Normalized accuracy: 0.3404 -> 0.3471 | Helped |
Surprising findings
- Phi-1.5-BeanCounter drops on several general benchmarks (e.g., BBH 0.3346 to 0.3124, GPQA 0.2732 to 0.2479), while Pythia-1.4B is similar or slightly better.
- Less than 0.1% of BeanCounter appears in Common Crawl-based datasets.
Limitations
- Continued pre-training ablation only on two small models (1.3B-1.4B); no tests on larger models or other data mixes.
- Business text may contain deceptive or misleading content despite regulation.
- Generalizability limited: some descriptors have different meanings in business text, and demographic analysis covers only a subset of axes.
Experimental setup
Datasets: BeanCounter C4-en Financial PhraseBank Fin NER RealToxicityPrompts ToxiGen
Benchmarks: Financial PhraseBank Fin NER RealToxicityPrompts SafeNLP BBH MMLU-Pro MuSR IFeval GPQA
Metrics: F-1 score toxicity score Perspective score safety score normalized accuracy average toxicity
Baselines: Pythia-1.4B Phi-1.5 C4-en
Builds on: Pythia Phi-1.5 RealToxicityPrompts ToxiGen EDGAR-CORPUS
Closest papers by meaning
- What's In My Big Data? (ICLR 2024 · similarity 0.86)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.86)
- The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (NeurIPS 2025 · similarity 0.85)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.85)
- The SEC Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data (NeurIPS 2026 · similarity 0.85)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.