Research map / Large language models / AI and society
On the Consideration of AI Openness: Can Good Intent Be Abused?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that open-source LLMs (komt-Mistral-7B, EEVE-10.8B, Qwen2-72B) fine-tuned on small sets of Korean criminal precedent QA pairs and unethical data become willing to give detailed criminal advice, and discusses the resulting legal liability.
Problem
Open-source AI models and legal datasets (court precedents) may be misused to produce detailed criminal guidance, raising the question of how likely such misuse is and whether builders could escape legal liability.
Approach
Fine-tune widely used open-source LLMs with very small sets (200 to 600) of QA pairs derived from Korean criminal precedents, plus an existing Korean unethical QA dataset, to measure how easily ethical refusal can be removed and informativeness about crimes increased; then analyze Korean and EU legal implications.
Key results
- Fine-tuning komt-Mistral-7B on 200 crime QA examples dropped its human-rated ethics score from 4.4 to 1.8 (GPT-4 rating: 4.7 to 2.4).
- On KO-Fraud-eval-100, Qwen2-72B informativeness rose from 1.56 to 4.84 (GPT-4o) after fine-tuning on fraud data plus KEQA.
- Informativeness and ethics scores were found to be negatively correlated across fine-tuned models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune KOMT-Mistral-7B on ours | KOMT-Mistral-7B · ours-eval-16 | Ethics: 4.4 -> 1.8 | Hurt |
| Fine-tune KOMT-Mistral-7B on ours | KOMT-Mistral-7B · ours-eval-16 | Informativeness: 1.1 -> 2.0 | Helped |
| Fine-tune KOMT-Mistral-7B on ours | KOMT-Mistral-7B · ours-eval-16 | Fluency: 5.0 -> 2.3 | Hurt |
| Fine-tune KOMT-Mistral-7B on KEQA | KOMT-Mistral-7B · ours-eval-16 | Informativeness: 1.1 -> 1.1 | No clear effect |
| Fine-tune KOMT-Mistral-7B on ours and KEQA | KOMT-Mistral-7B · ours-eval-16 | Informativeness: +2.1 points | Helped |
| Fine-tune KOMT-Mistral-7B on ours and KEQA | KOMT-Mistral-7B · ours-eval-16 | Fluency: -0.5 | Hurt |
| Fine-tune Qwen2-72B-Instruct on ourstwo and KEQA | Qwen2-72B-Instruct · ourstwo-eval-100 | Informativeness: 1.56 -> 4.84 | Helped |
| Fine-tune Qwen2-72B-Instruct on ourstwo instead of KEQA only | Qwen2-72B-Instruct · ourstwo-eval-100 | Informativeness: 3.90 -> 4.78 | Helped |
Surprising findings
- Only 200 precedent-derived examples were enough to remove refusal behavior in open-source models.
- GPT-4 gave relatively high (~3) informativeness scores to refusals, indicating judge miscalibration.
Limitations
- Experiments were intentionally minimal and the models are not fluent or powerful malicious LLMs.
- Evaluation relies on small test sets and GPT-4 judging, with partial lawyer verification.
- Focused on Korean language and Korean legal context, so generality beyond it is untested.
Experimental setup
Datasets: KoEthicalQA (KEQA) Korean legal precedents (200 and 600 QA pairs) KOLD SQUARE TOXIGEN KoTox KoSBi KoBBQ
Benchmarks: KO-Crime-eval-16 KO-Crime-eval-50 KO-Fraud-eval-100
Metrics: ethics score (1-5) informativeness score (1-5) fluency score (1-5)
Baselines: komt-mistral-7b-v1 (unfine-tuned) EEVE-Korean-Instruct-10.8B-v1.0 (unfine-tuned) Qwen2-72B-Instruct (unfine-tuned)
Builds on: LoRA QLoRA Mistral-7B many-shot jailbreaking
Closest papers by meaning
- Position: Near to Mid-term Risks and Opportunities of Open-Source Generative AI (ICML 2024 · similarity 0.84)
- Modeling the Economic Impacts of AI Openness Regulation (NeurIPS 2025 · similarity 0.83)
- Position: A Safe Harbor for AI Evaluation and Red Teaming (ICML 2024 · similarity 0.83)
- Position: Editing Large Language Models Poses Serious Safety Risks (ICML 2025 · similarity 0.83)
- Consent in Crisis: The Rapid Decline of the AI Data Commons (NeurIPS 2024 · similarity 0.83)
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.