Research map / Large language models / Language, safety and interpretability

ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ROK-FORTRESS, a bilingual English–Korean NSPS safety benchmark with a transcreation matrix separating language and U.S.–ROK geopolitical context effects, and evaluates 14 frontier and Korean models with LLM-judge panels.

Problem

Multilingual LLM safety is usually assessed with translation-only benchmarks that keep the scenario fixed, so it is unclear how language and geopolitical context jointly affect safety behavior in high-stakes national security and public safety (NSPS) scenarios.

Approach

A transcreation matrix varies language (English vs Korean) and geopolitical grounding (U.S. vs Korean entities) independently over matched adversarial intents, paired with dual-use benign prompts, scored by tier-weighted risk (TRS) and calibrated LLM-judge panels. Finds a consistent Korean suppression effect, with context interacting with language in a few models.

Key results

• Linguistic drop averages about 10 pp in TRS, roughly 2.5x the contextual drop of about 4 pp; Mistral Large 3 shows the largest linguistic drop at 28 pp.
• Korean culture-agnostic variants have lower TRS than English for 12 of 14 models.
• Korean context significantly mitigates language-driven suppression for K-EXAONE (+7.7 pp) and Kanana 2 (+7.0 pp) after Bonferroni correction.
• In direct-request ablation, closed-source models keep a positive linguistic drop (+4.5 pp mean) while open-source models show a reversed drop (-6.0 pp mean).

Surprising findings

• Korean variants are safer than English for most models, reversing the common translation-as-attack-vector finding.
• Under direct requests without jailbreak wrappers, open-source models become less safe in Korean (reversed linguistic drop), while closed-source models remain more conservative in Korean.

Limitations

• Findings limited to a single language pair (English-Korean) and one geopolitical axis (U.S.-ROK).
• Public release is a subset of 791 of 1,235 tasks.
• Harm-dimension classifier and judge agreement are only moderate (kappa 0.57 to 0.59 for dimension classification), and reasoning-trace analysis is observational.

Experimental setup

Datasets: FORTRESS ROK-FORTRESS

Benchmarks: ROK-FORTRESS

Metrics: tier-weighted risk score (TRS) over-refusal score (ORS) linguistic drop contextual drop F1 against human consensus compliance rate

Baselines: GPT-5.2 Claude Opus 4.5 Claude Sonnet 4.5 Gemini 3 Pro o4-mini DeepSeek V3.2 Kimi K2 Llama 4 Maverick Mistral Large 3 Qwen3 HyperCLOVA X Kanana 2

Builds on: FORTRESS MultiJail AdvBench CAGE

Closest papers by meaning

• SafeWorld: Geo-Diverse Safety Alignment (NeurIPS 2024 · similarity 0.87)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.84)
• On the Role of Attention Heads in Large Language Model Safety (ICLR 2025 · similarity 0.84)
• On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.84)
• RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios? (AAAI 2025 · similarity 0.83)
• SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.83)