Research map / Large language models / Language, safety and interpretability
ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ObscuraCoder, pre-training small-to-2.8B Llama-style code LMs with a bidirectional source-to-obfuscated-code translation objective on a 55M-pair multilingual dataset (ObscuraX), improving syntactic/semantic code understanding, library code generation, and multilingual completion.
Problem
Code language models' pre-training objectives have stayed largely unchanged; they are data-inefficient and struggle to disentangle code syntax from semantics, while high-quality code data is limited (the "data wall").
Approach
Use source-to-obfuscated-code translation pairs (identifiers and, for a fraction of samples, imports replaced by generic tokens) as a bidirectional pre-training objective mixed with plain language modelling on unpaired source and obfuscated code, trained from scratch for decoder-only Code-LMs, rather than masked de-obfuscation as in DOBF.
Key results
- ObscuraCoder 2.8B improves BigCodeBench pass@25 from 31.95 (CausalLM 2.8B) to 48.78 and ReCode function pass@1 from 14.72 to 20.11.
- ObscuraCoder 1.2B raises Multipl-E pass@1 from 12.07 to 18.34 over the same-size CausalLM.
- Post-hoc obfuscation continued pre-training (CausalLM-CP) does not improve over the causal baseline, and ObscuraCoder beats a size-matched decoder-only DOBF across zero-shot tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add obfuscation-grounded pre-training instead of vanilla causal LM pre-training | Llama 2.8B · CodeXGLUE Defect | F1-Score: 64.36 -> 65.42 | Helped |
| Add obfuscation-grounded pre-training instead of vanilla causal LM pre-training | Llama 2.8B · ReCode | pass@1: +10.59 Format; +9.77 Syntax; +5.38 Function | Helped |
| Add obfuscation-grounded pre-training with import obfuscation instead of vanilla causal LM pre-training | Llama 2.8B · BigCodeBench | pass@25: +16.83 | Helped |
| Add obfuscation-grounded pre-training instead of vanilla causal LM pre-training | Llama 2.8B · Multipl-E | pass@100: +14.39 | Helped |
| Add obfuscation-grounded pre-training instead of vanilla causal LM pre-training | Llama 2.8B · CommitChronicle | ROUGE-2: +1.26 | Helped |
| Use bidirectional source/obfuscated-code translation and train on obfuscated code, rather than identifier-map-only DOBF (ablation) | Llama 2.8B pre-training · ReCode Function | pass@1: 16.69 -> 20.11 | Helped |
| Use bidirectional source/obfuscated-code translation and train on obfuscated code, rather than identifier-map-only DOBF (ablation) | Llama 2.8B pre-training · BigCodeBench | pass@1: 13.18 -> 14.77 | Helped |
| Use bidirectional source/obfuscated-code translation and train on obfuscated code, rather than identifier-map-only DOBF (ablation) | Llama 2.8B pre-training · Multipl-E | pass@100: 51.27 -> 61.93 | Helped |
| Use bidirectional source/obfuscated-code translation and train on obfuscated code, rather than identifier-map-only DOBF (ablation) | Llama 2.8B pre-training · CommitChronicle | ROUGE-2: 13.39 -> 14.15 | Helped |
| Apply obfuscation translation post-hoc after causal pre-training rather than grounding representations early (ablation) | Llama 2.8B · ReCode Function | pass@1: 15.85 vs 20.11 | Hurt |
Surprising findings
- Post-hoc obfuscation training on already pre-trained causal LMs yields no gains, suggesting obfuscation grounding must be applied early in pre-training.
- Gains over causal LM widen with model size on zero-shot library-oriented and multilingual generation tasks.
Limitations
- Import obfuscation sacrifices strict semantic correctness for about 25% of samples.
- Models only trained up to 2.8B parameters, so scaling trends are extrapolated.
- Some corner cases (wildcard imports, header-only libraries, macros) may break correctness.
Experimental setup
Datasets: ObscuraX filtered-code-text filtered-source-code (The Stack) CommitChronicle CodeXGLUE defect detection
Benchmarks: CodeXGLUE defect detection ReCode BigCodeBench Multipl-E CommitChronicle
Metrics: accuracy F1-score pass@1 pass@10 pass@25 pass@100 ROUGE-1 ROUGE-2 ROUGE-L
Baselines: CausalLM (vanilla autoregressive) DOBF (decoder-only adaptation) StarCoderBase 1B StarCoderBase 3B DeepSeekCoder 1.3B StarCoder2 3B StableCode 3B CodeGemma 2B Phi-2
Builds on: DOBF Llama Megatron-LM LoRA
Closest papers by meaning
- Evaluation of Large Language Models on Code Obfuscation (Student Abstract) (AAAI 2024 · similarity 0.89)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.88)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.88)
- Emergent Representations of Program Semantics in Language Models Trained on Programs (ICML 2024 · similarity 0.88)
- Magicoder: Empowering Code Generation with OSS-Instruct (ICML 2024 · similarity 0.88)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.