Research map / 3D vision / NeRF and radiance fields
TextNeRF: A Novel Scene-Text Image Synthesis Method based on Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a NeRF-based pipeline that models real scenes, semantically labels text regions with semi-supervised geometric consistency, and renders multi-view scene text images with 3D pose annotations, releasing a 3D-pose scene text dataset.
Problem
Synthetic scene text images for training detectors lack realism and 3D control; existing synthesis uses 2D backgrounds or graphics engines that struggle to replicate real-world scene complexity, and no datasets provide 3D pose attributes for text.
Approach
Use instant-NGP NeRFs of real scenes with an added semantic renderer (positionally enhanced) trained semi-supervised via projected proxy labels, so text regions are delineated in 3D, their 3D poses are computed, and novel camera views and text edits are rendered for synthesis.
Key results
- DB and EAST fine-tuned after pre-training on the proposed synthetic data gained F-score improvements of 22.72%, 5.26%, 2.52% (DB) on IC13, IC15, MLT17 over real-only training (as reported).
- Under 10k-image synthetic-only training, the proposed dataset gave the highest F-score on IC15 (66.41 with DB) and IC13 (70.21), outperforming VISD, SynthText3D and UnrealText.
- Balanced viewpoint sampling improved DB instance-level F-score from 64.25 (random) to 82.09 on the multi-view test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on Our-10k instead of UT-10k | DB · ICDAR 2013 | F-score: 62.64 -> 70.21 | Helped |
| Train on Our-10k instead of UT-10k | DB · ICDAR 2015 | F-score: 50.42 -> 53.72 | Helped |
| Train on Our-10k instead of UT-10k | DB · MLT 2017 | F-score: 51.22 -> 53.22 | Helped |
| Train on Our-10k instead of UT-10k | EAST · ICDAR 2015 | F-score: 58.36 -> 68.73 | Helped |
| Train on Ours-10k + Real instead of UT-10k + Real | EAST · ICDAR 2013 | F-score: 73.09 -> 76.47 | Helped |
| Train on Ours-10k + Real instead of UT-10k + Real | EAST · ICDAR 2015 | F-score: 85.79 -> 85.88 | Helped |
| Train on Ours-10k + Real instead of UT-10k + Real | DB · ICDAR 2013 | F-score: 64.55 -> 64.93 | Helped |
| Train with balanced rather than random viewpoint sampling | DB · TextNeRF multi-view test set | F-score: 64.25 -> 82.09 | Helped |
| Train with balanced rather than random viewpoint sampling | DB · TextNeRF multi-view test set, scene level | F-score: 57.90 -> 77.28 | Helped |
| Train with balanced rather than random viewpoint sampling | EAST · TextNeRF multi-view test set | F-score: 62.74 -> 80.66 | Helped |
| Train with balanced rather than random viewpoint sampling | EAST · TextNeRF multi-view test set, scene level | F-score: 55.21 -> 75.16 | Helped |
| Add structural importance masks during style transfer (ablation) | TextNeRF appearance changing · Rendered scene-text images | text content authenticity | Helped |
Surprising findings
- Synthetic-only trained detectors on UnrealText reached higher precision than the proposed dataset on some benchmarks, while the proposed data gave higher recall and F-score.
Limitations
- A specific NeRF must be trained for each scene, limiting scalability.
- Style transfer can introduce color inconsistencies.
- Hard to capture text under extreme conditions such as highly stylized fonts or severe occlusion.
Experimental setup
Datasets: VISD-10k SynthText3D-10k UnrealText-10k ICDAR 2013 ICDAR 2015 MLT 2017 COCO
Benchmarks: ICDAR 2013 ICDAR 2015 MLT 2017
Metrics: precision recall F-score PSNR SSIM LPIPS
Baselines: VISD SynthText3D UnrealText EAST trained on real data DB trained on real data
Builds on: NeRF instant-NGP S3IM PhotoWCT Semantic-NeRF
Closest papers by meaning
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.89)
- HyperFields: Towards Zero-Shot Generation of NeRFs from Text (ICML 2024 · similarity 0.88)
- Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing (NeurIPS 2024 · similarity 0.86)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.86)
- Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model (AAAI 2024 · similarity 0.85)
- CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.