Research map / Image generation / Editing and style transfer
ACFun: Abstract-Concrete Fusion Facial Stylization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACFun, a one-shot facial stylization method that uses one style image and one face image, combining an abstract fusion module (textual-inversion-style CLIP embedding), a concrete fusion module (affine adapters in U-Net ResBlocks), and a face-and-style imagery alignment loss on Stable Diffusion 1.4.
Problem
Facial stylization from a single style image is hard: existing methods need many style images, or cause style degradation, limited deformation, or loss of facial identity when stylizing faces with strong geometric distortion.
Approach
Separately learn abstract style/face features (via a CLIP embedding fused with a facial description prompt) and concrete features (trainable affine parameters inserted into U-Net ResBlocks), then align style and face images in a combined "imagery" latent space using a weighted two-target diffusion loss.
Key results
- In a 50-volunteer user study, ACFun was preferred for face consistency (63.33% vs. 10.83% for InstantStyle SDXL) and overall (54.17% vs. 15.00% for InstantStyle SD1.5).
- Against InstantStyle SDXL in a pairwise study, ACFun won 52.27% on style consistency, 63.64% on face consistency and 71.59% on overall preference.
- Training takes about 3 minutes on one A6000 GPU with 40 diffusion steps at inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACFun instead of InstantStyle SDXL | facial stylization · Facial stylization user study | face consistency preference: 10.83% -> 63.33% | Helped |
| Use ACFun instead of InstantStyle SDXL | facial stylization · Facial stylization user study | overall preference: 10.83% -> 54.17% | Helped |
| Use ACFun instead of InstantStyle SD1.5 | facial stylization · Facial stylization user study | style consistency preference: 34.17% -> 29.17% | Hurt |
| Use ACFun instead of InstantStyle SDXL | facial stylization · Facial stylization user study vs InstantStyle SDXL | overall preference: 28.41% -> 71.59% | Helped |
| Remove CFun and generate using AFun abstract features only (ablation) | ACFun facial stylization · Facial stylization | visual quality | Hurt |
| Add CFun in the encoder (ablation) | ACFun facial stylization · Facial stylization | concrete visual feature extraction | Helped |
| Add CFun in the decoder (ablation) | ACFun facial stylization · Facial stylization | face structure and posture | Helped |
| Replace facial description prompt with a single pseudo-word prompt (ablation) | ACFun AFun module · Facial stylization | generation quality | Hurt |
| Set imagery alignment loss hyperparameters to γ:β = 0.6:1 (ablation) | ACFun facial stylization · Facial stylization | style intensity | Hurt |
| Set imagery alignment loss hyperparameters to γ:β = 1:1 (ablation) | ACFun facial stylization · Facial stylization | style leakage | Hurt |
| Insert trainable parameters into Attention Blocks instead of Resblocks (ablation) | ACFun CFun module · Facial stylization | style leakage | Hurt |
| Use Mean + Standard instead of Standard only in CFun (ablation) | ACFun CFun module · Facial stylization | stylization stability | Hurt |
Surprising findings
- Inserting the trainable parameters into attention blocks caused serious style leakage and overfitting, so ResBlock insertion was chosen.
- In user study, ACFun was not preferred on style consistency (29.17% vs. 34.17% for InstantStyle SD1.5).
Limitations
- Style leakage: stylized faces can change with the pose in the style image, and style ambiguity limits extraction of specific style elements.
- Performance depends on the pre-trained model; training-data imbalance causes semantic binding and style leakage for certain characters or styles.
- Evaluation relies on a small user study with 40 pairs and no error bars or statistical significance tests.
Experimental setup
Datasets: Internet-collected style and facial images
Benchmarks: user study on 40 style-face pairs (50 volunteers)
Metrics: style consistency preference face consistency preference overall preference
Baselines: InstantStyle DreamStyler Inst AesPA StyTr2
Builds on: Textual Inversion Stable Diffusion AdaIN DreamBooth
Closest papers by meaning
- StyO: Stylize Your Face in Only One-Shot (AAAI 2025 · similarity 0.88)
- Styl3R: Instant 3D Stylized Reconstruction for Arbitrary Scenes and Styles (NeurIPS 2025 · similarity 0.85)
- OmniConsistency: Learning Style-Agnostic Consistency from Paired Stylization Data (NeurIPS 2025 · similarity 0.85)
- Multi-StyleGS: Stylized Gaussian Splatting with Multiple Styles (AAAI 2025 · similarity 0.85)
- Puff-Net: Efficient Style Transfer with Pure Content and Style Feature Fusion Network (CVPR 2024 · similarity 0.85)
- Local Consistency Guidance: Personalized Stylization Method of Face Video (Student Abstract) (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.