Research map / Image generation / Autoregressive and latent generation
The challenge of realistic music generation: modelling raw audio at scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add an ADA conditioning code with hop size 8 | WaveNet audio reconstruction · Solo piano audio waveforms | reconstruction NLL (eval): 1.151 -> 0.734 | Helped |
| use one-hot rather than continuous encoder input (ablation) | VQ-VAE · Solo piano audio waveforms, hop size 8 | reconstruction NLL (eval): 0.734 -> 0.682 | Helped |
| replace VQ-VAE quantisation with AMAE quantisation (ablation) | ADA with continuous input · Solo piano audio waveforms, hop size 8 | reconstruction NLL (eval): 0.734 -> 0.759 | Hurt |
| replace VQ-VAE quantisation with AMAE quantisation (ablation) | ADA with one-hot input · Solo piano audio waveforms, hop size 8 | reconstruction NLL (eval): 0.682 -> 0.695 | Hurt |
| replace ReLU and divisive normalisation with softmax (ablation) | AMAE · Solo piano audio waveforms, hop size 8 | reconstruction NLL (eval): 0.695 -> 0.806 | Hurt |
| increase the ADA hop size from 8 to 64 (ablation) | VQ-VAE audio reconstruction · Solo piano audio waveforms | reconstruction NLL (eval): 0.734 -> 1.172 | Hurt |
| use AMAE instead of VQ-VAE with PBT (ablation) | second-level ADA · Solo piano code sequences, hop size 8 and decoder RF 64 | reconstruction NLL (eval): 4.191 -> 4.247 | Hurt |
| increase the second-level AMAE decoder RF from 64 to 128 (ablation) | second-level ADA · Solo piano code sequences, hop size 8 | reconstruction NLL (eval): 4.247 -> 4.143 | Helped |
| increase the second-level AMAE decoder RF from 64 to 256 (ablation) | second-level ADA · Solo piano code sequences, hop size 8 | reconstruction NLL (eval): 4.247 -> 4.159 | Helped |
| increase WaveNet depth, doubling its receptive field (ablation) | single-level WaveNet · Solo piano generation | musicality: $2.43 \pm 0.14$ -> $2.89 \pm 0.17$ | Helped |
| extend receptive field while making the WaveNet thin (ablation) | single-level WaveNet · Solo piano generation | musicality: $2.43 \pm 0.14$ -> $1.71 \pm 0.18$ | Hurt |
| add a hop-8 VQ-VAE code hierarchy before the large WaveNet | solo piano generation · Solo piano generation | musicality: $2.89 \pm 0.17$ -> $3.04 \pm 0.16$ | Helped |
Closest papers by meaning
- Fast Timing-Conditioned Latent Audio Diffusion (ICML 2024 · similarity 0.88)
- SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement (NeurIPS 2025 · similarity 0.88)
- An Independence-promoting Loss for Music Generation with Language Models (ICML 2024 · similarity 0.87)
- Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models (ICLR 2024 · similarity 0.87)
- Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling (ICLR 2025 · similarity 0.87)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.