Research map / Probabilistic and scientific modeling / Molecules and proteins

Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows

NeurIPS 2025 · Probabilistic and scientific modeling › Molecules and proteins · approach: Neuro-symbolic probabilistic layers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TarFlowLM, a transformer-based autoregressive normalizing flow over VAE-encoded token latents, with mixture-CDF and mixture-Rosenblatt coupling layers, for flexible language modeling in continuous space.

Problem

Autoregressive language models are tied to discrete tokens, unidirectional context and single-token decoding; the paper asks whether language modeling can be recast in a continuous latent space with more modeling flexibility.

Approach

Model token latents (from a VAE with a tied Gaussian codebook encoder/decoder) with stacked alternating-direction transformer autoregressive flows whose coupling layers are mixture-of-Gaussians CDF (1-D) or Rosenblatt (d-D) transforms, so that a mixture density is exactly an invertible flow layer.

Key results

• On text8, TarFlowLM Mix-d reaches test BPC of about 1.30 versus 1.23 for the Transformer AR baseline and 1.37 for the Mix-1 variant (reported as upper bounds).
• On OpenWebText, TarFlowLM Mix-d attains validation perplexity of about 22.64 (NELBO-based), compared with 17.54 for Transformer AR and 21.80 for GenMD4.
• Mixture coupling layers substantially improve NELBO over affine coupling on OpenWebText, and the internal mixture size of 64 or more matches the performance of the full GPT-2 vocabulary size.

Measured effects reported in the paper

Surprising findings

• The Transformer AR baseline still achieves the best reported BPC and perplexity, ahead of the flow-based model.
• Flexible internal mixture size: 64 or more mixture components match the performance of using the full 50257-token GPT-2 vocabulary.

Limitations

• Sampling is sequential and slower than parallel non-autoregressive or diffusion approaches.
• Likelihood gap versus the discrete Transformer AR baseline on text8 and OpenWebText.
• Sampling speed optimization and advanced sampling techniques were not explored.

Experimental setup

Datasets: text8 OpenWebText

Benchmarks: text8 OpenWebText

Metrics: bits per character perplexity negative ELBO generative perplexity

Baselines: Transformer AR Plaid BFN MAC SEDD Absorb MD4 MDLM Latent NF CNF Argmax Flow Gaussian Diffusion

Builds on: normalizing flows masked autoregressive flow variational autoencoder Rosenblatt transform TarFlow

Closest papers by meaning

• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
• Flow Map Language Models: One-step Language Modeling via Continuous Denoising (NeurIPS 2026 · similarity 0.89)
• Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.89)
• Continuous Diffusion Model for Language Modeling (NeurIPS 2025 · similarity 0.89)
• Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.89)
• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.89)