On the Unification of Text and Visual Modalities: Next-Token Prediction and Diffusion

Text tokens are discrete, while images and video live in a continuous space by design. After encoding, however, both become continuous embeddings. The bulk of the model is therefore modeling continuous variables either way, and language only needs an LM head at the end to map back to discrete tokens.

Multimodal modeling can, in principle, share one approach. This post looks at the trends toward that unification we see today.

1. Causality in Text and Video

Next-token prediction (NTP) for text is autoregressive by construction. Recent diffusion language models that generate text block by block are autoregressive as well: blocks are produced causally, and diffusion only denoises the tokens inside the current block.

Image and video generation took the opposite route. Earlier models denoised the whole image or clip bidirectionally. Recent models move to autoregression over chunks of several frames, with causal attention across chunks, and some go further to small causal blocks inside a single image. The motivation is twofold: causal generation makes the modeling problem easier, above all for global consistency, and it cuts latency for streaming inference.

Top row: NTP and chunk-causal AR video. Middle row: block-causal masks, with text blocks on the left and sub-frame video blocks on the right. Bottom row: the same masks with a doubled clean + noisy sequence, as used in teacher-forced training (video: S2PD).

Setting aside the doubled sequence that teacher-forced training requires, the attention masks of the two modalities have already converged to essentially the same shape.

2. Loop Transformers and Multi-Step Denoising Diffusion

A loop transformer runs a block of layers repeatedly with shared weights, so depth comes from iteration rather than parameters. The Universal Transformer looped a single layer. Recent models loop a multi-layer block, or the whole stack: Saunshi et al. (2025) show a $k$-layer block looped $L$ times nearly matches a $kL$-layer model.

Diffusion sampling has the same outer loop: one network, run $T$ times with shared weights. With x-prediction, as in JiT, the network outputs a clean estimate $\hat{x} = f_\theta(z_t, t)$ at every step, where $z_t = t\cdot x + (1-t)\cdot\varepsilon$ and $t$ runs from 0 (noise) to 1 (data). The sampler turns it into a velocity $v = (\hat{x} - z_t)/(1-t)$ and takes a step, moving $z$ part of the way toward $\hat{x}$. The new $z$ goes back in as input. The loop transformer feeds back a hidden state; diffusion feeds back a point in data space.

Left: the whole layer stack loops over hidden state $h$ (Ouro-style). Right: the denoiser predicts a clean $\hat{x}$ each step, and the sampler moves $z$ toward it.

Seen this way, the two loops do the same job. A looped language model makes the network deeper, and adds capacity, without adding weights. Multi-step diffusion does the same: one network applied $T$ times. In both cases, compute is amortized across several forward passes of the same model.

The difference is the extra role diffusion gives to noise. In text-to-image and text-to-video generation, noise reflects an information asymmetry between the two sides: the text prompt carries far less information than the image or video it describes, and the noise supplies what the condition leaves unspecified. For text generation itself, it is not settled whether noise has a role to play. Text diffusion models now work quite well, but that does not indicate that this type of modeling is optimal for text.

3. OPD and Self-Rollout in DMD

The same unification appears in the distillation recipes for text and video. In language models, on-policy distillation (OPD) trains a student against a stronger teacher: the student samples its own responses, the teacher scores every sampled token, and the loss is the per-token reverse KL (Agarwal et al., 2023; Thinking Machines, 2025). In autoregressive video, Self Forcing distills a many-step diffusion teacher into a few-step causal student with DMD. During training, the student rolls out the video chunk by chunk with its own KV cache, and the DMD loss matches the rolled-out video to the teacher's distribution.

The two communities developed these separately, but they are the same recipe for the same problem: a mismatch between training and inference.

Top row: the context comes from data or the teacher, and only the last block is the student's. Bottom row: every block in the context is the student's own output, and the teacher only grades it.

The mismatch comes from teacher forcing. In training, the model conditions on context from data or the teacher: ground-truth text, teacher-written responses, ground-truth frames. At inference, it conditions on its own earlier outputs. Small errors put the student in contexts it never saw in training, and they compound over long sequences: off-track responses in language models, drift and error accumulation in long videos.

OPD and self-rollout remove the mismatch the same way: the KV cache context is always produced by the student itself, never by the teacher. Training then sees the same context distribution as inference. The teacher only supplies the learning signal on the student's own outputs: per-token log-probabilities in OPD, a score estimate in DMD.

The two losses also share one objective: a reverse KL from student to teacher. They differ only in how the gradient passes through a sample, and each matches a standard policy-gradient estimator from RL.

DMD's output is continuous, so the gradient flows through the sample (reparameterization). Expanded, the DMD gradient weights the generator's Jacobian by a difference of two scores, i.e. gradients of log p:

This is the reparameterized gradient of the SAC algorithm in traditional RL. SAC's policy is a Gaussian, $a = f_\varphi(\varepsilon; s)$, and its update minimizes the reverse KL from $\pi$ to $\exp(Q)/Z$. Its gradient (Eq. 13) also weights $\partial f/\partial\varphi$ by a difference of log-density gradients:

OPD's output is discrete tokens, so there is no path through the sample. It uses the likelihood-ratio (score-function) gradient instead, with the per-token log-ratio as the weight (OPD's per-token form; the exact sequence-level gradient weights each token by the sum of the log-ratios from that token on):

The choice between the two estimators is set by the output space (continuous or discrete), and both have been well studied in the RL literature. Reparameterization needs a differentiable path from the parameters to the sample. Continuous outputs have one: SAC's Gaussian actions and DMD's generated frames. Discrete tokens have no such path, so OPD falls back to the likelihood-ratio estimator, the same split as between reparameterized methods such as SAC and likelihood-ratio methods such as REINFORCE and PPO in traditional RL.

4. Representation Alignment

Aligning representations across modalities has always been hard. A simple example: text cannot describe every detail of an image. Even after both sides are tokenized, the information they carry stays asymmetric.

The relation is many-to-many: one caption fits many images, and one image admits many captions. But the two directions are not symmetric. A caption discards most of the image's information, so the conditional entropy $H(\text{image} \mid \text{text})$, the uncertainty left about the image given its text, stays large, while $H(\text{text} \mid \text{image})$ is bounded by the far smaller information content of text. Some image details have no text counterpart at all. The two embedding spaces also differ in intrinsic rank.

In practice, each modality comes with its own encoding. Text uses BPE with learnable embeddings, or a text VAE. Images use a ViT or a VAE. These spaces are far from aligned, and their intrinsic ranks differ.

Transfusion runs both modalities through one set of weights. MoT keeps global attention but splits the FFN, attention projections and norms by modality. MoE shares attention and lets a router choose experts per token.

By design, Transfusion should yield a more unified latent representation than Mixture-of-Transformers (MoT). Yet MoT usually does better on each modality's tasks. Separate parameters and representations help, which is itself a sign that the representations do not align. The misalignment may come from fixed input encoders, such as a pretrained ViT or VAE; an encoder-free setting could ease it, at the price of expensive joint training.

MoE does not entangle modalities by itself. The router can learn to send each modality to different experts, but nothing pushes the modalities to fuse. In experiments, leaving expert routing entirely to the model does not seem to beat MoT's hand-set modality split. Unification may have to start further upstream, at the encoder.

Conclusion

Text and visual generation are converging on the same recipes along several axes:

• Causality. Text block diffusion and AR video generation have arrived at the same block-causal attention mask, up to the doubled sequence used in teacher-forced training.
• Iteration. Loop transformers and multi-step diffusion both amortize compute across repeated forward passes of one model. Diffusion adds noise, which in text-to-visual generation supplies what the condition leaves unspecified.
• Distillation. OPD and self-rollout DMD fix the same train-inference mismatch by filling the KV cache with the student's own outputs. Both minimize a reverse KL, with a likelihood-ratio or a reparameterized gradient depending on whether the output is discrete or continuous.
• Representation. This is the main remaining gap. Text and image embeddings relate many-to-many and differ in intrinsic rank, and architectures that split parameters by modality still tend to do better. Closing the gap may require unifying at the encoder.

Two questions remain open: whether text modeling needs the role of noise at all, and whether encoder-free training can let a fully shared model match modality-split ones at an acceptable cost.

References

1. Arriola et al. (2025). Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models.
2. Hu, Olmeda Reino and Tewari (2026). S2PD: Serial-to-Parallel Diffusion for Physically and Logically Consistent Video Generation.
3. Dehghani et al. (2018). Universal Transformers.
4. Saunshi et al. (2025). Reasoning with Latent Thoughts: On the Power of Looped Transformers.
5. Geiping et al. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Huginn).
6. Zhu et al. (2025). Scaling Latent Reasoning via Looped Language Models (Ouro).
7. Li and He (2025). Back to Basics: Let Denoising Generative Models Denoise (JiT).
8. Agarwal et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.
9. Lu and Thinking Machines Lab (2025). On-Policy Distillation.
10. Yin et al. (2024). One-step Diffusion with Distribution Matching Distillation (DMD).
11. Huang et al. (2025). Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion.
12. Haarnoja et al. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.
13. Schulman et al. (2017). Proximal Policy Optimization Algorithms.
14. Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model.
15. Liang et al. (2024). Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models.