Diffusion Models Explained

Introduction

Diffusion models are a class of generative models that learn to generate data by reversing a gradual noising process. They have achieved state-of-the-art results in image generation, surpassing GANs in many benchmarks.

Forward Process (Adding Noise)

The forward process gradually adds Gaussian noise to data $x_0$ over $T$ timesteps:

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} x_{t-1}, \beta_t I)$

where $\beta_t$ is the noise schedule. We can sample $x_t$ directly from $x_0$:

$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t)I)$

where $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$.

Reverse Process (Denoising)

The model learns to reverse the noising process:

$p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \sigma_t^2 I)$

Training Objective

The simplified training objective is:

$L_{\text{simple}} = \mathbb{E}_{t, x_0, \epsilon}\left[\|\epsilon - \epsilon_\theta(x_t, t)\|^2\right]$

where $\epsilon_\theta$ is a neural network (typically a U-Net) that predicts the noise.

Sampling

To generate new data, start from pure noise $x_T \sim \mathcal{N}(0, I)$ and iteratively denoise:

$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta(x_t, t)\right) + \sigma_t z$

where $z \sim \mathcal{N}(0, I)$.

Key Variants

• DDPM: Denoising Diffusion Probabilistic Models (Ho et al., 2020)
• DDIM: Faster deterministic sampling
• Score-based models: Score matching perspective (Song & Ermon)
• Latent Diffusion: Operate in latent space (Stable Diffusion)
• Classifier-Free Guidance: Conditional generation without a separate classifier

Applications

• Image Generation: DALL-E, Stable Diffusion, Midjourney
• Video Generation: Sora, Make-A-Video
• Audio: Music and speech synthesis
• Molecular Design: Drug discovery

Conclusion

Diffusion models provide a principled framework for generative modeling with stable training and high-quality outputs, making them one of the most important developments in modern generative AI.

References

1. Ho et al., "Denoising Diffusion Probabilistic Models," NeurIPS 2020.
2. Song et al., "Score-Based Generative Modeling through SDEs," ICLR 2021.
3. Rombach et al., "High-Resolution Image Synthesis with Latent Diffusion Models," CVPR 2022.