什么是扩散模型 (Diffusion Model)

What are Diffusion Models?

[Updated on 2021-09-19: Highly recommend this blog post on score-based generative modeling by Yang Song (author of several key papers in the references)]. [Updated on 2022-08-27: Added classifier-free guidance, GLIDE, unCLIP and Imagen. [Updated on 2022-08-31: Added latent diffusion model. So far, I’ve written about three types of generative models, GAN, VAE, and Flow-based models. They have shown great success in generating high-quality samples, but each has some limitations of its own.

https://lilianweng.github.io/posts/2021-07-11-diffusion-models/

什么是扩散模型

正向扩散过程

与随机梯度 Langevin 动力学的联系

反向扩散过程

Parameterization of Lt for Training Loss#

Simplification#

Connection with noise-conditioned score networks (NCSN)#

Parameterization of βt#

Parameterization of reverse process variance Σθ#

Speed up Diffusion Model Sampling

1. Generate higher-quality samples using a much fewer number of steps.

2. Have “consistency” property since the generative process is deterministic, meaning that multiple samples conditioned on the same latent variable should have similar high-level features.

3. Because of the consistency, DDIM can do semantically meaningful interpolation in the latent variable.

• KL-reg: A small KL penalty towards a standard normal distribution over the learned latent, similar to VAE.

• VQ-reg: Uses a vector quantization layer within the decoder, like VQVAE but the quantization layer is absorbed by the decoder.

Conditioned Generation#

Classifier Guided Diffusion#

Classifier-Free Guidance#

Scale up Generation Resolution and Quality#

• Truncated conditioning augmentation stops the diffusion process early at step $t > 0$ for low resolution.

• Non-truncated conditioning augmentation runs the full low resolution reverse process until step 0 but then corrupt it by z ∼ q(x|x) and then feeds the corrupted z s into the super-resolution model.
t
t
0
t

• A prior model P(c|y): outputs CLIP image embedding c given the text y.
i
i

• A decoder P(x|c, [y]): generates the image x given CLIP image embedding c and optionally the original text y.
i
i

1. Given a text y, a CLIP model is first used to generate a text embedding c(y). Using CLIP latent space enables zero-shot image manipulation via text.
t

2. A diffusion or autoregressive prior P(c|y) processes this CLIP text embedding to construct an image prior and then a diffusion decoder P(x|c, [y]) generates an image, conditioned on the prior. This decoder can also generate image variations conditioned on an image input, preserving its style and semantics.
i
i

• Static thresholding: clip x prediction to [ − 1, 1]

• Dynamic thresholding: at each sampling step, compute s as a certain percentile absolute pixel value; if $s > 1$, clip the prediction to [ − s, s] and divide by s.

• Shift model parameters from high resolution blocks to low resolution by adding more residual locks for the lower resolutions;

• Scale the skip connections by $1/\sqrt{2}$

• Reverse the order of downsampling (move it before convolutions) and upsampling operations (move it after convolution) in order to improve the speed of forward pass.

Quick Summary#

• Pros: Tractability and flexibility are two conflicting objectives in generative modeling. Tractable models can be analytically evaluated and cheaply fit data (e.g. via a Gaussian or Laplace), but they cannot easily describe the structure in rich datasets. Flexible models can fit arbitrary structures in data, but evaluating, training, or sampling from these models is usually expensive. Diffusion models are both analytically tractable and flexible

• Cons: Diffusion models rely on a long Markov chain of diffusion steps to generate samples, so it can be quite expensive in terms of time and compute. New methods have been proposed to make the process much faster, but the sampling is still slower than GAN.

Citation#

Weng, Lilian. (Jul 2021). What are diffusion models? Lil’Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

@article{weng2021diffusion,
  title   = "What are diffusion models?",
  author  = "Weng, Lilian",
  journal = "lilianweng.github.io",
  year    = "2021",
  month   = "Jul",
  url     = "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/"
}

References#