Image Diffusion Models
EstablishedDiffusion models conceptualize generation as a gradual denoising process. A model is trained to reverse a forward process that slowly destroys data by adding Gaussian noise.
Pixel vs. Latent Diffusion
Early models (like DDPM) operated in Pixel Space, predicting noise for every single RGB pixel. This was computationally prohibitive for high resolutions. Latent Diffusion Models (LDMs) use a pre-trained VAE to compress the image into a smaller latent grid (e.g., 8x downsampling). The diffusion process happens entirely in this cheaper latent space, and the result is decoded back to pixels at the very end.
Backbones: U-Net vs. DiT
Historically, the neural network predicting the noise was a U-Net (using convolutional layers to downsample and upsample spatial features). Modern architectures are shifting to the Diffusion Transformer (DiT), replacing convolutions with attention across patch tokens, yielding better scaling laws.
Classifier-Free Guidance (CFG)
Interpretive intuitionCFG is the mathematical lever for "prompt adherence." The model runs twice per step: once with the text prompt, once unconditionally. It then extrapolates away from the unconditional prediction towards the conditioned one. High CFG forces strict alignment but causes "deep-fried" over-saturation.
References
- Ho et al. (2020). Denoising Diffusion Probabilistic Models (DDPM). arXiv:2006.11239
- Rombach et al. (2021). High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752
- Song et al. (2020). Score-Based Generative Modeling through Stochastic Differential Equations. arXiv:2011.13456
- Peebles & Xie (2022). Scalable Diffusion Models with Transformers (DiT). arXiv:2212.09748