Video Generation

Video generation extends image diffusion by adding a temporal dimension, introducing immense computational and consistency challenges.

Spatial vs. Temporal Compression

Just as images are compressed spatially by a VAE, video models require a 3D VAE to compress across both space and time. A common strategy compresses spatial dimensions by 8x, but temporal dimensions by only 4x, recognizing that motion carries critical high-frequency detail.

Joint vs. Factorized Space-Time Models

• Factorized: The network alternates between processing spatial layers (treating frames independently) and temporal layers (mixing across frames). It is cheaper to train but struggles with complex object persistence.
• Joint (Full 3D): Uses unified 3D attention, letting any patch in space/time attend to any other. Highly compute-intensive but yields superior physical coherence (e.g., CogVideoX).

The Error Accumulation Problem

Current models primarily generate "short clips" (e.g., 5 seconds) simultaneously. Attempting to generate long video autoregressively (chunk by chunk) leads to error accumulation—small hallucinations in frame 40 become structural deformations by frame 120, leading to objects morphing or physics breaking.

Conditioning and Camera Control

Intervention in video generation often relies on conditioning the temporal blocks. Control signals can be injected via camera-motion vectors (Plucker coordinates) or temporal ControlNets guiding structural edges frame-by-frame.