Generative ML Architectures

Educational Baseline

This wiki is a technical, intuition-first guide to modern generative systems. It is tailored for HCI researchers looking to understand architecture differences, failure modes, and intervention points—moving beyond "AI as a black box API" into structurally aware interaction design.

Learning Path

Foundations → LLMs (Text) → Images → Video → World Models → WHAM (All paradigms) → Control Levers

The Shared Generative Loop

Despite vastly different modalities (text, image, video, interactive environments), nearly all modern generative systems share a structural loop:

Representation: Projecting raw data (pixels, characters, state) into a dense vector space (embeddings, latent space).
Prediction / Denoising: A backbone neural network (e.g., Transformer, U-Net) predicting the next token, or predicting the noise to subtract from a latent state.
Sampling: Choosing one trajectory or outcome from a learned distribution—such as selecting a token or taking a denoising step.
Iteration: Autoregressive systems feed an output back as context; diffusion systems repeatedly revise the whole latent. These are different generation loops, not interchangeable descriptions.

Cross-Modal Architecture Comparison

Modality	Representation Units	Backbone	Learning Objective	Generation Order	Conditioning Channel	Temporal State/Memory	Main Control Levers	Characteristic Failure Mode
Text (LLM)	Subword tokens	Decoder Transformer	Next-token prediction	Autoregressive (Left-to-right)	Prompt tokens in causal self-attention	KV Cache	Prompting, Activation steering, Logit bias	Hallucination, Context amnesia
Image	Continuous latent patches	U-Net or DiT	Score matching (Noise prediction)	Iterative denoising (All at once)	Cross-attention (Text), ControlNet	None	CFG, Inpainting, ControlNet	Anatomical distortion, Oversaturation
Video	3D Space-time latent patches	3D DiT / Temporal U-Net	Noise prediction (Space & Time)	Iterative denoising	Text cross-attention, First frame	Temporal attention blocks	Motion vectors, Temporal masks	Error accumulation, Object morphing
World Models	Compact Latent State	RNN / Transformer	Next-state prediction & Reward	Autoregressive state rollout	Action inputs	Recurrent hidden state	Action sequences, Policy tuning	Out-of-distribution collapse
WHAM	Visual/Gameplay tokens	Causal Transformer	Next-token prediction	Autoregressive (Frame-by-frame)	Controller action tokens	Context window	Controller inputs	Edit persistence and coherence can degrade over long horizons

Next: Foundations →