Core Concepts & Lexicon

Before analyzing specific modalities, we must define the fundamental geometry and vocabulary of representation learning.

Tokens & Embeddings

Neural networks do not read strings or look at jpegs; they perform matrix multiplications on vectors (lists of floating-point numbers). To bridge this gap:

• Tokens: Discrete chunks of data. In text, a token is often a sub-word (e.g., "un", "believ", "able"). In vision, a token might be a 16x16 pixel patch of an image.
• Embeddings: A look-up table that maps a discrete token ID to a high-dimensional continuous vector. Note carefully: Semantic proximity in embeddings is entirely learned and task-dependent. Words don't naturally exist close together; the training objective forces vectors into an arrangement that minimizes prediction error for that specific context.

Latent Spaces

A "latent space" is a compressed representation of the data. If a raw image has 3 million pixels, it might be compressed by an autoencoder into a much smaller, dense grid of "latent variables". Working in latent space is computationally cheaper. While it often forces the model to learn structural semantics rather than memorizing high-frequency noise, latent compression does not necessarily guarantee semantic disentanglement—features like color and shape might still be chaotically intertwined depending on the architecture.

Conditioning & Sampling

Generative models rarely run in a vacuum. Conditioning is the mechanism of steering generation using external signals (like a text prompt, a semantic map, or a previous frame). Mathematically, this often involves cross-attention, where the generated features query the conditioning features.

Sampling is the final step of turning continuous network outputs back into discrete user-facing data. Because models output probability distributions or noise gradients, sampling is where we introduce stochasticity (randomness) to resolve the prediction into a concrete choice (e.g. picking a specific pixel color or word).

Training vs. Inference

It is critical for HCI researchers to separate these phases:

• Training (Pretraining/Fine-tuning): The model's weights are updated via backpropagation. Massively expensive, batch-processed, and slow.
• Inference (Generation): The weights are frozen. The model runs forward passes to generate output. Interventions here (like prompting or activation steering) are cheap and interactive.