WHAM: World & Human Action Models

Note: This covers Microsoft Research's "World and Human Action Models towards gameplay ideation" (Nature 2025), completely distinct from the CVPR human-motion recovery model of the same acronym.

Tokenized Gameplay and Controllers

WHAM reframes interactive environment generation as a sequence modeling problem. Instead of predicting continuous states, it tokenizes both visuals (video/gameplay frames) and human controller inputs into a unified discrete vocabulary. The model generates the next sequence of visual tokens autoregressively, conditioned heavily on the interleaved sequence of controller action tokens.

Consistency, Diversity, and Persistency

Evaluating generative gameplay requires metrics beyond visual fidelity:

• Consistency: Do the visuals correctly reflect the physics of the controller input?
• Diversity: Does the same input sequence yield varied, plausible outcomes across runs?
• Persistency: Do user edits, environment changes, and states remain consistent over long horizons and subsequent generations? (For instance, if a player breaks a crate, does it stay broken when the camera pans away and returns?)

Limitations & HCI Ideation

WHAM was trained primarily on gameplay data from a single game (Bleeding Edge). Consequently, it faces severe generalization limits; it cannot inherently invent new game physics from scratch. For HCI, its primary value is gameplay ideation—allowing designers to "play" generated scenarios, acting as a highly advanced visualization tool rather than a robust engine.

Caveat: The real-time variant, WHAM-RT, introduces optimizations for playable frame rates, though detailed quality/latency tradeoffs between the variants are not fully documented publicly.