Skywork open-sources Matrix-Game 4.0: game engine owns state, a 5B causal video model renders playable games in real time on one GPU

Background

Most "interactive world models" so far extrapolate pixels frame by frame. World state stays implicit, objects can vanish once they leave the view, and game rules have to be re-inferred every frame. That makes for good demo reels, but not for games you can reliably play or modify. Matrix-Game 4.0 from Skywork AI (Kunlun Tech) stops asking the video model to also be the physics engine.

Architecture

Skywork calls the design a harness. An executable game engine (C++20/SDL2) owns state, physics and input, and draws only a cheap low-poly RGB proxy. The video model then just renders. The renderer is built on Wan2.2-Fun-5B-Control, with about 5B BF16 parameters per scene checkpoint. It is a causal DiT distilled to 1–2 denoising steps that emits one latent (four frames) per step, and a 21-latent rolling KV cache with a sink keeps appearance stable. A coding agent turns natural-language requests into new engine rules and prompts, so new gameplay means changing code, not retraining. One caveat: the README says the released shooter configuration uses fixed prompts, with no retrieved spatial memory or dynamic prompt generation. Only City Racing supports per-chunk dynamic captions.

Benchmarks

Skywork's launch post claims 480p at 60 FPS, 112 ms end-to-end and up to 83.4 FPS generation on a single RTX 5090. The repo's own numbers come from one A100-80GB (864×480, one step, CUDA graphs): ~38–39 FPS for Castle/Egypt, and ~38.5 FPS (fixed prompt) or ~28.9 FPS (dynamic prompt) for Racing. The README also says these are not browser presentation FPS or action-to-photon latency. No third-party evaluation exists yet, and the paper is still marked "soon".

Getting started

You need Linux, an NVIDIA GPU and Python 3.12. Clone the repo, cd Matrix-Game-4, install PyTorch 2.12 / CUDA 13.2, and fetch one scene with matrix-game download, which uses a pinned Hub revision and checks SHA256 for every file. matrix-game render --preset realtime renders low-poly recordings offline. scripts/serve.sh plus MediaMTX runs the single-user WebRTC browser demo, and there is an optional rtx5090 FP8 preset. scripts/play.sh runs the games without the model, so you can play, record, or have a coding agent edit terrain and NPC rules. The demo binds to loopback by default; do not expose it to the internet.