Gaussian splatting for games
A new graphics technique with enormous promise and a few real problems still to solve before games can lean on it the way they lean on polygons.
Gaussian Splatting is one of the most exciting graphics techniques to emerge in years. Instead of describing a 3D scene as a mesh of polygons with textures, GS represents it as millions of tiny semi-transparent "splats" called Gaussians. Each one a 3D Gaussian with a position, a color, and an orientation. Render them all together and you get a photoreal scene that's effectively a 3D photograph.
For games, that's a paradigm shift. Today, building a 3D environment is one of the most expensive parts of game development. Modelers, texture artists, lighting passes, weeks of iteration to get something that looks "real enough." The result is typicall too large to load in web games and too heavy to run on mobile phones. With GS you can capture a real space (or a generated one) from photos or video and bring it into the engine in minutes. The resulting scene runs at 60 fps on a mobile phone, and it looks like a photograph because it is one.
I've been experimenting with this in Bitmagic, which natively supports Gaussian Splat scenes. You can hand the engine a scene and prompt a game inside it: I've made an interactive level out of a single photo of a Supercell's London office breakroom, a haunted-cabin ghost hunt from a World Labs scene, and a full street scene from a Red Dead Redemption 2 video clip... all running in real time on the Bitmagic engine.
The two open problems
Authoring is the bigger one. To capture a useful scene you currently need either a sequence of carefully-shot photographs, a generative tool like World Labs Marble, or you accept the blurry edges of a single-photo capture. Tooling for editing splats (moving them, retopologizing them, recoloring them) is in its infancy. For many games, it's too limiting if the game world can't be edited during the design iteration phase. Games get better by playing and tuning and iterating, but with GS based scene you find yourself working around the fact that the scene cannot be edited once it's been generated. Multi-camera capture for large spaces is being figured out in real time. We'll get there fast but right now the gap is real.
Realtime manipulation is the second. A traditional game world runs polygon colliders against polygon meshes, and physics engines have been tuned to that for decades. There's no Gaussian-Splatting-native physics engine yet. The pragmatic workaround is to voxelize the splats and discard the transparent ones to filter out dust and flare, then build polygon colliders around the resulting voxels. At ~0.25-meter voxel size you get accuracy that's good enough for most gameplay, and you can wrap convex colliders around groups of voxels to keep collider counts manageable. It works, but it's a workaround. Native GS physics is the natural next step.
Why I think it's worth the bet
As recently as two years ago, "take a screenshot, get a playable world" sounded like science fiction. It works today. The trajectory on capture quality, authoring, and runtime tooling has been steep and it's accelerating. Whatever isn't possible this month will probably be possible in a few. If you're building games and you haven't looked at Gaussian Splatting yet, this is the moment to start.