How to generate voxel colliders from Gaussian Splatting scenes

I posted a video showing some gameplay with a Gaussian Splatting scene, and a lot of people have asked me how collisions are handled with GS. We use polygon colliders, because our physics engine is polygon based and I'm not aware of any Gaussian Splatting based physics engine.

But how do you auto-generate polygon colliders from a GS scene? Gaussian Splatting tends to introduce a lot of noise, and the shapes are not exactly accurate.

I've tried various ways, and the most reliable solution so far is by voxelizing the Gaussians and then generating colliders around the voxels. You'll need to extract the alpha from each Voxel (which is compressed in SPZ files) and discard those voxels that are almost transparent, to eliminate dust and flares from the collisions.

Here's a video that shows some preliminary work on this - you get a fairly accurate representation with voxels that are about 0.25 meters in size, and by merging the voxels you get a reasonable number of colliders in the end. It would probably be OK to wrap convex colliders around groups of voxels to get a much simpler scene, and it might also be necessary to shrink the resulting colliders by some amount, but you get the idea.

In my tests this accuracy is enough for most in-game situations (remember - the voxels won't be visible and it's hard to see small collision discrepancies in a 3D world). The voxels don't need to be colored, but coloring them makes it easier to visualized them.

Generating a voxel scene like this out of 2 million Gaussians produces about 10-15k colliders without optimization. When done on the client in Typescript it takes about a second, so it's something that can be done in realtime if needed. I think ultimately we will process these at upload time on the server using compiled scripts, instead of doing it in realtime on the client.

This allows for much heavier processing (for example, floors should be fully flag) and smaller voxel size (halving the voxel size produces 8x more voxels), to ensure there are no performance issues on mobile devices.

Once you have produced minimized voxel-based colliders, you will also be able to use them as shadow casters and shadow receivers, which is a great way to increase immersion. I think you should also be able to fast lighting effects on them and blend the result with the underlying Gaussians for explosions other bright but temporary lighting.

So, there you have it! I've had this question asked so many times, I thought I'd write a post about it to make it easier to share. Please share this to anyone who is working with Gaussian Splatting. | 35 comments on LinkedIn