GaussianRenderFormer renders 3D Gaussian splats with a transformer, not a rasterizer

arXivarXivPapers

GaussianRenderFormer renders 3D Gaussian splats with a transformer, not a rasterizer
Levy and Benaim / GaussianRenderFormer project page

Shahaf Valencia Levy and Sagie Benaim present GaussianRenderFormer, a transformer that takes a set of 3D Gaussians and a camera and outputs an image without a rasterizer in the renderer. Each Gaussian is a token in a view-independent encoder, and a view-dependent decoder turns patches of camera rays into pixels by attending to those tokens.

Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, the model reaches 38.08 dB PSNR on 300 unseen objects against the full unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. The authors say the gap is consistent with a bottleneck in the view decoder rather than a failure to generalize, and they have submitted the work to TMLR.