SceneAgent turns 3DGS and 4D captures into interactive simulation environments

arXivarXivPapers

SceneAgent turns 3DGS and 4D captures into interactive simulation environments
arXiv / Luke Hollis, Tianxing Fan and Heng Yang

Luke Hollis, Tianxing Fan and Heng Yang present SceneAgent, an agentic pipeline that converts visual captures into interactive simulation environments through simulator-based visual inspection and repair. It processes 3D Gaussian Splatting, photogrammetry, LiDAR, AI-generated 3DGS scenes and 4D videos, combining object segmentation, physical parameter estimation, part decomposition, articulation and deformable simulation, and can generate “digital sisters” and scripted demonstrations for policy learning.

On a cluttered bin scene from GraspClutter6D, all agent configurations exceed SimFoundry’s F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%.