SceneAgent turns 3DGS and 4D captures into interactive simulation environments

Luke Hollis, Tianxing Fan and Heng Yang present SceneAgent, an agentic pipeline that converts visual captures into interactive simulation environments through simulator-based visual inspection and repair. It processes 3D Gaussian Splatting, photogrammetry, LiDAR, AI-generated 3DGS scenes and 4D videos, combining object segmentation, physical parameter estimation, part decomposition, articulation and deformable simulation, and can generate “digital sisters” and scripted demonstrations for policy learning.
On a cluttered bin scene from GraspClutter6D, all agent configurations exceed SimFoundry’s F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%.