MIT’s SceneSmith builds richer robot training worlds

Robots need experience to learn, but collecting that experience in the real world is slow and expensive. Researchers at MIT CSAIL and Toyota Research Institute say they have a way to speed that up: a system called SceneSmith that generates detailed 3D indoor environments from text prompts, giving robots more realistic places to practice before they are deployed.

According to the team, SceneSmith uses three agents powered by the vision-language model GPT-5.2. A designer creates the scene, a critic checks whether it looks realistic, and an orchestrator manages the back-and-forth and decides when the scene is finished. The output can then be loaded into physics simulation software.

Russ Tedrake, the Toyota Professor of Electrical Engineering and Computer Science, Aeronautics and Astronautics, and Mechanical Engineering at MIT, and a principal investigator at CSAIL, said simulation is a natural training ground for robots, but creating rich enough virtual worlds has remained a challenge.

Recommended reading

Some AI problems stay unsolvable, even with infinite data

Nicholas Pfaff, an MIT EECS Ph.D. student, CSAIL researcher, and lead author on the paper, said the team created more than 1,300 scenes with a leading VLM using internet-scale priors.

“We’ve found that the system can construct 3D scenes the way a human designer would. We made over 1,300 scenes using a leading VLM that has internet-scale priors, and it made insanely creative and diverse arrangements. I hadn’t taught the system to do that in the prompts; it just improvised.”

SceneSmith can generate spaces such as restaurants, bedrooms, hotels, garages, private offices, pottery stores, and even a Minecraft-themed gaming room. The researchers say its scenes contain up to six times more items per scene than prior methods, which helps robots rehearse tasks like:

• putting a cup in the sink
• placing fruit on plates
• moving a soda can from a shelf to a table

In tests, the researchers generated 100 unique spaces and used a VLM agent to evaluate robot action plans. That agent flagged many plans as faulty, and humans agreed with its verdicts more than 99 percent of the time. In another test, a pre-trained robot policy that had never seen a SceneSmith environment successfully carried out the instruction to “take the apple from the bowl and place it onto the cutting board.”

The team also teleoperated robots in the simulated spaces to open cabinets, put away bottles, and move between rooms. Compared with scene-generation baselines including HSM and Holodeck, the researchers said SceneSmith produced denser and more realistic environments. In a study with more than 200 users, its visuals were judged more realistic more than 90% of the time.

The trade-off is speed: generating a single scene can take multiple hours. Still, Jeremy Binagia, an applied scientist at Amazon Robotics who was not involved in the work, called SceneSmith “a significant advance” for creating simulation-ready indoor environments from simple text prompts.

The paper, “SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes,” by Nicholas Pfaff et al, is posted on arXiv with DOI 10.48550/arxiv.2602.09153.