University of Waterloo

From Monocular 3D Reconstruction to Simulation-Ready Digital Twins

Daniel (Quanyun) Wu

November 20th, 2026 – 12:00-12:30 pm, EC4-2101A

Embodied AI systems learn by interacting with the world, but collecting that experience on real robots is slow, costly, and risky. Simulation can scale this training, yet only if the virtual scene faithfully matches the real one. Existing 3D methods fall short: feed-forward reconstruction networks produce scale-free, fused surfaces, while generative models create clean object meshes with no sense of real-world scale or placement.  This talk presents two frameworks that turn an ordinary monocular RGB video into a simulation-ready digital twin that is metric-scaled, object-separable, and watertight. KitchenTwin uses VLM-guided scale recovery and geometry-aware registration to place generated object meshes accurately into a metric scene. OfficeTwin handles large offices with many similar cubicles, where each cubicle is only partly visible. It uses the repeated cubicles to fill in each other’s missing parts and rebuilds 23 complete, closed cubicles, which are tested with robot navigation and physics simulation in Isaac Sim. The talk concludes with plans for sim-to-real robot training using these twins.