Most scene reconstruction systems either fuse everything into a single monolithic surface or leave occluded regions incomplete. WorldSculpt's core insight is that a strong single-object 3D generative prior can be grounded to multi-view, posed observations per object and then reused compositionally to produce large, cluttered scenes of individually addressable meshes—without scene-level retraining.
Key Findings
- Object-centric generation: Each object is reconstructed as an individual mesh in an anchor-aligned canonical frame, then placed into a shared world frame via a canonical-to-world transform, enabling selection, manipulation, and simulation.
- Multi-view grounding + generative completion: Per-view DINOv3 features are lifted into canonical voxel volumes and fused with an IBR-style, permutation-invariant aggregator; the aggregated 3D condition is injected into a frozen Pixal3D prior via zero-initialized projection layers while LoRA adapters let the prior exploit multi-view evidence to plausibly complete unobserved geometry.
- No scene-level training: The generative prior is finetuned only on single-object canonical data yet generalizes at test time to scenes with hundreds of densely occluded objects.
- Benchmark and evaluation: Introduces UE-MeshyScene, a photorealistic, large-scale benchmark with per-object ground-truth meshes and annotations; WorldSculpt outperforms prior methods, with larger gains as occlusion increases.
Who it's for and tradeoffs
Great fit if you need editable, per-object meshes from multi-view captures (games, AR/VR, simulation, robotics) and want plausible completion in heavily occluded clutter. Look elsewhere if you require exact, physically verified scene-level joint optimization or end-to-end scene learning—WorldSculpt relies on per-object anchoring, posed observations, and a strong single-object prior. It also requires reliable object localization and posed multi-view inputs; quality depends on the prior and the coverage of object views.
Where it fits
Positions between geometry-only multi-view reconstruction (which leaves occluded holes) and monolithic generative worlds (which are not object-separable). It offers a practical path to compositional scene assets by marrying generative priors with grounded multi-view conditioning.
Method highlights
- Anchor-aligned canonical frames built per detected object
- Per-view feature extraction (DINOv3) and lifting into canonical voxels
- Permutation-invariant IBR-style feature fusion supporting variable inputs
- Zero-init projection layers + LoRA to inject multi-view condition into a pretrained Pixal3D prior
- Outputs individual meshes that are transformed back into the world frame for composition
This combination yields scalable compositional reconstruction in scenes with hundreds of objects and severe mutual occlusion while preserving direct usability for downstream rendering and simulation.