Spatial Foundry: A Multimodal Dataset for Spatial Intelligence

Spatial learning requires correspondence between visual observations and the underlying scene structure. Explicit geometry, object identities and camera calibration provide supervision beyond image appearance alone.

Floor plans, point clouds and editable meshes provide complementary representations of scene structure. Calibrated perspective and panoramic RGB are associated with metric depth, object IDs and 3D bounding boxes. Video frames are paired with camera trajectories and per-frame poses.

Residential layout estimation, metric 3D reconstruction, instance-level scene understanding, embodied navigation and camera-conditioned world modeling.

A common 3D reference links supervision across views, modalities and temporal sequences.

Limited environmental diversity can bias models toward familiar layouts and object configurations. Variation in scene structure and furnishings supports evaluation of generalization across residential environments.

Residential scene configurations are paired with images, depth, camera calibration, object annotations and 3D geometry. Camera trajectories and corresponding videos preserve the relationship between local observations and global scene structure. Scene counts include furnishing variants of the same layout.

Cross-scene generalization for depth estimation, reconstruction and object perception; scene retrieval; and camera-conditioned generation across diverse residential layouts.

Layout and furnishing variation provide distinct axes for studying environmental generalization.

Geometric learning depends on consistent camera models and metric scale. Misalignment between appearance, depth and camera calibration introduces ambiguity into both supervision and evaluation.

Perspective RGB and depth are rendered directly from the source scene, with camera intrinsics and extrinsics retained. Equirectangular panoramas and explicit 3D geometry provide complementary representations of the same environment.

Perspective and panoramic depth estimation, RGB-D fusion, multiview reconstruction, novel view synthesis and geometric evaluation of neural scene representations.

Consistent scene coordinates support comparisons between image-space predictions and explicit 3D structure.

Instance-level scene understanding requires object identity to remain consistent across viewpoints. Explicit 3D bounds and dimensions provide geometric references for localization and spatial relationships.

Scene meshes, RGB observations and point clouds share object IDs and metric 3D bounds. Projected annotations establish correspondence between visible image regions and the underlying scene instances.

3D object detection, visual grounding, cross-view instance association and structured scene prediction from video.

Shared instance identities enable consistent supervision across multiple observations of the same environment.

Textual motion instructions do not fully specify a camera trajectory. Explicit pose sequences define position, orientation and timing, enabling reproducible evaluation of camera control and spatial consistency.

Video frames are associated with camera poses, intrinsics and timestamps. Routes through furnished 3D scenes support controlled changes in viewing direction, height and speed while preserving the underlying environment.

Camera-conditioned video generation, visual navigation, controllable scene exploration and spatial-memory evaluation through revisitation.

Rendered reference sequences permit comparison under explicitly specified camera conditions.

Multi-agent world modeling requires consistent state across observers. A scene update initiated by one participant should be reflected in other viewpoints and retained in subsequent observations.

Independent participant cameras are linked to shared object identities, transforms and edit events. Synchronized observations provide references for distinguishing viewpoint changes from changes in scene state.

Collaborative exploration, multi-agent spatial reasoning, shared-state prediction and consistency evaluation for online world models.

A shared 3D state provides a reproducible reference for multi-observer interaction.

Spatially consistent editing requires changes to persist beyond a single image. Updated object geometry must remain coherent with occlusion, illumination and subsequent camera viewpoints.

Editable object meshes, materials and transforms support explicit scene-state changes. Before-and-after observations are linked to the corresponding edits and camera poses.

Interactive scene editing, instruction-guided 3D manipulation, action-conditioned world models and counterfactual scene synthesis.

Recorded scene-state changes provide reproducible supervision for editing and action-effect prediction.