Cross-cue reconstruction of perceived 3D object structure from human visual cortex

# Cross-cue reconstruction of perceived 3D object structure from human visual cortex Preprint: https://www.biorxiv.org/content/10.64898/2026.06.08.730830 ## Abstract The human brain assembles 3D percepts from qualitatively different depth cues, yet the perceived 3D structure the brain builds — a representation shared across cues — has been difficult to measure directly. We show this cue-invariant 3D structure can be externalized as explicit 3D objects from human brain activity: fMRI responses are decoded into the latent features of a pretrained 3D point-cloud autoencoder, and a generator maps these features back to a point cloud. A decoder trained only on responses to 2D rendered objects passed three increasingly stringent tests: (i) it generalized to novel object categories; (ii) it generalized across depth cues to random dot stereograms (RDSs), which evoke 3D percepts through binocular disparity but share no pictorial shape information with the training images; and (iii) it tracked the 3D slant of contour-matched RDSs whose 2D outlines were identical but whose disparity-defined slants varied. Cross-cue generalization was strongest in higher visual areas, particularly along the dorsal stream. ## Dataset overview Whole-brain fMRI from five subjects (S1–S5) viewing 3D objects presented as 2D rendered images (monocular pictorial cues) or random dot stereograms (RDSs; binocular disparity). The data support a decoding pipeline in which fMRI responses are mapped to the latent features of a pretrained 3D point-cloud autoencoder and then reconstructed as explicit 3D point clouds. Decoders are trained only on responses to 2D rendered images and tested on novel object categories, RDSs, and contour-matched RDSs. See the preprint for full stimulus, presentation, and analysis details. Note: Stimulus labels in task event files (`stimulus_name`) differ from those in the preprocessed fMRI data shared at [figshare](https://doi.org/10.6084/m9.figshare.c.8508462). Task event files will be updated to match the stimulus names in a future release. ## Tasks | Task label | Stimuli | |---|---| | `3dNaturalObjectsTrain` | 2D rendered natural objects, 2,000 stimuli (20 categories), decoder training; 120 runs over 3 repeats | | `3dNaturalObjectsTest` | 2D rendered natural objects, 80 test stimuli (40 trained + 40 novel categories) | | `3dArtificialObjectsImage` | 2D rendered artificial objects (geometric primitives), 30 stimuli | | `3dArtificialObjectsRDS` | The same 30 artificial objects as RDSs (+ 5 2D control stimuli, excluded) | | `3dContourMatchedRDSV1` | Contour-matched RDS slant bars: thin (horizontal), thick, cylinder at ±60/±45/±30/±15° (+ 24 2D controls) | | `3dContourMatchedRDSV2` | Contour-matched RDS slant bars: thin bars, horizontal and vertical, at the same eight slants (+ 16 2D controls) | Contour-matched RDSs (`V1`, `V2`) share an identical cyclopean (fronto-parallel) 2D contour across slants while the disparity-defined 3D slant varies. Per-bar-type slant slopes use `V1` for the horizontal thin bar, thick bar, and cylinder, and `V2` for the vertical thin bar; `V2` presents both thin-bar orientations together for the matched horizontal-vs-vertical comparison. ## Acquisition 3.0-T Siemens MAGNETOM Verio (Kyoto University). Functional EPI 2 mm isotropic, TR 2000 ms, multiband factor 4; MP-RAGE 1 mm isotropic anatomical. Full parameters are in the preprint (Methods). ## Ethics All subjects gave informed consent. Approved by Kyoto University (KUIS-EAR-2017-002) and the ATR International Ethics Committee (no. 106); conducted per the Declaration of Helsinki. ## Related resources - Preprint: https://www.biorxiv.org/content/10.64898/2026.06.08.730830 - Processed data & stimulus 3D structures (figshare): https://doi.org/10.6084/m9.figshare.c.8508462 - Analysis code (GitHub): https://github.com/KamitaniLab/3dReconstruction ## How to cite Aoki, S.C., Tsukasa, R., Yang, S., Tanaka, M., Doi, E., Nakamura, T., Ho, J.-K., and Kamitani, Y. Cross-cue reconstruction of perceived 3D object structure from human visual cortex. bioRxiv. https://doi.org/10.64898/2026.06.08.730830