AI-guided immersive exploration of brain ultrastructure for collaborative analysis and education

We introduce NeuroVerse, a framework for exploring 3D nanometric-scale reconstructions of neural and glial cellular processes in the central nervous system. Using image stacks from volume electron microscopy, NeuroVerse generates 3D mesh models through a SAM2-based segmentation pipeline and integrates absorption signals for deployment in a Metaverse environment. The framework includes a SAM2 adapter optimized for biological microscopy imaging, adapted with feature enhancement blocks and dual decoders to improve the segmentation of complex cellular structures. An interactive virtual AI agent, powered by Heygen and OpenAI models with domain-specific knowledge, provides semi-real-time assistance. NeuroVerse supports education and collaborative analysis for neuroanatomy and neuroscience. It includes a pipeline for the creation of 3D models, automated segmentation, mesh reconstruction, and heatmap computation, optimized for the Spatial.io ecosystem. Contributions include a virtual anatomy lab for neuroanatomy education and collaborative sessions on spatial morphology correlation and neuroenergetic absorption models. Evaluations show that the SAM2 adapter preserves fine cellular details and manages irregular boundaries. Preliminary sessions indicate potential to enhance neuroscience education, improve remote collaboration among scientists, and provide access to advanced neuroscientific data and tools. Evaluation of the virtual AI agent confirms its ability to provide context-aware support, interpret complex cellular structures, and facilitate understanding through semi-real-time assistance for students analyzing neural and glial reconstructions. NeuroVerse combines imaging, segmentation, and AI technologies within an immersive Metaverse platform for neuroscience education and research.

Other Information

Published in: Computers & Graphics
License: http://creativecommons.org/licenses/by/4.0/
See article on publisher's website: https://dx.doi.org/10.1016/j.cag.2025.104239