Interpretable Visual Neural Decoding with Unsupervised Semantic Disentanglement

In the field of brain decoding research, reconstructing visual perception from neural recordings is a challenging but crucial task. With the use of superior algorithms, many methods have been dedicated to enhancing decoding performance. However, these models that map neural activities onto semantically entangled feature space are difficult to interpret. It is hard to understand the connections between neural activities and these abstract features. In this paper, we propose an interpretable neural decoding model that projects neural activities onto a semantically disentangled feature space with each dimension representing distinct visual attributes, such as gender and facial pose. A two-stage algorithm is designed to achieve this goal. First, a deep generative model learns semantically-disentangled image representations in an unsupervised way. Second, neural activities are linearly embedded into the semantic space, which the generator uses to reconstruct visual stimuli. Due to modality heterogeneity, it is challenging to learn such a neural embedded high-level semantic representation. We induce pixel, feature, and semantic alignment to ensure reconstruction quality. Three experimental fMRI datasets containing handwritten digits, characters, and human face stimuli are used to evaluate the neural decoding performance of our model. We also demonstrate the model interpretability through a reconstructed image editing application. The experimental results indicate that our model maintains a competitive decoding performance while remaining interpretable.