Yi-Jun Zhang, Zhao-Fei Yu, Jian. K. Liu, Tie-Jun Huang. Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches. Machine Intelligence Research. https://doi.org/10.1007/s11633-022-1335-2
Citation: Yi-Jun Zhang, Zhao-Fei Yu, Jian. K. Liu, Tie-Jun Huang. Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches. Machine Intelligence Research. https://doi.org/10.1007/s11633-022-1335-2

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

doi: 10.1007/s11633-022-1335-2
More Information
  • Author Bio:

    Yi-Jun Zhang received the B. Eng. degree in communication engineering from Yingcai Honors College, University of Electronic and University of Electronic Science and Technology of China, China in 2017. He is currently a Ph. D. degree candidate in computer science at Department of Computer Science and Engineering, Shanghai Jiao Tong University, China. His research interests include artificial intelligence and computational neuroscience. E-mail: yijzhang@sjtu.edu.cn ORCID iD: 0000-0003-2289-2372

    Zhao-Fei Yu received the B. Sc. degree in automation from Hong Shen Honors School, College of Optoelectronic Engineering, Chongqing University, China in 2012, and the Ph. D. degree in automation from Automation Department, Tsinghua University, China in 2017. He is currently an assistant professor with Institute for Artificial Intelligence, Peking University, China. His research interests include artificial intelligence, brain-inspired computing, and computational neuroscience. E-mail: yuzf12@pku.edu.cn (Corresponding author) ORCID iD: 0000-0002-6913-7553

    Jian. K. Liu received the Ph. D. degree in mathematics from University of California at Los Angeles, USA in 2009. He is currently a lecturer with School of Computing, University of Leeds, UK. His research interests include computational neuroscience and brain-inspired computing. E-mail: j.liu9@leeds.ac.uk ORCID iD: 0000-0002-5391-7213

    Tie-Jun Huang (Senior Member, IEEE) received the B. Sc. and M. Sc. degrees in computer science from Wuhan University of Technology, China in 1992 and 1995, respectively, and the Ph. D. degree in pattern recognition and image analysis from Huazhong (Central China) University of Science and Technology, China in 1998. He is currently a professor with Department of Computer Science, School of Electronics Engineering and Computer Science, Peking University, China, and the director of Beijing Academy of Artificial Intelligence, China. He has published two books and more than 200 peer-reviewed articles in leading journals and conferences. He is a co-editor of four ISO/IEC standards, five national standards of China, and four IEEE standards. He holds more than 50 granted patents. He is also a fellow of Chinese Association for Artificial Intelligence (CAAI) and China Computer Federation (CCF). He received the National Award for Science and Technology of China (Tier-2) three times in 2010, 2012, and 2017. He is also the secretary-general of the Artificial Intelligence Industry Technology Innovation Alliance and the vice-chair of the China National General Group on AI Standardization. His research interests include visual information processing and neuromorphic computing. E-mail: tjhuang@pku.edu.cnORCID iD: 0000-0002-4234-6099

  • Received Date: 2022-02-07
  • Accepted Date: 2022-04-27
  • Publish Online: 2022-07-14
  • Vision plays a peculiar role in intelligence. Visual information, forming a large part of the sensory information, is fed into the human brain to formulate various types of cognition and behaviours that make humans become intelligent agents. Recent advances have led to the development of brain-inspired algorithms and models for machine vision. One of the key components of these methods is the utilization of the computational principles underlying biological neurons. Additionally, advanced experimental neuroscience techniques have generated different types of neural signals that carry essential visual information. Thus, there is a high demand for mapping out functional models for reading out visual information from neural signals. Here, we briefly review recent progress on this issue with a focus on how machine learning techniques can help in the development of models for contending various types of neural signals, from fine-scale neural spikes and single-cell calcium imaging to coarse-scale electroencephalography (EEG) and functional magnetic resonance imaging recordings of brain signals.

     

  • loading
  • [1]
    M. O. Ernst, M. S. Banks. Humans integrate visual and haptic information in a statistically optimal fashion. Nature, vol. 415, no. 6870, pp. 429–433, 2002. DOI: 10.1038/415429a.
    [2]
    A. Borst, F. E. Theunissen. Information theory and neural coding. Nature Neuroscience, vol. 2, no. 11, pp. 947–957, 1999. DOI: 10.1038/14731.
    [3]
    R. Q. Quiroga, S. Panzeri. Extracting information from neuronal populations: Information theory and decoding approaches. Nature Reviews Neuroscience, vol. 10, no. 3, pp. 173–185, 2009. DOI: 10.1038/nrn2578.
    [4]
    S. S. Jia, X. Y. Li, T. J. Huang, J. K. Liu, Z. F. Yu. Representing the dynamics of high-dimensional data with non-redundant wavelets. Patterns, vol. 3, no. 3, Article number 100424, 2022. DOI: 10.1016/j.patter.2021.100424.
    [5]
    J. K. Liu, H. M. Schreyer, A. Onken, F. Rozenblit, M. H. Khani, V. Krishnamoorthy, S. Panzeri, T. Gollisch. Inference of neuronal functional circuitry with spike-triggered non-negative matrix factorization. Nature Communications, vol. 8, no. 1, Article number 149, 2017. DOI: 10.1038/s41467-017-00156-9.
    [6]
    J. K. Liu, D. Karamanlis, T. Gollisch. Simple model for encoding natural images by retinal ganglion cells with nonlinear spatial integration. PLoS Computational Biology, vol. 18, no. 3, Article number e1009925, 2022. DOI: 10.1371/journal.pcbi.1009925.
    [7]
    J. J. DiCarlo, D. Zoccolan, N. C. Rust. How does the brain solve visual object recognition? Neuron, vol. 73, no. 3, pp. 415–434, 2012. DOI: 10.1016/j.neuron.2012.01.010.
    [8]
    G. B. Stanley, F. F. Li, Y. Dan. Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus. Journal of Neuroscience, vol. 19, no. 18, pp. 8036–8042, 1999. DOI: 10.1523/JNEUROSCI.19-18-08036.1999.
    [9]
    O. Marre, V. Botella-Soler, K. D. Simmons, T. Mora, G. Tkačik, M. J. Berry II. High accuracy decoding of dynamical motion from a large retinal population. PLoS Computational Biology, vol. 11, no. 7, Article number e1004304, 2015. DOI: 10.1371/journal.pcbi.1004304.
    [10]
    T. Yoshida, K. Ohki. Natural images are reliably represented by sparse and variable populations of neurons in visual cortex. Nature Communications, vol. 11, no. 1, Article number 872, 2020. DOI: 10.1038/s41467-020-14645-x.
    [11]
    V. Botella-Soler, S. Deny, G. Martius, O. Marre, G. Tkačik. Nonlinear decoding of a complex movie from the mammalian retina. PLoS Computational Biology, vol. 14, no. 5, Article number e1006057, 2018. DOI: 10.1371/journal.pcbi.1006057.
    [12]
    Z. F. Yu, J. K. Liu, S. S. Jia, Y. C. Zhang, Y. J. Zheng, Y. H. Tian, T. J. Huang. Toward the next generation of retinal neuroprosthesis: Visual computation with spikes. Engineering, vol. 6, no. 4, pp. 449–461, 2020. DOI: 10.1016/j.eng.2020.02.004.
    [13]
    D. K. Warland, P. Reinagel, M. Meister. Decoding visual information from a population of retinal ganglion cells. Journal of Neurophysiology, vol. 78, no. 5, pp. 2336–2350, 1997. DOI: 10.1152/jn.1997.78.5.2336.
    [14]
    N. Brackbill, C. Rhoades, A. Kling, N. P. Shah, A. Sher, A. M. Litke, E. J. Chichilnisky. Reconstruction of natural images from responses of primate retinal ganglion cells. eLife, vol. 9, Article number e58516, 2020. DOI: 10.7554/eLife.58516.
    [15]
    J. W. Pillow, J. Shlens, L. Paninski, A. Sher, A. M. Litke, E. J. Chichilnisky, E. P. Simoncelli. Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature, vol. 454, no. 7207, pp. 995–999, 2008. DOI: 10.1038/nature07140.
    [16]
    T. Naselaris, R. J. Prenger, K. N. Kay, M. Oliver, J. L. Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, vol. 63, no. 6, pp. 902–915, 2009. DOI: 10.1016/j.neuron.2009.09.006.
    [17]
    S. Nishimoto, A. T. Vu, T. Naselaris, Y. Benjamini, B. Yu, J. L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, vol. 21, no. 19, pp. 1641–1646, 2011. DOI: 10.1016/j.cub.2011.08.031.
    [18]
    R. J. Ellis, M. Michaelides. High-accuracy decoding of complex visual scenes from neuronal calcium responses. [Online], Available: https://biorxiv.org/content/10.1101/271296v1, 2018.
    [19]
    S. Garasto, A. A. Bharath, S. R. Schultz. Visual reconstruction from 2-photon calcium imaging suggests linear readout properties of neurons in mouse primary visual cortex. [Online], Available: https://biorxiv.org/content/10.1101/300392v1, 2018.
    [20]
    S. Garasto, W. Nicola, A. A. Bharath, S. R. Schultz. Neural sampling strategies for visual stimulus reconstruction from two-photon imaging of mouse primary visual cortex. In Proceedings of the 9th International IEEE/EMBS Conference on Neural Engineering, IEEE, San Francisco, USA, pp. 566–570, 2019.
    [21]
    B. C. McCann, M. M. Hayhoe, W. S. Geisler. Decoding natural signals from the peripheral retina. Journal of Vision, vol. 11, no. 10, Article number 19, 2011. DOI: 10.1167/11.10.193.
    [22]
    N. Parthasarathy, E. Batty, W. Falcon, T. Rutten, M. Rajpal, E. J. Chichilnisky, L. Paninski. Neural networks for efficient Bayesian decoding of natural images from retinal neurons. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 6437–6448, 2017.
    [23]
    Y. C. Zhang, S. S. Jia, Y. J. Zheng, Z. F. Yu, Y. H. Tian, S. W. Ma, T. J. Huang, J. K. Liu. Reconstruction of natural visual scenes from neural spikes with deep neural networks. Neural Networks, vol. 125, pp. 19–30, 2020. DOI: 10.1016/j.neunet.2020.01.033.
    [24]
    S. B. Ryu, J. H. Ye, Y. S. Goo, C. H. Kim, K. H. Kim. Decoding of temporal visual information from electrically evoked retinal ganglion cell activities in photoreceptor-degenerated retinas. Investigative Ophthalmology &Visual Science, vol. 52, no. 9, pp. 6271–6278, 2011. DOI: 10.1167/iovs.11-7597.
    [25]
    Y. Kamitani, F. Tong. Decoding the visual and subjective contents of the human brain. Nature Neuroscience, vol. 8, no. 5, pp. 679–685, 2005. DOI: 10.1038/nn1444.
    [26]
    E. Yargholi, G. A. Hossein-Zadeh. Brain decoding-classification of hand written digits from fMRI data employing Bayesian networks. Frontiers in Human Neuroscience, vol. 10, Article number 351, 2016. DOI: 10.3389/fnhum.2016.00351.
    [27]
    K. N. Kay, T. Naselaris, R. J. Prenger, J. L. Gallant. Identifying natural images from human brain activity. Nature, vol. 452, no. 7185, pp. 352–355, 2008. DOI: 10.1038/nature06713.
    [28]
    T. Horikawa, Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, vol. 8, no. 1, Article number 15037, 2017. DOI: 10.1038/ncomms15037.
    [29]
    D. Kleinfeld, L. Luan, P. P. Mitra, J. T. Robinson, R. Sarpeshkar, K. Shepard, C. Xie, T. D. Harris. Can one concurrently record electrical spikes from every neuron in a mammalian brain? Neuron, vol. 103, no. 6, pp. 1005–1015, 2019. DOI: 10.1016/j.neuron.2019.08.011.
    [30]
    A. L. Orsborn, S. Dangi, H. G. Moorman, J. M. Carmena. Closed-loop decoder adaptation on intermediate time-scales facilitates rapid BMI performance improvements independent of decoder initialization conditions. IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 20, no. 4, pp. 468–477, 2012. DOI: 10.1109/TNSRE.2012.2185066.
    [31]
    A. L. Orsborn, H. G. Moorman, S. A. Overduin, M. M. Shanechi, D. F. Dimitrov, J. M. Carmena. Closed-loop decoder adaptation shapes neural plasticity for skillful neuroprosthetic control. Neuron, vol. 82, no. 6, pp. 1380–1393, 2014. DOI: 10.1016/j.neuron.2014.04.048.
    [32]
    M. M. Shanechi, A. L. Orsborn, H. G. Moorman, S. Gowda, S. Dangi, J. M. Carmena. Rapid control and feedback rates enhance neuroprosthetic control. Nature Communications, vol. 8, no. 1, Article number 13825, 2017. DOI: 10.1038/ncomms13825.
    [33]
    A. V. Nurmikko. Approaches to large scale neural recording by chronic implants for mobile BCIs. In Proceedings of the 6th International Conference on Brain-Computer Interface, IEEE, Gangwon, Korea, pp. 1–2, 2018. DOI: 10.1109/IWW-BCI.2018.8311503.
    [34]
    N. A. Steinmetz, C. Koch, K. D. Harris, M. Carandini. Challenges and opportunities for large-scale electrophysiology with Neuropixels probes. Current Opinion in Neurobiology, vol. 50, pp. 92–100, 2018. DOI: 10.1016/j.conb.2018.01.009.
    [35]
    C. Pandarinath, D. J. O′Shea, J. Collins, R. Jozefowicz, S. D. Stavisky, J. C. Kao, E. M. Trautmann, M. T. Kaufman, S. I. Ryu, L. R. Hochberg, J. M. Henderson, K. V. Shenoy, L. F. Abbott, D. Sussillo. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature Methods, vol. 15, no. 10, pp. 805–815, 2018. DOI: 10.1038/s41592-018-0109-9.
    [36]
    M. Rácz, C. Liber, E. Németh, R. Fiáth, J. Rokai, I. Harmati, I. Ulbert, G. Márton. Spike detection and sorting with deep learning. Journal of Neural Engineering, vol. 17, no. 1, Article number 016038, 2020. DOI: 10.1088/1741-2552/ab4896.
    [37]
    M. Emami, M. S. Ardakan, P. Pandit, A. Fletcher, S. Rangan, M. Trumpis, B. Bent, C. H. Chiang, J. Viventi. Low-rank nonlinear decoding of μ-ECoG from the primary auditory cortex. In Proceedings of Conference on Cognitive Computational Neuroscience, Cognitive Computational Neuroscience, Philadelphia, USA, Article number 1276, 2018.
    [38]
    Allen Institute for Brain Science. Allen Brain Atlas SDK: Visual Coding Neuropixels, 2019. [Online], Available: https://allensdk.readthedocs.io/en/latest/visual_coding_neuropixels.html. Accessed on: Feb. 2022.
    [39]
    N. Steinmetz, M. Pachitariu, C. Stringer, M. Carandini, K. Harris. Eight-probe neuropixels recordings during spontaneous behaviors, 2019. [Online], Available: https://figshare.com/articles/dataset/Eight-probe_Neuropixels_recordings_during_spontaneous_behaviors/7739750#:~:text=Eight-probe%20Neuropixels%20recordings%20in%20three%20mice%20of%20spontaneous,times%20and%20the%20cluster%20identity%20of%20each%20spike. Accessed on: Feb. 2022.
    [40]
    S. Musall, M. T. Kaufman, A. L. Juavinett, S. Gluf, A. K. Churchland. Single-trial neural dynamics are dominated by richly varied movements. Nature Neuroscience, vol. 22, no. 10, pp. 1677–1686, 2019. DOI: 10.1038/s41593-019-0502-4.
    [41]
    N. A. Steinmetz, P. Zatka-Haas, M. Carandini, K. D. Harris. Distributed coding of choice, action and engagement across the mouse brain. Nature, vol. 576, no. 7786, pp. 266–273, 2019. DOI: 10.1038/s41586-019-1787-x.
    [42]
    A. Iqbal, P. Dong, C. M. Kim, H. Jang. Decoding neural responses in mouse visual cortex through a deep neural network. In Proceedings of International Joint Conference on Neural Networks, IEEE, Budapest, Hungary, 2019. DOI: 10.1109/IJCNN.2019.8852121.
    [43]
    Y. J. Kim, N. Brackbill, E. Batty, J. Lee, C. Mitelut, W. Tong, E. J. Chichilnisky, L. Paninski. Nonlinear decoding of natural images from large-scale primate retinal ganglion recordings. Neural Computation, vol. 33, no. 7, pp. 1719–1750, 2020. DOI: 10.1162/neco_a_01395.
    [44]
    Q. Xu, J. R. Shen, X. M. Ran, H. J. Tang, G. Pan, J. K. Liu. Robust transcoding sensory information with neural spikes. IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 5, pp. 1935–1946, 2022. DOI: 10.1109/TNNLS.2021.3107449.
    [45]
    M. A. B. Brazier. A History of the Electrical Activity of the Brain: The First Half-Century. Oxford, UK: Macmillan, 1961.
    [46]
    E. S. Kappenman, S. J. Luck. ERP components: The ups and downs of brainwave recordings. The Oxford Handbook of Event-Related Potential Components, E. S. Kappenman, S. J. Luck, Eds., New York, USA: Oxford University Press, pp. 3–30, 2012.
    [47]
    S. J. Luck. An Introduction to the Event-related Potential Technique, 2nd ed., Cambridge, UK: MIT Press, 2014.
    [48]
    B. J. Roach, D. H. Mathalon. Event-related EEG time-frequency analysis: An overview of measures and an analysis of early gamma band phase locking in schizophrenia. Schizophrenia Bulletin, vol. 34, no. 5, pp. 907–926, 2008. DOI: 10.1093/schbul/sbn093.
    [49]
    A. S. Al-Fahoum, A. A. Al-Fraihat. Methods of EEG signal features extraction using linear analysis in frequency and time-frequency domains. International Scholarly Research Notices, vol. 2014, Article number 730218, 2014. DOI: 10.1155/2014/730218.
    [50]
    S. Aggarwal, N. Chugh. Signal processing techniques for motor imagery brain computer interface: A review. Array, vol. 1–2, Article number 100003, 2019. DOI: 10.1016/j.array.2019.100003.
    [51]
    R. Boostani, F. Karimzadeh, M. Nami. A comparative review on sleep stage classification methods in patients and healthy individuals. Computer Methods and Programs in Biomedicine, vol. 140, pp. 77–91, 2017. DOI: 10.1016/j.cmpb.2016.12.004.
    [52]
    E. P. Torres, E. A. Torres, M. Hernández-Álvarez, S. G. Yoo. EEG-based BCI emotion recognition: A survey. Sensors, vol. 20, no. 18, Article number 5083, 2020. DOI: 10.3390/s20185083.
    [53]
    Q. Xiong, X. M. Zhang, W. F. Wang, Y. H. Gu. A parallel algorithm framework for feature extraction of EEG signals on MPI. Computational and Mathematical Methods in Medicine, vol. 2020, Article number 9812019, 2020. DOI: 10.1155/2020/9812019.
    [54]
    P. C. Sen, M. Hajra, M. Ghosh. Supervised classification algorithms in machine learning: A survey and review. In Proceedings of IEM Graph 2018, Springer, Singapore, pp. 99–111, 2020.
    [55]
    S. Kumar, F. Yger, F. Lotte. Towards adaptive classification using Riemannian geometry approaches in brain-computer interfaces. In Proceedings of the 7th International Winter Conference on Brain-computer Interface, IEEE, Gangwon, Korea, 2019.
    [56]
    F. Lotte, L. Bougrain, M. Clerc. Electroencephalography (EEG)-based brain–computer interfaces. Wiley Encyclopedia of Electrical and Electronics Engineering. J. G. Webster, Ed., New York, USA: John Wiley & Sons, Ltd, pp. 1–20, 2015.
    [57]
    S. Y. Chung, H. J. Yoon. Affective classification using Bayesian classifier and supervised learning. In Proceedings of the 12th International Conference on Control, Automation and Systems, IEEE, Jeju, Korea, pp. 1768–1771, 2012.
    [58]
    V. Anand, S. R. Sreeja, D. Samanta. An automated approach for task evaluation using EEG signals. [Online], Available: https://arxiv.org/abs/1911.02966, 2019.
    [59]
    M. Čukić, M. Stokić, S. Simić, D. Pokrajac. The successful discrimination of depression from EEG could be attributed to proper feature extraction and not to a particular classification method. Cognitive Neurodynamics, vol. 14, no. 4, pp. 443–455, 2020. DOI: 10.1007/s11571-020-09581-x.
    [60]
    J. V. Riquelme-Ros, G. Rodríguez-Bermúdez, I. Rodríguez-Rodríguez, J. V. Rodríguez, J. M. Molina-García-Pardo. On the better performance of pianists with motor imagery-based brain-computer interface systems. Sensors, vol. 20, no. 16, Article number 4452, 2020. DOI: 10.3390/s20164452.
    [61]
    S. Garofalo, C. Timmermann, S. Battaglia, M. E. Maier, G. Mediofrontal negativity signals unexpected timing of salient outcomes. Journal of Cognitive Neuroscience, vol. 29, no. 4, pp. 718–727, 2017. DOI: 10.1162/jocn_a_01074.
    [62]
    J. J. Foster, D. W. Sutterer, J. T. Serences, E. K. Vogel, E. Awh. Alpha-band oscillations enable spatially and temporally resolved tracking of covert spatial attention. Psychological Science, vol. 28, no. 7, pp. 929–941, 2017. DOI: 10.1177/0956797617699167.
    [63]
    J. J. Foster, D. W. Sutterer, J. T. Serences, E. K. Vogel, E. Awh. The topography of alpha-band activity tracks the content of spatial working memory. Journal of Neurophysiology, vol. 115, no. 1, pp. 168–177, 2016. DOI: 10.1152/jn.00860.2015.
    [64]
    J. Samaha, J. J. Barrett, A. D. Sheldon, J. J. LaRocque, B. R. Postle. Dissociating perceptual confidence from discrimination accuracy reveals no influence of metacognitive awareness on working memory. Frontiers in Psychology, vol. 7, Article number 851, 2016. DOI: 10.3389/fpsyg.2016.00851.
    [65]
    T. C. Sprague, S. Saproo, J. T. Serences. Visual attention mitigates information loss in small-scale and large-scale neural codes. Trends in Cognitive Sciences, vol. 19, no. 4, pp. 215–226, 2015. DOI: 10.1016/j.tics.2015.02.005.
    [66]
    D. W. Sutterer, J. J. Foster, J. T. Serences, E. K. Vogel, E. Awh. Alpha-band oscillations track the retrieval of precise spatial representations from long-term memory. Journal of Neurophysiology, vol. 122, no. 2, pp. 539–551, 2019. DOI: 10.1152/jn.00268.2019.
    [67]
    H. Keren, G. O′Callaghan, P. Vidal-Ribas, G. A. Buzzell, M. A. Brotman, E. Leibenluft, P. M. Pan, L. Meffert, A. Kaiser, S. Wolke, D. S. Pine, A. Stringaris. Reward processing in depression: A conceptual and meta-analytic review across fMRI and EEG studies. The American Journal of Psychiatry, vol. 175, no. 11, pp. 1111–1120, 2018. DOI: 10.1176/appi.ajp.2018.17101124.
    [68]
    J. Radua, A. Schmidt, S. Borgwardt, A. Heinz, F. Schlagenhauf, P. McGuire, P. Fusar-Poli. Ventral striatal activation during reward processing in psychosis: A neurofunctional meta-analysis. JAMA Psychiatry, vol. 72, no. 12, pp. 1243–1251, 2015. DOI: 10.1001/jamapsychiatry.2015.2196.
    [69]
    A. C. Connolly, J. S. Guntupalli, J. Gors, M. Hanke, Y. O. Halchenko, Y. C. Wu, H. Abdi, J. V. Haxby. The representation of biological classes in the human brain. Journal of Neuroscience, vol. 32, no. 8, pp. 2608–2618, 2012. DOI: 10.1523/JNEUROSCI.5547-11.2012.
    [70]
    C. Allefeld, J. D. Haynes. Multi-voxel pattern analysis. Brain Mapping, A. W. Toga, Ed., Amsterdam, Netherlands: Academic Press, pp. 641–646, 2015.
    [71]
    C. J. Burke, M. Baddeley, P. N. Tobler, W. Schultz. Partial adaptation of obtained and observed value signals preserves information about gains and losses. Journal of Neuroscience, vol. 36, no. 39, pp. 10016–10025, 2016. DOI: 10.1523/JNEUROSCI.0487-16.2016.
    [72]
    C. D. Du, C. Y. Du, L. J. Huang, H. G. He. Reconstructing perceived images from human brain activities with Bayesian deep multiview learning. IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 8, pp. 2310–2323, 2019. DOI: 10.1109/TNNLS.2018.2882456.
    [73]
    T. W. Chen, T. J. Wardill, Y. Sun, S. R. Pulver, S. L. Renninger, A. Baohan, E. R. Schreiter, R. A. Kerr, M. B. Orger, V. Jayaraman, L. L. Looger, K. Svoboda, D. S. Kim. Ultrasensitive fluorescent proteins for imaging neuronal activity. Nature, vol. 499, no. 7458, pp. 295–300, 2013. DOI: 10.1038/nature12354.
    [74]
    A. Giovannucci, J. Friedrich, P. Gunn, J. Kalfon, B. L. Brown, S. A. Koay, J. Taxidis, F. Najafi, J. L. Gauthier, P. C. Zhou, B. S. Khakh, D. W. Tank, D. B. Chklovskii, E. A. Pnevmatikakis. CaImAn an open source tool for scalable calcium imaging data analysis. eLife, vol. 8, Article number e38173, 2019. DOI: 10.7554/eLife.38173.
    [75]
    J. T. Vogelstein, A. M. Packer, T. A. Machado, T. Sippy, B. Babadi, R. Yuste, L. Paninski. Fast nonnegative deconvolution for spike train inference from population calcium imaging. Journal of Neurophysiology, vol. 104, no. 6, pp. 3691–3704, 2010. DOI: 10.1152/jn.01073.2009.
    [76]
    T. Deneux, A. Kaszas, G. Szalay, G. Katona, T. Lakner, A. Grinvald, B. Rózsa, I. Vanzetta. Accurate spike estimation from noisy calcium signals for ultrafast three-dimensional imaging of large neuronal populations in vivo. Nature Communications, vol. 7, no. 1, Article number 12190, 2016. DOI: 10.1038/ncomms12190.
    [77]
    J. Friedrich, P. C. Zhou, L. Paninski. Fast online deconvolution of calcium imaging data. PLoS Computational Biology, vol. 13, no. 3, Article number e1005423, 2017. DOI: 10.1371/journal.pcbi.1005423.
    [78]
    B. F. Grewe, D. Langer, H. Kasper, B. M. Kampa, F. Helmchen. High-speed in vivo calcium imaging reveals neuronal network activity with near-millisecond precision. Nature Methods, vol. 7, no. 5, pp. 399–405, 2010. DOI: 10.1038/nmeth.1453.
    [79]
    P. Rupprecht, S. Carta, A. Hoffmann, M. Echizen, A. Blot, A. C. Kwan, Y. Dan, S. B. Hofer, K. Kitamura, F. Helmchen, R. W. Friedrich. A database and deep learning toolbox for noise-optimized, generalized spike inference from calcium imaging. Nature Neuroscience, vol. 24, no. 9, pp. 1324–1337, 2021. DOI: 10.1038/s41593-021-00895-5.
    [80]
    L. Theis, P. Berens, E. Froudarakis, J. Reimer, M. R. Rosón, T. Baden, T. Euler, A. S. Tolias, M. Bethge. Benchmarking spike rate inference in population calcium imaging. Neuron, vol. 90, no. 3, pp. 471–482, 2016. DOI: 10.1016/j.neuron.2016.04.014.
    [81]
    Z. Q. Wei, B. J. Lin, T. W. Chen, K. Daie, K. Svoboda, S. Druckmann. A comparison of neuronal population dynamics measured with calcium imaging and electrophysiology. PLoS Computational Biology, vol. 16, no. 9, Article number e1008198, 2020. DOI: 10.1371/journal.pcbi.1008198.
    [82]
    S. M. Tang, Y. M. Zhang, Z. H. Li, M. Li, F. Liu, H. F. Jiang, T. S. Lee. Large-scale two-photon imaging revealed super-sparse population codes in the V1 superficial layer of awake monkeys. eLife, vol. 7, Article number e33370, 2018. DOI: 10.7554/eLife.33370.
    [83]
    S. Schoenmakers, M. Barth, T. Heskes, M. Van Gerven. Linear reconstruction of perceived images from human brain activity. NeuroImage, vol. 83, pp. 951–961, 2013. DOI: 10.1016/j.neuroimage.2013.07.043.
    [84]
    Y. Fujiwara, Y. Miyawaki, Y. Kamitani. Modular encoding and decoding models derived from Bayesian canonical correlation analysis. Neural Computation, vol. 25, no. 4, pp. 979–1005, 2013. DOI: 10.1162/NECO_a_00423.
    [85]
    U. Güçlü, M. A. J. Van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, vol. 35, no. 27, pp. 10005–10014, 2015. DOI: 10.1523/JNEUROSCI.5023-14.2015.
    [86]
    H. G. Wen, J. X. Shi, Y. Z. Zhang, K. H. Lu, J. Y. Cao, Z. M. Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral Cortex, vol. 28, no. 12, pp. 4136–4160, 2018. DOI: 10.1093/cercor/bhx268.
    [87]
    R. M. Cichy, A. Khosla, D. Pantazis, A. Torralba, A. Oliva. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific Reports, vol. 6, no. 1, Article number 27755, 2016. DOI: 10.1038/srep27755.
    [88]
    K. Fukushima, S. Miyake. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Proceedings of Competition and Cooperation in Neural Nets, Springer, Kyoto, Japan, pp. 267–285, 1982.
    [89]
    M. Riesenhuber, T. Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience, vol. 2, no. 11, pp. 1019–1025, 1999. DOI: 10.1038/14819.
    [90]
    K. Cho, B. Van Merrienboer, D. Bahdanau, Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST@EMNLP 2014, the 8th Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, pp.103–111, 2014.
    [91]
    D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, J. Schmidhuber. Flexible, high performance convolutional neural networks for image classification. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Spain, pp. 1237–1242, 2011.
    [92]
    I. Goodfellow, Y. Bengio, A. Courville. Deep Learning, Cambridge, UK: MIT Press, 2016.
    [93]
    S. Hochreiter, J. Schmidhuber. Long short-term memory. Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. DOI: 10.1162/neco.1997.9.8.1735.
    [94]
    A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet classification with deep convolutional neural networks. Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017. DOI: 10.1145/3065386.
    [95]
    Y. LeCun, Y. Bengio. Convolutional networks for images, speech, and time series. The Handbook of Brain Theory and Neural Networks. M. A. Arbib, Ed., Cambridge, USA: MIT Press, pp. 255–258, 1998.
    [96]
    D. H. Hubel, T. N. Wiesel. Receptive fields of single neurones in the cat′s striate cortex. The Journal of Physiology, vol. 148, no. 3, pp. 574–591, 1959. DOI: 10.1113/jphysiol.1959.sp006308.
    [97]
    B. A. Richards, T. P. Lillicrap, P. Beaudoin, Y. Bengio, R. Bogacz, A. Christensen, C. Clopath, R. P. Costa, A. De Berker, S. Ganguli, C. J. Gillon, D. Hafner, A. Kepecs, N. Kriegeskorte, P. Latham, G. W. Lindsay, K. D. Miller, R. Naud, C. C. Pack, P. Poirazi, P. Roelfsema, J. Sacramento, A. Saxe, B. Scellier, A. C. Schapiro, W. Senn, G. Wayne, D. Yamins, F. Zenke, J. Zylberberg, D. Therien, K. P. Kording. A deep learning framework for neuroscience. Nature Neuroscience, vol. 22, no. 11, pp. 1761–1770, 2019. DOI: 10.1038/s41593-019-0520-2.
    [98]
    A. Al-Saegh, S. A. Dawwd, J. M. Abdul-Jabbar. Deep learning for motor imagery EEG-based classification: A review. Biomedical Signal Processing and Control, vol. 63, Article number 102172, 2021. DOI: 10.1016/j.bspc.2020.102172.
    [99]
    S. Bhattacharya, P. K. R. Maddikunta, Q. V. Pham, T. R. Gadekallu, S. R. Krishnan S, C. L. Chowdhary, M. Alazab, J. Piran. Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey. Sustainable Cities and Society, vol. 65, Article number 102589, 2021. DOI: 10.1016/j.scs.2020.102589.
    [100]
    K. Fiok, W. Karwowski, E. Gutierrez, M. Saeidi, A. M. Aljuaid, M. R. Davahli, R. Taiar, T. Marek, and B. D. Sawyer. A study of the effects of the COVID-19 pandemic on the experience of back pain reported on twitter® in the United States: A natural language processing approach. International Journal of Environmental Research and Public Health, vol. 18, no. 9, Article number 4543, 2021. DOI: 10.3390/ijerph18094543.
    [101]
    K. T. Schütt, H. E. Sauceda, P. J. Kindermans, A. Tkatchenko, K. R. Müller. SchNet – A deep learning architecture for molecules and materials. The Journal of Chemical Physics, vol. 148, no. 24, Article number 241722, 2018. DOI: 10.1063/1.5019779.
    [102]
    S. Roy, I. Kiral-Kornek, S. Harrer. ChronoNet: A deep recurrent neural network for abnormal EEG identification. In Proceedings of the 17th Conference on Artificial Intelligence in Medicine in Europe, Springer, Poznan, Poland, pp. 47–56, 2019.
    [103]
    B. Krause, I. Murray, S. Renals, L. Lu. Multiplicative LSTM for sequence modelling. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 2016.
    [104]
    Y. J. Miao, M. Gowayyed, F. Metze. EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding. In Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding, IEEE, Scottsdale, USA, pp. 167–174, 2015. DOI: 10.1109/ASRU.2015.7404790.
    [105]
    W. F. Wiggins, F. Kitamura, I. Santos, L. M. Prevedello. Natural language processing of radiology text reports: Interactive text classification. Radiology:Artificial Intelligence, vol. 3, no. 4, Article number e210035, 2021. DOI: 10.1148/ryai.2021210035.
    [106]
    J. H. Tan, Y. Hagiwara, W. Pang, I. Lim, S. L. Oh, M. Adam, R. S. Tan, M. Chen, U. R. Acharya. Application of stacked convolutional and long short-term memory network for accurate identification of CAD ECG signals. Computers in Biology and Medicine, vol. 94, pp. 19–26, 2018. DOI: 10.1016/j.compbiomed.2017.12.023.
    [107]
    Y. J. Zheng, S. S. Jia, Z. F. Yu, J. K. Liu, T. J. Huang. Unraveling neural coding of dynamic natural visual scenes via convolutional recurrent neural networks. Patterns, vol. 2, no. 10, Article number 100350, 2021. DOI: 10.1016/j.patter.2021.100350.
    [108]
    M. Mirza, S. Osindero. Conditional generative adversarial nets. [Online], Available: https://arxiv.org/abs/1411.1784, 2014.
    [109]
    A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2016.
    [110]
    E. L. Denton, S. Chintala, A. Szlam, R. Fergus. Deep generative image models using a Laplacian pyramid of adversarial networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS, Montreal, Canada, pp. 1486–1494, 2015.
    [111]
    X. T. Wang, K. Yu, S. X. Wu, J. J. Gu, Y. H. Liu, C. Dong, Y. Qiao, C. C. Loy. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision, Springer, Munich, Germany, pp. 63–79, 2018.
    [112]
    S. Bozinovski, A. Fulgosi. The influence of pattern similarity and transfer learning upon training of a base perceptron b2. In Proceedings of Symposium Informatica, Gdańsk, Poland, pp. 3–121, 1976.
    [113]
    S. Bozinovski. Reminder of the first paper on transfer learning in neural networks, 1976. Informatica, vol. 44, no. 3, pp. 291–302, 2020.
    [114]
    L. Y. Pratt, S. Thrun. Machine Learning-Special Issue on Inductive Transfer. Berlin, Germany: Springer, 1997.
    [115]
    L. Y. Pratt. Discriminability-based transfer between neural networks. In Proceedings of the 5th International Conference on Neural Information Processing Systems, Morgan Kaufmann Publishers Inc., Denver, Colorado, pp. 204–211, 1992.
    [116]
    Y. J. Zhang, T. Bu, J. Y. Zhang, S. M. Tang, Z. F. Yu, J. K. Liu, T. J. Huang. Decoding pixel-level image features from two-photon calcium signals of macaque visual cortex. Neural Computation, vol. 34, no. 6, pp. 1369–1397, 2022. DOI: 10.1162/neco_a_01498.
    [117]
    G. H. Shen, K. Dwivedi, K. Majima, T. Horikawa, Y. Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, vol. 13, Article number 21, 2019. DOI: 10.3389/fncom.2019.00021.
    [118]
    G. H. Shen, T. Horikawa, K. Majima, Y. Kamitani. Deep image reconstruction from human brain activity. PLoS Computational Biology, vol. 15, no. 1, Article number e1006633, 2019. DOI: 10.1371/journal.pcbi.1006633.
    [119]
    K. Han, H. G. Wen, J. X. Shi, K. H. Lu, Y. Z. Zhang, D. Fu, Z. M. Liu. Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex. NeuroImage, vol. 198, pp. 125–136, 2019. DOI: 10.1016/j.neuroimage.2019.05.039.
    [120]
    G. St-Yves, T. Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, IEEE, Miyazaki, Japan, pp. 1054–1061, 2018. DOI: 10.1109/SMC.2018.00187.
    [121]
    K. Gerven, U. Güçlü, L. Ambrogioni, Y. Güçlütürk, M. A. J. Van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage, vol. 181, pp. 775–785, 2018. DOI: 10.1016/j.neuroimage.2018.07.043.
    [122]
    Y. Gerven, U. Güçlü, K. Seeliger, S. Bosch, R. Van Lier, M. Van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 4249–4260, 2017.
    [123]
    R. VanRullen, L. Reddy. Reconstructing faces from fMRI patterns using deep generative neural networks. Communications Biology, vol. 2, no. 1, Article number 193, 2019. DOI: 10.1038/s42003-019-0438-y.
    [124]
    C. J. Markiewicz, K. J. Gorgolewski, F. Feingold, R. Blair, Y. O. Halchenko, E. Miller, N. Hardcastle, J. Wexler, O. Esteban, M. Goncavles, A. Jwa, R. Poldrack. The OpenNeuro resource for sharing of neuroscience data. eLife, vol. 10, Article number e71774, 2021. DOI: 10.7554/eLife.71774.
    [125]
    J. L. Teeters, K. D. Harris, K. J. Millman, B. A. Olshausen, F. T. Sommer. Data sharing for computational neuroscience. Neuroinformatics, vol. 6, no. 1, pp. 47–55, 2008. DOI: 10.1007/s12021-008-9009-y.
    [126]
    Z. T. Lu, Y. X. Ku. NeuroRA: A Python toolbox of representational analysis from multi-modal neural data. Frontiers in Neuroinformatics, vol. 14, Article number 563669, 2020. DOI: 10.3389/fninf.2020.563669.
    [127]
    G. Berto. Benchmark Bundle Segmentation. [Online], Available: https://doi.org/10.25663/BRAINLIFE.PUB, 2021.
    [128]
    M. Hanke, Y. O. Halchenko, P. B. Sederberg, S. J. Hanson, J. V. Haxby, S. Pollmann. PyMVPA: A Python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics, vol. 7, no. 1, pp. 37–53, 2009. DOI: 10.1007/s12021-008-9041-y.
    [129]
    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, É. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, vol. 12, no. 85, pp. 2825–2830, 2011.
    [130]
    S. Sonnenburg, H. Strathmann, S. Lisitsyn, V. Gal, F. J. I. García, W. Lin, S. De, C. Y. Zhang, frx, tklein23, E. Andreev, JonasBehr, sploving, P. Mazumdar, C. Widmer, P. D. Zora, G. De Toni, S. Mahindre, A. Kislay, K. Hughes, R. Votyakov, khalednasr, S. Sharma, A. Novik, A. Panda, E. Anagnostopoulos, L. Pang, A. Binder, serialhex, B. Esser. Shogun-toolbox/shogun: Shogun 6.1.0. Zenodo, 2017. [Online], Available: https://zenodo.org/record/1067840#.Yn9UufkyZ_p.
    [131]
    S. Huang, L. Sun, M. Yousefnezhad, M. L. Wang, D. Q. Zhang. Temporal information guided generative adversarial networks for stimuli image reconstruction from human brain activities. IEEE Transactions on Cognitive and Developmental Systems, to be publised, 2021.
    [132]
    R. Beliy, G. Gaziv, A. Hoogi, F. Strappini, T. Golan, M. Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 6514–6524, 2019.
    [133]
    D. Li, C. D. Du, H. G. He. Semi-supervised cross-modal image generation with generative adversarial networks. Pattern Recognition, vol. 100, Article number 107085, 2020. DOI: 10.1016/j.patcog.2019.107085.
    [134]
    C. D. Du, C. Y. Du, H. Wang, J. P. Li, W. L. Zheng, B. L. Lu, H. G. He. Semi-supervised deep generative modelling of incomplete multi-modality emotional data. In Proceedings of the 26th ACM International Conference on Multimedia, Lisboa, Portugal, pp. 108–116, 2018. DOI: 10.1145/3240508.3240528.
    [135]
    X. Cai, F. P. Nie, W. D. Cai, H. Huang. Heterogeneous image features integration via multi-modal semi-supervised learning model. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, pp. 1737–1744, 2013. DOI: 10.1109/ICCV.2013.218.
    [136]
    Z. X. Zhang, F. Ringeval, B. Dong, E. Coutinho, E. Marchi, B. Schüller. Enhanced semi-supervised learning for multimodal emotion recognition. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, Shanghai, China, pp. 5185–5189, 2016. DOI: 10.1109/ICASSP.2016.7472666.
    [137]
    K. Ibayashi, N. Kunii, T. Matsuo, Y. Ishishita, S. Shimada, K. Kawai, N. Saito. Decoding speech with integrated hybrid signals recorded from the human ventral motor cortex. Frontiers in Neuroscience, vol. 12, Article number 221, 2018. DOI: 10.3389/fnins.2018.00221.
    [138]
    M. Schrimpf, J. Kubilius, H. Hong, N. J. Majaj, R. Rajalingham, E. B. Issa, K. Kar, P. Bashivan, J. Prescott-Roy, F. Geiger, K. Schmidt, D. L. K. Yamins, J. J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv, Article number 407007, 2020.
    [139]
    N. Kriegeskorte, J. Diedrichsen. Peeling the onion of brain representations. Annual Review of Neuroscience, vol. 42, pp. 407–432, 2019. DOI: 10.1146/annurev-neuro-080317-061906.
    [140]
    J. V. Haxby, A. C. Connolly, J. S. Guntupalli. Decoding neural representational spaces using multivariate pattern analysis. Annual Review of Neuroscience, vol. 37, pp. 435–456, 2014. DOI: 10.1146/annurev-neuro-062012-170325.
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(3)

    用微信扫码二维码

    分享至好友和朋友圈

    Article Metrics

    Article views (107) PDF downloads(92) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return