Fully integrated multi-mode optoelectronic memristor array for diversified in-sensor computing | Nature Nanotechnology

Nature Nanotechnology (2024)Cite this article

Metrics details

In-sensor computing, which integrates sensing, memory and processing functions, has shown substantial potential in artificial vision systems. However, large-scale monolithic integration of in-sensor computing based on emerging devices with complementary metal–oxide–semiconductor (CMOS) circuits remains challenging, lacking functional demonstrations at the hardware level. Here we report a fully integrated 1-kb array with 128 × 8 one-transistor one-optoelectronic memristor (OEM) cells and silicon CMOS circuits, which features configurable multi-mode functionality encompassing three different modes of electronic memristor, dynamic OEM and non-volatile OEM (NV-OEM). These modes are configured by modulating the charge density within the oxygen vacancies via synergistic optical and electrical operations, as confirmed by differential phase-contrast scanning transmission electron microscopy. Using this OEM system, three visual processing tasks are demonstrated: image sensory pre-processing with a recognition accuracy enhanced from 85.7% to 96.1% by the NV-OEM mode, more advanced object tracking with 96.1% accuracy using both dynamic OEM and NV-OEM modes and human motion recognition with a fully OEM-based in-sensor reservoir computing system achieving 91.2% accuracy. A system-level benchmark further shows that it consumes over 20 times less energy than graphics processing units. By monolithically integrating the multi-functional OEMs with Si CMOS, this work provides a cost-effective platform for diverse in-sensor computing applications.

This is a preview of subscription content, access via your institution

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

$29.99 / 30 days

cancel any time

Subscribe to this journal

Receive 12 print issues and online access

$259.00 per year

only $21.58 per issue

Buy this article

Prices may be subject to local taxes which are calculated during checkout

The data that support the findings of this study are available from the corresponding authors on request. More data are also presented in the Supplementary Information. Source data are provided with this paper.

All the codes that support the findings of this study are available from the corresponding authors upon reasonable request.

Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).

Article PubMed CAS Google Scholar

Wu, N. Neuromorphic vision chips. Sci. China Inf. Sci. 61, 060421 (2018).

Article Google Scholar

Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671 (2020).

Article Google Scholar

Dai, S. et al. Emerging iontronic neural devices for neuromorphic sensory computing. Adv. Mater. 35, 2300329 (2023).

Article CAS Google Scholar

Du, J. et al. A robust neuromorphic vision sensor with optical control of ferroelectric switching. Nano Energy 89, 106439 (2021).

Article CAS Google Scholar

Li, G. et al. Photo-induced non-volatile VO2 phase transition for neuromorphic ultraviolet sensors. Nat. Commun. 13, 1729 (2022).

Article PubMed PubMed Central CAS Google Scholar

Jang, H. et al. An atomically thin optoelectronic machine vision processor. Adv. Mater. 32, 2002431 (2020).

Zhang, Z. et al. In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array. Nat. Commun. 13, 6590 (2022).

Article PubMed PubMed Central CAS Google Scholar

Zhao, R. et al. A framework for the general design and computation of hybrid neural networks. Nat. Commun. 13, 3427 (2022).

Article PubMed PubMed Central CAS Google Scholar

Wang, Y. et al. Optoelectronic synaptic devices for neuromorphic computing. Adv. Intell. Syst 3, 2000099 (2021).

Article Google Scholar

Zhang, J., Dai, S., Zhao, Y., Zhang, J. & Huang, J. Recent progress in photonic synapses for neuromorphic systems. Adv. Intell. Syst. 2, 1900136 (2020).

Song, S. et al. Recent progress of optoelectronic and all-optical neuromorphic devices: a comprehensive review of device structures, materials, and applications. Adv. Intell. Syst. 3, 2000119 (2021).

John, R. A. et al. Reconfigurable halide perovskite nanocrystal memristors for neuromorphic computing. Nat. Commun. 13, 2074 (2022).

Article PubMed PubMed Central CAS Google Scholar

Wang, T. et al. Reconfigurable neuromorphic memristor network for ultralow-power smart textile electronics. Nat. Commun. 13, 7432 (2022).

Article PubMed PubMed Central CAS Google Scholar

Pi, L. et al. Broadband convolutional processing using band-alignment-tunable heterostructures. Nat. Electron. 5, 248–254 (2022).

Article CAS Google Scholar

Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

Article PubMed CAS Google Scholar

Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2022).

Article PubMed Google Scholar

Sun, L. et al. In-sensor reservoir computing for language learning via two-dimensional memristors. Sci. Adv. 7, eabg1455 (2021).

Article PubMed PubMed Central CAS Google Scholar

Wu, X. et al. Wearable in-sensor reservoir computing using optoelectronic polymers with through-space charge-transport characteristics for multi-task learning. Nat. Commun. 14, 468 (2023).

Article PubMed PubMed Central CAS Google Scholar

Lao, J. et al. Ultralow-power machine vision with self-powered sensor reservoir. Adv. Sci. 9, 2106092 (2022).

Article Google Scholar

Zhong, Y. et al. A memristor-based analogue reservoir computing system for real-time and power-efficient signal processing. Nat. Electron. 5, 672–681 (2022).

Article Google Scholar

Liang, X. et al. Physical reservoir computing with emerging electronics. Nat. Electron. 7, 193–206 (2024).

Article Google Scholar

Moon, J. et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. Nat. Electron. 2, 480–487 (2019).

Article Google Scholar

Portner, K. et al. Analog nanoscale electro-optical synapses for neuromorphic computing applications. ACS. Nano. 15, 14776–14785 (2021).

Article PubMed CAS Google Scholar

Hu, L. et al. All-optically controlled memristor for optoelectronic neuromorphic computing. Adv. Funct. Mater. 31, 2005582 (2021).

Tan, H. et al. An optoelectronic resistive switching memory with integrated demodulating and arithmetic functions. Adv. Mater. 27, 2797–2803 (2015).

Article PubMed CAS Google Scholar

Chen, J. Y. et al. Dynamic evolution of conducting nanofilament in resistive switching memories. Nano Lett. 13, 3671–3677 (2013).

Article PubMed CAS Google Scholar

Simanjuntak, F. M., Panda, D., Wei, K. H. & Tseng, T. Y. Status and prospects of ZnO-based resistive switching memory devices. Nanoscale Res. Lett. 11, 368 (2016).

Xu, N. et al. Characteristics and mechanism of conduction/set process in TiN/ZnO/Pt resistance switching random-access memories. Appl. Phys. Lett. 92, 232112 (2008).

Zhou, Z., Pei, Y., Zhao, J., Fu, G. & Yan, X. Visible light responsive optoelectronic memristor device based on CeOx/ZnO structure for artificial vision system. Appl. Phys. Lett. 118, 191103 (2021).

Wang, T.-Y. et al. Reconfigurable optoelectronic memristor for in-sensor computing applications. Nano Energy 89, 106291 (2021).

Wang, W. et al. CMOS backend-of-line compatible memory array and logic circuitries enabled by high performance atomic layer deposited ZnO thin-film transistor. Nat. Commun. 14, 6079 (2023).

Article PubMed PubMed Central CAS Google Scholar

Wang, Z. et al. Vacancy driven surface disorder catalyzes anisotropic evaporation of ZnO (0001) polar surface. Nat. Commun. 13, 5616 (2022).

Article PubMed PubMed Central CAS Google Scholar

Lanza, M. et al. Recommended methods to study resistive switching devices. Adv. Electron. Mater. 5, 1800143 (2019).

Kuzum, D., Yu, S. & Wong, H. S. Synaptic electronics: materials, devices and applications. Nanotechnology 24, 382001 (2013).

Russo, P., Xiao, M., Liang, R. & Zhou, N. Y. UV-induced multilevel current amplification memory effect in zinc oxide rods resistive switching devices. Adv. Funct. Mater. 28, 1706230 (2018).

Oh, I., Pyo, J. & Kim, S. Resistive switching and synaptic characteristics in ZnO/TaON-based RRAM for neuromorphic system. Nanomaterial 12, 2185 (2022).

Seo, S. et al. Artificial optic-neural synapse for colored and color-mixed pattern recognition. Nat. Commun. 9, 5106 (2018).

Article PubMed PubMed Central Google Scholar

Yilmaz, A., Javed, O. & Shah, M. Object tracking. ACM Comput. Surv. 38, 1–45 (2006).

Article Google Scholar

Wang, S. et al. Networking retinomorphic sensor with memristive crossbar for brain-inspired visual perception. Natl Sci. Rev. 8, nwaa172 (2020).

Article PubMed PubMed Central Google Scholar

Download references

This work was in part supported by National Natural Science Foundation of China 92264201 (J.T.), 62025111 (H.W.) and 62174095 (Y.W.), National Key R&D Programme of China 2021ZD0109901 (L.F.), China Postdoctoral Science Foundation 2021M701845 (H.H.), the XPLORER Prize (H.W.), Tsinghua University Initiative Scientific Research Programme and the Centre of Nanofabrication, Tsinghua University. We are also grateful to J. Chen from the University of Zurich and J. Tang from Boston College for their valuable suggestions on the manuscript.

School of Integrated Circuits, Beijing Innovation Center for Integrated Circuits, Tsinghua University, Beijing, China

Heyi Huang, Xiangpeng Liang, Yuyan Wang, Jianshi Tang, Yuankun Li, Yiwei Du, Wen Sun, Peng Yao, Xing Mou, Feng Xu, Yuyao Lu, Zhengwu Liu, Zhixing Jiang, Ruofei Hu, Ze Wang, Qingtian Zhang, Bin Gao, He Qian & Huaqiang Wu

Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China

Heyi Huang, Yuyan Wang, Jianshi Tang, Bin Gao, Lu Fang, Qionghai Dai & Huaqiang Wu

Integrated Circuit Advanced Process R&D Center and Key Laboratory of Fabrication Technologies for Integrated Circuits, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, China

Heyi Huang & Huaxiang Yin

Department of Electronic Engineering, Tsinghua University, Beijing, China

Jianing Zhang, Jinzhi Zhang & Lu Fang

Beijing National Laboratory for Condensed Matter Physics, Institute of Physics, Chinese Academy of Sciences, Beijing, China

Jianlin Wang & Xuedong Bai

You can also search for this author in PubMed Google Scholar

H.H., Y.W. and J.T. conceived and designed the experiments. H.H. contributed to the OEM system fabrication. W.S., H.H., J.W. and X.B. contributed to the TEM analysis. H.H., X.M., Z.J. and R.H. participated in the measurements. Y. Li performed a simulation of image noise reduction under the supervision of Q.Z.; Jianing Zhang and Jinzhi Zhang performed a simulation of object tracking under the supervision of L.F. and Q.D.; and X.L., Y. Li and H.H. performed experiments on RC. F.X. and Y. Lu provided theoretical support. Y.D., P.Y., Z.L., Z.W., B.G., H.Y. and H.Q. analysed the data and discussed the results. H.H., X.L., Y.W. and J.T. wrote the manuscript. All authors discussed the results and commented on the manuscript. Y.W., J.T. and H.W. supervised the project.

Correspondence to Yuyan Wang, Jianshi Tang or Huaqiang Wu.

The authors declare no competing interests.

Nature Nanotechnology thanks Jang-Sik Lee and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Figs. 1–30, Notes 1–7 and Tables 1–3.

Real-time object tracking for the moving Tsinghua University school bus, based on the OEM system, achieving an accuracy of 96.1%.

The targets are the school buses (B1, B2) and motorcycle (M). Both buses and motorcycle can be well tracked during the driving process, and the motorcycle can also be well re-identified after being obscured by the school bus.

Raw datasets for the multi-functional OEM mode. Figure 2b–d shows the D-OEM mode dataset. Figure 2f–h shows the EM mode dataset. Figure 2j–l shows the NV-OEM mode dataset.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

Huang, H., Liang, X., Wang, Y. et al. Fully integrated multi-mode optoelectronic memristor array for diversified in-sensor computing. Nat. Nanotechnol. (2024). https://doi.org/10.1038/s41565-024-01794-z

Download citation

Received: 23 May 2023

Accepted: 26 August 2024

Published: 08 November 2024

DOI: https://doi.org/10.1038/s41565-024-01794-z

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative