Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Gradient Decomposition Methods for Training Neural Networks with Non-Ideal Synaptic Devices



Junyun Zhao, Siyuan Huang, Osama Yousuf, Yutong Gao, Brian Hoskins, Gina Adam


While promising for high capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average out gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on MNIST with only 3 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementations in memristor accelerators.
Frontiers in Neuroscience


artificial intelligence, resistive switch, matrix decomposition


Zhao, J. , Huang, S. , Yousuf, O. , Gao, Y. , Hoskins, B. and Adam, G. (2021), Gradient Decomposition Methods for Training Neural Networks with Non-Ideal Synaptic Devices, Frontiers in Neuroscience, [online],, (Accessed May 29, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created November 22, 2021, Updated November 29, 2022