Hardware Demonstration of Feedforward Stochastic Neural Networks with Fast MTJ-based p-bits
Feedforward networks form the backbone of deep learning, used in deep multilayer perceptrons, convolutional neural networks, and deep belief networks. Even though stochastic activations of deep neural networks are highly desired, they are often avoided due to their heavy computational costs in traditional hardware. This paper presents the hardware implementation of inference in such deep feedforward stochastic networks with the fastest nanodevice-based probabilistic bits (p-bit) demonstrated to date. The stochasticity of low-barrier stochastic magnetic tunnel junctions (sMTJ) is used to create p-bits, routed back to a Field Programmable Gate Array (FPGA) to build a hybrid CMOS+sMTJ computer. Unlike commonly implemented Boltzmann-Ising type undirected networks, feedforward networks require carefully ordered updates. We achieve such ordered updating by generating sequenced signals from the FPGA back to the p-bits. In addition, each neuron in a given layer is updated concurrently by the fluctuations of sMTJs, achieving layer-by-layer parallelism. Fast sMTJ-based p-bits are demonstrated using specially engineered sMTJs with in-plane anisotropy with microsecond fluctuations, three orders of magnitude faster than all previous demonstrations. We experimentally perform inference on two feedforward Belief Networks including a medical diagnosis example. We improve earlier projections based on the experimentally-demonstrated sMTJ parameters illustrating the potential of scaled-up versions of our system.