Deep Neural Networks (DNNs) have been adopted in many systems because of their higher classification accuracy. While progress in achieving large scale, highly accurate DNNs has been made, however, the huge time and energy requirement for both inference and training pose a challenge to any DNN implementation. The large DNN size causes significant energy and area due to massive memory accesses and computations. In this thesis, we propose a new DNN architecture, LightNN, which replaces the multiplications to one shift or a constrained number of shifts and adds. Thus, LightNN inference uses hardware-efficient shift-based operations, and needs fewer memory accesses due to the fewer bits to represent each weight. Our experiment using image datasets show that LightNNs can achieve much higher hardware efficiency with a slight accuracy loss.

LightNNs constrain all the weights of DNNs to be a limited combination (denoted as $k\in\{1,2\}$) of powers of 2, and by varying the $k$ one can obtain a group of LightNNs with different levels of accuracy and energy cost. To provide even more design flexibility, the $k$ for each convolutional filter can be optimally chosen instead of being fixed for every filter. In this thesis, we formulate the selection of $k$ to be differentiable, and describe model training for determining $k$-based weights on a per-filter basis. Over 46 FPGA-design experiments involving eight configurations and four data sets reveal that lightweight neural networks with a flexible $k$ value (dubbed FLightNNs) fully utilize the hardware resources on Field Programmable Gate Arrays (FPGAs), our experimental results show that FLightNNs can achieve $2\times$ speedup when compared to lightweight NNs with $k=2$, with only $0.1\%$ accuracy degradation. Compared to a 4-bit fixed-point quantization, FLightNNs achieve higher accuracy and up to $2\times$ inference speedup, due to their lightweight shift operations. In addition, our experiments also demonstrate that FLightNNs can achieve higher computational energy efficiency for ASIC implementation.

Binarized Neural Networks (BNNs) quantize the weights and activations to 1 bit, which has lower precision than LightNNs and FLightNNs, and therefore can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN’s energy efficiency. In this thesis, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.

Terms of Use

The supplemental file or files you are about to download were provided to ProQuest by the author as part of a dissertation or thesis.
The supplemental files are provided "AS IS" without warranty.
ProQuest is not responsible for the content, format or impact on the supplemental file(s) on our system.
in some cases, the file type may be unknown or may be a .exe file.
We recommend caution as you open such files.

Copyright of the original materials contained in the supplemental file is retained by the author and your access to the supplemental files is subject to the ProQuest Terms and Conditions of use.

Depending on the size of the file(s) you are downloading, the system may take some time to download them. Please be patient.