There’s been a great increase in interest recently on the topic of deep neural networks (DNNs). The term DNN is general and somewhat ambiguous. A regular neural network has one layer of input nodes, one layer of hidden processing nodes, and one layer of output nodes. Deep neural networks have multiple hidden layers of processing nodes.

There are specific kinds of DNNs that have complicated architectures. They have names like convolutional neural networks (CNNs), recurrent neural networks (RNNs), residual neural networks (ResNets), and many more. In order to understand these exotic DNNs, in my opinion it’s necessary to have a solid understanding of basic DNNs — neural networks with multiple hidden layers but nothing fancy with regards to connection architecture.

So I spent a few hours coding up basic deep neural network from scratch, using the C# language. It wasn’t too difficult, but it was time-consuming, and there were many, many different design choices.