Neural Arithmetic Logic Unit

Share

Neural networks are being widely adopted across disciplines with regards to computation. One fundamental flaw with neural networks is that they are unable to count. The neural arithmetic logic unit (NALU) was created to extend the neural accumulator model (NAC) to allow for better arithmetic computation.

Neural networks have been shown to compute very well with the training sets they are given. However, outside of the training sets, even simple tasks like learning the scalar identity function is impossible outside of the training sets. However, with the introduction of NACs and NALUs, the ability for neural networks to train and retain mathematically complex models outside of their original training sets has been increased.

Building a NALU involves building a NAC. NACs outputs the linear transformation of the inputs. Simple arithmetic operations such as additions and subtractions are performed. In Python, the following code is provided for the NAC. W is the transformation matrix, M is the matrix of interest, A is the accumulation vector, and G is the learned sigosmodial gate. For some freedom, one can determine the standard of deviation for the layers. Note that one will need TensorFlow and NumPY for the implementation:

Now for the NALU code. The NALU code will take the output from the NAC cells where the layers and standard of deviation as well as the number of inputs is given. The NALU will learn the weighted sums of the two subcells. The NALU will output what is passed through the NACs.

This NALU can now be used in Python-based applications for any type of training and then used in general. The NALU implemented comes from this Python implementation. The NALU can be trained and tested to do the scalar identity function. If you want to learn more about NALUs, check out Andrew Trask’s paper. These concepts may be useful for challenges related to machine learning and neural networks.