RETHINKING COMMON PRACTICES IN DEEP LEARNING

Rethinking Common Practices in Deep Learning

Deep learning has revolutionized the way machine learning is used in various domains, improving previous approaches substantially in many cases. However, as theory is still lacking, practitioners often base their models on heuristics and intuition.
We will examine and shed new light on several common practices and beliefs in Deep Learning: the effect of batch size on generalization, the purpose of the classifier in the last layer, and the role of batch-normalization. Both theoretical and empirical arguments will be used to show that commonly used practices are often misguided ― and how can they be improved.
* PhD candidate in EE under the supervision of Prof. Daniel Soudry and Prof. Nir Ailon.