Yann Lecun’s lectures at the Collège de France : n°2

This is a short report of the second course of Yann Lecun at the Collège de France in Paris. Don’t look for the N°1, unfortunately, we were not there … But now, and until the end of his class, there will be each week a report of his lesson and of the seminar which is following.

This class is a very special event for Deep Learning in France, since the Collège de France is probably the most prestigious French place for research. It is a very old institution where all fields are represented. It does not deliver diplomas, and is open to everyone. The fact that Yann Lecun was invited to give lessons this year show that all the academic field recognize Deep Learning as a central domain in research. We were not the only one realizing that this was a special event, and the largest lecture hall was full.

Last Friday, for the second lesson, the theme was « Convolutional Networks ». It was followed by a very interesting conference by Stéphane Mallat (ENS, Paris) about « Mathematical mysteries of the Convolutional Neural Networks »

Convolutional Networks : Yann leCun

The aim of this course was first to present the main ideas of the convolution operator and the convolutional networks, and then to present the different improvements to this field from the 90s until now.

Yann LeCun in the Collège de France

Yann LeCun explained why the convolution is a relevant mathematical operation for image, because of invariance by translation. He showed the effects of the different convolutional masks on the original pictures. And he explained how this could be useful to recognize the MNIST handwritten digits. We reproduce here some of his figures.

On the middle of these figures we can see the input. On the left, we see six pictures which are the first hidden layer, the convolution of the input with six different masks. Then on the right of these one we see the Layer-3 of the network, and then the Layer-5 which is used to recognize the digit.

He explained how being able to recognize a digit makes us able to find all the digits in an input by moving the convolutional masks. Here, we see how the networks recognize the two digits, whatever their positions are, if they are not one over an other.

He showed the architecture of the most powerfull networks, performing the best scores on the ImageNet competitions, which are huge.

Finally, he talked about all the applications of these methods : image labelling, medical imaging, generative models.

The video of this lesson will be available on : Video (in French). For the moment, only the audio track is available.

Mathematical mysteries of Convolutional Networks : Stéphane Mallat

Stéphane Mallat, well known for his research on wavelet transform, tried to analyze what are the mathematical reasons of the effectiveness of Convolutional Networks.

Stéphane Mallat in the Collège de France

According to him, the challenge of a lot of scientific problems is to understand very complex problems (particles, molecules, …), and it is well-known that the global properties of these systems are consequences of its invariants (like in the Noether theorems in physics).

Recognizing a pattern on a picture is a very complex problem : the picture can be seen as a high dimensional vector, and all the picture containing a given pattern, let’s say a face, are not close at all (for the metric of this space), or separable from the other pictures by a linear classifier.

But even if this problem seems to be highly complex, there are a lot of transformations that can change the image such that the patterns are still recognizable. Some of these transformations are simple : translations, rotations, re-scaling, … But some of them are much more complex (distortions, …).

These transformations are the invariants of the system, as the invariant of a physical system. There are the structure of the system, and the reason why we are able to classify the pictures.

If we are able to describe these transformations, then a classifier should learn to be classify in the same way an original picture and a picture transformed by one of these invariants.

Stéphane Mallat tried to study these transformations, and then showed that both Deep Convoluional Neural Networks and some algorithms using wavelets transform and scattering are using the same invariants.

This would be a great progress in research in Deep Learning since it would be the beginning of a theory explaining the effectiveness of these networks. Indeed, the wavelet transform is a well knwon mathematical topic, and showing a similarity between these two methods would be very interesting.

The video of this lesson will be available on : Video (in French). For the moment, only the audio track is available.

The next lesson will be Friday at 11am, and will be about using neural networks in practice. It will be followed by a talk of Yann Ollivier about « Optimizing and training Recurrent Networks »