TSCL Is The Newest Addition To Deep Learning Algorithms

One of the newest developments in deep learning is the curriculum learning, where algorithms are trained to learn on a meaningful order in increasing complexity rather than just examples being fed to them.

A new study by researchers who worked with OpenAI has dug deep into curriculum learning. Their model, called as Teacher-Student Curriculum Learning (TSCL), aims to be a game changer in learning subtasks associated with major deep learning tasks. This article looks into the technicality surrounding TSCL.

What Is TSCL?

In the context of curriculum learning, a Teacher algorithm gives subtasks to the Student algorithm in the increasing order of complexity, while the Student performs them and returns a score. This is gradually repeated until all tasks are performed by the Student successfully.

So, as and when the Student learns and masters a particular task, the Teacher assigns more probability on the subsequent task ahead thus focussing less on the current one as it has been learnt completely.

Repetition of performing tasks here makes learning faster. One important point to be noted here is the Teacher algorithm also learns information simultaneously along with the Student algorithm. This forms the basis for TSCL algorithm.

Matiisen et.al, the creators of TSCL, say “While an obvious choice for optimization criteria would have been the performance in the final task, initially the Student might not have any success in the final task and this does not provide any meaningful feedback signal to the Teacher. Therefore we choose to maximize the sum of performances in all tasks. The assumption here is that in curriculum learning the final task includes the elements of all previous tasks, therefore good performance in the intermediate tasks usually leads to good performance in the final task”

On this front, PODMPs are generally solved using RL but the training itself takes time and becomes iterative. Thus, researchers derive insights from the popular ‘multi armed non-stationary bandit problem’ and bring out the following new algorithms to incorporate in TSCL.

Online Algorithm

Naive Algorithm

Window Algorithm

Sampling Algorithm

All these algorithms are tweaked with respect to improving scores as well as keeping a check on the number of times a task has been performed in the Teacher-Student setup.

Thus, Matiisen et.al, consider this problem for analysis. As mentioned earlier, batch training POMDP is taken as the TSCL method here. The addition is carried along two parameters: 1-dimensional curriculum teaching and 2-dimensional curriculum teaching. In the former, tasks related to finding the maximum number of digits in the number obtained after addition, while the latter includes another criterion i.e., taking the length of numbers separately on top of finding decimal digits.

Results show that TSCL fares better than conventional LSTM. In fact, with more expected return, it is even faster.

Popular video game Minecraft, was also experimented with respect to reinforcement learning strategies. By using Microsoft’s Project Malmo with OpenAI Gym, a 5-step Curriculum Learning is created by Matiisen and team. This generates random mazes in Minecraft where the learning agent carefully navigates and learns the maze environment. (A detailed account of the Minecraft training can be found here.)

Even in this case, Minecraft agent learns faster with every run iteration. If a five-step curriculum is performed without considering each step, the agent terribly fails in learning the environment, which supports the researchers’ critique on selecting only the final task.

Comment

While this study has paved way for using TSCL in a handful of applications, it is again yet to stand with standard reinforcement learning and supervised learning algorithms. Nonetheless, TSCL will alleviate complexities in algorithms at every stage (dividing into sub-tasks etc.) thus reducing the burden on computing power and similar resources.

Provide your comments below

I research and cover latest happenings in data science. My fervent interests are in latest technology and humor/comedy (an odd combination!). When I'm not busy reading on these subjects, you'll find me watching movies or playing badminton.