Around time when I was finishing the school, I attended a course on Scientology (Applied Religious Philosophy). What was interesting is that Ron Hubbard advised to visualize all unknown phrases – to make a clay demo, draw a picture (use pictorial dictionaries), make a movie/cartoon etc…

As I was studying Chinese, this sounded double-interesting to me (some hieroglyphs look like what they represent, say, hieroglyph Sun developed from sun picture).

Anyway, some time passed, and I had an idea that it would be cool if some computer program could do that – visualize text, or, say, convert it to a movie/cartoon automatically. I went to Kyiv Institute of Cinema and Theatre, department of animation and talked with students about the idea. I said that as there is machine translation programs (services) like Google Translate, so it could be great to make a program which converts a text into a … movie … automatically.

It was a “Boom”, “Chain reaction” started. Students retold the idea to teachers. After some time Russian movie company took this idea. Bazelevs Innovations in collaboration with ABBYY Compreno got 5 000 000 dollars from Skolkovo Fund to make a program to convert text into audiovisual sequence. Technology was called “FilmLanguage”. On their site FilmLanguage.net they were looking for Talented Unity3d Programmer. This is how I got interested in Unity3d. Bazelevs Innovations decided that such program works best for comedy scenarios, so they collaborated with Paramount Comedy on site obosmeysia.ru (translated as “laughing out loud”) where a game using FilmLanguage was run.

Users could type Russian phrases – commands for two robots, and they would perform actions like text-to-speech synthesis or playing animations like walk, wave hand etc. When a button was pressed, a text was converted to audiovisual sequence, and then, supposedly, automatically published on Youtube (this is their channel – Chromosapiens ). Soon, this project disappeared and seems to be closed now.

But I took the idea to use Unity3d and made a small research to find out what can be done to make such program using free Unity3d assets.

To make Unity3d program for text conversion to audio-visual sequence we would require:

• Unity3d game engine, of course

• SharpNLP. It can do part-of-speech tagging and similar simple but useful tasks for English texts. SharpNLP needs some changes to work with Unity3d.

• Carnegie Mellon University Motion Capture library. It consists of over 2500 animations (many are similar) for Mecanim/humanoid rig (though I’ld expect to see these in LEGO-technic style).

• Text-to-speech library. I use this one.

• Code. Now, having all these, are you able to make a program which converts text into audio-visual sequence?