Natural language processing tools for tamil

01 Aug 2018

Language is a beautiful thing, Tamil as a Language is even more beautiful. Need for learning other languages like English is for commerce, communication among other things. Languages that have written form and grammar have survived longer than languages that are without it. Technology is playing the role of written form and grammar now. English being the most spoken and written language in the world has the benefit of lot of technology and tools developed to understand language better. English has lot of good systems developed as open source as well as by the companies like Google,Facebook, Microsoft etc for Speech to Text, Syntactic Parser, Stemmer, Lemmatizer, Parts of Speech Tagger etc. This was due to years of data collection, research, availability of computing power and to great extent due to Deep Learning. We have to develop most of these Natural Language Understanding tools for Tamil, there are some that exist but they are in very very early stages. When a language is beautiful it is complicated too, English has 26 alphabets and Tamil has 247 alphabets, and the grammar rules are different between English and Tamil. In most cases we are limited by the lack of available data for us(tech community) to build these tools. These tools when built should be open sourced under Apache 2.0 license, so it is available for public to use for free. I have listed few key activities that have to be performed for this to be successful

Website to be created to list out mission statement and project status. This can be as simple as github pages.

Android app to be developed for data collection and even gamify it. For e.g. for Speech to Text similar to how Mozilla recently asked people to donate their voices to develop open source Speech to Text System for English. They open sourced the code and the trained model.

Data Access. There are research libraries that have digitized lot of books, dailies, news papers etc. If that data is publicly available, it can spawn multiple research similar to how Project Gutenberg did. One low hanging fruit for algorithm would be a vectorization engine for Tamil words.

Community involvement. This effort will not succeed without community involvement, particularly without student and teacher community for data collection.

Tech community involvement to develop this as open-source.

Eminent linguistic advisors to advise the correct approach for Tamil.

Technology advisors on right technology choices to solving problems.

Advisors to help registering and managing this as a non-profit and may be even seeking grants and sponsorship. This can start as a small thing, but if it has to be taken seriously it has to be registered as a non-profit organization.