README.md

Goal

The main goal of the project is extending Ukrainian tonal dictionary. At first, we tried to achieve it by looking at words similar to ones with known tonality. Word2vec and LexVec models are used to find similar words. Then we built NN classifier and used word embeddings and existing tonal dictionary to train it.

General

split_to_chunks/subsample.py - is used to take a piece of files so it can be read with notepad:utils.py - has some useful methods and folders paths

Split to sentences:

We have different sources of text: csv, txt and wiki. There are different files to preprocess them.

for csv: (each item is a news or article)

split raw csv data to chunks and save as chunks: split_to_chunks/csv_to_pd_chunks.py
Result items are saved in data\chunks

Using LexVec and word2vec models to predict the tone of the word

predict/build_joined_vect_dict.py - is used to concatenate two models: LexVec and word2vecpredict/predict.py - predict, save the whole set, save subsamplepredict/save_best.py - take best negative and positive candidates

Credits

Oleksandr Marykovskyi, Vyacheslav Tykhonov provided the seed dictionarySerhiy Shehovtsov wrote the code and ran numerous experimentsOles Petriv created and trained neural network modelVsevolod Dyomkin proof-read the result and prepared it for publishingDmitry Chaplinsky led the project :)