On Diffusion Kernels, Histograms, and Arabic Writer Identification

We catch up with Yanir Seroussi, a graduate student in Computer Science, on how he took third place in the ICFHR 2012 - Arabic Writer Identification Competition. After signing up for a Kaggle account over a year ago, he finally decided to give one of the competitions 'just a quick try'. Famous last words...

What was your background prior to entering this challenge?

I'm currently in the final phases of my PhD, which is in the areas of natural language processing and user modelling. Even though I address some predictive modelling problems in my thesis, I've never done any image processing work, though it did help to have some background knowledge in machine learning and statistics.

What made you decide to enter?

I signed up to Kaggle over a year ago but never used my account. Recently, I started thinking about what I want to do once I graduate, and somehow bumped into Phil Brierley's blog. This inspired me to give one of the smaller competitions "just a quick try", which ended up consuming a lot of my free time...

What preprocessing and supervised learning methods did you use?

My most successful submission was based on SVMs. I joined the competition quite late and didn't have time to play with blending techniques, which seem to be a key component of many winners' approaches.

As to preprocessing, I don't have any prior knowledge about image processing, so I only briefly experimented with one idea that didn't require much knowledge: converting all the images to texts with freely-available OCR software, which is based on the idea that the same OCR errors would appear for the same writers. While using this as a standalone feature yielded some interesting results, it didn't improve accuracy when used in conjunction with the provided features.

What was your most important insight into the data?

At first I played around with SVMs and the commonly-used kernels (linear, polynomial, RBF and sigmoid). Then I remembered a recent paper that was presented at the ACL conference, about using character histograms for authorship attribution (http://aclweb.org/anthology-new/P/P11/P11-1030.pdf). Since the provided features were given in the form of histograms, I figured that the same techniques would be applicable here. And indeed, using SVMs with a diffusion kernel proved to yield the most significant performance boost.

Were you surprised by any of your insights?

I wouldn't call it "surprised", but I was a bit frustrated by the apparent lack of correlation between cross-validation results on the training data and the accuracy on the validation set. This is probably because each of the 204 writers had only two training paragraphs (all containing the same text), while the test instances were a third paragraph with different content. So any form of cross validation yielded a training subset that was very different from the full training set, and a test subset that obviously couldn't contain the "third paragraphs".

Which tools did you use?

Mostly LibSVM and SVMLight (for some brief experiments with transductive SVMs that didn't go well). I used Python to parse the feature files, run the libraries, and produce the final results.

What have you taken away from this competition?

A better understanding of SVMs and the conclusion that I still have a lot to learn 🙂