Projects

Over the years, I’ve been involved in many cool projects, and I can not possibly list them all on this page. However, I have summarized a few of my more public projects below.

SpeechRec

From 2013 to 2015, I researched speech-based recommender systems at the Graz University of Technology. The final prototype of the developed system was called “SpeechRec”, a conversational natural-language knowledge-based recommender system – in other words: a virtual shopping assistant.

SpeechRec was designed to work similarly to a humanoid shopping assistant, and used extensive domain knowledge to extract nuanced information from a wide array of inputs (more than 800 key phrases). In parallel to natural language processing, which allowed SpeechRec to distinguish e.g., between “a little cheaper” and “a lot cheaper”, the system employed paralinguistic analysis to react to the speaker’s “tone” to detect priorities and further enhance conflict resolution.

Internally, SpeechRec used a modified version of Simon (see below) employing CMU’s PocketSphinx decoder to provide speech recognition, and the OpenEAR framework for paralinguistic analysis. The avatar was designed and rendered in Blender 3D (a custom avatar library was written to animate real-time conversations) and voiced using MARY TTS. Among the many tools and libraries used to develop SpeechRec were MongoDb (to store domain knowledge), Stanford NLP and OpenNLP (various pre-processing, knowledge engineering), and Heritrix (knowledge engineering).

Available command plug-ins contain the basics of keyboard- and mouse simulation and starting programs but also a complete dialog system with voice output, an integration with KDE’s PIM layer, both JSON and D-Bus connectors to third party software and many more.

Elaborate context layer allows to employ contextual information to improve recognition accuracy or change commands based on the current situation.

Mobile clients for Meego and BlackBerry.

An acoustic model generator for power users called Sam, a distributed tool for large sample acquisitions, SSC, and more.

Due to it’s modular design, Simon also lends itself to integration in other projects and has been used successfully in various research projects including an EU project investigating the use of a voice controlled caregiving robot.

Sadly, after swapping the projector for a much more conventional large TV as part of moving to a much smaller apartment, Melissa was replaced with a slightly customized installation of XBMC and is thus no longer maintained.
If you are interested, the code is available on request.