Vending Machine Interface Design Supported by Image Processing

I propose a user interface - which is suitable for vending machines - , that does not need the user to get in physical contact with the device. The user can choose a product by rotating his head. He is notified about the current state of the selection process by a LED matrix. The confirmation method for the selection can be selected from the followings: hand raising, mouth opening, changing facial expression, blinking. After that, smiling in the cameras will make the machine deliver into delivery box

I use a system based on image processing, in which a PC and an MCU is used for recognizing the user's intentions and to control the vending machine. This system can recognize the user's intention and control the machine based on that. The software, running on the PC, processes the image stream using the OpenCV framework and gives order to the MCU, that handles the machine control.

The two camera is needed to compute the distance of the user. I wrote a program that makes the calibration process easier and the theoretical background of calibration is also presented.

The face of the user is a very important information source, thus the program fits a model on it. The changes in the model tells about the user's emotions, his head's orientation, his blinking. I tried out several algorithms, built models based on different facial databases, but in the end FaceTracker, which gave the best fitting results, was built into the system.

To utilize the model points an SVM was used, trained and tested on different annotated facial databases.

I examined numerous user interaction methods for possible implementation, particularly the ones based on the face model and the hand.

The hardware that implements the machine control is an AVR based embedded system with USB connectivity that communicates with software running on the PC. It controls the machine by imitating button pressings.

The main software that processes the images was written in C++ and Qt under LINUX. It has multiple threads as it has different tasks and needs a lot of computing capacity. I present the software in details like its construction, performance, data flow and installation.

In the development phase numerous small programs were created in order to try out different algorithms. These are also presented with their purpose and a picture.