User:Aps43

Robot Lab Spring 2012. Worked on OpenCV and object tracking. Below is a description of the work I've done, how to run my code, and some points on future development. I would suggest running the code alongside reading the descriptions, as this is very visual in nature and seeing the demos will help in understanding.

Contents

Summary

I have worked on various methods of analyzing input from a single camera, such as a webcam. All of my implementations have been in C++ using OpenCV. The algorithms implemented are as follows:

Background subtraction

Optical flow

Color tracking

Blob tracking

Histogram of Oriented Gradients (untested)

I have made a demo of each of the algorithms. Data from each of these demos can be pulled and used for various robot tasks, like movement, mapping, etc. Each demo can simply be run with no arguments. They will pop up one or more windows with a camera image and some other data. Hit the escape key to exit.

Installing cvBlob

Optical Flow

My implementation of this algorithm uses edge detection to find "points of interest" and then tracks these points frame to frame. It also keeps track of how far each point moved between each pair of frames, which is an indicator of perceived speed.

For each tracked point, the demo will draw a line from where the point was the previous frame to where the point is in the current frame. The color of the line depends on how far the point moved, red being not far, green being medium distance, and blue meaning very far.

Assuming all of the tracked points are the same distance from the camera, the length of the line drawn (meaning how far the point traveled) is an indicator of speed. More movement means faster. However, if the points are all different distances from the camera, the line length is a function of speed and distance because distant objects will travel less per frame than close ones moving at the same speed.

Using these properties, it is possible to do some image segmentation based on approximate distance. For example, assuming everything but the camera is stationary, if a set of ten points are moving quickly while the rest of the points are moving slowly, those ten points are likely part of the same object and that object is close to the camera. The same principal can be applied to moving objects. If a set of points are moving in the opposite direction than all of the other points, then they are part of an object moving in a different direction.

Color Tracking

A very simple method of determining the location of an object of a certain color. A color range is given, and any pixel within that range is tracked. The location of the object is given as the center of all of the tracked pixels (the average of their x and y coordinates). This can lead to some wonky behavior, such as when there are two trackable objects. In that scenario, it will claim there is a single object with a location directly between the two objects.

The demo will display two windows of interest. One will display a black and white image where the white indicates a pixel within the tracked range and black is an untracked pixel. The second window will paint a line wherever the tracked object is located. The tracked object is the paintbrush.

Blob Tracking and Location Estimation

A more advanced color tracking algorithm. This one segments the tracked color into "blobs". It then tracks each blob individually. This algorithm also has some approximate location estimation, based on distance from the camera and angle from the camera.

The distance calculation is based on a few factors. Firstly, the size of the object to be tracked needs to be known, and this value is currently hard coded in the demo. Secondly, some calibration needs to be done. The calibration is done by holding the object a set distance from the camera. By knowing how large the object appears at a set distance from the camera, the focal length of the camera can be calculated. The focal length needs to be known in order to perform the distance calculation.

The angle estimation is much simpler, but also needs claibration. By holding the object at a set angle from the front of the camera, we can convert the location of the object to the angle. The program assumes there's a linear rate of change between angle and pixel location.

The demo displays each blob on the screen with a rectangle around it. To calibrate the camera for distance estimation, hold the object one meter away from the camera and press the d key. To calibrate for angle estimation, hold the object at a 20 degree angle to the side of the camera and press the a key.

Once both distance and angle are calibrated, object tracking will start. The tracking is connected to ROS_Interface.cpp. The ROS interface is currently just stubbed out with print statements. It will print how far the robot has to move forward to reach the object or how far it has to turn to be directly facing the object. These stubs can later be changed to fully integrate with ROS, robot simulation software, or some other code.

If the ROS interface stubs are modified to directly control a robot and the blob code is unchanged, the robot will have some simple "follow the object" behavior.

Background Subtraction

In a nutshell, this algorithm detects movement by looking at recent changes in the image.

When started, the demo will show a completely black window. When the image changes (meaning there was movement), the parts of the image that changed will show up as white. Over time, it will "learn" a new background, so if the camera turns and the whole screen goes white (because everything moved), then it will adjust to the new background and turn the window black again.

Histogram of Oriented Gradients

This is a method of tracking multiple people within a scene. Due to specific technical issues, this code was not working on my computer. It is a known issue with the HOGDescriptor class. This code may very well work on your machine.

Because it was not working for me, I cannot provide much documentation. The code is provided as is and for reference purposes.