Support of Intelligent Rooms by Image Processing

Fig.1 - Illustration of an intelligent room with a CCD camera at the ceiling

Fig.2 - Extraction of finger region

Fig.3 - Recognition of the number of fingers

Fig.4 - Recognition of pointing object

Fig.5 - Example of recognized number of fingers

Fig.6 - Recognizing pointing direction

Fig.7 - Recognized pointing direction

In recent years, rooms with intelligence are studied. This kind of rooms is becoming important not only for medical or welfare use, but also at offices or ordinary houses. Obviously, the man-machine interface to communicate human intention is a substantial part for building up intelligent rooms. This report shows an example of applying image processing to support intelligent rooms.

Outline of Gesture Recognition for Intelligent Room

In order to realize a human-friendly interface, it is necessary that an intelligent room finds a person autonomously and understands his or her gesture. So as to make the procedures easier and more robust, we set a CCD camera at the ceiling.

We focus on the gesture by a hand. First the system finds a person with the CCD camera, and then extracts the hand region using color information. Then, the finger regions are extracted from the hand region and the number of the fingers is recognized. Pointing can be defined that the number of extracted fingers is one and the hand is not moving.

Extraction of a Hand Region

We recognize a person by detecting his or her head. In order to reduce calculation cost, the search area of the hand is set up from the center of a head region. We convert an image to hsv space to decrease the influence of brightness, and extract a hand region in the 2-dimensional space of hue and saturation.

Recognition of the Number of Fingers

The number of fingers serves as the important element of the method of intention transfer. The procedure for recognizing the fingers is illustrated in Fig.2. The extracted hand region (1) is reduced and finger regions are eliminated (2). Then the region is expanded (3) to the original size. The difference of two regions (1)-(3) represents finger regions (4). It is considered that the number of finger regions is the number of fingers.

Recognition of Pointing

We suppose that an object is pointed when the number of fingers is one and the hand stops. The direction of the principal axis of the finger region is obtained as the pointing direction. Suppose that the positions of target objects in the room are given. Then the direction for each object from the hand is calculated. The angle between the direction of each object and obtained pointing direction is calculated. The object with smallest angle is recognized as the pointed object, if the angle is less than some threshold value. For example, object A is recognized as the pointed object in Fig.3.

The experiment for extracting the hand region was performed as shown in Fig.4. Head region and hand region are extracted, and the number of fingers is recognized as one.

In our experiments with 200 samples for three subjects the recognition rates for the number of fingers (0-5) are as high as 96.0% on average (Fig.5).

The experiments to point to two objects were performed for 200 images as shown in Fig.6 and 7. The angle between the objects was set to 60°, 45°, 30° and 15°, and the average recognition rate was 87.1%. When the angle was larger than 30°, the recognition was almost correct.

Future Plans

So far, methods for recognizing the number of fingers and the pointing direction were developed and evaluated by experiments. The future works include the construction of a more practical gesture recognition system, and an intelligent room itself.