Abstract

A new projection pattern control technique is presented in an attempt to solve the problem whereby an image having an ideal intensity distribution cannot be photographed when measurement conditions, such as object color or object surface reflection, change. The proposed technique can adjust the intensity distribution of a projection pattern automatically, according to changes in the measurement conditions. An image with an ideal intensity distribution can then be obtained in a short time, approximately three projections on average. Thus, the speed, robustness, and practicality of 3-D image measurement can be improved.

In order to achieve high measurement accuracy by these methods, it is important to detect stripe order correctly. Binary methods have a high detection accuracy of stripe order, but to compute the stripe order, many projections are needed and the measurement takes time. Non-binary methods can obtain 3-D information of objects in a shorter time than binary methods, because a large amount of information is acquired in one projection. In order to detect the stripe order of a projection pattern correctly in a non-binary method, it is necessary that the observation pattern image being photographed has a large intensity distribution and not be saturated. However, measurement conditions, such as surface color or the surface reflection factor of the object, can change the ideal observation pattern image and an ideal intensity may not be obtained. As such, the detection accuracy of stripe order decreases or measurement becomes impossible.

Either the projected light automatic control method or the camera parameter automatic control method may be used to automatically obtain a reflection pattern image having an ideal intensity distribution [12

]. The automatic mode of the camera, which adjusts the shutter speed and iris diaphragm, for example, automatically, is considered. In automatic mode, the intensity value of one specific point (or two or more points) specified beforehand can be controlled, and the intensity distribution of the entire image can be distributed within ideal limits. However, since the camera cannot extract the measurement object automatically from the observation pattern image, the intensity of the stripes reflected by the measurement object cannot be distributed within required limits. As such, it is difficult to automatically obtain the ideal reflection pattern image for 3-D measurement.

The present paper proposes a technique for the automatic control of projection pattern intensity distribution in addition to camera parameter regulation. Using this technique, the intensity distribution of a projection pattern can be adjusted automatically according to surface color or reflection factor distribution, and the ideal projection pattern for 3-D measurement is generated automatically.

2. Measurement principle

The intensity distribution of an ideal observation pattern image is given in Fig. 1. In this figure, Imax is the maximum and Imin is the minimum intensity value of the image system. For such an intensity distribution, highly precise stripe order detection is possible, because most of the intensity information is used.

Fig. 1. Ideal histogram of the observation pattern image

However, an image with an ideal histogram might not be obtained, due to the influence of the measurement environment, such as the color and surface reflection factor of the measurement object. Figure 2 is an example of this phenomenon. Figure 2(a) is the original image of two objects with different surface reflection factors. Figure 2(b) is the observation pattern image obtained when an intensity-modulation stripe pattern is projected on the objects. Figure 2(c) is the intensity distribution of the AA’ line on the left object, and Fig. 2(d) is the intensity distribution of the BB’ line on the right object. Figures 2(e) and 2(f) are histograms of the left and right objects, respectively. The intensity range of the stripes on the left object is from grade 30 to grade 200, which is ideal for measurement. In comparison, the intensity range of the right object is from grade 0 to grade 90, which is very narrow. Detection of the stripe order from the stripe intensity change in the narrow intensity range shown in Fig. 2(d) is very difficult. Therefore, in order to obtain the intensity distribution of the observation pattern image in the required range, it is necessary to adjust the intensity distribution of the projection pattern.

Fig. 2. Observation images and intensity distributions of two objects with different surface reflection factors: (a) Original image, (b) Observation image, (c) Intensity distribution of line AA’, (d) Intensity distribution of line BB’, (e) Histogram of left object, and (f) Histogram of right object.

In order to obtain an ideal image within the shortest time, an intensity control algorithm of the projection pattern, shown in Fig. 3, is proposed. The explanation of each process is as follows:

Step 1: The initial projection pattern is a uniform-intensity no-stripe pattern. The intensity takes the mean value of the projection intensity range (which is set at 128 in this paper).

Step 2: The initial projection pattern reflected by the object is photographed as an RGB color image. This is called the observation pattern image.

Step 3: The measurement object is extracted from the observation pattern image using the background comparison method.

Step 4: In every pixel of the extracted object image, the color and intensity distributions are detected [14

]. The channel of the observation pattern image intensity maximum is chosen to be the measurement channel of the pixel by

Il(i,j)=Max{IR(i,j),IG(i,j),IB(i,j)}

(1)

where l is the measurement channel, l∊{R,G,B} ; (i, j) are the image coordinates of the current pixel; and IR, IG and IB are the intensities of the R, G and B channels, respectively, of the current pixel.

Fig. 3. Intensity control of projection pattern flowchart

Since the frequent changes in the measurement channel reduces the measurement stability in a certain area, the choice of the measurement channel is corrected so that distribution of the measurement channel is changed as little as possible. For example, when l=R, the correction can be written as

where δ is the size of the area around the pixel (i, j), which influences the channel choice of the pixel (i, j), the value is settled by image size. For the case in which the image size is 1024×768 pixels, we generally set δ=3~7. λ is a coefficient used to compare the intensity value of the measurement channel to that of the other channel. For example, R channel was chosen as the measurement channel by formula (1) for pixel (i, j). However, most of the measurement channel of the pixels around pixel (i, j) are G, and the intensity value of the G channel of pixel (i, j) is not smaller than λ times the intensity value of the R channels. In this case, the measurement channel of the pixel (i, j) is changed into G from R. Since the value of λ is small, the stability of the choice of the measurement channel is higher. However, if the value of λ is too small, the intensity range of the measurement image will become small. We generally set λ=0.5~1.

Step 5: The purpose of camera parameter regulation is to roughly adjust the intensity range of the observation pattern image. Fine regulation is realized by intensity control of the projection pattern. The intensity distribution of the observation pattern image in the measurement channel is compared with an ideal intensity distribution, and the camera parameter is adjusted if the difference exceeds a certain value.

Step 6: In the present study, the iris diaphragm is fixed and the shutter speed is adjusted as follows:

Vn=Vn−1−C(S−Sn−1)

(5)

where n is the number of projections (n=0 when projecting the initial pattern), Vn is the shutter speed when carrying out the n-th photograph, C is a constant, Sn is the stripe maximum intensity value of the observation pattern image for the n-th projection, and S is the desired value of the stripe maximum intensity in the observation pattern image. In the present paper, the position at which the area of the histogram reaches 2% from the right is defined as the stripe maximum intensity value for control. This is done in order to suppress the influence of noise or highlights.

When using a general-purpose camera, the camera parameter cannot be set to an arbitrary value. For this reason, a shutter speed value near the value calculated using formula (5) is used. The error produced by this approximation is corrected by the k2 clause of formula (6).

Step 7: The maximum intensity of stripes of the projection pattern is calculated as follows:

The maximum intensity of stripes of the 1st projection pattern is decided by the clauses of k1 and k2. The clause of k1 is a simple proportionality control component. The clause of k2 is a compensation component accompanying regulation of the shutter speed of the camera, the value of which is 0 when there is no change in the camera shutter speed. The maximum intensity of stripes of the 2nd projection is decided by the clause of k3, which is a simple proportionality control. In order to shorten the control time and minimize the regular error, the maximum intensity of stripes of the 3rd projection henceforth uses the original proportionality integral control, which is shown in the clause of k4. The technique of differential control is often employed when the goal is high-speed control, but since the influence on stability is large, this control method is not employed herein.

Steps 8–10: The processes from 8 to 10 of Fig. 3 are straightforward and therefore their explanation is not included herein.

Step 11: When the object has a nonuniform surface, the stripe intensity of the observation pattern images of the same order are not necessarily the same. In this case, it is necessary to use the technique of Ref. [9

where Il(i, j) and I0(i, j) are the intensities of the measurement image and the initial observation pattern image, respectively, M(n) is the intensity modulation function of the projection pattern, O(x,y) is the reflection function of the object, and k and k’ are constants. The stripe intensity of the corrected image is dependent only on the intensity modulation function of the projection pattern.

Step 12: Finally, the 3-D representation can be obtained by the fringes intensity of corrected measurement image.

3. Experimental results and evaluation

Figure 4 shows an example of a measurement using the abovementioned intensity control technique. The measurement object is a wooden box with complicated surface color and discontinuous form. Experimental data are listed in Table 1. The projector is a liquid crystal projector having a spatial resolution of 1024×768 pixels and an intensity resolution of 8 bits. The camera is an 8-bit 1024×768 pixels 3-CCD color camera, and the photography speed is 7.5 frames per second. The personal computer used in the present study had a 2-GHz Athlon™ 64 CPU, 1 GB of RAM and the OS of Microsoft Windows XP Professional.

Table 1. Experimental Data for Intensity Control of Projection Pattern

n

Vn

Pn

Sn

|S-Sn|

0

1/30

128

141

69

1

1/15

186

191

19

2

1/15

205

212

2

Table 1. Experimental Data for Intensity Control of Projection Pattern

Figure 4(a) is the initial projection pattern, the intensity of which is 128. Figures 4(b), 4(c), and 4(d) are the initial observation pattern image, the measurement channel image and the single-channel measurement image, respectively. Figure 4(i) is the histogram of the measurement image. The maximum intensity value S0 (red line) of the object is approximately 141, and the difference between S0 and the ideal value S (green line, S=210) is fairly large. Figure 4 (e) is the projection pattern adjusted using the proposed technique. The projection intensity is from 40 to 205. Figure 4(f) is the observation pattern image measured using the projection pattern of Fig. 4 (e). Figure 4(g) is the measurement channel image, and Fig. 4(h) is the measurement image of Fig. 4(f). Figure 4(j) is the histogram of Fig. 4(h), and S2 (red line) and S (green line) are in approximate agreement. Figure 4(k) is the corrected measurement image, and Figs. 4(l) and (m) are the intensity distributions of line AA’ and line BB’, respectively, of Fig. 4(k). Both are approximate linearity distributions in the range from 50 to 210. By analyzing the above intensity distribution, the detection accuracy of the stripe order reaches 100%, and the 3-D representation can be obtained as shown in Fig. 5.

For comparison, the experimental results obtained by camera automatic-mode are shown in Fig. 6. Figures 6(a) and 6(b) show the reflected pattern and measurement image, respectively. The measurement object is the same as the object shown in Fig. 4 and the projection pattern is as shown in Fig. 4(e). Figure 6(c) shows the histogram for Fig. 6(b). Since there is no portion with a large intensity value of the reflection pattern on the object, the range of the stripe intensity reflected from the object is insufficient.

In order to evaluate the validity of the proposed technique, a total of 12 objects of different materials, color distributions, textures, and surface reflectances were measured. The measurement objects are shown in Fig. 7, and the measurement results are listed in Table 2. The proposed technique was effective for all of the measurement objects. The average number of adjustments of the projection pattern intensity was two and the control error of the intensity was less than 2 (1%).

Fig. 7. Measurement objects for the evaluation experiment

Table 2. Experimental Results of Color-Intensity Control of Projection Pattern

No

Object

Projection intensity

Observation image intensity

|S-Sn|

Shutter speed (Second)

Measurement Times (SecondNo.)

Sort

Material

P1

P2

P3

S0

S1

S2

S3

Process

Total

1

Pot

Ceramic

151

255

212

2

1/30⇒1/60

0.383

0.649

2

Pot

Ceramic

188

199

170

198

211

1

1/30

0.487

0.886

3

Cake case

Japanese paper

170

174

208

2

1/30

0.435

0.701

4

Vase

Plastics

232

219

150

222

212

2

1/30

0.416

0.815

5

Body of car

Metal

224

217

153

216

209

1

1/30

0.570

0.969

6

Pen stand

Wood

153

140

255

229

210

0

1/30⇒1/60

0.337

0.736

7

Fancy box

Plastics

87

97

255

189

210

0

1/30⇒1/60

0.452

0.851

8

Paper box

Paper

138

129

125

255

224

213

209

1

1/30⇒1/60

0.616

1.148

9

Ornament

China

188

223

212

2

1/30⇒1/60

0.413

0.679

10

Ornament

Plastics

255

244

97

219

212

2

1/30⇒1/15

0.417

0.816

11

Face

Skin

118

150

255

167

211

1

1/30

0.546

0.945

12

Toy

Cotton cloth

240

147

208

2

1/30

0.505

0.771

Table 2. Experimental Results of Color-Intensity Control of Projection Pattern

The measurement time is shown in the last two columns of Table 2. The process time contains the times for multiple projection pattern generations, the times for multiple pattern projection and reflection image processing, and 3-D coordinate computation time. The total time is the total of the process time, the image capture time, and the image data transference time. The total time is the comprehensive measurement time from measurement object discovery until the three-dimensional information calculation. The measuring time is mainly dependent on the measurement algorithm, the camera, and the computer. Although the process time reflects the measurement speed of the proposed algorithm, it changes with the calculation speed of the computer and the response speed of the projector, for example. The image capture time and the image data transference time depend on the performance of the camera and computer. By using a high speed camera, the measurement time can be shortened further.

4. Conclusion

A technique for photographing an image that has an ideal intensity distribution using a small number of projections is greatly needed for 3-D optical measurement based on pattern projection. Therefore, a robust measurement technique that can automatically adjust the intensity distribution of the projection pattern according to changes in the surface color, surface reflective distribution of the measured object, etc., is required.

The stage formula projection pattern intensity control method was proposed herein. Using this method, observation pattern images with ideal intensity distributions can be obtained in a short time with an average of three projections on objects of various materials and colors. Using the proposed technique, a 3-D image measurement system can be built using standard equipment, such as a general-purpose liquid crystal projector and a CCD color camera.

High-accuracy detection of stripe order of projection patterns is an important technology for non-binary 3-D image measurement methods. The technique proposed in the present paper increases the detection accuracy of stripe order, particularly when used in conjunction with the previously proposed optimal combination technique of projection pattern stripe intensity [15

Cited By

OSA is able to provide readers links to articles that cite this paper by participating in CrossRef's Cited-By Linking service. CrossRef includes content from more than 3000 publishers and societies. In addition to listing OSA journal articles that cite this paper, citing articles from other participating publishers will also be listed.