Abstract: Vision systems are used more and more in recent industrial robot applications for enhancingrobot flexibility and intelligence. This paper presents the integration of a three-dimensional (3D)binocular stereo vision system into an industrial robot system. Examples are used to illustrate thedeveloped method and its implementation including vision system setup and calibration, robot hand-eyeintegration, and vision measurement analysis. The results show that a 3D binocular stereo vision systemis able to aid an industrial robot in performing robot motions to a given 3D part through accurate visionmeasurements and integrated robot programming.

I. Introduction

Industrial robots are often equipped with vision systems for recognizing objects in a robot workenvironment. Among various vision technologies, a three-dimensional (3D) binocular stereo visionsystem utilizes two cameras to obtain the positioning information (x, y, z, roll) of a 3D object. Thistechnology is used in industrial robot applications due to the simple setup of lighting conditions and theefficient image processing of the vision system. Fig. 1 shows an industrial robot workcell where theFANUC M6i robot is assisted by the FANUC VisLOC 3D vision system in performing robot motions toa given 3D part on the table. However, in order to make the 3D vision application successful, the robotprogrammer must correctly set up the 3D measuring functions of the vision system and integrate theprecise vision measurements into the robot application programs. Industrial practice shows that vision-guided industrial robot applications not only use the technology of vision cameras, image processing,system calibration, and communication [1] [2] [3] [4][5]

but also require the robot programmer to have asolid understanding of other issues including camera positions and views, vision measurements andaccuracy, reference frames and frame transformations, and robot operations and programming.

This paper presents the method for integrating the measuring functions of a 3D binocular stereo visionsystem into an industrial robot system. The study deals with the design techniques and proceduresrelated to vision system setup, robot hand-eye integration, and vision measurement analysis. Thethe Technology Interface Journal/Winter Special Issue 2009 Cheng and Chenselected paper from the Proceedings of the IAJC-IJME 2008 Conference

discussion includes the concepts of Epipolar geometry and perspective calibration used by 3D binocularvision systems, the procedures for unifying the coordinates of vision measurements and robot points,and the variations of vision depth measurements due to pattern distortion. With the developed methodand solution, the robot programmer is able to better apply the 3D binocular stereo vision technology forenhancing flexibility and intelligence of industrial robot operations.

II. 3D Binocular Stereo Vision and Calibration

The geometry of 3D binocular stereo vision is referred to as Epipolar geometry. When two camerasview a 3D scene from two distinct positions as shown in Fig. 2, there are a number of geometricrelationships between a 3D point and its projection in a 2D image as shown in Fig. 3. The concept of3D binocular stereo vision is illustrated in Fig. 4 where two cameras look at point P and the projectionproblem is simplified by placing a virtual image plane in front of the focal point of each camera toproduce an unrotated image. Specifically, OLand ORrepresent the focal points of the two cameras.PointspLandpRare the projections of point P on the left and right image planes, respectively. Eachcamera captures a 2D image of the 3D world. This conversion from 3D to 2D is referred to as aperspective projection. Since the two cameras are at distinct locations, the focal point of one cameraprojects a distinct point on the image plane of the other camera. These two projected image points arecalled epipoles and denoted as ELand ER, respectively. It is clear that epipoles, ELand ER, and focalpoints, OLand OR, lie on a single line. Although the left camera sees line POLas a point because it isdirectly aligned with its focal point, the right camera sees the same line as epipolar linepRERon itsimage plane. Symmetrically, the left camera sees line PORas epipolar linepLELon its own image plane.Alternatively, each epipolar line is actually the intersection of the corresponding camera image planeand the epipolar plane formed by points P, OLand OR. Thus, all epipolar lines intersect thecorresponding epipoles regardless of P point locations. If projection pointpLis known, thencorresponding epipolar linepRERis known and projection pointpRmust lie on this particular epipolarline. This epipolar constraint is applied to all corresponding image points. This means that if imagethe Technology Interface Journal/Winter Special Issue 2009 Cheng and Chenselected paper from the Proceedings of the IAJC-IJME 2008 Conference

Vision camera calibration is a process that establishes the vision frame, Vis (x, y, z), and the pixel valuefor a vision system. A 3D binocular stereo vision system conducts the perspective calibration throughtwo 2D camera view pictures of a common calibration grid sheet as shown in Fig. 5, where the largecircles define the x- and y-axes of vision frame Vis and the small circles calibrate the camera pixel value(mm) in Vis.

(a) Calibration view planes from both cameras

(b) Calibration Accuracy

Figure 5: 3D binocular stereo vision calibration

Visxythe Technology Interface Journal/Winter Special Issue 2009 Cheng and Chenselected paper from the Proceedings of the IAJC-IJME 2008 Conference

A 3D binocular stereo vision system also utilizes the user-trained pattern of an actual object to identifysimilar objects from the camera view. Fig. 6 shows a real block and the corresponding trained patternsfor the two cameras. The robot programmer must define the object frame, OBJ (x, y), for each objectpattern by choosing a point that represents a common point on the real object. Through patterncomparison and matching, the vision software finds the similar object from the camera view andmeasures the corresponding OBJ frame location OBJ[n] as frame transformationVisTOBJ[n]as shown inFig. 7.

Figure 6: The trained patterns of an object in two camera views

Figure 7: Vision-found object and its OBJ frame location

III. Robot Hand-Eye Integration

An industrial robot system employs Cartesian reference frames and frame transformations to representthe position and orientation of the robot hand as shown in Fig. 8 [6]. Among them, the robot base frameR(x, y, z) is a fixed one and the default tool-center-point frame Def_TCP (n, o, a) is located at the centerof the robot’s wrist faceplate.

To integrate the vision-measured OBJ frame location of the object into the robot system, the robotprogrammer needs to setup a UF[i] that coincides with vision frame Vis as shown in Fig. 9a. The taskrequires the robot programmer to follow the UF Setup procedure provided by the robot system and teachthree non-collinear points as shown in Fig. 8 by using the same camera calibration grid sheet that is atthe same location in vision calibration as shown in Fig. 5. With the coincidence of frames UF[i] and Vis,object frame location OBJ[n] measured in vision frame Vis actually represents robot point P[n] asshown in Eq. (1):

]n[OBJVis]k[UT]i[UF]i[UF]k[UTTT]n[P . (1)

The robot programmer may directly use a vision measurement as a robot point as shown in Eq. (1).However, after reaching to the vision-defined point, the robot cannot use the robot points that are taughtvia the similar object located at a different position OBJ[m] within the camera view. To avoid thisproblem, the robot programmer usually determines the position offset of UF[i],UF[i]TUF[i]’, through twovision measurementsVisTOBJ[n]andVisTOBJ[m]as shown in frame transformation Eq. (2) and Fig. 9b:

where frames Vis’ and UF[i]’ represent the locations of frames Vis and UF[i] after object frame OBJchanges its location from OBJ[n] to OBJ[m] and (Vis’TOBJ[m])-1is the inverse of transformationVis’TOBJ[m]. Usually, the vision system obtainsVisTOBJ[n]when the object is found at a reference positionwithin the camera view during the vision setup and acquiresVisTOBJ[m]when the object is found from theactual view picture.

With the vision-measuredUF[i]TUF[i]’, the robot programmer is able to transformrobot point][][][iUFkUTnPthat is taught via the object at OBJ[n] into robot point]i[UF]'k[UT]m[Pfor the similarobject at a different location OBJ[m] in the robot program as shown in frame transformation Eq. (3):the Technology Interface Journal/Winter Special Issue 2009 Cheng and Chenselected paper from the Proceedings of the IAJC-IJME 2008 Conference

where UT[k]’ represents the location of UT[k] after UT[k] moves to object frame location OBJ[m].

In the FAUNU robot workcell in Fig. 1, the FANUC VisLOC vision system [7] measuresVisTOBJ[n]viathe object located at the reference position where the robot programmer also teaches the required robotpoints]i[UF]k[UT]1[Pand]i[UF]k[UT]2[Pvia the same object. Executing the VisLOC micro function “Snap & Find”in the FANUC robot Teaching Pendant (TP) program [8] allows the vision system to obtainVisTOBJ[m]

for the vision-found object within the camera view and calculate offset valueUF[i]TUF[i]’for the robotsystem as shown in Eq. (2). The vision system sends the determinedUF[i]TUF[i]’to the specified robotposition register PR[x]. The robot programmer then utilizes the “offset” value in the following threeFANUC robot TP program instructions to modify]i[UF]k[UT]1[Pand]i[UF]k[UT]2[Pinto the corresponding onesfor the vision-found object as shown in Eq. (3):1: Offset Conditions PR[x], UFRAME[i];2: J P[1] 100% Offset;3: J P[2] 100% Offset;the Technology Interface Journal/Winter Special Issue 2009 Cheng and Chenselected paper from the Proceedings of the IAJC-IJME 2008 Conference

The accuracy of vision calibration and pattern matching is important in vision-guided industrial robotapplications. Due to the fact that during perspective calibration the camera lens in a 3D binocular stereovision system cannot be perpendicular to the calibration grid sheet, certain degrees of distortion occurwhen the vision software records all the corresponding locations of the circles in the camera view. Forexample, Fig. 5b shows that the perspective calibration may result in an average error of 0.2 pixels, amaximum error of 0.45 pixels, and a scale of 0.76 mm/pixel. Based on these data, we can obtain thefollowing error range of the vision measurement:

Due to pattern distortion, the depth measurement (i.e. z value) of an object varies when the object isaway from the center point of the 3D vision view plane as shown in the following test results. During thetest, the block, as shown in Fig. 6, was moved 10 mm each time along the x- and y-axes of the cameraview plane, respectively, and the corresponding depth value (mm) of the block was measured by usingthe FANUC VisLOC Snap & Find function. Then, the relationship between the average depth and thelocation of the block at each axis was statistically formulated.

The diagram in Fig. 10 shows that the Z – X relationship is a declining curve along the positive directionof the x-axis. Its model is:

z = -0.0096x + 36.108 with R² = 0.7948

The diagram in Fig. 11 shows that the Z – Y relationship is almost a non-fluctuating curve. Its model is:

z = -0.0002y + 35.912 with R² = 0.0018

These relationships illustrate that due to decreased identical features in both camera views the locationvariation of the object along the x-axis produces some degrees of pattern distortion and mismatch,resulting in an inaccurate depth measurement of the object. However, the depth measurement remainsconsistent in the y-axis due to more overlapped features in both camera views.

V. Conclusion

This paper provided the robot programmer with the method for using a 3D binocular vision system in anindustrial robot application. The study has shown that the camera setup, calibration of robot and visionsystems, vision measurement acquisition, and robot hand-eye system integration were the key designtasks in 3D vision-guided industrial robot applications and the developed method addressed thethe Technology Interface Journal/Winter Special Issue 2009 Cheng and Chenselected paper from the Proceedings of the IAJC-IJME 2008 Conference

solutions for successfully completing these design tasks. The Epipolar geometry allows the 3D binocularstereo vision system to conduct the perspective calibration by using two 2D camera view pictures of acommon calibration grid sheet. The robot programmer integrates the vision measurement into the robotsystem by setting up a robot user frame UF[i] that coincides with vision frame Vis. The robot programuses the vision-determined “offset” value to transform all pre-defined robot operation points to thecorresponding ones for the vision-found object. The results also show that the 3D binocular stereo visionsystem is able to provide the robot system with the accurate depth measurement for the 3D objectlocated in the center of the camera view. The variation of depth measurement occurs as the vision-foundobject is away from the center of the camera view.

Frank S. Cheng received his Ph.D. degree from the University of Cincinnati in 1995. He is currently anassociate professor in the department of Engineering and Technology at Central Michigan University.Dr. Cheng is the instructor for courses in the areas of robotics and automation for both engineering andengineering technology programs and is responsible for the Robotics and Automation Laboratory. Hisresearch interests include robotics, mechatronics, controls, and industrial automation. Dr. Cheng haspublished his research developments in refereed journals, proceedings, and book chapters. He is amember of ASEE, IEEE, and IAJC.

Xiaoting Chen is a graduate student in the department of Engineering and Technology at CentralMichigan University. He holds a B.S. degree in Electrical Engineering from the Shanghai Institute ofTechnology, China. His research interests include robotics, control systems, and industrial automation.