Figure 20. Results of 3D reconstruction of the weld pool. (a) shows the 3D coordinates of all the projected laser dots on the weld pool surface, (b) is the weld pool surface interpolated using the laser dots in (a), (c) shows the reflected pattern from imaging process and that from the reconstructed weld pool surface.

Figure 22. Simulation results of the proposed reconstruction scheme. (a) The 3D coordinates of all the projected laser dots; b) The reconstructed surface of the bench mark; (c) The boundary of the reconstructed bench mark; (d) The height errors between the actual and reconstructed bench marks row by row and the static height error. Row 1 is defined as the most left reflected dots row in (a). The absolute errors between height errors and the corresponding static errors are shown in (e).

Real-Time Measurement of Three Dimensional Weld Pool Surface in GTAW

Wei Jie Zhang, Yu Kang Liu and Yu Ming Zhang

[1] Department of Electrical and Computer Engineering Institute for Sustainable Manufacturing, University of Kentucky, Lexington, USA

1. Introduction

Gas tungsten arc welding (GTAW) [1] is the primary process used by human welders for critical applications. In this process as shown in Fig. 1, an arc is established between the non-consumable tungsten electrode and base metal. The base metal is melted by the arc forming a liquid weld pool that joins the two pieces of base metal together after solidification. An optional filler metal (not shown in the figure) can be added if necessary but it is melted by the arc column, rather than directly by an arc spot as in gas metal arc welding (GMAW) where the anode can much more efficiently melt a continuously-fed wire than the arc column to increase the melting productivity. However, the detachment and impact of the associated droplets on the weld pool compromise the controllability of the process and limit its use in precision applications.

Figure 1.

Illustration of GTAW

Because GTAW is primarily used in applications where appropriate degree of full penetration (if and how much the liquid metal has fully penetrated the entire thickness of the base metal) is critical for the service, the process should be mechanized or automated as long as it can be justified for production cycle, cost, and quality. However, there are a number of issues which adversely affect the automation significantly. The first one is the accessibility. That is, in many applications there is no sufficient space to allow a mechanized system's torch head to access. Second, mechanized systems require significant amount of time for on-site installation and joints be prepared with great precision. The production cycle in many applications is adversely affected substantially. The third issue is the assurance of the weld quality. In manual welding, welders who observe the weld pool can assure the desired full penetration is produced. However, in mechanized welding, no welder has the capability to interference with the system; they are not required or allowed in robotic welding to observe the welding process with the similar level of concentration as in manual operation. Mechanized/automated systems rely on precision control of joint fit-up and welding conditions and tedious programming of welding parameters to produce repeatable results. However, precision control of joints and welding conditions is very costly and not always guaranteed. Up to date, there are no satisfactory sensors/ways that can conveniently/ automatically monitor the penetration depth (how far the liquid metal penetrates along the thickness of the base metal) or the degree of the full penetration like a skilled welder.

The difficulty is primarily due to the invisibility of the liquid metal bottom surface underneath the weld pool and the extreme brightness of the arc and various methods have been studied, including pool oscillation, ultrasound, infrared sensor, and vision-based sensing method, etc. In the following subsections, each sensing method is briefly reviewed.

1.1. Pool oscillation method

Sensing the weld penetration by monitoring weld pool oscillation behavior is based on the fact that a weld pool can be brought into natural oscillation and the oscillation frequency of the weld pool is related to the weld pool geometry. This phenomenon may be used to monitor the weld pool in a feedback control system. The pioneering work in pool oscillation was conducted by Kotecki [2], and Richardson [3].

Hardt [4] and their co-workers proposed a method to determine the back-side bead width by measuring the natural frequency of pool motion when driven by a time varying arc plasma force. The method was developed analytically and verified by experiments. However, the results were obtained for stationary weld pools, and it was unclear if similar results occur when the welding torch was moving. G. den Ouden found an abrupt change in the oscillation frequency of the pool during the transition from partial to full penetration [5, 6].

Andersen [7] developed a synchronous weld pool oscillation method for controlling the weld pool dimensions and state of penetration. The approach used to induce pool oscillations was to excite the weld pool with current pulses synchronized to the natural oscillations of the pool. An optical sensor was utilized to detect the pool oscillations. A model of the weld pool was also developed using a fluid droplet formulation for the relation of weld pool geometry and other physical parameters to the natural frequencies of the weld pool. Comparison of the weld pool's natural frequency as predicted by the developed weld pool geometry models and measurements of the pool width thus allowed the assessment of the penetration state. Hartman [8] further evaluated this synchronous excitation method and developed a control system that regulated the total heat input to maintain constant fusion zone geometry by monitoring the arc light reflection from the oscillation of the molten metal surface.

Ju [9] proposed a new vibration method: the Pulse Shielding Gas (PSG) oscillating method. A control system was constructed by controlling the welding current based on the natural vibration frequency measurements from an arc sensor. It was found that spectrum analysis using the Fast Fourier Transformation (FFT) was effective for detecting peculiar frequency of the molten pool.

Yudodibroto [10] implemented the weld penetration control based on weld pool oscillation sensing method during GTAW process with cold filler wire addition. The frequency of the weld pool oscillation was obtained from the arc voltage variation via analysis. It was found that the weld pool oscillation approach is suitable for penetration control during cold wire GTAW when the metal transfer occurs in an uninterrupted bridging manner.

1.2. Ultrasonic sensing method

Ultrasonic sensors[11-17] are widely used to determine the boundaries of the liquid and metal in the weld pool.

In [15] the developed ultrasonic sensing system could locate and track the welding seam ensuring correct positioning of the welding head relatively to the joint preparation. The system was able to monitor the joint profile of the molten weld pool and modified the relevant heat input parameters ensuring consistent penetration, joint filling and acceptable weld bead shape. It also made use of both the above information to reconstruct 3D images of the weld pool silhouettes providing in-process inspection capabilities of the welded joints.

At Georgia Institute of Technology, Ume leaded the development of non-contact ultrasonic penetration sensors based on laser-phased array techniques [13, 14]. Recently, in order to overcome the contact requirement of the ultrasonic sensing method, various non-contact ultrasonic sensing methods have been developed, such as laser ultrasonic sensing [11, 17], electromagnetic acoustic transducer (EMAT) ultrasonic sensing [12], and laser-EMAT ultrasonic sensing [16], etc.

Mi [17]developed a ultransonic sensing system to monitor the weld penetration. The sensing system was based on using a laser phased array technique to generate focused and steered ultrasound, and an EMAT as a receiver. Both the ultrasound generation by the laser phased array and the reception by the EMAT were non-contact, which could thus eliminate the need for a couplant medium. This made the system capable of operating at high temperatures involved in the welding process. A signal processing algorithm based on a cross-correlation technique was further developed to estimate the time-of-flight (TOF) of the ultrasound.

1.3. Infrared sensing method

Infrared sensing is a type of non-contact thermal measurement technique which has been widely used in various applications. Because the temperature distribution in the weld zone contains abundant information about the welding process, infrared sensing of welding processes has drawn considerable attentions from various research institutions.

Chin at Auburn University [18-21] developed a thermal imaging system to measure the variations in weld process parameters such as bead width, penetration depth, and torch offset. The penetration depth has been correlated with the infrared characteristics of the infrared image. The interference of arc radiation was reduced by selecting scanner with specific wavelength region.

At MIT, Hardt used an infrared camera to view the temperature field from the back-side [22]. The penetration depth was precisely estimated from the measured temperature distribution and then controlled [23]. In particualr, a discrete time transfer function matrix empirical model for gas metal arc welding process was proposed, which took the common dynamics for each output and inherent process and measurement delays into account. The adaptation mechanism employed in the control system rendered this model useful over a wide operating range.

In [24] infrared sensor was used to monitor weld process parameters including the weld bead width, penetration depth, and torch position. Analysis of the computed ellipse showed that the temperature gradient or heat energy distribution (minor axis of the ellipse) and the heat input (volume under the temperature profiles) varied with the penetration depth.

1.4. Vision-based sensing method

Based on the observation of the weld pool, a skill welder can assure the desired full penetration. The weld pool thus should contain abundant information of weld penetration. To this end, vision-based systems have been applied to monitor the weld poo by emulating human welders' visual sensory ability. Continued advances in computational capabilities and reduction in cost have recently led to an increase in researches and applications of vision-based systems for the weld pool measurement and welding process control. In the following subsections, vision-based sensing methods are extensively reviewed, including 2D weld pool sensing, and 3D weld pool sensing methods.

1.4.1. 2D weld pool sensing

2D weld pool geometry contains certain information of the welding process, and has been used to monitor the welding process and control the weld penetration [25-27].

Fan et.al [26] studied 2D visual sensing and penetration control in aluminum (Al) alloy pulse GTAW process. A three-optical-route visual sensor was designed. The sensor could capture the weld pool from three directions at the same time. The authors finally used PID and a multiplex controller to control the penetration.

Ma [27] used two normal CCD cameras for capturing clear images from two directions: one of them was used to measure the root gap and another one was used to measure the geometric parameters of the weld pool. Seam tracking and penetration control of robot welding process was simultaneously established based on the proposed binocular vision sensor.

1.4.2. 3D weld pool sensing

Although 2D weld pool geometry has been obtained with above different techniques, the convexity/deformation of the weld pool is not yet fully explored. Early researchers have found that important information such as weld defects and penetration are contained in the surface deformation of the weld pool [28, 29]. A recent study suggests that compared with the 2D weld pool geometry, the 3D geometry can better predict the weld penetration which is measured by the backside weld bead width[30]. Therefore, numerous methods have been developed to reconstruct the 3D weld pool surface.

The measurement of 3D surface has been recently studied extensively with techniques which can be roughly categorized into three branches: 1) reflectometry/deflectometry with fringe reflection technique [31-33]; 2) phase shifted digital fringe projection technique for diffuse objects [34, 35]; 3) shape from shading technique [36]. Unfortunately, the dynamic and specular nature of the weld pool and the interference from the strong arc radiation complicate the observation and deteriorate the effectiveness of most of those methods.

The most popular techniques currently being studied for 3D weld pool measurement can be divided into four categories:

1. Model-based reconstruction

The 3D weld pool surface was partially reconstructed based on a simple model proposed in [37]. The 2D weld pool images were captured under the base-current period in GMAW. The proposed model then used the capturing angle of the camera and the 2D weld pool profile to calculate the weld pool width, the length of the pool tail, the height of the rear of the pool, etc. The reconstruction algorithm was further applied in [38] for the control of weld pool shape. A fuzzy logic controller was constructed to control weld penetration. It was found that the correlation was nonlinear and thus suitable to employ the proposed fuzzy controller. Simulation and control experiments were carried out to verify the effectiveness of the proposed control algorithm.

Although this model-based reconstruction algorithm is simple and fast, it can only measure the height of the weld bead that is solidifying or have already solidified at the rear of the weld pool. The 3D geometry of head of the weld pool cannot be acquired using this method. Further, the model-based reconstruction algorithm only suits for thin work piece welding application.

2. Stereovision measurement

In [39], two cameras were synchronized to capture the two images of weld pool surface simultaneously in the short circuit period during the Surface Tension Transfer (STT) process and external illumination was used. The paired images were rectified using calibration parameters obtained through the stereo calibration procedure. As the weld pool surface was highly patterned in the experiment, an image correlation-type measure was used to match points between the two rectified images. Then by using stereo image processing algorithms the weld pool shape was rendered in 3D. A closed-loop control system was further developed using the technique for robot welding process [40]. However, the shape of the bright part in the head of the weld pool cannot be acquired by using this method. Further, the accurate reconstruction of the weld pool requires both precise synchronization of the two cameras and high quality of the captured images.

To avoid the synchronization problem, the biprism stereo vision sensing was proposed in which one camera was used with a biprism attached on its head [41]. However, only the height of the weld pool boundary was extracted in real-time, the 3D geometry inside the weld pool was missed. Furthermore, the reconstruction accuracy might be an issue since the visual differences are comparatively small between the two simultaneously captured images. A similar reconstruction algorithm has been utilized in a stereo sensing system using single camera with a stereo adapter developed to reconstruct the 3D weld pool for tracking particle flow on the weld pool surface [42].

3. Shape from shading (SFS) reconstruction

3D weld pool reconstruction algorithms have also been proposed based on shape from shading method [43-47]. Zhao et al. [46] useSFS algorithm to reconstruct the surface from one single weld pool image. Two-dimensional shape parameters were extracted from a 2D image processing algorithm. Finally, a SFS algorithm on a single image was used to recover the surface height from a single weld pool image. The extracted three-dimensional parameters for the weld pool surface were verified and used for double-sided shape control.

However, SFS algorithms are usually complex and thus used for off-line reconstruction of the 3D weld pool surface. Furthermore, the reconstruction algorithms are based on two assumptions: 1) The object surface is a Lambertian surface which reflects light with equal intensity in all directions; 2) The camera and the light source are at the infinite far distance from the object surface. The weld pool, on the other hand, is a specular surface which is not a Lambertian surface. The camera and light source in the experiment systems are not far enough from the weld pool such that the infinite far position assumption is invalid. Therefore, the 3D weld pool reconstruction using SFS might not be an ideal solution.

4. Structured-light based sensing

A structured-light vision system was developed in [48] projecting a pulsed laser on the weld pool surface through a special grid. A high shutter-speed camera was used to capture the laser stripe pattern reflected by the weld pool surface. First, to eliminate the influence of the bright arc light, a short duration pulsed laser was projected onto the weld pool surface. The camera shutter was synchronized with the pulse duration. Second, a frosted glass was used to allow each laser ray as a new point light source which disperses light with a certain diffuse angle. The camera viewed the grid openings through their reflection from the weld pool surface and obtained image consisted of bright strips deformed by the weld pool surface deformation. The proposed method could obtain specular reflection from the weld pool under the presence of the bright arc. An iterative algorithm was used to calculate the surface of the weld pool. The time cost of the reconstruction was about $1$s. However, the synchronization of the laser and high-speed shutter required specific, high-costly, and sophisticated equipment. The boundary of the weld pool was also hard to extract using this sensing method.

Follow-up study [49] provided a measurement system based on a mathematical model of weld pool surface. The captured image from [48] was applied as an example in the study. Although this work did not propose a new reconstruction algorithm, it provided some novel insights of 3D weld pool surface measurement.

A laser grating sensing technique was proposed in [50]. The reflected grating was captured by a two-lens system. The depth of weld pool was determined based on the phase changes of the deformed grating image [51]. However, using this method the boundary of the weld pool was hard to be determined. Further, it was only a primarily study since there is no detailed quantitative analysis of the reconstruction.

A novel reconstruction algorithm using the slope field and point tracking of the dot matrix was proposed in [52]. A single laser line was projected onto the weld pool surface from a known position with a certain angle. The reflected laser beam from the weld pool surface was captured by a calibrated compact CCD sensor. From the acquired images, the profile of the weld pool surface can be extracted according to ray-tracing technique and the parameters of the CCD sensor. If the line was projected onto the center of the weld pool, the depth of weld pool could also be extracted.

In this technique incorporated use of a calibrated CCD sensor and structured light made it possible to extract the depth of pool from captured images. Although the height reconstruction error was small, the point tracking procedure was complex such that the point matching for each frame requires to process three consecutive frames. It was thus only suitable for off-line reconstruction of 3D weld pool surface. Also, the boundary information of the weld pool in the reconstruction was not addressed.

A laser pattern reflected from the weld pool surface has been intercepted/imaged by/on a diffusive imaging plane placed with a distance from the weld pool [53]. The camera aimedat the imaging plane (rather than the weld pool illuminated by the extremely strong arc) to acquire the reflected laser pattern. Its uniqueness lied in its simultaneous use of the distance and specular nature of the weld pool surface to significantly decay the arc radiation but not the intensity of the laser reflection from the specular weld pool surface despite the distance. To compute the weld pool surface from the reflected patterns, an iterative algorithm has been proposed using the slope field of the projected dot matrix. The slope differences between the neighborhood laser dots were used to find the estimated height of the weld pool surface.

However, this slope error based algorithm requires numerous iterative loops till the estimated surface approaches the actual weld pool surface resulting in relatively large reconstruction errors. Similarly, this imaging method and reconstruction algorithm have been used to image and reconstruct the weld pool surface in gas metal arc welding (GMAW) using a five line laser pattern [54].

This chapter focuses on the development of a procedure of image processing algorithms and an analytical solution that allows the 3D weld pool surface in GTAW be reconstructed in real-time using the aforementioned innovative imaging principle [53]. The effectiveness, time cost, accuracy and robustness of the proposed algorithm are quantitatively studied. The accuracy and speed are tested using objects with known geometry and compared with those from previous studies. In particular, the chapter is organized as follows: Section 2 details the vision-based monitoring system. The proposed image processing algorithm procedure is presented in Section 3. The proposed analytic reconstruction algorithm is detailed in Section 4. In Section 5 one object with known 3D geometry is used to emulate the weld pool surface. By comparing the reconstruction surface of the object with its actual surface, the effectiveness and accuracy of the proposed algorithms are verified. The time cost of the reconstruction algorithm is then analyzed. Section 6 presents the summaries of this chapter.

2. Vision-based monitoring system

2.1. Monitoring system

The configuration of the sensing system and the 3D rectangular coordinate systems oxyz are shown in Fig. 2.

A 20 mW illumination laser generator at a wavelength of 685 nm with variable focus is used to generate a structure light pattern, i.e., a 19×19 dot matrix pattern (Lasiris SNF-519×0.77-685-20). The laser pattern is projected onto the area under the torch electrode and covers the whole possible weld pool region. During the welding process, the base metal is melted by the arc forming a liquid weld pool which has a mirror-like specular surface. It can reflect the majority of the incident laser rays. Therefore, only the dots projected on the weld pool are reflected by its specular surface [49].

Figure 2.

Monitoring system

In order to capture the reflected dot matrix, an imaging plane made by a sheet of glass attached with a piece of paper is installed with a distance of 100 mm approximately from the electrode. A camera (Pointgrey Flea 3 FL3-FW-03S1C-C) is located behind the imaging plane directly aiming at it. The camera captures the images of the reflected pattern from the imaging plane. The captured image is 8-bit monochrome with a resolution of 640×480 or 480×640. A band-pass filter of 20 nm band-width centered at a wavelength of 685 nm is attached to the camera to block the majority of the arc radiation. A computer connects with the camera using a 9-pin 1394b interface. With a maximal frame rate of 200 fps (frame per second), the high transfer rate from the camera to PC (maximum rate 800Mbit/s) makes possible the real-time monitoring and measurement of the 3D weld pool surface in GTAW.

The projection pattern, the dot matrix, is shown in Fig. 3a. The reflection patterns at the solution 480×640 and 640×480 are presented in Fig. 3b and 3c, respectively. The reference dot, i.e., the center ray of the dot matrix is intentionally missed. Please note that the brightness of reflected patterns in the captured images is intentionally enhanced for readability. The original images captured from welding process are much darker than the presented ones.

Figure 3.

2.2. Experiment conditions

The welding process used is direct-current electrode-negative (DCEN) GTAW. The material of the pipe is stainless steel (4 inch normal, stainless T-304/304L, schedule 5). The pipe rotates during welding while the torch’s orientation, imaging plane, laser projector, and camera are stationary. The rotation speed and the distance from the tungsten tip to the pipe surface are controlled by a computer to weld at required welding speed and arc length.

Ranges of parameters selected to conduct the welding experiments and acquire images in this chapter are shown in Table 1. The full penetration, i.e., the liquid weld pool extends from the front to the back face of the work piece, can be produced on the work piece with those welding parameters. Shielding gas is pure argon. The 2% ceriated ground tungsten electrode (3/32 ×7'') grinding to 30º is used.

Welding Parameters

Current/A

Welding speed/mm/s

Arc length/mm

Argon flow rate/L/min

65

1.5

4.5

11.8

Monitoring Parameters

Project angle/°

Laser to weld pool distance/mm

Imaging plane to weld pool distance/mm

35.5

24.7

101

Camera Parameters

Shutter speed/ms

Frame rate/ fps

Camera to imaging plane distance/mm

4

30

57.8

Table 1.

Major experiment parameters

3. Image processing scheme

Using the vision-based monitoring system, images of laser reflection pattern, as shown in Fig. 3b and 3c, can be captured during welding process. However, those two images are deliberately enhanced in brightness such that clear reflection patterns can be seen. The original captured image corresponding to Fig. 3b is shown in Fig. 4a. Fig. 3b is also presented in Fig. 4b for a comparison.

Figure 4.

A human with naked eyes can identify the distorted reflection pattern from captured images as shown in Fig. 4b, while the dots in Fig. 4a can hardly be seen. Images captured during the experiments with different conditions (see Table 1) might obtain even lower brightness and contrast. Furthermore, even in the same captured image, Fig. 4 for example, the gray levels of reflected dots are of great difference. The dots located in the lower part of the image are brighter than those in the upper part of the image. The reflected laser dots are highly coupled with the background in gray scale, especially those in the upper part of the image. In addition, the background of the image, i.e., the part other than the reflection pattern in the captured image, has a severely unbalanced brightness. It can be clearly observed that the brightness of the lower part of the background is much stronger than that of the upper part.

In order to reconstruct the 3D weld pool surface based on the reflection pattern, the reflected dots in the pattern should be extracted from the captured image first. Based on those features of the captured image, noise reduction operation should be conducted first to smooth the image before unifying the brightness. To this end, the flowchart for the proposed image processing scheme is shown in Fig. 5. In particular, a Wavelet-based method is used here for noise reduction which is a pre-processing step to remove certain noises from the image to assure the effectiveness of subsequent processing steps shown in the chart. A Top-hat operation is then performed to unify the background brightness of the captured image while enhancing the reflected dots in the meantime. After a binary thresholding the reflected dots are extracted from the image along with certain noises that are considered as ``fake dots". An adaptive identification algorithm is thus proposed to distinguish the reflected laser dots from the “fake dots”. In order to calculate the weld pool surface, each reflected dot is matched with its corresponding incident ray as defined by its row and column numbers in the projected laser matrix. This is done through the row and column pattern recognition.

Figure 5.

Flowchart for the proposed image processing scheme

In the following sections, the image Fig. 4a is taken as an example to demonstrate the proposed image processing scheme. In particular, each block of image processing operation/algorithm in Fig. 5 is detailed. The effectiveness and robustness of the proposed procedure will be verified in Section 5.

3.1. Noise reduction

Effective noise reduction is a prerequisite for high quality image segmentation. To this end, a wavelet thresholding method is employed to reduce noises in the image. Wavelet noise reduction procedures rely on the recurrent fast wavelet transform (FWT) algorithm proposed by Mallet [55]. The principle of a wavelet-based noise reduction can be described as

where x^ is an estimation ofx, Tλ(⋅)is the thresholding operation, FW(⋅)and FW−1(⋅) are the forward and inverse FWT respectively.In order to avoidloss of useful information in the captured image, a soft thresholding is applied [56] with a 8th Symlet Wavelet with 3 levels used for the FWT [57]. After performing noise reduction as shown in Eq. 1 to Fig. 4a, the resultant image is shown in Fig. 6.

Figure 6.

Image after wavelet noise reduction, brightness of the image is enhanced for readability

It can be observed that the lower part image is much brighter than that of the upper half as shown in Fig. 6. To balance the uneven brightness distribution of captured images a top-hat operation is performed. The resultant image is shown in Fig. 7. It can be observed that the unbalance of background in gray scale is much less after the top-hat operation. The reflected dots are clearly seen, and the gray level of background is low enough such that the reflected dots are in good contrast with their local areas.

Figure 7.

Result image for top-hat operation

3.2. Adaptive segmentation

A binary thresholding is performed to Fig. 7, and the resultant image is shown in Fig. 8. From the figure, the sizes of the identified dots (including both the reflected dots and the fake dots/noise) can be calculated; the size histogram for all the dots thus can be obtained, as shown in Fig. 9. It can observed that most reflected laser dots' sizes are roughly similar while fake dots' sizes are much smaller than those of the laser dots. Therefore, a bimodal histogram is obtainedin Fig. 9 in which most fake dots are in area F, while the majority of the reflected laser dots are concentrated in area R.

Figure 8.

Resultant image after the binary thresholding

To distinguish the reflected laser dots from fake dots an adaptive threshold is required. To this end, Otsu adaptive thresholding method is applied to the size histogram to find the optimal threshold such that the reflected laser dots can be identified from the fake dots [58].

Figure 9.

Histogram of dots' sizes, fake dots' size is small, thus they are concentrated in F. The laser dots' size is comparatively large; therefore they are focused in R

Taking the dots in Fig. 8 into calculation, the resultant threshold is 23. Using this threshold, the reflected laser dots are identified, and their positions are shown in Fig. 10.

Figure 10.

Positions of the identified laser dots

One can find all the fake dots are filtered out after the thresholding, while majority of the reflected dots is preserved. However, it can also be observed that a few reflected laser dots, 5 out of 86 dots in this case, are misjudged as fake dots. This is understandable the adaptive thresholding might not be “intelligent” enough to be able to identify all the reflected dots in each image captured during the welding process with different experiment conditions. However, although a small portion of reflected dots are temporally lost, they can be retrieved in the next section.

3.3. Row/column recognition of the reflection pattern

In order to apply the reflection pattern of laser dots to reconstruct the 3D weld pool surface, each reflected dot should be matched to its corresponding incident ray from the dot matrix first. This subsection first develops the recognition process to identify the row number of the corresponding of each laser dot in the reflection pattern. The column numbers are extracted latter in this subsection.

One can observed that the laser dots in the reflection pattern are well distributed in several smooth top-convex curves which can be roughly considered as quadratics. Fig. 11 is the illustration of the second-order polynomial fitting for the rows of the reflected dots. It can be observed that all rows of reflected dots can be modeled as the following second-order polynomial, as shown in Eq. 2, where a pixel location is presented by coordinate (x, y), and a, b, c>0. Row rα can be denoted by rα(aα,bα,cα), where variables aα,bα,andcα are what represent the row using Eq. 2, where α=1, …,R, and R is the number of the rows in the reflection pattern, in the case shown in Fig. 11, R=7.

Figure 11.

Since the correspondence of the mapping is known, the recognition process is to first assign all the reflected dots to different rows, second match to the rows to the corresponding rows in dot matrix. In particular, the first step is to define the 7 rows using Eq. 2; Second is to find the a row for each reflected dot with a pre-defined offset. Then the 7 rows are thus formed; Last step is use the reference dot (in 10th row and 10th column shown in Fig. 3) to map the rows into the dot matrix.

Using the Hugh transform method, all the rows in the reflection pattern in Fig. 10 can be identified, as shown in Fig. 12. The curves present the identified row. The reflected dots are distributed around those rows.

Figure 12.

Results of row identification

It can be found some of the reflected dots are missed in some of the rows. That is because those reflected dots are mistakenly filtered out in Section 3.2. Since all the rows are extracted, those reflected dots can be retrieved. The process is, for one specific row, 1) the median distance between two neighbor dots in the row is first calculated; 2) all the dots are visited to find out the abnormal large distance between neighbor dots. It is considered that there is for a reflected dot missed; 3) starting from the largest size, all the filtered out dots (including the fake dots, i.e., the noise) are visited to find the fittest to fill in the position of the missed reflected dot. The process goes through recursively for every row, and the filtered out reflected dots can be retrieved. The result is shown in Fig. 13. Reflected dots are marked with different shapes in different rows.

Figure 13.

Reflected dots in different rows

It can be found in Fig. 13 that there is one point missed in the second row of reflected dots from the top of the reflection image. That intentionally absent dot is the center dot (in 10th row and 10th column of the 19×19 dot matrix). It serves as the reference dot (see Fig. 3) to facilitate the row/column identification and the corresponding match between reflected dots and incident rays. To this end, the row pattern recognition can be accomplished. The result is shown in Fig. 14. The numbers in the image indicate the corresponding row match between the row in the reflection image and the incident rows in the dot matrix.

After the reference dot found in one row in the reflection image, the column number for each dot in the row can be identified, shown in Fig. 15. The distortion of the laser pattern in the vertical direction is much less severe than that in the horizontal direction. Therefore, a center line y=kx+b is fitted using reference dot together with its nearest adjacent dots in the neighbor rows.

Figure 14.

Reflected rows matching with the corresponding incident rows in the dot matrix. The numbers represent the incident rows in the dot matrix that the reflected rows match respectively.

Figure 15.

Column extraction with fitted line

4. Weld pool reconstruction

This section focuses on the development of an analytical solution that allows the 3D weld pool surface in GTAW be reconstructed in real-time. In particular, the boundary of the weld pool is extracted in section 4.1 ; the reconstruction of the 3D weld pool surface is detailed in section 4.2; An reconstruction example is given in section 4.3.

4.1. Extraction of weld pool boundary

Before the 3D surface is reconstructed, the boundary of the weld pool should be determined first.The model used to fit the boundary of the weld pool is from literature [59] and demonstrated in Fig. 16:

Figure 16.

Illustration of the weld pool boundary model.

Figure 17.

Modeling of the weld pool boundary

Fig. 3c is taken as an example to demonstrate the boundary fitting. The resultant boundary of the weld pool is shown in Fig. 17. Red stars and blue crosses show the laser dots projected on the specular weld pool surface and the raw estimates of the boundary points respectively. The rays and boundary shown in the figure are the projection of them on the oxy plane. The red stars in each row are curved because the projected rays intercept the pipe surface rather than a plate. The blue crosses are the raw estimates of the boundary points as defined above and the blue lines are the fitted boundary using Eq. 3. It can be seen that the weld pool is about 7 mm long and 6 mm wide.

4.2. Reconstruction of 3D weld pool surface

The formation of the image on the imaging plane is governed by the law of specular reflection. A reconstruction scheme thus is required to extract the 3D geometry of the weld pool surface by solving an inverse problem of the reflection law. The dot matrix reflection from the weld pool surface is demonstrated in Fig. 18.

Figure 18.

Demonstration of dot matrix reflection

The equations for all incident rays projected from laser to the weld pool surface are known. For ray Lpi, j, the ithrow jthcolumn in the pattern, the position of its reflection image ri, jin oxyz coordinate in Fig. 18 can be obtained using its position in the imaging plane (which has been extracted by the image processing algorithm) and the equation of the imaging plane in oxyz coordinate. The goal of the surface reconstruction algorithm is to calculate the coordinates of each pi, j in oxyz coordinate. Then the 3D weld pool surface can be interpolated using the coordinates from all pi, j 's.

However, without further constraints or assumptions, pi,j 's positions cannot be directly determined. Fortunately, the weld pool surface in GTAW is smooth such that two reasonable assumptions can be made:

For the most left and right dots in a row, i.e., edge dots, reflected from the weld pool surface that are close to the boundary of the weld pool surface, their deviations from the original pipe surface as measured by their z-coordinates are approximately zero.

zi,j=zi(xi,j(j)) can be modeled as a polynomial of the column numberj wherezi,j andxi,j are thez and xcoordinate respectively forpi,j. Based on the observation of the extracted pattern, a second order polynomial should be sufficient to meet the reconstruction accuracy requirement defined later in the paper.

A procedure can thus be proposed to reconstruct the 3D weld pool surface from the extracted image, i.e., the extracted laser reflection pattern, obtained from the image processing:

Step 1.Determination of Intersection Points:The surface of work piece or a previous estimate if available is used as the “previously estimated weld pool surface”. With this known surface together with known knowledge for the origin, projection angle, and internal angle of the laser pattern, the positions of the intersections of the projected rays with it can be easily acquired. That is, the coordinates for all pi, j 's in the oxyz coordinate system shown in Fig. 18 are obtained.

Step 2.Updating of Slope Field at Intersection Points:With the known equations of the incident rays and equations of the estimated reflection rays pi,jri,j’s, slopes and components∂z∂x’s and∂z∂y’sat all pi,j’s, i.e., the row and column slopes are obtained/updated.

Step 3.Updating of Intersection Points:The projection of the ith row of the incident rays in the dot matrix on the oxz plane is illustrated in Fig. 19. Updating from pi,j(0)′s(on the previously estimated or initial surface obtained in Step 1) requires to find a new set of points pi,j’s along with this row of incident rays to satisfy the row slopes∂z∂x’s obtained in Step 2 from the previously estimated or initial surface. Fig. 19 shows the case when the previously estimated surface is the initial surface (the surface of the work piece).

Figure 19.

Projection of ith row (an arbitrary row) in dot matrix to oxz plane

To analytically solve for pi,j’s, assumption (2) is used such that pi,j’s are constrained in a 2nd order polynomial:

where a0, a1 and a2 are the parameters. (Higher order polynomials may be used if a 2nd order one is not sufficient.) To be convenient, denote the edge point by pa(xa,za)and inner point bypb(xb,zb), where xm and zm is the x coordinate and z coordinate of pm. Denote their row slopes as Sa and Sb, respectively. The slopes of the two incident rays Lpaand Lpb in Fig. 19 are denotedaska and kb. The distance of the laser projection origin to the x-axis is denoted as b. Since the internal angle of the laser pattern is fixed (0.77°), the slope for each incident ray can be easily determined.

At point pm(xm,zm) wherem=a, b, the geometric relationships between incident rays and the row constraint polynomial Eq. (5) are:

From Eq. (8)-(13), the parameters of the row constraint polynomial are completely solved. However, this requires the position of the dot pa. According to the assumption (1), the z coordinate of each edge dot on the weld pool surface can be considered zero, that is za=0. Itsxacan be calculated by intersecting the corresponding incident ray with z=0. The coordinates for pa is thus considered known.

With this constraint polynomial, the intersections of the incident rays (projections on the oxz planes shown in Fig. 19) with it are the projection of the updated pi, j's. The updated pi, j's in the oxyz coordinate system as shown in Fig. 18 can thus be obtained through the inverse projection.

Step 4.Interpolating Surface:With all the coordinates of the pi, j's, a triangle-based cubic interpolation [60] method is applied to interpolate the 3D weld pool surface. With the obtained surface, the slopes of the surface at the pi, j's can be easily acquired. Therefore, the resultant reflection pattern using the reconstructed surface is obtained.

4.3. Reconstruction example

Taking Fig. 3c as an example, the results of the weld pool reconstruction using the proposed reconstruction scheme are shown in Fig. 20.

Figure 20.

Results of 3D reconstruction of the weld pool. (a) shows the 3D coordinates of all the projected laser dots on the weld pool surface, (b) is the weld pool surface interpolated using the laser dots in (a), (c) shows the reflected pattern from imaging process and that from the reconstructed weld pool surface.

Fig. 20a shows the position for each projected dot on the weld pool surface. The interpolation result for the 3D weld pool in Fig. 20b gives a better view of the weld pool surface. The comparison between the two reflection patterns (calculated from the reconstructed surface and extracted from the acquired image) is shown in Fig. 20c. An acceptable match of the reflection patterns is obtained using the proposed reconstruction algorithm.

In order to verify the accuracy of the proposed reconstruction algorithm, the results in this study are compared with the previous work in [52, 53]. The reason the two studies are selected for comparison is that high reconstruction accuracies are acquired through detailed quantitative analysis in these two studies. In order to compare the reconstruction accuracy with the previous work [53] which uses the similar sensing system, the same reconstruction error measurement parameters are adopted here, i.e., the average reflection error (ARE) and maximum reflection error (MRE):

whereWr and Lr are the horizontal and vertical ranges of the actual reflected dots extracted from image processing, respectively and Wp and Lp represent the horizontal (X axis) and vertical (Y axis) ranges of the corresponding projected dots on the work piece, which are the width and length of the weld pool boundary.

Using the same example (Fig.3c),Ek can be calculated using the data shown in Fig. 4 and Fig. 7. The ARE and MRE of the reconstruction are thus obtained: ARE=0.03mm,MRE=0.20mm. The minimal ARE and MRE obtained is [53] are 0.08 mm and 0.22 mm, respectively. Therefore, using the proposed algorithm, the error measurement parameters are 0.05 mm smaller in ARE and 0.02 mm smaller in MRE than those parameters obtained in [53]. The reconstruction accuracy increases 62.5 % in ARE and 9% in MRE.

5. Experiments results and analysis

In Section 3, the reflected laser dots have been identified from the reflection pattern. Then they have been used to reconstruct the profile of the 3D weld pool surface in Section 4. To verify the accuracy with respect to ground truth, the proposed algorithm needs to be tested by reconstructing objects with known geometry. Also, the real-time performance of the proposed scheme is not validated. In this section, using a spherical bench mark with a known geometry having a specular surface to emulate a weld pool, a simulation has been conducted to evaluate the effectiveness and accuracy of the proposed algorithm; the time cost of the algorithm was evaluated to see if it is suitable for real-time welding process; During simulation, the error measurement parameters ARE and MRE are used to evaluate the reconstruction accuracy.

To evaluate the accuracy and effectiveness of the proposed algorithm, the 3D weld pool is characterized using the three geometric parameters, i.e., the length, the width and the height, in which the length is the distance from the head to the tail of the weld pool; the width can be obtained using Eq. 4; the height is defined as the maximum height of the weld pool.

5.1. Simulation and results

The verification of the proposed reconstruction scheme starts with a bench mark with a known geometry to imitate the weld pool surface which is shown in Fig. 21. The top cap of the sphere is obtained by using a plane to intercept a sphere. It is used to imitate the weld pool surface in the experiment. Its height (d) is 0.5 mm and the diameter (r) is 5 mm.

In Fig. 22b the boundaries of the reconstructed bench mark fitted by the weld pool boundary model illustrated in Fig. 16 and a circle are presented. For the dimension of the boundary fitted by the weld pool boundary model, its length is 10.024 mm and its width is 10.32 mm which lead to 0.024 mm error in length and 0.32 mm error in width. Based on the dimension of the actual bench mark, the reconstruction errors are 0.48% of the length and 3.2% of the width respectively. Having a circle to fit the reconstructed bench mark boundary using the least squared method, the boundary diameter is 9.989 mm which is only 0.11% of reconstruction error. It can be seen that the circle can fit the boundary of the reconstructed bench mark better than the weld pool boundary model. This is understandable since the model only can fit the moving weld pool which rarely has a circular boundary.

Figure 21.

The illustration of the imitation of the weld pool

A simulation was designed to project dot matrix on the bench mark. The projected dots on the spherical surface were reflected and intercepted by an imaging plane. The results of the simulation for the reconstruction of the bench mark are shown in Fig. 22.

Figure 22.

Simulation results of the proposed reconstruction scheme. (a) The 3D coordinates of all the projected laser dots; b) The reconstructed surface of the bench mark; (c) The boundary of the reconstructed bench mark; (d) The height errors between the actual and reconstructed bench marks row by row and the static height error. Row 1 is defined as the most left reflected dots row in (a). The absolute errors between height errors and the corresponding static errors are shown in (e).

Fig. 22d show the height errors of the reconstruction at the all the x, y coordinates of the reflected dots. The sphere function for the bench mark is:

where zk(b)is the height of bench mark at position (xk,yk) which is the x, y coordinate of corresponding projected laser dotpk(xk,yk,zk). It should be noted that there is another possible solution for the height from Eq. 17, which is−R2−xk2−yk2+(d−R). However, according to the definition in Fig. 21, the height in Eq. 18 is the right solution. Therefore, the height error at position (xk,yk) is:

The height errors of the reconstruction obtained for all the projected laser dots are shown in Fig. 22(d). From Row 1 to Row 8, the errors can be observed turning from positives to negatives. That indicates static errors in the reconstruction. The static errors are obtained by fitting the height errors into a straight line as shown in Fig. 22(d). In general, static errors might be caused (in real system) by the noises from the image processing, welding machine, the data acquisition process and calibration errors of experiment set-up. Static error also occurs when the laser projection direction does not aim at the center of the weld pool. (This may also occur in simulation). Hence, the calibration is needed such that the point where the tungsten axis intersects with the surface of the work piece is also the point where it interests with the reference incident ray, i.e., the central ray or 10th row 10th column ray, in order to minimize this possible static error.

In this simulation, the experiment condition is considered ideal. The static error shown in Fig. 22(d) is thus comparatively small. Having the height errors subtracted by the corresponding static errors, the resultant absolute errors are shown in Fig. 22(e).

It can be seen in Fig. 22(e) that maximum height error is 3.20x10-3 mm at 1st point of Row 6, and the minimum height error is 1.93x10-5 mm at 11th point of Row 6. The heights of the bench mark at the corresponding positions are 0.104 mm and 0.468 mm. That is that the height error is 0.3% and 0.004% of the two heights respectively. The height error at the maximum height (8th point of Row 6) is 1.2 x10-4 mm which is 0.024% of the maximum height (0.5 mm). In comparison with the best result in study [52] which is 0.16%, the proposed algorithm achieves higher reconstruction accuracy in addition to being analytic.

The obtained results are then used to calculate the ARE and MRE (see Eq. 14 and Eq.15) to evaluate the reconstruction accuracy. The resultant ARE and MRE for the reconstruction of the bench mark are 0.06 mm and 0.21 mm, respectively. Compared with the best results in [53] which are 0.08 mm of ARE and 0.22 mm of MRE, the accuracy of the proposed reconstruction algorithm in this study is improved by 25% in ARE and 4.55% in MRE.

5.2. Time cost of the reconstruction algorithm

Since there is no difference for the reconstruction scheme for a bench mark or an actual weld pool in a GTAW experiment, the bench mark reconstruction thus was used to verify the computation time cost for the proposed reconstruction scheme. To this end, the reconstruction process of the bench mark is performed 2000 times in this study to evaluate the time cost for each reconstruction process.

Figure 23.

Computation time obtained in the reconstruction for the bench mark in 2000 attempts

The scheme is programmed by C++ in Visual Studio 2008. The computer used is a desktop with Intel i7, 2.8 GHz processor with 4 GB RAM. The computation time is recorded and shown in Fig. 23. The minimum time of one loop is 3 ms, the maximum is 20 ms, and the average of time cost is 3.04 ms.It is believed that an average time cost of 3.04 ms for the reconstruction of 3D weld pool surface is fast enough for the real-time monitoring a GTAW process, and for the future control of GTAW process.

6. Summary

The monitoring and measurement of three-dimensional weld pool surface provides the foundation for a possible advanced control for the welding process. An analytic reconstruction algorithm based on the slope field of the reflected laser pattern is proposed in this chapter to measure the 3D weld pool surface in real-time. The reconstruction is mathematically formulated. The virtual spherical convex surface, considered as similar in shape of a weld pool surface which cannot be precisely measured by any existing methods, were used to test the effectiveness and accuracy of the proposed algorithm. The following are concluded:

The proposed algorithm can effectively reconstruct the weld pool surface.

Reconstruction accuracy is improved. The boundary reconstruction error in the simulation of the bench mark is 0.11%; the height reconstruction error at the maximum height is 0.024%. The two reconstruction error parameters ARE (average reconstruction error) and MRE (average reconstruction error) are 0.06 mm and 0.21 mm. Compared with the previous works, the reconstruction accuracy is significantly improved.

The time cost for the reconstruction algorithm is 3.04 ms on average in the simulation and 3.22 ms on average obtained in the real-time welding experiment. The proposed algorithm is thus considered to be real-time for GTAW application.