Polygonal Approximation of Closed Curves across Multiple Views

Transcription

1 Polygonal Approximation of Closed Curves across Multiple Views M. Pawan Kumar Saurabh Goyal C. V. Jawahar P. J. Narayanan Centre for Visual Information Technology International Institute of Information Technology Gachibowli, Hyderabad Abstract Polygon approximation is an important step in the recognition of planar shapes. Traditional polygonal approximation algorithms handle only images that are related by a similarity transformation. The transformation of a planar shape as the viewpoint changes with a perspective camera is a general projective one. In this paper, we present a novel method for polygonal approximation of closed curves that is invariant to projective transformation. The polygons generated by our algorithm from two images, related by a projective homography, are isomorphic. We also describe an application of this in the form of numeral recognition. We demonstrate the importance of this algorithm for real-life applications like number plate recognition and aircraft recognition. 1. Introduction We recognize a large number of familiar and novel objects every day with little effort. We can also recognize many objects that may vary significantly in form when viewed from different view points. Objects can be recognized even when they are partially obstructed. The recognition of objects from different views is a well studied problem in computer vision. Most of the literature, however, is confined to similarity transformations between different views, and not the general case. Planar objects form a subset of all objects but are of great practical interest. We use several planar objects every day, such as books, drawings, cloth, etc. Many non planar objects can be approximated by planar ones when viewed from sufficiently far. Many planar object recognition algorithms represent the boundary object using a single parameter vector. Polygonal approximation i.e, approximating a given closed curve as a 2D polygon provides a simple representation of the planar object boundary. Parameterizing the boundary using a polyline representation makes recognition easy. Polygonal approximation has also been used as an intermediate step in various applications such as volume rendering and multiresolution modeling [3, 12]. Let be a set of points on the boundary of a planar object to be approximated using a polygon. Polygonal approximation can be defined as a partitioning of the set into mutually exclusive and collectively exhaustive subsets such that each of the subsets can be approximated using a linear sequence of points. This can be achieved by minimization of an objective function of the form:! #$ % (1) where is the linear structure which approximates the points in and ( from. The deviation ( can be the perpendicular distance or another measure of how the linear structure distorts the real point. Thus, the deviation can be considered as the error is a measure of deviation of in fitting the linear structure to the set of points. Given, problem of identification of can be addressed using classical regression techniques or eigen vectors of the covariance matrix [4]. A general optimization of the objective function may be computationally expensive and prone to get stuck in local minima. Therefore, most of popular polygonal approximation algorithms look for an optimal solution with the help of a greedy algorithm [6, 9]. They primarily exploit the advantage that the points in the set are connected and ordered. Dynamic Strip Algorithm is one such algorithm [6] which is a fast polygonal approximation algorithm. Hopfield neural network based algorithms also have been reported for polygonal approximation [2]. Some methods do not exploit the connectedness of points [1]. They group points on the boundary into linear clusters. Some of the algorithms emphasize the optimality and efficiency of the polygon approximation [11]. The above mentioned algorithms are primarily designed to handle similarity image transformations mostly, transformations involving translation, rotation, and scaling for recognition. They do not perform well if the two boundaries being compared differ more than by a similarity transformation. When a planar object is imaged from multiple viewing positions the image-to-image transformation is projective, a lot richer than the similarity transformation [5]. In this paper, a polygonal approximation algorithm for closed curves which is invariant to projective transformation is proposed. In Section 2, we formally introduce the 1

2 problem. In Section 3, a description of a polygonal approximation algorithm which is invariant to projective transformation is presented. We discuss the implementation details of the algorithm in Section 4. Section 5 presents the results of the algorithm and describes real-life applications of such an algorithm. 2. Preliminaries 2.1. Image-to-Image Homographies When an object is imaged from multiple viewpoints, the points on the planar object boundary (and also the points inside) undergo a transformation. The transformation that the coordinates of each point of a plane undergoes from one image to the other can be mathematically described as a general projective or linear operation in homogeneous coordinates. That is, where T is a general matrix and T and T are the images of the same world point in view 1 and 2 respectively. In some special cases, the relation could have simpler forms of affine or similarity transformation. Similarity transformation deals with only the case of the camera rotating about its principal axis and/or a change in focal length. Most reported polygonal approximation algorithms are primarily designed to be invariant under similarity transformation. That is, they provide similar polygonal approximation even if the boundary of a planar object is translated, rotated or scaled. They will fail to match the boundaries if the transformation is more general, as is often the case when the object is viewed from radically different viewpoints. An affine homography between two images is represented by T matrix with its last row as. Affine transforms deform the shape of an object beyond rigid motions. They preserve parallelism and map points at infinity to other points at infinity. Affine transforms form a subset of the general projective group. A projective transformation T between two planes is represented as a non-singular matrix and is a linear mapping on homogeneous points. A mapping of four points between two planes, of which no three points are collinear, is sufficient to determine the transformation matrix T. Projective transformations, in general, do not preserve parallelism of lines and can map points at infinity to finite points and vice versa. They, however, preserve properties such as collinearity of points, and several cross ratios. It has been shown that a planar object viewed from multiple viewing positions results in a projective image-to-image homography. Consider an aerial image of an aircraft. Since the height of the aircraft is considerably less than the altitude from which it is imaged, the object can be considered planar and aircraft recognition problem can be addressed using planar object recognition algorithms. Consider an aircraft shown as in Figure 1(a). This aircraft has undergone a similarity transformation (translation, rotation, scaling) to result in Figure 1(b). Most of the existing polygonal approximation algorithms can generate same polygonal approximation (with a scale factor) for this pair of images. In Figure 1(c) and 1(d) we respectively show affine and projectively transformed versions of the same aircraft. Euclidean distances between points, angles between lines, etc., are not preserved under these transformations. In this paper, we primarily aim at developing an algorithm which can result in isomorphic polygonal approximations of a planar boundary even under the general projective transformation. (a) (c) Figure 1: An aircraft image and its transformed versions. (a) Original (b) Translated,rotated and scaled (c) Affine (d) Projective 2.2. Problem Formulation Let be the points on the boundary of an object in the first view. To approximate these points using a polygon, one can follow the same procedure discussed in the previous section by minimizing an objective function same or similar to Equation 1. be the point corresponding to in another view. Let Then,. The polygonal approximation algorithm in the second view minimizes the objective function where is the linear structure which approximates the points in. For the polygonal approximation to be invariant to projective transformation, there should be a one-toone mapping from to. We address this problem using projective invariants. % (b) (d) (2) 2

3 % 2.3. Projective Invariants An invariant,, of a geometric structure described by a parameter vector subject to a linear transformation T of the coordinates, is transformed according to [8]. Here is the function of the parameters after the linear transformation. Invariants for which are referred to as scalar invariants [8]. Consider four points. Neither distances nor ratio of distances are preserved if a projective transformation is applied to all of them, though collinearity is preserved. Below we describe two projective scalar invariants. Cross-ratio of four collinear points: points is defined as The cross-ratio of where represent the corresponding positions of each point along the line, e.g. is the distance between points and. This is invariant to a general projective transformation [8]. Cross-ratio of areas of five points: areas is defined by # (3) The cross-ratio of the! # $!! # $! where! is the area of the triangle formed by points. Clearly, the five points have to be in general positions to avoid degenerate cases. This is invariant to general linear or projective transformations [8]. 3. Projective Invariant Polygon Approximation In this section, we derive an algorithm for polygonal approximation that is invariant to a projective transformation. This algorithm can result in isomorphic polygonal approximation under projectively transformed versions of the same image. We first define a ratio of cross-ratios of areas, denoted by. Consider points and. We have, ( ( We see that the area base *),+ height.-0/21, where *),+ is the length of the line segment defined by points and and.-0/21 is the perpendicular distance of point from the line defined by points and. We (4) define a ratio of cross-ratios as % ( ( )2+ *),+ )2+ 3-0/21 *),+.-0/21 3-0/21.-0/21 3-0/ 1 3-0/21 3-0/ 1 3-0/21 which can be rewritten as %.-0/ /21.-0/ /21 We first establish three useful properties before proving the existence of projective invariant polygon approximation. Lemma 1: An invariant polygonal approximation algorithm can exist only for a transformation where collinearity is preserved. Proof: We prove this by contradiction. Say we have a polygonal approximation algorithm invariant to a transformation which does not preserve collinearity. Under such a transformation, not every line gets transformed as a line. Applying such an algorithm to the boundaries of two transformations of a planar object would result in subsets which have a oneto-one mapping. Consider one such pair of corresponding subsets, and. Let and be the linear structures for and respectively. However, under such a transformation, there would exist cases when cannot be transformed as a line. So there may not be a corresponding to. Therefore, there cannot exist a polygonal approximation algorithm which is invariant to such a transformation. 6 Since projective transformation preserves collinearity, there can exist a polygonal approximation algorithm invariant to this transformation. We now establish that the measure, ratio of cross-ratios of area, can be used as the measure in Equation 1. of deviation Lemma 2: The ratio of cross-ratios of areas can give us a measure of deviation in the objective function of polygonal approximation given by Equation 1. Proof: It is clear from Equation 5 that % can be interpreted as the ratio of the ratio of perpendicular distances of the points and +7) from the two lines - +7) and -. If this quantity is equal to 1, the two ratios of perpendicular distances are equal. +7) The locus of all such points is the moves away from straight line -. When the point this line, % moves away from 1. Thus, % gives a measure of the collinearity 8 % of the points and. A simple measure ( can then be used as the measure of deviation as given in Equation 1. 6 (5) 3

4 Another constraint for the existence of a polygonal approximation algorithm invariant to projective transformation is that the distance function should be invariant to projective transformation. Lemma 3: The ratio of cross-ratios of area is invariant to projective transformation. Proof: The result follows since the cross-ratios of areas are projectively invariant. 6. We now construct an algorithm which will minimize the optimization function with ( as defined above. This algorithm may be computationally expensive. We can additionally exploit the connectedness and the + order of the points of the boundary. This results in an algorithm for polygonal approximation where + is the number of points on the boundary. We traverse the boundary in a clockwise direction and successively insert points into the set until we encounter a point which deviates too much from the current. linear structure, measured using % Theorem 1: There exists an algorithm for polygonal approximation which is invariant to projective transformation. Proof: From the above lemmas, it is clear there is a function which is invariant to projective transformation. Also, if is invariant, the above algorithm would result in a oneto-one mapping between the subsets. Therefore, the above algorithm is invariant to projective transformation Algorithm, Implementation and Discussion We develop the algorithm in the following manner. 1. Choose point 1 as the starting point and point 5 as the point next to point 1 on the curve. Points 2, 3, and 4 may or may not lie on the curve. 2. We look at the point adjacent to 5 in the clockwise direction on the curve. Call this point 6. We measure the % quantity. 3. If 1 where 1 is a suitable tolerance threshold, the line joining points 1 and 5 can approximate the curve up to this point. We move the point 6 forward and repeat the process from step 2. +7) 4. Otherwise, the point 6 has deviated from the - sufficiently. We, therefore, approximate the curve from 1 to the point before 6 by the line joining points 1 and 5. We then repeat the procedure by taking point 6 to be the new point 1. % The algorithm described above was implemented to work with real images. Real images, however, pose many challenges. Points 3, 4, and 5 have to chosen carefully in order to overcome errors that are present in real images. The most prominent error is due to the discretization or pixelization of the curve because of which the cross-ratios calculated may differ in different views. The error associated with computing the cross-ratios from real images is discussed in [8]. In order to decrease the relative error, points 3 and 4 were taken significantly far from point 1 and from each other. Point 5 was chosen to be a fixed number of points after point 1 because for the points near point 1, cross-ratios tend to vary considerably due to discretization. The reference points should not be collinear as the area based cross-ratios may not be defined for them. Such configurations have to be avoided carefully. Due to discretization, points collinear in one view may not be collinear in another. Therefore, a different algorithm for checking collinearity was used which does not require points to be strictly collinear. Another problem arises due to the small differences in the location of point 3 and 4 in different views. However, as long as the threshold is low, this results in a difference of only 1-2 pixels in the boundary points. The choice of the threshold should depend on the curvature of the section that we are trying to approximate. For example, for a linear section the threshold should be low to retrieve the same linear section as the approximation. 5. Results The results of the proposed algorithm on two planar boundaries are shown in the Figure 2. The results were obtained on images of size. We considered only the boundary pixels for our algorithm. The original image is shown on top with two projectively transformed versions below it. The boundary pixels are drawn in black colour. The red lines in the figure show the polygonal approximation of the boundary using our algorithm. The polygon approximation has twenty sides. The number of sides of the polygon and their relative position with respect to the object remain same across all projective transformations we have considered. To quantitatively compare the results, we projectively transformed the polygonal approximation of the original image by the same projective transformations shown as green lines. The green and red polygons in the other figures are identical in all cases except one or two nodes which got shifted by 1-2 pixels due to discretization. Another example is shown on the right in Figure 2. Here an aircraft image is polygonal approximated with a twenty five sided polygon. The boundary of the aircraft is shown in black and the red lines show the polygonal approximation of the boundary. Our experience with other planar boundaries is also very good. 4

5 Figure 3: Polygonal approximation of the numeral 5 used as input to the numeral recognition system reference images of each class. The test image was assigned to the nearest class. The polygonal approximation of one of the numerals used as an input to the recognition system is shown in Figure 3. Recognition results for the numeral data are presented in Table 1. In a dataset of size 340 images, we could achieve an accuracy of 94.70%. Figure 2: Polygon approximation of two original curves and after applying two projective transformations Numeral Recognition An important application of polygonal approximation is planar object recognition [7, 10]. We demonstrate the applicability of this algorithm for recognition of numerals across multiple views. Numeral recognition has been conventionally addressed among the document image processing community. There are many other situations in image and video processing where the numerals are to be recognized under projective transformation. Conventional OCRs are not designed to address this problem. We considered numeral images of size for this experiment. Thirty four dif- ferent views of each numeral were considered for experimentation. Thirty of these images used as inputs are shown in Figure 4. The boundaries of these numerals were extracted and were approximated by a polygon using our algorithm. A feature vector was then formed by taking the cross-ratio of areas of every five consecutive boundary points. In order to classify the test images, we employed a nearest neighbour classifier. We calculated the Euclidean distance of the feature vector of the test image from the feature vectors of the Digit Number of Images Accuracy(%) Total Table 1: Recognition accuracy for the numerals Application Domains Number Plate Recognition: There are many number plate detection algorithms for images and videos. Coupled with this we need a module which can recognize alphanumerals under projective transformation. In a number plate recognition system, the input is a projectively transformed image of numerals and alphabets, as shown in Figure 5.It s objective is to identify the number on the number plate. To test the feasibility of number plate recognition using our algorithm, we analyzed a few real images of number plates. The test was limited to recognition of digits and can be easily extended for alphabets as well. A sample polygonal approximation is shown in Figure 6. We are yet to test this system on a large database of number plates. 5

6 Figure 4: Some of the inputs used in the numeral recognition system References Figure 5: Two projectively transformed images of a number plate Figure 6: Polygon approximations of real images of 4 and 8 extracted from a number plate Aircraft Recognition: In an aircraft recognition system, we are given projectively transformed images of aircrafts of various aircraft models for recognition. The system should be able to identify the aircrafts correctly by distinguishing between various models. The polygon approximation of one such aircraft is shown on right side of Figure 2. The system has not been tested with a large database of aircraft images. 6. Conclusions and Future Work We presented an approach to generate a projectively invariant polygonal approximation using invariant properties of the cross-ratio of areas. We demonstrated how planar shape recognition can be achieved using this algorithm. This can be applied to real-life problems such as number plate recognition and aircraft recognition. We plan to use polygon approximation for object recognition with occlusion. Also, other applications have to be explored like shape from texture. Repeated texture elements can be polygon approximated and may be used for reconstruction of shape. [1] I. Anderson and J. Bezdek. Curvature and tangential deflection of discrete arcs. T-PAMI, 6:27 40, [2] P. Chung. Polygon approximation using a competitive hopfield neural network. Pattern Recognition, 27(1): , [3] M. DeHaemer, Jr. and M. J. Zyda. Simplification of objects rendered by polygonal approximations. Computers and Graphics, 15(2): , [4] R. O. Duda and P. E. Hart. Pattern Classification and Scene Analysis. John Willey Sons, New York, [5] R. Hartley and A. Zisserman. Multiple View Geometry. Cambridge University Press, [6] M. K. Leung and Y. Yang. Dynamic strip algorithm in curve fitting. In Computer Vision,Graphics and Image Processing, volume 23, pages 69 79, [7] M. Lourakis, S. Halkidis, and S. Orphanoudakis. Matching disparate views of planar surfaces using projective invariants. In British Machine Vision Conference, volume 1, pages , [8] J. L. Mundy and A. Zisserman. Geometric Invariance in Computer Vision. MIT Press, [9] T. Pavlidis and S. Horowitz. Segmentation of plane curves. In IEEE Transactions on Computers, volume C-23, pages , [10] C. Rothwell, A. Zisserman, D. Forsyth, and J. Mundy. Planar object recognition using projective shape representation. IJCV, 16:57 99, [11] M. Salotti. An efficient algorithm for the optimal polygonal approximation of digitized curves. Pattern Recognition Letters, 22(2): , February [12] P. Shirley and A. A. Tuchman. Polygonal approximation to direct scalar volume rendering. In Proceedings San Diego Workshop on Volume Visualization, Computer Graphics, volume 24, pages 63 70,

1 2 The calibration problem was discussed in details during lecture 3. 3 Once the camera is calibrated (intrinsics are known) and the transformation from the world reference system to the camera reference

Geometry Performance Level Descriptors Limited A student performing at the Limited Level demonstrates a minimal command of Ohio s Learning Standards for Geometry. A student at this level has an emerging

MA 323 Geometric Modelling Course Notes: Day 02 Model Construction Problem David L. Finn November 30th, 2004 In the next few days, we will introduce some of the basic problems in geometric modelling, and

Face Recognition using Principle Component Analysis Kyungnam Kim Department of Computer Science University of Maryland, College Park MD 20742, USA Summary This is the summary of the basic idea about PCA

Section 12.1 Translations and Rotations Any rigid motion that preserves length or distance is an isometry (meaning equal measure ). In this section, we will investigate two types of isometries: translations

Epipolar Geometry We consider two perspective images of a scene as taken from a stereo pair of cameras (or equivalently, assume the scene is rigid and imaged with a single camera from two different locations).

Geometry: Unit 1 Vocabulary 1.1 Undefined terms Cannot be defined by using other figures. Point A specific location. It has no dimension and is represented by a dot. Line Plane A connected straight path.

NEW YORK STATE TEACHER CERTIFICATION EXAMINATIONS TEST DESIGN AND FRAMEWORK September 2014 Authorized for Distribution by the New York State Education Department This test design and framework document

CHAPTER 3 Truss Element 3.1 Introduction The single most important concept in understanding FEA, is the basic understanding of various finite elements that we employ in an analysis. Elements are used for

A Short Introduction to Computer Graphics Frédo Durand MIT Laboratory for Computer Science 1 Introduction Chapter I: Basics Although computer graphics is a vast field that encompasses almost any graphical

John Vince Essential Mathematics for Computer Graphics fast Springer Contents 1. MATHEMATICS 1 Is mathematics difficult? 3 Who should read this book? 4 Aims and objectives of this book 4 Assumptions made

Solving Simultaneous Equations and Matrices The following represents a systematic investigation for the steps used to solve two simultaneous linear equations in two unknowns. The motivation for considering

13 Groups of Isometries For any nonempty set X, the set S X of all one-to-one mappings from X onto X is a group under the composition of mappings (Example 71(d)) In particular, if X happens to be the Euclidean

Terms and Definitions Absolute Value the magnitude of a number, or the distance from 0 on a real number line Additive Property of Area the process of finding an the area of a shape by totaling the areas

Algebra I Common Core Standards Algebra 1 Overview Number and Quantity The Real Number System Extend the properties of exponents to rational exponents. Use properties of rational and irrational numbers.

Geometric Camera Parameters What assumptions have we made so far? -All equations we have derived for far are written in the camera reference frames. -These equations are valid only when: () all distances

MATHEMATICS: THE LEVEL DESCRIPTIONS In mathematics, there are four attainment targets: using and applying mathematics; number and algebra; shape, space and measures, and handling data. Attainment target

Senior Secondary Australian Curriculum Mathematical Methods Glossary Unit 1 Functions and graphs Asymptote A line is an asymptote to a curve if the distance between the line and the curve approaches zero

I The Real and Complex Number Systems 1. Identify subsets of complex numbers, and compare their structural characteristics. 2. Compare and contrast the properties of real numbers with the properties of

Chapter 4 The Scientific Data Mining Process When I use a word, Humpty Dumpty said, in rather a scornful tone, it means just what I choose it to mean neither more nor less. Lewis Carroll [87, p. 214] In

Big Ideas in Mathematics which are important to all mathematics learning. (Adapted from the NCTM Curriculum Focal Points, 2006) The Mathematics Big Ideas are organized using the PA Mathematics Standards

Grade 8 Overview View unit yearlong overview here The unit design was created in line with the areas of focus for grade 8 Mathematics as identified by the Common Core State Standards and the PARCC Model

Classifying Manipulation Primitives from Visual Data Sandy Huang and Dylan Hadfield-Menell Abstract One approach to learning from demonstrations in robotics is to make use of a classifier to predict if

Course Project (Introduction to Reflection Groups) Construction and Properties of the Icosahedron Shreejit Bandyopadhyay April 19, 2013 Abstract The icosahedron is one of the most important platonic solids

Chapter 27 Solutions PSS 27.2 The Electric Field of a Continuous Distribution of Charge Description: Knight Problem-Solving Strategy 27.2 The Electric Field of a Continuous Distribution of Charge is illustrated.

Infinite Algebra 1 Kuta Software LLC Common Core Alignment Software version 2.05 Last revised July 2015 Infinite Algebra 1 supports the teaching of the Common Core State Standards listed below. High School

2.3 Scheduling jobs on identical parallel machines There are jobs to be processed, and there are identical machines (running in parallel) to which each job may be assigned Each job = 1,,, must be processed

Homework Exam 1, Geometric Algorithms, 2016 1. (3 points) Let P be a convex polyhedron in 3-dimensional space. The boundary of P is represented as a DCEL, storing the incidence relationships between the

SOL 8.1 exponents order of operations expression base scientific notation Represents repeated multiplication of the number. 10 4 Defines the order in which operations are performed to simplify an expression.

ÌÖ Ò ÓÖÑ Ø ÓÒ Ó Ø ÈÐ Ò A transformation of the Euclidean plane is a rule that assigns to each point in the plane another point in the plane. Transformations are also called mappings, or just maps. Examples

Geometric Transformations Definitions Def: f is a mapping (function) of a set A into a set B if for every element a of A there exists a unique element b of B that is paired with a; this pairing is denoted

Math 366 Lecture Notes Section. Basic Notions (of Geometry) The fundamental building blocks of geometry are points, lines, and planes. These terms are not formally defined, but are described intuitively.

Math 5. Selected Solutions for Week 2 Section. (Page 2). Let u = and A = 5 2 6. Is u in the plane in R spanned by the columns of A? (See the figure omitted].) Why or why not? First of all, the plane in

11 Vibration Analysis 11.1 Introduction A spring and a mass interact with one another to form a system that resonates at their characteristic natural frequency. If energy is applied to a spring mass system,

Isometries of the Plane Teacher s Notes Henri Picciotto This unit is intended to be consistent with the Common Core State Standards for Mathematics (CCSSM), but it does go quite a bit further than is required

Math 497C Sep 17, 2004 1 Curves and Surfaces Fall 2004, PSU Lecture Notes 4 1.9 Curves of Constant Curvature Here we show that the only curves in the plane with constant curvature are lines and circles.

Coefficient of Potential and Capacitance Lecture 12: Electromagnetic Theory Professor D. K. Ghosh, Physics Department, I.I.T., Bombay We know that inside a conductor there is no electric field and that

Collinearities in Kinetic Point Sets Ben D. Lund George B. Purdy Justin W. Smith Csaba D. Tóth Abstract Let P be a set of n points in the plane, each point moving along a given trajectory. A k-collinearity

Outline 1. Short introduction: points and lines Epipolar Geometry Prof. D. Stricker 2. Two views geometry: Epipolar geometry Relation point/line in two views The geometry of two cameras Definition of the