MACHINE VISION APPLICATIONS

Successful examples from many industries

Formula Student: autonomous driving using 3D imaging

April 2020

Universities from all over the world participate in the Formula Student Driverless, a challenging engineering competition to develop the best driverless racing car. The municHMotorsport team relies on machine vision from STEMMER IMAGING to identify and evaluate track markings.

There is a hint of Formula 1 in the air watching the Formula Student Racing Team, members of the Munich University of Applied Sciences, diligently working together to develop and optimise their driverless racing car. Since 2005, students from various study programmes including automotive, mechanical and business engineering as well as computer science, design, business administration and many more, have been preparing for their future professional life in a fun, hands-on challenge.

Universities from all over the world participate in the Formula Student Driverless, a challenging engineering competition to develop the best driverless racing car. The municHMotorsport team relies on machine vision from STEMMER IMAGING to identify and evaluate track markings.

“There are currently around 120 students and many of us are investing a great deal of time and effort in this project”, explains Timo Socher, a computer science student who has already been involved with the project for three years. In his role as CTO Driverless for the current season, he is responsible for all technical aspects of the driverless racing car that municHMotorsport is sending to the race in 2020. At the end of the project, which is organised completely independently, a six-figure sum will have been invested in this racer, which will navigate autonomously on a race track at speeds of up to 65 km/h.

In the top discipline, track drive, the autonomous vehicles must complete ten laps on an unknown circuit of up to 500 metres in length. Source: municHMotorsport

Unknown circuit

Lap times and top speed are not the only criteria for the assessment of the cars in the battle for the winner's podium (see text box). However, the new developments have to prove their skills on the race track in three dynamic disciplines. The first one evaluates acceleration on a straight, 75-metre track from zero to a complete stop. In the second task, the so-called skidpad, the cars race on a well-known figure-eight circuit. The top discipline is the track drive: In this test, the cars have to complete ten laps on an unknown circuit that is up to 500 metres in length.

The course of all three disciplines is marked by blue and yellow traffic cones, which are placed at maximum distances of 5 metres apart, left and right along the boundary lines. "The autonomous vehicles follow these markings using the latest hardware and software technologies. They control acceleration, braking and steering movements of the racing cars to optimise the entire system for the best possible driving performance," explains Timo Socher.

Machine vision is one of the key components in our system and plays a decisive role in detecting the cones. As a sensor, it provides the subsequent evaluation systems with the basic data for the vehicle’s reactions. Hence, the quality of the environment and cone detection forms the basis for all other vehicle modules and is essential for the entire system’s stability and performance.

Timo Socher, CTO Driverless, municHMotorsport

Timo Socher explains: "Machine vision is one of the key components in our system and forms the basis for all other modules of our autonomous racing car."

Demanding object recognition

In their current racing car, the Munich students have integrated two vision systems forming the base for the track detection. One vision system is attached to the safety bar and detects the cones at greater distances at a range of five to 20 metres. For the upcoming season, this range is to be extended to up to 30 metres in order to have more calculation time for the required measurements and to be able to react even more proactively. The second system consists of two Intel RealSense cameras mounted under the front nose of the racing car, each with overlapping angles of 80 degrees. They ensure the recording of the images to evaluate the boundary markers at close distances of between 1.5 and 8 metres.

The image data is transferred to the CPU located at the racer’s rear and processed by a Jetson Xavier embedded computer in combination with other sensor data. The system uses technologies such as the merged sensor data and visual odometry which calculates the vehicle’s position and orientation by analysing the recorded images. This technology is often used in industry as the basis for robot positioning –further proof of the close connection between the municHMotorsport project and real industrial applications.

A racing car should, of course, be as fast as possible. For this reason, the images from the camera systems must be quickly and efficiently processed in the CPU. The purpose of machine vision is to locate and classify the coloured cones in the colour images and to estimate their position in relation to the camera. Object recognition is hampered by a number of unpredictable factors such as weather and light conditions, or the actual state of the race track, which can deviate from the ideal state due to potholes, unevenness or the elevation profile. Also, the background may vary from race to race due to spectator stands and other objects.

Two Intel RealSense 3D cameras in the vehicle nose providing high speed stereoscopic vision capture the race track and the marking cones at short distances.

Best results with Deep Learning

In order to be optimally prepared for all the eventualities that may arise in terms of object recognition, the team uses state-of-the-art techniques, explains Socher: "There is a large number of algorithms and neural networks for object recognition and classification in colour images. In addition, there are of course, still more conventional imaging methods, such as edge detection using special colour filters. However, we chose a deep learning approach, which proved to be particularly robust to different weather and environmental conditions and therefore promised the best results for our purpose."

Timo Socher's colleagues took the required images with traffic cones in front of different backgrounds and environments under different conditions and further enriched the training data sets using data augmentation techniques. The main idea behind this approach: The more various training data is available, the better the accuracy of the trained models. Using data augmentation techniques, existing images are easily modified by software, for example, by adding random pixel values, softening images, slightly rotating them or changing contrasts to generate a larger number of training images.“In addition, we exchanged training data with other teams in the competition and used about 3,000 images from the so-called KITTI data set, which provides images of roads without traffic cones”, Timo Socher explains.

The entire training data set now includes several thousand images and serves as the basis for the simulations used by the student team to save time in optimising the autonomous racing car.

Support from the machine vision experts

Initially, machine vision was unknown territory for many students of the Formula Student Racing team at the Munich University of Applied Sciences.

In order to gain more expertise, some of us attended training courses at the renowned STEMMER IMAGING European Imaging Academy (EIA). The training was particularly helpful for new project members to better understand the big picture and to quickly become familiar with the necessary machine vision basics.

Timo Socher, CTO Driverless, munichMotorsport

The European Imaging Academy offers a wide range of hands-on training courses, videos and events around machine vision. Beginners are introduced to the basics, while advanced users gain a more in-depth insight and useful tips and tricks to improve the efficiency of their solutions. Highly skilled experts conduct the training sessions in STEMMER IMAGING's customer centres throughout Europe. STEMMER IMAGING CTO Martin Kersting comments: “Formula Student combines theory with hands-on experience. Acquiring skills and key competencies such as interdisciplinary thinking, problem solving and business management know-how is excellently put into practice.”

The Munich University of Applied Sciences and the machine vision experts share a long-standing close relationship, the reason why STEMMER IMAGING has become a sponsor of municHMotorsport. "When developing the vision systems in our racing cars, we were able to benefit from many years of experience in the industry while being provided with powerful machine vision components such as 3D stereoscopic cameras", enthuses Timo Socher. These high-speed cameras are ideally suited for use as the eyes of autonomous racing cars due to their compact, robust design and great depth of field.

STEMMER IMAGING’s practical expertise also paid off when it came to transferring the image data recorded by both camera systems to the race car’s CPU: The cabling had to meet special requirements, as it was laid through the entire vehicle and partly alongside the power electronics."For this reason, STEMMER IMAGING produced special EMC shielded cables for our racers, which was a great help to us".

The students use a training data set of several thousand images to simulate various racing scenarios and optimise the autonomous racer’s performance.

Best conditions for the winner’s podium

For the coming racing season, the students are working flat out to develop a competitive vehicle that will represent their team, the Munich University of Applied Sciences and their sponsors in the best possible way. “We want to rank in the top three this year with our autonomous racer“, says Timo Socher, setting out the team’s goal. In any case, the team’s ambition and the machine vision systems used offer the best conditions for the winner’s podium.

International competition

The Formula Student Event is one of the world's largest interdisciplinary student competitions with teams from all over the world. Since 2017, they have been competing in three categories, building vehicles with combustion engines, electric drive systems or autonomous racing cars, each of which must comply with strict rules to ensure the safety of the racing car and the driver. Before the vehicles enter the actual competition, they are carefully checked by experts.

Once this hurdle has been taken, each car is allowed to participate in various static and dynamic disciplines in order to collect as many of the maximum 1,000 points as possible. The main discipline is Engineering Design, with a maximum of 325 points. In this discipline, the concept of the car is judged by industry representatives. The component or subsystem managers must be able to back up their decisions with data and detailed reports, thus demonstrating that an engineering approach has been followed. For the autonomous competition, there are additional auditors who evaluate only the software for the autonomous stack, from the sensors to the simulation procedures.

In each of the three dynamic disciplines, the teams can score another 350 points.

Short portrait: municHMotorsport

municHMotorsport is a student project of the Munich University of Applied Sciences and was founded in 2005 as FHM Racing. The project serves the promotion, training and further education of students from different faculties. Subject-related and interdisciplinary topics provide optimal preparation for professional life. In addition to technical skills, the focus lies on social competencies such as teamwork, project work and interdisciplinary cooperation. About 60 active students take part in international competitions of the Formula Student within this project, in order to compare their work results with other universities and colleges and at the same time to establish contacts for further sharing of experiences.

STEMMER IMAGING has been one of the leaders in the machine vision market since 1987. It is one of Europe's largest technology providers in this field. In 1997 STEMMER IMAGING presented Common Vision Blox (CVB), a powerful programming library for fast and reliable development and implementation of vision solutions, which has been deployed successfully throughout the world in more than 80,000 imaging applications in various industries.

When developing the vision systems in our racing cars, we were able to benefit from many years of experience in the industry while being provided with powerful machine vision components such as 3D stereoscopic cameras.