Vision Sensors Hit the Road

No longer confined to the factory floor, machine vision technology is going for a spin. The same technology that determines whether a label is missing is being pressed into service to manage traffic, to classify cars for the purpose of collecting fines or tolls, and to help drivers avoid each other and pedestrians. This could be a significant market for machine vision vendors, but there are some problems, ranging from technical issues to cost, to be overcome. A look at several examples shows how vision sensors are not getting stuck in traffic.

Mounted on a gantry above passing traffic, several vision systems check license plates and classify vehicles so that the proper tolls can be assessed across multiple lanes. Because of the speed with which this can be done, the traffic flows without stopping, and road capacity therefore is not affected. Courtesy of Vitronic GmbH.Paying the toll

Unlike other machine vision companies, Vitronic GmbH of Wiesbaden, Germany, has deep roots in transportation. The company got its start in the mid-1980s with the development of a laser-based navigation system for automatic vehicle guidance. Today it has two traffic product families — one for speed and the other for toll enforcement — that use machine vision technology such as illumination, software and specialized cameras with resolutions of many megapixels.

“We have a wide range of cameras, from low- to high-resolution high-definition cameras,” said Vitronic’s international sales manager Daniel Scholz.

It’s not the case that one configuration fits all, he noted. “It also depends on the specifications of what the customer is looking for.”

In the company’s speed enforcement products, measurements of a vehicle’s velocity are performed using lidar. The camera captures identifying information about a particular vehicle; for example, it might image the license plate so that optical character recognition technology can extract letters and numbers. However, laws govern how this is done. In some regions, images of the driver and the front license plate are taken. In others, only a rear view of the license and no driver is captured.

A lidar system in the tower shown here determines the speed of passing vehicles. If it is above the posted limit, a vision system captures the information needed by authorities to issue a ticket. This data usually includes the license plate and sometimes an image of the driver as well. Courtesy of Vitronic GmbH.When multiple lanes must be surveyed, a multicamera system can be used. Because ambient conditions can vary from full sunlight to an overcast night, the system supplies its own lighting, often with a nondistracting infrared flash providing illumination. The cameras operate in the near-infrared, although they may provide only black-and-white images.

In a few places, the situation is a bit more complex. One such is in the United Arab Emirates, where taxis, government vehicles and average cars have license plate numbers that are colored differently. That can create problems for a black-and-white imager, explained Scholz. “You can have the same number but in a different color.”

The company’s solution is to extract the information with an appropriate color camera and visible light source. A trigger based upon the vehicle’s location helps in such traffic imaging because the system can take the clearest possible shot.

This image of a car was captured during a test of an automated speeder-catching unit that is deployed in France. When triggered by radar, a vision system snaps an image of a speeding car. If the system is fast enough and has high enough resolution, the license plate can be seen easily. Letters and numbers can be extracted with optical character recognition, and a ticket is mailed to the offending driver. Courtesy of Allied Vision Technologies GmbH.Scholz noted that the systems often pay for themselves. When the police in Wiesbaden installed one system, he said, they caught several hundred people a week speeding. Thus, the setup quickly paid for itself in traffic fines.

In toll enforcement, regional differences also play a role. Germany, for example, has a regulation whereby trucks and other haulers of heavy goods must pay to use the roads. The toll depends on the dimensions of the vehicle, on the number of axles and on the presence or absence of trailers. Thus, trucks must be identified and classified in free-flowing traffic and at traditional toll plazas for proper charges to be assessed.

Again, Vitronic uses cameras as well as optical and pattern recognition technology to execute this task.

Keeping cool

Based in Stadtroda, Germany, Allied Vision Technologies GmbH has had some exposure to transportation, but this domain is not the majority of the company’s business, said marketing director Henning Staerk. However, the company does see this segment as having the potential for growth.

For this market, as for others, the company makes machine vision cameras that its customers use in their own products. These applications include inspection of road surfaces, railway equipment and railways themselves. A notable transportation-related example is a large-scale French traffic safety program launched early this decade. Part of the effort included automated speed control devices on major roadways. Since 2004, the company’s digital cameras have been installed in these devices.

One problem confronting such applications is the distance between cameras and the processing computer, Staerk noted. “To address that issue, we offer a range of solutions using glass optical fiber technology to bridge up to several hundred meters’ distance with a FireWire digital interface.”

This approach is possible because of the advent of IEEE 1394b, one of several FireWire incarnations. Using that and glass optical fiber, the company’s products can support runs of up to 500 m, with the added benefit that such a cable is immune to electromagnetic interference. Just as importantly, the cable doesn’t produce any interference, which can be important in some situations. All of the company’s products use a FireWire interface.

Another problem that any outdoor application faces is temperature. For example, a camera may be situated inside a casing along with a radar unit, with the latter determining speed while the former takes an image for identification. On a hot summer day, the temperature inside such a casing can be high.

Although solving that problem is largely an issue for the application developers, Allied Vision Technologies does try to help. Its latest camera family, the Stingray, was designed to dissipate heat from the CCD sensor, which can be up to 2 megapixels. At that size, the camera can capture 14 fps. Staerk said that, in the future, the passive cooling could be augmented with an active, fanless approach.

“We also plan to offer an optional Peltier cooling system to be triggered by the camera itself when required, thanks to the built-in temperature sensor,” he said.

Watching the flow

Vision systems are being used for more than just handing out tickets and collecting tolls. They also are being employed to smooth the flow of traffic. Anaheim, Calif.-based and privately held Econolite Control Products Inc. uses digital video cameras with CCD sensors to image traffic intersections at tens of thousands of locations. Among other things, the company’s products provide the signals that turn a red light green when a car is waiting. Such switching depends upon the ability of the camera and associated software to determine when a car is stopped at the light. This identification must be made in good and bad weather and without responding to images that are not from cars, such as passing clouds or shadows.

Protected against the elements and mounted high for a good view, digital video camera-based systems identify cars so that signals can be adjusted to keep traffic flowing smoothly. Courtesy of Econolite Control Products Inc.“There could be other objects moving across the field of view. We work very, very hard to distinguish the vehicle,” said Econolite spokesman Frank Provenzano.

The company has been helped in this by years of algorithm development with strategic partner Image Sensing Systems Inc. of St. Paul, Minn. An upgrade of hardware from black and white to color several years ago also helped. Even with all those technology advances, a key factor is the camera’s positioning, which must account for challenging circumstances; for example, a small car in a turn lane sitting next to a large truck, which hides it from certain points of view.

Econolite’s products come as a package, with an enclosure to keep out the environment and communications links to transfer data. The resolution is either NTSC or PAL, two analog television broadcast standards. Thus, the number of effective pixels tops out at about 400,000, with the NTSC configuration slightly below that number and the PAL somewhat above. The systems offer a zoom function, and magnifications of up to 22× are available in Econolite’s Autoscope Solo Terra sensor, for example.

Besides acting as part of the traffic management system, the digital image processing in the video camera allows the cameras to be used for detection of speed and incidents as well as for collection of traffic flow data. They can, for instance, capture how many cars are turning at a given location. Such information can be used to set the timing of signals or even to justify the installation of a turn lane.

This type of camera also is used by state departments of transportation to show road and traffic conditions to the public via the Internet. This use is part of an overall trend toward open standards, higher bandwidth and increased connectivity, Provenzano said.

As for the future, he added that one consequence of the move toward high-definition television will be new demands placed on traffic video cameras. Both the public and traffic management professionals will want high-resolution video streamed out at full frame rates. “The demand is definitely going to be there. We’re already starting to see that.”

Staying on track

Not all vision systems are installed near roadways. Some are on the road itself, packaged into systems found in cars. An example can be found in products offered by the Troy, Mich.-based Delphi Corp., a global supplier of electronics for the automotive industry. Earlier this year, the company announced that its active safety system was being incorporated into several new models from a European automaker. This system fuses radar and CMOS vision sensors, with the first detecting an object’s range and its rate of approach and the second sorting objects into categories such as pedestrians or cars, with the actions taken depending on that classification. Vision systems also handle other tasks, such as determining whether the lights or wipers should be on. Optical sensors also play a part in lane departure warning systems, determining when this happens much as a person does. The vision sensor looks for the stripes in the highway and alerts the driver if these are crossed by the car.

Vision sensors provide lane departure warnings and detect pedestrians (top) by using pattern recognition to identify lane markings painted in the road and by classifying people based on their shape. Related technology can be useful at night (bottom). For example, a sensor mounted in or near the rearview mirror (bottom left with camera indicated by arrow) can detect whether it is night so that the lights should be on, as well as switch from high to low beams based upon the absenceor presence of oncoming traffic. Using infrared illumination and the near-infrared sensitivity of cameras can enable active night vision (bottom right). Courtesy of Delphi Corp.Because many sensors have some sensitivity in the near-infrared, the company can exploit this quality by using infrared illuminators. These can beam down the road without blinding other drivers, with the sensors capturing the image. This information then can be presented to the driver, improving nighttime safety.

William G. Shogren, chief engineer for advanced vehicle safety systems at Delphi, said that vision sensors in the future will remove one excuse for speeding. “We plan to use cameras for things like traffic sign recognition so we can provide the information for the driver before they get to that speed zone,” he explained.

However, any vision sensor in a car faces some serious constraints. One is the environment, which tends to be dusty, subject to temperature extremes and full of vibration. There also is the issue of electromagnetic noise, either produced by or received by the camera. Such interference can play havoc with data transmission between the sensor and a processor.

There are other problems for vision sensors. The best place to put the sensors, said Shogren, is behind the rearview mirror, a very confined space. For space and cost reasons, then, the smallest number of sensors should be used.

On the other hand, the demands of the various vision applications conflict, making this merger difficult. For example, lane departure warning systems need a wide but not necessarily deep field of view. A night vision system, in contrast, must look far down the road, but it doesn’t necessarily have to cover a wide span.

That combination of environment, cost and performance is a challenge for today’s standard camera components, and the solution often requires special sensors or packaging. The situation also means that in-car vision sensors lag behind those found in consumer or even other machine vision products in terms of resolution.

Typically, the cameras used in a car today are VGA, or 640 × 480 pixel, resolution. That’s a little less than a third of a megapixel — far short of the many megapixels common in other vision applications. This deficit, when compared with other areas, is not a matter of choice, Shogren said. “The issue for us is to get an automotive-grade imager with the same resolution.”