Taking The Next Step

During a recent large-scale field test of a new stereo vision system for detecting pedestrians at street crossings, FHWA focused on the visually impaired.

Two individuals in wheelchairs wait to cross an intersection in Portland, ME, one of four test sites for a new stereo camera system for detecting visually impaired pedestrians.

According to the 2011 census, people 65 and older make up more than 13 percent of the U.S. population. Perhaps you have seen an elderly or visually impaired person attempting to finish crossing a street as the pedestrian phase of the traffic signal ended? It can be unnerving if impatient drivers start honking or, worse still, start driving. On the other hand, perhaps you have stopped your own vehicle at an intersection and waited and waited for the traffic signal’s pedestrian phase to end -- with no pedestrians in sight.

In an effort to demonstrate to traffic engineers that it is operationally safe to install pedestrian sensors to eliminate these problems, the Federal Highway Administration (FHWA) conducted a large-scale field test of a stereo vision system for detecting pedestrians at signal-controlled intersections. Stereo vision, also called stereopsis and binocular vision, uses two views of a scene viewed with two cameras to compare slightly different pixel images to extract information that can be used to calculate depth.

In March 2010 an FHWA contractor, Migma Systems, Inc., conducted the field test in four cities in the Southwest and Northeast under FHWA’s Small Business Innovation Research (SBIR) program. These cities were carefully selected based on criteria such as the volume of sighted and visually impaired pedestrians and weather patterns.

Proving that stereo vision will reliably detect pedestrians in real-world environments is critical to convincing engineers that using these systems is safe for improving intersection crossings. The real world, of course, means under all types of weather conditions (sunny, cloudy, foggy, rainy, and snowy), including extreme temperatures (very high in southern parts of the United States and very cold in northern regions). It is critical to determine that stereo vision systems effectively actuate pedestrian calls and trigger the audible locator tones on accessible pedestrian signals. Compared to regular pedestrian pushbuttons, accessible pedestrian signal pushbuttons are designed specifically for pedestrians who are visually impaired and blind to cross safely through a combination of functions such as locator tone, audio messages, vibro-tactile, and tactile arrows. Section 4E.09 of the Manual on Uniform Traffic Control Devices (MUTCD) defines accessible pedestrian signals as providing “information in nonvisual formats” and provides information on their correct usage. The locator tones help pedestrians who are visually impaired and blind quickly locate the pushbuttons that activate pedestrian calls.

The FHWA contractor, in cooperation with the local departments of transportation (DOTs), selected test sites where a number of pedestrian fatalities had occurred in recent years. The sites selected were ones that are used by a significant number of pedestrians daily, especially pedestrians who are visually impaired. The researchers conducted the tests in Tucson, AZ; Somerville, MA; Portland, ME; and Manchester, NH -- sites selected for their temperature extremes and wide variations in weather conditions.

Pinpointing the Problem

Pedestrians who are visually impaired can easily recognize that they are at an intersection with an accessible pedestrian signal when they hear the locator tone, which helps them find the pushbutton. After locating the button (the same one used by all pedestrians), they are typically required to push it to request the traffic signal’s pedestrian phase.

In the absence of a locator tone, pedestrians who are visually impaired can find it challenging to locate pushbuttons, and most will not attempt to do so unless they already know a crossing has a pushbutton and they know its location. Instead, they will listen for traffic flow.

The MUTCD requires locator tones to operate constantly. To evaluate the effectiveness of stereo pedestrian detection and consequent actuation of locator tones and pedestrian calls, all stereo detection units installed in the field test were capable of actuating the pushbuttons based on detection of pedestrians in an area near the pushbutton. Because locator tones can be loud and can disturb nearby residents, they are supposed to be audible only to pedestrians who are within 12 feet (3.7 meters) from the pushbuttons, as defined by the MUTCD.

The Field Test

The contractor conducted the field test in two stages. In stage I, the stereo units installed at the test sites were stand-alone, not interfaced with the traffic signal system. The units merely detected pedestrians and recorded the images saved on external hard drives that were periodically swapped with new external drives before their storage capacities became full.

Continuing stage I, the contractor’s research personnel manually scanned the saved images in two categories: regular images saved every second and specific images saved only whenever pedestrians were detected. To evaluate the system’s performance, the researchers subsequently measured both the correct detection rate of pedestrians that were there and the number of detections of pedestrians that were not there (false calls). These were captured on the external drive. For example, if an image in the detection category does not have any pedestrians, this detection is treated as a false call. Similarly, by comparing the detection images with regular images, the system detection accuracy can be estimated as well. The combination of detection accuracy and false call rate represents the overall performance of a pedestrian detector.

A stereo camera, indicated by a red circle, is mounted on an intersection’s signal pole, as part of a stereo vision system for detecting pedestrians waiting to cross the street.

When performance at all field test sites was consistently satisfactory with high detection accuracy and few false calls per day, the test entered stage II. The local DOT engineers connected the stereo units directly to the pushbuttons and traffic signals, so that service calls on behalf of detected pedestrians could be placed as they waited to cross intersections. When the stereo units are connected to the pushbuttons and traffic signals, false calls place service calls and potentially delay traffic flow and reduce the trust that people have in the system, both of which can increase impatience and risky behavior.

The locator tones performed in typical fashion, sounding once per second during the “Don’t Walk” and “Flashing Don’t Walk” intervals. During the walk interval, only the “Walk” message conveying the information on which approach is safe to cross is conveyed audibly to the pedestrian, and the locator tone is not on. All systems were in operation 24/7, and local DOT personnel in all four cities carefully monitored the systems’ performance and gathered comments from the public through communication with pedestrians at the test site and public feedback channels, such as phone calls.

“The intersection is quiet until someone walks up to the crosswalk, which I think is great given the complaints we get of the constant beeping at night,” says Kevin Thomas, traffic operations supervisor with the DOT in Portland, ME. “Overall, I think it’s working fine.”

Shown here is a typical two-zone detection architecture. The actuation zone for the automated pushbutton is shown in yellow, and the triggering zone for the pushbutton locator tone is shown in red.

To better understand the benefits to pedestrians who are visually impaired and blind, a followup stage took place in Portland. The researchers selected this site because of the numerous visually impaired pedestrians living nearby and using the intersection daily. A group of researchers conducted a human factors study of the interaction of the system with 18 pedestrians who are visually impaired or blind. Overall feedback from the participants of this study was positive.

Stereo Detection System

At each of the four sites, the system used in the field test consisted of a stereo camera with two lenses; an industrial-grade, single-board computer; and video cables. The stereo camera’s two lenses were equipped with 24 infrared LEDs for low-light operation. The infrared LEDs enabled the cameras to detect pedestrians up to 80 feet (24 meters) away in areas that had few or no streetlights.

Stereo cameras take two images of the same scene from two slightly different viewing angles. From the images, algorithms extract three-dimensional features in order to detect human bodies. With stereo imaging, the system can achieve a higher detection rate (greater than 98 percent) than single imaging and a lower number of false calls (fewer than five per day). The detection algorithm filters out most problems caused by changing illumination conditions, clouds, sun, and background and shading due to moving objects.

To detect pedestrians waiting at an intersection, the stereo camera is often mounted nearby at a height approximately 15 feet (4.6 meters) above the ground. The detection zone can be drawn through a software configurator, and pedestrians will be detected anywhere in the zone.

For the field tests, the local traffic engineers placed the computers inside the traffic controller cabinets. Video cables transmitted the camera images to the computers, which processed them to detect pedestrians. The engineers wired separate relay cables between the computers and the accessible pedestrian signal pushbuttons for the visually impaired that activate voice messages. For a typical intersection in the field test, there were eight accessible pedestrian signal pushbuttons, four computers, and eight stereo cameras to cover all four crosswalks.

The contractor installed this stereo camera system outside its research laboratory to create a simulated pedestrian detection area for testing the equipment prior to the large field test.

The field test had a total of 17 computers and 34 stereo cameras installed at locations in four cities: 8 systems at Tucson, 4 systems at Somerville, 5 systems at Portland, and finally, 2 systems at Manchester.

The stereo pedestrian detection system automatically triggers the locator tone and actuates the pushbutton based on the presence and location of pedestrians. There are two detection zones: the first for automated triggering of the locator tone and the second for actuation of the automated pushbutton. The pushbutton locator detection turns on the locator tone for a period of time to assist visually impaired pedestrians. The pushbutton actuation places a call for the pedestrian phase.

Preparation for The Field Test

At the start of this large-scale field test, the contractor’s engineers constructed the detection units and tested them for stereo alignment and reliability. The researchers then installed and tested the software in each unit. All stereo cameras and electronics went through rigorous tests at a laboratory test site located outside the contractor’s office before being placed in the field. The laboratory test site simulated a typical intersection corner and accommodated varying distances between the stereo camera and the intersection’s pedestrian ramp.

To prepare for the field test, the contractor’s researchers collected and evaluated detection images and once-per-second “ground-truth” images containing pedestrians. They developed a folder structure on the external drives and naming scheme to facilitate logging and organizing the data for analysis during the field tests.

Test Site Selection

Each participating local DOT recommended candidate sites based on the number of pedestrian incidents in recent years and pedestrian volume at the sites. Local DOT engineers and the contractor’s researchers then inspected each test site. Using selection criteria such as volume of sighted and visually impaired pedestrians, they chose the final sites and then scheduled subsequent field installations.

Tucson has a total of five test sites, which include one signalized intersection and four signalized midblock crossings. The researchers installed walktime extension systems, which can grant more walking time for slow pedestrians to cross a long crosswalk, at two of the midblock crossings: East Broadway Boulevard at Fellowship Square and West Kelso Street/North Oracle Road. Both midblock crosswalks are 120 feet (37 meters) long. Instead of extending the crossing time to the maximum duration for each pedestrian phase, the systems at these two sites can extend walktime based specifically on a request from the pedestrian sensor identifying the presence of slow-moving pedestrians and sending a request for a walktime extension to the traffic signal controller, if it has that feature.

This aerial photo shows the extra-long crosswalk on East Broadway Boulevard at Fellowship Square in Tucson, where researchers installed equipment to extend pedestrian walktime phases.

This aerial photograph shows another extra-long crosswalk at West Kelso and North Oracle in Tucson that also required walktime extension capabilities.

The test site in Somerville is a very large signalized intersection in Massachusetts. Located next to a recreational park and residential buildings, the intersection experiences high pedestrian volumes, both in the daytime and at night.

The local DOT in Portland recommended two signalized intersections: one at Deering and Park Avenues and the other at Commercial and Franklin Streets. The Deering Avenue site is within walking distance of an education and rehabilitation center for people who are blind or have low vision.

Shown here from above is the Deering and Park Avenues test site in Portland.

The test site in Manchester is an intersection next to the entrance to the Catholic Medical Center. The pedestrian volume is extremely high, with many patients and visitors having vision and mobility impairments.

The intersection test site near the entrance to the Catholic Medical Center in Manchester, NH, experiences heavy use by pedestrians with vision and mobility impairments. The pedestrian sensor is on the signal pole marked by a red circle.

Standard Lab Tests for Hardware Components

Past experience in State and city DOTs has shown that the surface street environment in which traffic control equipment must survive is harsher on equipment than the unchanging vacuum of interstellar space. The Massachusetts Department of Transportation (MassDOT) recommended that the equipment for the field test meet the standards specified in the U.S. Department of Defense Test Standard for Environmental Engineering Considerations and Laboratory Tests document (MIL-STD-810F).

A series of lab tests by an independent firm that specializes in traffic equipment tests confirmed that the specified hardware met the requirements for high and low temperatures, humidity, rain, icing, vibration, and Federal Communications Commission (FCC) guidelines. The cycles of high and low temperatures included three sets of 24 hours each. The hardware met the requirement of withstanding temperatures ranging from -31°C to 70°C (-24°F to 158°F). The resistance of equipment to warm, humid atmosphere was proven during an 11-day test period with a series of 48-hour test cycles. The temperature varied between 25°C and 40°C (77°F to 104°F) at 95 percent humidity.

The rain test verified the effectiveness of protective covers, cases, and seals in preventing damaging penetration of water. The simulated rain had a droplet size in the 0.5-millimeter to 4.5-millimeter (0.02-inch to 0.18-inch) range at approximately 267 kilopascal, kPa (40 pounds per square inch, psi). Because rain, drizzle, fog, and splash in cold weather can produce icing or freezing rain, the researchers tested for these conditions as well.

Roadways experience significant levels of vibration due to passing heavy vehicles, such as dump trucks and tractor trailers. The researchers successfully tested the equipment using the techniques described by method 514.5 of the MIL-STD-810F document.

As a final step, the researchers needed to ensure that the equipment used would not interfere with other electronics. Therefore, they tested the equipment for compliance with the FCC standards for unintentional radiators to ensure that it would not affect nearby heart pacemakers, electronics, or radio and computer equipment. The study’s equipment passed all of these tests successfully.

Performance Results Based on Full-Day Scan

During the field test, the stereo unit captured images of the detection zone at a rate of one per second to ground-truth the images flagged as detected pedestrians. Engineers compared the ground-truth images with the pedestrian-detection images to determine how many pedestrians were correctly detected by the system and how many nonexistent pedestrians caused false calls.

One major challenge related to manual scanning was the sheer quantity of images. One single board computer with two stereo cameras will accumulate 15,552,000 images over a period of 3 months. With 17 computers installed for the field test, more than 260 million images accumulated in 3 months. Evaluating these images by hand would have been prohibitively time-consuming.

Therefore, to obtain an accurate estimation of system performance, two scanning methods were employed: full-day scans and random samplings. The full-day scans were needed for a portion of the data to understand the performance of the system. To scan all of the data within the limited time and budget, a random sampling approach was used. The random sampling approach provided a statistical estimation of the system performance, while the full-day scans provided a precise, or better, performance evaluation based on the data scanned.

For the full-day scan, the engineers manually reviewed 28 days’ worth of data from the four cities and a variety of weather conditions. They used the following criteria for evaluating the performance of the pedestrian detection system:

False detection count per day, defined as the number of false calls occurring during each 24-hour period

A detailed discussion of the data with results for all 17 sites in the four cities will be published in Large-Scale Field Test of New Stereo Detection System for the Pedestrian Phase for the Visually Impaired (FHWA-HRT-12-074). Provided here is a sample of the data for one of the cities, Somerville. The scanning results for the test site at Somerville covered 9 days, including snowy, rainy, cloudy, and sunny weather conditions. Somerville had the worst false detection error rate of the four cities but a typical positive detection rate.

Nine-Day Results for Somerville

Automated Actuation Zone

Locator Tone Triggering Zone

Ground-Truth Count

407

878

Total Detection Count

445

997

Missed Detection Count

2

12

False Detection Count

40

131

Overall Error Rate

38/407 = 9.3%

119/878 = 13.6%

Missed Detection Error Rate

2/407 = 0.5%

12/878 = 1.4%

False Detection Error Rate

40/407 = 9.8%

131/878 = 14.9%

Positive Detection Rate

100% â€“ 0.5% = 99.5%

100% â€“ 1.4% = 98.6%

Average False Count per Day

40/9 = 4.4/day

131/9 = 14.6/day

Source: FHWA

The overall positive detection rate for the automated actuation zone for the four cities was 98.5 percent, and the average number of false calls per day was 2.7. This performance implies that the stereo pedestrian detection system will detect almost every pedestrian waiting to cross a street while causing minimal delay to the traffic flow. This performance was confirmed during stage II of the large-scale field test. By May 2012, seven systems had been operational, 24/7, for over 6 months.

Performance Results Based On Statistical Sampling

Because it was impractical to manually scan all of the images saved for performance evaluation, the FHWA contractor applied statistical sampling to reduce the number scanned. This approach is similar to public opinion polls, which survey a small number of people selected randomly from a large population.

The FHWA contractor created confidence intervals of 90, 95, and 99 percent, as typically used for statistical testing. Confidence intervals show how confident the statistician is that the performance from a random sampling is close to the true performance if all the data were examined. For a margin of error of 3 percent, they derived a sample size of 1,067 scans out of 21,340 (that is, 1,067/0.05) images. The results will have a margin of error of 3 percent with 95 percent confidence. Each of the 21,340 images must have at least one pedestrian, however, which requires a manual scan prior to the statistical sampling.

In this approach, the team divided all of the images into temporal categories as follows: First, they divided the year into the four seasons: winter (December, January, and February), spring (March, April, and May), summer (June, July, and August), and fall (September, October, and November). Then, they divided each season into days consisting of three subcategories: morning (6 a.m.–12 p.m.), afternoon (12 p.m.–6 p.m.), and night (6 p.m.–6 a.m.). Finally, they divided each subcategory into 5-minute intervals.

Statistical Sampling Results for Somerville

Total Number of Pedestrians

Number of Pedestrians Detected

Number of Pedestrians Missed

Positive Detection Rate

Automated Actuation Zone

553

538

15

97.3%

Locator Tone Triggering Zone

930

910

20

97.8%

Source: FHWA

The researchers randomly sampled a number of images from each category to form a large pool of images. From these samples, they manually and randomly selected 1,090 images for ground-truth evaluation. The results were similar to what was estimated from the full-day scanning. Because of resource constraints, the number of false calls from random sampling was not estimated.

“Tucson is experiencing wonderful results with the equipment, and it has made our Puffin operations very effective,” says Richard B. Nassi, P.E., Ph.D., consultant to the Pima Association of Governments in Tucson. Puffin is an acronym for Pedestrian User Friendly INtelligent crossing, a mid-block signal-controlled crossing for pedestrians that adjusts waiting and crossing times in response to demands.

Integration with Traffic Signal Control System

Shown here is the integration of pedestrian sensor hardware into a traffic signal control system.

A key part of testing a sensor is to evaluate its interaction with real-world traffic signal systems and accessible pedestrian signal pushbuttons. The researchers evaluated the integration during stage II of the field test after the local traffic engineers had determined that the system was correctly detecting pedestrians.

At the intersection of Commercial and Franklin Streets in Portland, accessible pedestrian signal buttons were not installed due to irregular distances between the signal poles and the ramps. Instead, the engineers installed audible confirmation devices. This inexpensive device informs pedestrians that they have been detected and that a service request has been made automatically.

Initial feedback from the public and the visually impaired community is encouraging and suggests that the integration has worked well. After participating in a test of the system, James Phipps, executive director of the Iris Network, whose mission is “helping people who are visually impaired or blind attain independence and community integration,” said, “I like not having to go out of my way to push a button. I don’t want to be distracted from going straight across the street. This is especially true if I’m expected to cross without parallel traffic.” Parallel traffic is defined as the traffic on a street that parallels the travel direction of blind pedestrians.

Efforts are being made by the contractor to bring the stereo pedestrian sensor to the market in a timely fashion. The contractor is also working with distributors in the traffic industry and is pursuing the process of obtaining State approvals for the stereo pedestrian detection products.

Conclusion

When pedestrians are detected near the pushbuttons, the stereo pedestrian detection system actuates either regular or accessible pedestrian signal pushbuttons to make requests for the pedestrian crossing phases. The overall positive detection rate for the automated pushbutton actuation zone at all four cities is close to 98 percent, and the average number of false calls per day is less than 3. To date, all of the systems continue to be operational, 24/7, in complex outdoor environments.

Now that a stereo vision system is available that can identify pedestrians and turn on locator tones, much additional research is needed to fine-tune the system to optimize it for the needs of visually impaired individuals.

Ultimately, the stereo camera system’s ability to reliably detect pedestrians and automate actuation of pushbuttons based on the presence of pedestrians promises to help reduce the likelihood of conflicts between vehicles and pedestrians. The system also can extend walktimes, as needed, for seniors and wheelchair pedestrians. The end result is safer crossings and, ideally, fewer fatalities.

These children are actuating a stereo pedestrian sensor at an intersection in Portland, ME.

David R.P. Gibson, P.E., M.S.C.E., is a highway research engineer on the Transportation Enabling Technologies Team in FHWA’s Office of Operations Research and Development, working with traffic and pedestrian sensing and control.

Paul Burton, TSOS, has been the traffic signal supervisor for Tucson, AZ, since 2004. He supervises the installation, programming, maintenance, and repairs of all of the city’s traffic signals and was responsible for the hardware design of the first pedestrian hybrid beacons used in the United States.

Neil Boudreau is the State traffic engineer at MassDOT, Highway Division. He has been with MassDOT since 1995, served in many roles within the Traffic Operations and Safety Management groups, and is a member of the National Committee on Uniform Traffic Control Devices.

Michael J. Bobinsky, MPA, is director of the Department of Public Services, Portland, ME. He has been involved with public works and city transportation activities since 1981, and currently manages all departmental functions including administration, engineering, streets, and construction.

Jim Hoben is deputy traffic director for Manchester, NH, where he has been involved in traffic systems and their installation and maintenance since 1974. He is responsible for traffic signs and signals, pavement markings, and streetlights.

Bo Ling, Ph.D., is the founder of Migma Systems and is a researcher in sensing and tracking for the U.S. Department of Defense and U.S. Department of Transportation. He specializes in image and signal processing.

Billie Louise Bentzen, Ph.D., is an orientation and mobility specialist, having taught visually impaired individuals to travel independently for more than 30 years. She is best known for her human factors research and is a member of the Signals Technical Committee of the National Committee on Uniform Traffic Control Devices.

The authors would like to thank the many traffic engineers of Tucson DOT, MassDOT, Portland DOT, and Manchester DOT. Their dedicated, hard work made this project successful.