Sports Make Final Call on Technology

My print column this week examines how technology is aiding sports officiating by using cameras and advanced software to zero in on the location of batted, kicked and smashed balls. The use of that technology raises the question of how much to acknowledge about the technology’s limitations — and different sports are answering it in different ways.

At big tennis tournaments such as the World Tour Finals in London this week, players dissatisfied with calls made by line judges or umpires can appeal to a technology created by Hawk-Eye, a Basingstoke, U.K., company owned by Sony. Hawk-Eye uses a set of cameras to estimate where the ball hit the court and whether it was on, inside or outside of the lines.

Tennis has used the technology in matches since 2006. Cricket has used a different system from Hawk-Eye, to adjudicate leg-before-wicket calls, starting soon after. And this year, England’s soccer Premier League is using Hawk-Eye cameras and software to aid referees in deciding whether shots on a goal crossed the goal line and should count as scores.

Tests in all three sports show that the camera technology is more accurate than the unaided human eye, according to the company and the sports leagues. Human perception of a ball’s position, said George Mather, a psychophysical researcher at the U.K.’s University of Lincoln, is the combination of sensory data and decisionmaking, and the latter can be biased — in tennis, for instance, by information such as how well the player hit the ball. “I think Hawk-Eye is more precise than umpires and players,” he said.

Stuart Miller, executive director of the science and technical department of the International Tennis Federation, said Hawk-Eye’s performance keeps improving in the ITF’s tests — though it doesn’t publish the results of new tests. “When we look over time at all of the testing we have done, I can categorically say that in general they have gotten better,” Miller said.

The ITF originally estimated an average error of 3.6 millimeters when testing Hawk-Eye for use in matches, in 2005. Hawk-Eye founder Paul Hawkins said that average error fell to 2.2 millimeters by the last time he was involved in testing. And even that is an overstatement of the average error, he said, because the system is designed to perform better the closer the ball gets to the edge of a line. Larger errors farther from the line, where they don’t affect the call, artificially skew the average error upward.

“What we’ve shown consistently is, we get more and more accurate the closer the ball is to the back end of the line,” Hawkins said.

Hawk-Eye’s technology — and the other three systems approved by FIFA, soccer’s governing body, to aid referees in deciding whether soccer shots crossed the goal line — “is much more accurate” than the human eye, according to a FIFA spokeswoman.

Harry Collins, a sociologist of science at Cardiff University in Wales, doesn’t dispute any of that. But he’d like Hawk-Eye to acknowledge that its readings are estimates, like all scientific measurements, rather than exact depictions of where balls bounced. “What I’m worried about is people getting a misleading view of how science and technology works,” he said. Undue faith in technology, he said, reminds him of overblown claims for the effectiveness of the Star Wars and Patriot missile-defense systems. “It’s important for the public to know that it’s much harder to make a scientific device really, really accurate than to give the impression it is really, really accurate,” Collins said.

“The real way to do it is to show some error bars on the ball: Make the ball fuzzy edged, in such a way as to reflect the true accuracy of the device,” he said. “That seems to us to be the right way to do it because it acknowledges Hawk-Eye is not exactly accurate.”

“I think they have a valid point,” Mather said of the argument made by Collins and his co-author and Cardiff University colleague, Robert Evans, in papers about Hawk-Eye. Showing the margin of error “might be a more honest representation of data presented by Hawk-Eye,” added Mather, who wrote a paper based on Hawk-Eye data.

Fans interviewed this week at the O2 Arena, site of the men’s tour’s World Tour Finals tournament, said they understood the technology’s limitations and still preferred it to the alternative. “I believe it’s as accurate as it can be,” said Phyl Lake, 80, from Kent. “It’s not perfect.”

Collins also worries the replay images shown to fans courtside and to television audiences look too realistic, which accentuates the impression that Hawk-Eye is identifying the exact location of impact. “That’s just a mathematical reconstruction,” Collins said. “That’s not a photograph of anything. But it looks like a photograph.” He thinks if the sport depicted the Hawk-Eye finding in a more exaggerated, less realistic setting, it would help mitigate the false impression that the technology is perfectly accurate.

Hawk-Eye is heading in the opposite direction. Luke Aggas, director of tennis for Hawk-Eye, said the company has developed technology allowing it to overlay its re-creations of a ball’s trajectory on top of video of the play. “We’re trying to be more transparent,” Aggas said, “to show how complex, and how sophisticated, the technology is.”

There are practical considerations that might delay adoption of the technology, Aggas said. First of all, a video replay might inadvertently reveal an issue other than a missed line call, such as a distraction in the arena that a player might argue hindered him or her. Second, incorporating video would require upgrading some electronic display boards at tennis venues and the added expense could discourage use of Hawk-Eye, which isn’t on every court at most tennis events. “We don’t want to take a backward step,” Aggas said. “We want to move forward.”

On the plus side of the ledger, overlaying Hawk-Eye animation on video would enhance the entertainment value of the challenge system, which Mather said is already high. “I get the feeling in the case of tennis, part of the motivation is the spectacle of the game,” Mather said. “It adds interest and drama to the occasion.”

Hawkins never cared much for the Collins-Evans argument. “I just felt it lacked common sense and was typical of people who spent a lot of time in universities rather than on the tennis circuit,” he said. He and others who support Hawk-Eye’s current implementation in tennis say a wrong call is possible but in practice likely very, very rare.

He and others in tennis say the system’s advantage of quickly rendering a definitive decision would be neutered if the presentation of Hawk-Eye findings included an acknowledgment of its limitations. “The benefit of having a definitive decision outweighs any inaccuracy there might be in the system,” Miller said.

“The fact that there might be a margin of error on some of these things is a little bit moot,” said former American pro Leif Shiras, now a tennis commentator for Sky Sports in the U.K. “As long as you have an ultimate voice, right or wrong, it tends to put everyone at ease.” He added, “It’s pretty well-proven that it’s darn close. That’s pretty satisfying.”

Shiras thinks the sport should consider expanding use of Hawk-Eye technology, perhaps to review contentious foot-fault calls. (Hawkins said the system isn’t currently equipped to do so.)

“At some point you’ve got to make a decision,” Hawkins said. “At some point you’ve got to decide who’s won the point, and the technology is by far the most accurate.”

“If you feel the ball was in and it was called out, someone has to make the call,” said former French pro Guy Forget. “It might as well be a machine.” Forget called Hawk-Eye “great” and “wonderful.”

“I think it’s something really good for the game, for players, for umpires,” said world No. 8 Stanislas Wawrinka. “I trust the Hawk-Eye.”

The importance of having a universally accepted call has, ironically, held back expansion of Hawk-Eye onto clay courts. There, tradition reigns: Umpires check ball marks and players generally accept the ruling. Hawk-Eye testing indicates ball marks can mislead, because of how the ball interacts with the clay on impact. The company presented its findings to the sport’s governing bodies this summer but hasn’t heard back.

“Using Hawk-Eye would improve officiating on clay,” said Luke Aggas, director of tennis for Hawk-Eye, “but is it significant enough to open the Pandora’s box of now telling players the mark is inaccurate?” He pointed out that players have grown up with the understanding that ball marks are accurate. Telling them otherwise “is a concern,” and also necessitates installing Hawk-Eye on every court at a tournament, lest players start questioning the accuracy of ball marks on courts without the technology.

“Ball marks on clay are a well entrenched and well-accepted method for making definitive decision,” Miller said. “Irrespective of accuracy, its well-accepted.”

About The Numbers

The Wall Street Journal examines numbers in the news, business and politics. Some numbers are flat-out wrong or biased, while others are valid and help us make informed decisions. We tell the stories behind the stats in occasional updates on this blog.