What's Next

For a Squeeze Play, Software Seeks Out Game Highlights

By IAN AUSTEN

Published: April 29, 2004

BASEBALL fans who watch a televised game after it has been processed by video analysis software developed by Sharp Laboratories of America can forget about peanuts and Cracker Jack. There are no ritualistic warm-ups on the screen, no shots of boisterous fans or commercials.

Instead there's a nonstop assault of hits, runs and outs. Succumbing to any distraction like eating is enough to leave a viewer behind. Given that the system, called HiMpact Sports, reduces a three-hour baseball game to an eight-minute experience, there may not even be enough time to get hungry.

"We're trying to give the possibility to users to concentrate on what they prefer," said Ibrahim Sezan, who is the director of information systems technologies at Sharp Labs and developed the software. "We want to allow people to consume more sports because there is an insatiable appetite for sports."

While Sharp has yet to sell the software, Jon Clemens, who heads Sharp Technology Ventures, a company trying to license it, said there were several potential markets. Broadcasters could use it to pull game highlights automatically for sports and news programs, a task now performed by video editors. Web sites or cable and satellite television services could start selling the micro-games to subscribers. And because the software works on standard PC's, Dr. Clemens said, it may be offered directly to consumers or integrated into hard-drive digital video recorders like TiVos. A variation of the system, HiMpact Coach, has been created for football-coaching videos.

Dr. Sezan, who has worked for Eastman Kodak in motion picture post-production research and has also developed medical imaging processing systems, said his research group started by asking the basic question of how an important event in sports should be defined.

As Sharp's researchers began posing the question to broadcasters and experts in several sports, they discovered that answering it was not always easy. In the case of some sports - kickboxing and basketball in particular - no obvious answer emerged.

"In basketball, there's passes all the time, there's dribbling all the time, there's shooting all the time," Dr. Clemens said. "You can define all of these, but what you do with it is a difficult issue."

The predictability of sports and the way they are broadcast enable the software to sort out crucial moments. For the initial version of the software, the researchers were able to distill the essence of baseball, football, soccer and sumo wrestling.

Dr. Sezan said that one of the first steps in developing the software was defining the color, shape and appearance of the playing field. In the case of baseball, the researchers also had to develop general rules that accommodated variations among stadiums.

Repetitive camera positions and patterns in the way games are presented by television producers were the next important element. Then the group had to define when action started and stopped.

Finding the start of action in baseball, football and sumo was a relatively simple job, Dr. Sezan said. With those sports, there is generally no camera motion or player activity at the start of play. Similarly, changes in camera angles usually follow the end of a play.

Once the descriptions were laid down, the problem became a mathematical one. "You can't tell a computer what a pop-up is," Dr. Clemens said. "You have to define mathematically."

That, Dr. Sezan said, involves pulling together "clues from the image's color, structure, motion and camera angles to determine if there is an event or there isn't an event."

Soccer, which Dr. Clemens defines as a "continuous-action sport," required a different approach. In soccer, for example, there is usually no obvious starting point for a series of passes, dribbles and kicks that result in a goal. But the researchers found that every action that soccer fans regard as important is replayed by broadcasters. So for soccer games, the HiMpact software only searches for replays. (To a lesser degree, the software relies on football and baseball replays to identify highlights in those sports as well.)

How much the system accelerates the action varies by sport and by individual games. Sumo wrestling has the greatest speed gains - the pushing, grunting, grabbing and shoving of an hourlong tournament flashes past in just three minutes. Football, by contrast, is relatively sedate. A full National Football League game shrinks to about 12 minutes.

In its current form, HiMpact replays the action on computer monitors. A progress bar under the video picture gives an indication of where the highlight fell within the full length of the event. Software controls also enable the profoundly impatient viewer to further speed the action.

Watching the condensed games can be somewhat jarring and confusing. To help keep things straight in viewers' minds, the system can also synchronize with text descriptions provided by a company called SportsTicker.

Making that work with baseball, however, was tricky. "In baseball, there is no clock concept," Dr. Sezan said. Additional software was required to analyze text and images. The addition of text also makes searching the condensed games for specific players or events relatively simple.

The software technology in HiMpact can be adapted to analyze other moving images. While in the sports mode it looks for patterns, but the software can just as easily search out the unpredictable. For example, Dr. Clemens said, the software could be used with a camera to monitor automated machinery. When any variation from a routine occurred, the software could alert a human operator.

In addition, a surveillance version of the software could track athletes of a less celebrated variety. "You could define someone jumping over a turnstile as an event you'd want to find," Dr. Clemens said.