At last, PBS’s new distribution system nears completion

Hodgepodge of equipment in system causes bumps; upgrades burden stations

Published in Current, September 10, 2012

In August 2005, when PBS and CPB leaders announced the launch of their Next Generation Interconnection System (NGIS), a $120 million project adopting Internet-based technologies for pubTV program distribution, technical outlines for the project were hailed as leading-edge and a major advance for the public broadcasting system. Its target completion date was late 2006.

Public TV’s interconnection system will operate much more efficiently with non–real-time delivery of programs, McCoskey says. But the “multitude of station equipment configurations” has been a huge challenge in making NGIS work for all. (Photo: Aaron Clamage for PBS.)

Seven years, several generations of technology and a change of management later, the main components of NGIS are finally moving toward full implementation. The push to upgrade real-time satellite transmissions to a new digital standard and to complete the rollout of a non–real-time (NRT) video-distribution system will save PBS money and make public TV’s interconnection system more reliable, network officials say. But for some stations, the transition poses hefty technical and financial challenges.

If the latest deadline sticks, by year’s end two new sets of technology will be in use at each of the 180 or so local master-control points for PBS member stations around the country.

Sept. 17 is the target shutoff date for the satellite system running on older MPEG2 video-compression technology. Engineers at PBS will cut over to a real-time transmission system based on the MPEG4 (also known as H.264) standard. Already in common use for Internet video streams as well as Blu-ray discs, European DTV and the ATSC-M mobile DTV system in the U.S., the MPEG4 standard packs more video data into a smaller bitstream than MPEG2 does.

“MPEG4 is a much more efficient technology,” says PBS Chief Technology Officer John McCoskey. When the switch to new MPEG4 receivers is complete, the public TV system will save more than $1 million annually from reduced demand for satellite capacity.

PBS will need only three of the four satellite transponders it now uses to handle distribution on behalf of its members and other program providers. It plans to invest the savings in upgrades aimed at improving the quality of transmissions and adding a fifth high-definition feed providing additional programming and more time-shifting options, McCoskey says.

By itself, the move to MPEG4 would be just the latest in a string of technology upgrades that extend back to PBS’s pioneering C-band analog satellite interconnection system of the 1970s. While the underlying technology has improved, the basic architecture has remained the same: Programming is prepared for air and sent in real time from the PBS Network Operations Center (NOC). Individual stations can either broadcast the feed live or record programs for later.

This is where the second technical advance of the NGIS project comes into play — NRT delivery of programs as digital files instead of as live transmissions. Over the last few years, nearly 70 percent of local PBS outlets have been brought into the NRT system. Full rollout to the remaining stations is to be completed by year’s end.

The move to NRT requires a change in workflow for technicians at local stations, McCoskey says. Under the current system, stations record programs as they’re distributed in linear channels, and technicians have to trim excess material marking the beginning and end of each program.

Under NRT delivery, McCoskey says, “You don’t have to do that, because it’s a file that starts at the right time and ends at the right time. You don’t have to do visual checks for quality because it’s being sent as a file that’s already been checked.”

NRT is well-suited for public TV because most programs are produced and prepared for transmission three days before their scheduled airdate, according to McCoskey. Only two series, PBS NewsHour and Nightly Business Report, are transmitted live.

Devil in the details

The advantages of PBS’s NRT conversion and adoption of the MPEG4 standard are clear, but the process of moving local stations to both new systems simultaneously has been challenging on many levels.

“Sending a file over satellite is trivial,” explains Alan Popkin, chief engineer at KLCS in Los Angeles. “Getting it to play at every station is not.”

The difficulty of transmitting digital video to 180 local master-control sites, each with its own equipment and established workflow, has stymied PBS since before the NGIS project began. A decade ago, PBS attempted to address the wide variety of local standards by creating its own automation system.

But this push to standardize operations at local stations ended in 2006, when the PBS execs behind it, Ed Caleca and Andre Mendes, left the network. After McCoskey arrived at PBS in 2007, NGIS was redesigned around delivering standardized digital files that could be used by stations’ existing automation systems.

The concept drew from an earlier era of TV technology, according to Popkin — standardization of one-inch videotape under the Type C format of the mid-’70s. Because the format was interchangeable among a wide variety of tape machines, it became standard in TV broadcasting and remained so for almost two decades.

The PBS digital equivalent of Type C, a file specification called ASO3, was released in 2009. It has not proved anywhere near as universal.

“The devil is in the details,” Popkin says. In 2009, he was part of the team that drew up the 70-page specification for ASO3, which was supposed to deliver a file that was “station-ready” when it arrived. But that goal was lost in translation as the NRT system evolved beyond the six stations that began testing it in 2007.

What went wrong? As with ACE, which Popkin calls “a good idea, poorly executed,” it couldn’t bridge the complexity of modern broadcast systems and the decentralized nature of the public TV system. A simple idea became extraordinarily complex in its application to the real world of the master control room.

“Stations are kind of perplexed, too, because other networks have done projects like this successfully,” says Bruce Jacobs, chief technologist at Twin Cities Public Television.

“The challenge with non–real-time,” McCoskey says, “has been the multitude of station equipment configurations in the field. It’s been really challenging to make a one-size-fits-all service.”

Popkin, whose station was one of the alpha-test sites for NRT, says issues with the vendors supplying hardware and software for the project have also contributed to the delays.

“Some of it was a lack of understanding of how things worked on the station side,” he says. But the public TV system is not large and wealthy enough to command the attention of equipment manufacturers. “They’ve already sold the server, so what’s in it for them?” Popkin says.

Buggy automation

While the $120 million congressional appropriation backing NGIS has paid for MPEG4 transmission equipment at PBS and receivers for each station, abandonment of the ACE project left stations on their own to select and pay for local automation servers. McCoskey’s team at PBS and station engineers acting as advisors to NGIS have spent most of the last few years working to adapt NRT delivery so that it plays nicely with all those local systems.

“Over the summer, we did a lot of testing just to push the system and see how much we could distribute, and we were pushing over 40 hours a day of content in non–real-time,” McCoskey says. “I would say we’re still in beta-test only because not all stations have the gear yet. As soon as all the stations have the gear, we’ll transition to full production status.”

Before NGIS: Vermont Public Television used some automation in its master control room in early 2008, but the workflow still incorporated tape machines. (Photo: Scott Fybush.)

Vermont Public Television is among the outlets that have yet to take delivery of NRT equipment, says Joe Tymecki, director of engineering. VPT’s completed conversion to MPEG4 real-time delivery well ahead of the deadline. “For us, it’s just a matter of using the Ericsson [MPEG4] receivers instead of the older Sencore [MPEG2] receivers,” Tymecki said. “Other than some software upgrades, it’s been okay.”

WNIN-TV in Evansville, Ind., has had the opposite experience, according to David Dial, general manager. The station has already switched to NRT program delivery, but its Omneon automation system isn’t reliably processing the new MPEG4 files. The automation system is five years from the end of its expected lifespan, and Dial says there are no funding sources available to pay for a replacement.

“We don’t want to have to replace our Omneon five years early,” Dial says. He’s looking for a solution that will allow WNIN to ingest MPEG4 content automatically. “At 11 p.m., our master-control guy goes home, and we’re automated overnight, so if they’re sending programs for recording over MPEG4, we have no way to record them.”

While Dial would eagerly move his entire station over to non–real-time delivery if he could, he praises PBS for moving slowly on the rollout.

After NGIS: VPT’s new master control is driven by Omneon’s server-based digital automation. Omneon is the dominant automation vendor for local pubTV outlets, but not all stations have adopted its technology. (Photo: Ron Clark/VPT.)

“PBS gets a gold star for what they’ve done,” Dial says. “They’ve got a help desk in place, which is great, and they have an online chat with quick responses and follow-up, so in terms of a working system, we have no complaints about the NRT.”

Neither does C.C. Copeland, assistant director of engineering at Louisiana Public Broadcasting, another early NRT test site.

“We’re rocking and rolling” with both NRT and MPEG4, Copeland says. He praised the system’s capacity to resend files when transmissions are disrupted. When heavy rains from Hurricane Isaac impaired live satellite feeds of PBS programming in late August, “I had a file sitting there on the NRT server waiting for me,” Copeland says.

NGIS conversion hasn’t ended as happily at KLCS, a station that signed on as an alpha tester five years ago. “I still can’t use [NRT] on the air,” Popkin says. The problems he encountered involve yet another bit of technological complexity. KLCS’s automation system can air the ASO3 files as they’re delivered over NRT, but the station’s digital archive system can’t ingest those files.

“It still doesn’t fit in my workflow,” Popkin says. After years of trying to make NRT work, he’s still searching for solutions.

“I felt for a while like I had two jobs. I was spending 40 to 50 hours a week on my regular job here, and then another 20 or 30 doing NRT,” Popkin says.

PBS’s McCoskey acknowledges that rollout of NGIS has taken much longer than anyone expected, but he describes the delays as a conscious attempt to “not waste the stations’ time.”

“We didn’t want to push something out until it was ready to go,” McCoskey says. “If we saw something was not ready for prime time, we’ve been very careful to make sure we weren’t putting something out there that was going to put stations at risk,” he says.

Outpaced by technology

Beyond the technological challenges, many small stations have been concerned about the cost of the NGIS upgrades, says WNIN’s Dial, a former chair of the PBS Interconnection Committee who also headed pubTV’s affinity group for small stations.

Local outlets spent millions of dollars on new equipment to prepare for the DTV transition, and now the federal programs that subsidized their technical upgrades — the Public Telecommunications Facilities Program and CPB’s digital distribution fund — have been eliminated, Dial said. Many stations are struggling to keep up with the capital costs of maintaining their technical plants.

“Most of us don’t know how we’re going to replace the plants we have, so continuing to move ahead and forcing stations to move on a path they’re not prepared to follow is not the solution. PBS needs to make sure stations are protected,” Dial says.

Former PBS vice president of distribution and operations Gwen Wood agrees. Wood, who left the network in 2008, says PBS needs to consider whether technology initiatives such as NGIS and the ACE automation system that stalled in 2006 shifted too much of the technological burden to stations. The trend toward centralized master-control operations, such as the hubs now being built in Jacksonville, Fla., and Syracuse, N.Y., could point PBS toward a distribution model that’s more cost-effective than the current NGIS, she says.

“New technology is always a moving target, and they’re spending so much money aiming at a moving target,” Wood says. “If I were in charge, I would stop everything and bring in an outside, neutral broadcast-engineering consultant to do a complete evaluation of where they’ve been and what they’re planning to do.”

“The NRT was a great model in 2002, but in 2012 it’s kind of obsolete,” says KLCS’s Popkin. In the decade since the concept of IP video delivery over satellite was first demonstrated, he says, the cost of delivering video data over the Internet has dropped dramatically.

Instead of delivering NRT data in one format over an expensive satellite transponder, then putting the burden on stations to transcode the data themselves, PBS could adopt an architecture that would put transcoding “at the center of the system,” serving files over the Internet in multiple formats to better meet each station’s local needs. Moving much of the transmission load from satellite to the Internet would also enable more peer-to-peer program sharing among stations, Popkin says.

While stations and PBS debate the future of distribution, the goal now is simply to make NGIS work after years of delay. TPT’s Bruce Jacobs says there’s a sense of more transparency in recent years under McCoskey.

“It’s worth saying that management under John has made a difference in addressing these issues honestly, and that has been refreshing,” he says.

McCoskey says the final phase of the ten-year NGIS program involves working out the remaining problems, not revamping the system now in place. The final years of NGIS will be focused on making sure the system works as well as it can with the plethora of different vendors’ equipment in place at local stations, he says.

“We’ve got to stop changing things,” he says. “We’ve gone through a lot of changes, and it’s taken a lot on the station side to implement this.”

The writer, Scott Fybush, a news staffer of WXXI-AM in Rochester, N.Y., writes frequently on broadcast technology and edits NorthEast Radio Watch, a weekly online publication. He also publishes the annual Tower Site Calendar.