Video Restoration Innovation Restores Holocaust Testimony

Image: brunkfordbraun/Flickr

When Steven Spielberg came up with the idea of collecting testimony from survivors and witnesses of the Holocaust after he made “Schindler’s List” in 1993, time was not on his side. The number of those who endured the horrors of World War II was dwindling at an alarming rate and the team he assembled to collect their stories knew that speed had to take precedence over production quality.

The crews that crossed the globe in the early days of what became the USC Shoah Foundation – The Institute for Visual History and Education performed a seemingly impossible task: In just a few years, they had taped nearly 52,000 testimonies in 56 countries.

But there were unfortunate consequences of capturing so many testimonies so quickly; over the years, the post-production team discovered quite a few technical problems. Everything from misaligned cameras to malfunctioning microphones left pockets of static and flicker on a fraction of the 235,000 tapes containing irreplaceable testimony. Some of the flaws were so severe that the faces of the survivors were not even visible in parts of their testimony.

It was a maddening thought that after a survivor lived through the Holocaust and poured his or her heart into their testimony that parts of it would be lost due to technical problems during the recording process.

The original testimonies were recorded on Beta SP tapes starting in 1994. But knowing those tapes would degrade over time — and the technology behind them would become obsolete — we undertook an ambitious preservation project in 2009 that transferred the entire 105,000 hours of testimony of to a digital format.

We used robotic migration technology for the sake of efficiency. While that left no time for tape operators to intervene, we were able to flag about 12,000 tapes that had technical problems on the source recordings during the quality-assurance process.

We finished the preservation program in 2012 — two years ahead of schedule. We then received funding for what has become known as the Restoration Project to manually re-migrate the flagged tapes using whatever techniques we could harness to correct the video and audio problems, which would rescue damaged testimony.

I was tasked with researching existing restoration techniques used in the motion-picture industry. I soon discovered that while there are sophisticated film restoration tools, there were very few options for correcting thousands of hours of tape-based material. And even those options could do very little to correct our toughest video problems.

Our archive was recorded using what is known as interlaced video, which means that every frame is actually composed of two separate fields, which are recorded by separate recording heads in the camera. Many of the damages were caused when one of the heads failed to record properly. For many years, we had been using video hardware to remove the problematic field. While this tool was successful at correcting many of the problems, it can only work when one of the fields is damaged. It cannot correct problems in both fields.

While trying to figure out a way to solve this problem, I had an epiphany. If we could break up the video fields into a sequence of still images, we might be able to isolate the “bad” images and replace them with the nearest previous “good” image.

But a typical half-hour of tape contains more than 100,000 individual fields, so it would obviously be an overwhelming task to sift through so many images in order to identify problems. But one day while using Google’s Picasa to sort family photos, I noticed that the facial-recognition software was so powerful that it was able to identify my then 6-year-old daughter all the way back from the time she was a baby. I realized then that if we could automate the process of identifying the “good” and “bad” images using image-recognition software, then we could correct some of the most difficult video problems.

I began collaborating with Sindhu Jagadeesh, a graduate student who had been working with us in Information Technology Services. She was able to write the software the filled in the missing images. She also began to sort through some of the difficult workflow challenges of managing so many images.

Sindhu and I worked together for a semester and we successfully produced our first video and a prototype for the process. She graduated and now works for Cisco.

Last semester, Ivan Alberto Trujillo Prigeo, who is earning a master’s degree in biomedical engineering, has been an invaluable member of our restoration efforts. He would stand in our machine room for hours to troubleshoot and correct video. We were finding that the facial-recognition software tool in Picasa could help with some of the videos, but for many, we still needed to manually sift through the images. In some cases, it took us several days just to correct one half-hour video.

But Ivan told us that he had used a powerful image-recognition system during his undergraduate thesis and suggested it could do a better job than Picasa of sifting through the images. Ivan and I worked together and are seeing some very promising results.

So far, Ivan has replicated the workflow within the image recognition environment of National Instruments Vision Builder and we now have three successful videos that require almost no manual labor. We hope to go into production with system early this year.

Ryan Fenton-Strauss earned his B.A. from Brandeis University and his M.F.A. in Film Production from Chapman University. He has more than a decade of experience in post production at USC Shoah Foundation.