This site may earn affiliate commissions from the links on this page. Terms of use.

In the 1990s, the USC Shoah Foundation conducted video interviews with thousands of Holocaust survivors, so that their stories are never forgotten. The nonprofit's digital library currently houses 53,000 video testimonies, and in recent years has expanded to capture testimony from those who witnessed the 1994 Rwandan Tutsi Genocide, the 1937 Nanjing Massacre, and the Armenian Genocide that coincided with World War I.

But in an era that's seeing the rise of virtual and augmented reality, not to mention robots and virtual assistants, the organization has been exploring more modern ways to capture testimony: interactive, hologram-like representations of survivors that can interact with and answer questions from future generations.

The foundation is collaborating with the USC Institute of Creative Technologies (ICT) and Conscience Display on the project, known as New Dimensions in Testimony. It's been in the works since 2012, and will eventually be available at museums and learning institutions around the world. PCMag traveled to Playa Vista, California, recently to watch Auschwitz survivor Eva Kor give her testimony on a soundstage.

Project head Dr. Stephen D. Smith, executive director of the USC Shoah Foundation, said he "wanted the chance to talk to the people who had experienced genocide themselves, to hear directly from witnesses, rather than see them consigned to become statistics of history."

For five days Smith's team interviewed Kor, who was eight when WWII started. "Through recording her responses, we can create a living resource for future generations," he said.

Heather Maio, managing director of Conscience Display, had been experimenting with different technologies for museums and education environments to share history before she learned about the work at ICT.

"I'd done a test with 3D—using a two camera setup—but I was really looking for a technology that was furthest along in holographic type technology. I found ICT in 2010 and brought the whole project to the USC Shoah Foundation."

Maio wanted to go beyond the traditional narrative testimony and decided to research natural language processing so museum visitors could have an actual dialog with recorded witnesses.

Much of speech recognition is still in development. "We've plugged in Google's ASR (Automatic Speech Recognition), which is the best language software out there, specifically because it has the largest natural conversational vocabulary and phrasing, and has an understanding of context," Maio said. "But our witnesses are using words, specifically Polish and Yiddish terms, that are less used today, from the WWII era. However, I've noticed that it learns really quickly and is picking up new words, and understanding more every day."

It was time to go into the digital capture studio. A crew was setting up a vast rig, with a green-screen backdrop draped across two-thirds of the space, so that the interviewee could later be placed within any environment or backdrop.

The team films in 360 degrees via a rig with 112 cameras, a combination of Ximea MQ042MG-CMs, Red Epic M Dragons, and Panasonic HC-X900s. "It's a complex project, so ICT has developed custom systems running on top of Vegas Pro, by Sony, for the editing process incorporating software created by the Natural Language team at ICT," Maio said.

With everyone in place, Eva Kor took her seat in the center of the camera setup. Eva was chosen because she founded the organization C.A.N.D.L.E.S. (Children of Auschwitz Nazi Deadly Lab Experiments Survivors) in 1984 and opened the C.A.N.D.L.E.S. Museum in 1995. Now 83 years old, and still sharp as a tack, she listened carefully to the instructions on how to keep her torso in the same position, so her digital render will have continuity later on. Then it was just like a real film set.

"Quiet on set."

"Sound!"

"Roll Panasonics."

"Checking all Panasonics!"

"Roll Red."

"Slate, please."

The clapperboard slammed down to give the edit team their notations, everyone looked down at the script for the day, and Smith leaned into the microphone outside the camera rig.

"What did your parents tell you about Hitler and the Occupation?" he asked.

Kor took a deep breath, and detailed how Hungarian soldiers came to her family's small farm, in a village in Transylvania, Romania.

After a witness account is laid down, audio triggers will be applied so future audiences can ask questions in many different ways and get a cogent response from Kor. This meant Kor had to answer the same question in various ways, just to ensure that the software categorized her responses correctly and applied keywords with semantic comprehension.

The questions that elicited the most emotion from those watching were the sort children would ask. In fact, they had. The 2,000+ questions were crowdsourced from research trials, and many were about the everyday details of life in those days. Kor recalled childhood memories, growing up before her family was taken to Auschwitz. She described her mother's cooking, problems with bullying at school, and her twin sister Miriam's favorite pet.

It's because Eva and Miriam are twins, and so part of the horrific experiments carried out in the camp, that they both survived. The rest of the family was killed upon arrival.

After an hour's recording, Kor needed a rest. So, during a pause in the proceedings, Smith took us into a side room to show us a completed testimony. ICT's output allows the interviewees to be projected in 2D HD, 3D, within VR, or via any display monitor or device.

On a laptop, Smith pulled up a browser window and there was Pinchas Gutter, another Holocaust survivor, his hands resting gently on his legs, eyes alert, waiting for us, it seemed.

Smith put his finger on a laptop key to trigger Gutter to respond. Gutter's face lit up. Smith asked a question, released his finger, and Gutter gave his reply, just as if he was sitting there on a teleconference in another town.

We all asked a question—several of us with our different nationalities and thus accents. He got every one and gave cogent answers. It was compelling. The suspension of belief was complete.

Eva Kor was ready for her next 100 questions back in the studio. Everyone filed back in. But what's amazing about this project is that Kor will be answering people's questions about what happened to her long after she's left us in the physical realm.

Sophia Stuart is an award-winning digital strategist and technology columnist. Voted one of the "Top 21 Social Media Superstars" by Min Online in 2009, Sophia was an executive at Hearst from 2006 - 2013, winning a Webby Award for Cosmo Mobile and an MVA for Cosmo International Digital Strategy. Sophia now lives in Los Angeles and runs TheDigitalCheckUp.com consultancy. She was a judge for both the SheSays global awards (2014) and the Bookmarks, South Africa (2013). She has written for many publications including Esquire Mexico, Harper’s...
More »