/CSE Ph.D. Student at GT

A Viz of Ice and Fire: Exploring Entertainment Video Using Color and Dialogue

The interactive visualization, displaying the show's Season 2, Episode 9: "Blackwater."
The Color Representation plot (A) shows the palettes extracted for the entire series, each season, and each episode as labelled stacked columns, e.g. S2E9 corresponds to Season 2, Episode 9.
The dialogue plot (B) shows how much of each textual category is present in the dialogue at a given time.
Notice that the bottom color plot in (A) shares the horizontal time axis with the bubbles directly below in (B).
Metadata for each season and episode is displayed in (C).
The frames view (D) and dialogue view (E) display the video frames and spoken dialogue for the current time slice and update as users interact with the data in (A) and (B).
The word frequency histogram (F) displays the top words spoken in the episode.

Abstract

Films and television shows provide a rich source of cultural data and form an integral part of modern life.
However, the video medium remains difficult to analyze at scale effectively and its study has generally attracted limited research attention.
We propose a method of summarizing the audio and visual aspects of entertainment videos, through the automatic extraction of dominant colors from video frames and textual categories from dialogue.
The colors and dialogue are displayed in a visualization that allows the user to explore the video, highlighting both high-level and low-level patterns from the data.
Focusing on the hit television series Game of Thrones, we show how our visualization supports the detection of scene changes and plot points, providing a new perspective for both scholars and casual viewers.