README.md

goto-cassandra-spark

A Spark job that outputs the ranked list of episode to watch based on one preference "keyword". Results of previous searches are stored in Cassandra.

What this Job is supposed to do

This Spark job is a prototype of a trivial recommendation system for TV-show episodes to watch based on your preference.

Input requirements

A directory with text files containing dialogie scripts for each episode of the show. Current scripts are from the 1st season of "The Game of Thrones". It is absolutely okay to replace existing files in the moviescript directory with dialogue scripts from the TV-show of your choice, or even a series of book scripts.

A keyword of preference. It can be a favorite show character name, or a favorite thing that gets mentioned throughout the dialogue. For example, tyrion, arya, nymeria or cersei would all be valid things to have as a keyword of preference.

Cassandra Stateful Set running in your Kubernetes cluster.

Get Cassandra Stateful Set running

You may use the following instructions to get Cassandra Stateful Set running before following next steps:

Prepare Spark container image

Expected output

When the job is completed, it should output a ranked list of episodes (or books, or etc.) corresponding to the specified keyword of preference sorted from most preferable to least preferable. For example, in case the analysis is done for tyrion as a keyword of preference, and the moviescript directory contains dialogies for episodes of the first season of Game of Thrones, the ranked list of episodes will look similar to following:

In case the keyword of preference isn't mentioned in every episode, the ranked list wil me smaller than the original list.

Steps

There are many ways to run this Spark job, depending on resource scheduler and other criteria. These instructions are based on what I've done to run the job using the Spark's new option for Kubernetes scheduler (starting from 2.3 release) with Cassandra Stateful Set. There are more options how to run it than described here!