Should I Use Scalding or Scoobi or Scrunch?

In the past year there has been a tremendous amount of activity on Scala APIs for Hadoop. In this talk we`ll talk about writing Map/Reduce jobs in a more functional manner and explore the three most popular Scala packages for Hadoop: Scalding, Scoobi and Scrunch. Detailed usage examples will be provided for each along with some real world use cases.

You should also have a look at Stratosphere (stratosphere.eu). If also offers a Scala API that is similar to the ones presented here (but the execution is probably faster since stratosphere natively supports operators such as join or union and data flow graphs)

Similar to Spark, Stratosphere does not run on top of MapReduce, it has its own operators. But you can use your existing YARN cluster (and HDFS) and Stratosphere also has a Map and Reduce operator.

Transcript of "Should I Use Scalding or Scoobi or Scrunch? "

1.
SHOULD I USE
SCALDING OR SCOOBI
OR SCRUNCH?
CHRIS SEVERS @ccsevers
eBay SEARCH SCIENCE
Hadoop Summit
June 26th, 2013

2.
Obligatory Big Data Stuff
•Some fun facts about eBay from Hugh Williams’ blog:
–We have over 50 petabytes of data stored in
our Hadoop and Teradata clusters
–We have over 400 million items for sale
–We process more than 250 million user queries per day
–We serve over 100,000 pages per second
–Our users spend over 180 years in total every day looking
at items
–We have over 112 million active users
–We sold over US$75 billion in merchandize in 2012
•http://hughewilliams.com/2013/06/24/the-size-and-scale-
of-ebay-2013-edition/
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 2

5.
THANK YOU AND GOOD NIGHT
•Questions/comments?
•Thanks to Avi Bryant, @avibryant for settling this issue.
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 5

6.
NO REALLY, WHICH ONE SHOULD I USE?!
•All three (Scalding, Scoobi, Scrunch) are amazing projects
•They seem to be converging to a common API
•There are small differences, but if you can use one you will
likely be productive with the others very quickly
•They are all much better than the alternatives.
•Scalding: https://github.com/twitter/scalding, @scalding
–Authors: @posco @argyris @avibryant
•Scoobi: https://github.com/NICTA/scoobi
–Authors: @bmlever @etorreborre
•Scrunch: http://crunch.apache.org/scrunch.html
–Author: @josh_wills
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 6

7.
THE AGENDA
1. Quick survey of the current landscape outside
Scalding, Scoobi, and Scrunch
2. A light comparison of Scalding, Scoobi, and Scrunch.
3. Some code samples
4. The future
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 7

10.
PIG
•Apache Pig is a really great tool for quick, ad-hoc data
analysis
•While we can do amazing things with it, I’m not sure we
should
•Anything complicated requires User Defined Functions
(UDFs)
•UDFs require a separate code base
•Now you have to maintain two separate languages for
no good reason
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 10

11.
APACHE HIVE
•On previous slide: s/Pig/Hive/g
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 11

12.
STREAMING
•Can be concise
•Easy to test
–cat myfile.txt | ./mymapper.sh | sort | ./myreducer.sh
•Same problems as vanila MR when it comes to
multistage flows
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 12

13.
CASCADING/CRUNCH
•Higher level abstractions are good
•Tell the API what you want to do, let it sort out the actual
series of MR jobs
•Very easy to do cogroup, join, multiple passes
•Still a bit too verbose
•Feels like shoehorning a fundamentally functional notion
into an imperative context
•If you can’t move away from Java, this is your best bet
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 13

15.
FEATURE COMPARISON
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 15
Scalding Scoobi Scrunch
Data model Tuple or
distributed
collection
Distributed
collection
Distributed
collection
Has Algebird
baked in
Yes No No
Is Java-free No Yes No
Backed by
Cloudera
No No Yes
Free Yes Yes Yes

20.
ADVANTAGES
•Type checking
–Find errors at compile time, not at job submission time (or
even worse, 5 hours after job submission time)
•Single language
–Scala is a fully functional programming language
•Productivity
–Since the code you write looks like collections code you can
use the Scala REPL to prototype
•Clarity
–Write code as a series of operations and let the job planner
smash it all together
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 20

21.
BREAD AND BUTTER
•You can be effective within hours by just learning a few
simple ideas
–map
–flatMap
–filter
–groupBy
–reduce
–foldLeft
•Everything above takes a function as an argument.
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 21

28.
MONOIDS: WHY YOU SHOULD CARE ABOUT
MATH
•From Wikipedia:
–a monoid is an algebraic structure with a single associative
binary operation and an identity element.
•Almost everything you want to do is a monoid
–Standard addition of numeric types is the most common
–List/map/set/string concatenation
–Top k elements
–Bloom filter, count-min sketch, hyperloglog
–stochastic gradient descent
–moments of distributions
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 28

29.
MORE MONOID STUFF
•If you are aggregating, you are probably using a monoid
•Scalding has Algebird and monoid support baked in
•Scoobi and Scrunch can use Algebird (or any other
monoid library) with almost no work
–combine { case (l,r) => monoid.plus(l,r) }
•Algebird handles tuples with ease
•Very easy to define monoids for your own types
•Algebird: https://github.com/twitter/algebird @algebird
–Authors: Lots!
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 29

31.
SPARK
•Spark is a system for executing general computation
graphs, not just MR
•The syntax looks very much like Scalding, Scoobi
and, Scrunch
–It inspired the API on a couple of them
•Spark runs on YARN as of the latest release
•Can cache intermediate data
–Iterative problems become much easier
•Developed by the AMPLab at UC Berkeley
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 31

32.
GREAT, NOW I HAVE TO LEARN 4 THINGS
INSTEAD OF 3
•Scalding, Scoobi, and Scrunch seem to have all sprung
into being around the same time and independently of
each other
•Spark was around a little before that
•Do we really need 3 (or 4) very similar solutions?
•Wouldn’t it be nice if we could just pick one and all get
behind it?
•I was prepared to make a definitive statement about the
best one, but then I learned something new
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 32

33.
HAVE CAKE AND EAT IT
•There is currently working being done on a common API
that spans Scalding, Scoobi, Scrunch and Spark
•Not much is implemented yet, but all 4 groups are talking
and working things out
•The main use case is already done
–After word count everything else is just academic
•Check it out here: https://github.com/jwills/driskill
•In the future* you’ll be able to write against this common
API and then decide which system you want to execute
the job
–Think of choosing a list, a buffer or a vector
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 33

34.
HOW CAN WE HELP?
•Get involved
•If something bothers you, fix it
•If you want a new feature, try and build it
•Everyone involved is actually quite friendly
•You can build jars to run on your cluster and no one will
know there is Scala Inside™
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 34

36.
THINGS TO TAKE AWAY
•Mapreduce is a functional problem, we should use
functional tools
•You can increase productivity, safety, and maintainability
all at once with no down side
•Thinking of data flows in a functional way opens up
many new possibilities
•The community is awesome

37.
THANKS! (FOR REAL THIS TIME)
•Questions/comments?
SHOULD I USE SCALDING OR SCOOBI OR
SCRUNCH? 37