Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

First time accepted submitter sagecreek writes "Hadoop is an open-source, Java-based framework for large-scale data processing.
Typically, it runs on big clusters of computers working together to crunch large
chunks of data. You also can run Hadoop in "single-cluster mode" on a
Linux machine, Windows PC or Mac, to learn the technology or do testing and
debugging. The Hadoop framework, however, is not quickly mastered. Apache's
Hadoop wiki cautions: "If you do not know about classpaths,
how to compile and debug Java code, step back from Hadoop and learn a bit more about Java before
proceeding." But if you are reasonably comfortable with Java, the well-written
Hadoop Beginner's Guide by Garry Turkington can help you start mastering this
rising star in the Big Data constellation." Read below for the rest of Si's review.

Hadoop Beginner's Guide

author

Garry Turkington

pages

374

publisher

Packt Publishing

rating

9/10

reviewer

Si Dunn

ISBN

9781849517300

summary

Explains and shows how to use Hadoop software in Big Data settings.

tester dataDr. Turkington is vice president of data engineering and lead architect for London-based Improve Digital. He holds a doctorate
in computer science from Queens University of Belfast in Northern Ireland. His
Hadoop Beginner's Guide provides an effective overview of Hadoop and hands-on guidance in how to use
it locally, in distributed hardware clusters, and out in the cloud.

Packt Publishing provided a review copy of the book. I have reviewed one other Packt
book previously.

Much of the first chapter is devoted to "exploring the
trends that led to Hadoop's creation and its enormous
success." This includes brief discussions of Big Data, cloud computing, Amazon
Web Services, and the differences between "scale-up" (using increasingly larger
computers as data needs grow) and "scale-out" (spreading the data processing
onto more and more machines as demand expands).

Dr. Turkington writes, "One of the most confusing aspects of Hadoop to a newcomer is its various components, projects, sub-projects,
and their interrelationships."

His 374-page book emphasizes three major aspects of Hadoop: (1) its common
projects; (2) the Hadoop Distributed File System (HDFS); and (3) MapReduce.

He explains, "Common projects comprise a set of libraries
and tools that help the Hadoop product work in the real world."

The HDFS, meanwhile, "is a filesystem unlike most you may have encountered before."
As a distributed filesystem, it can spread data storage across many nodes.
"[I]t stores files in blocks typically at least 64 MB in size, much larger than
the 4-32 KB seen in most filesystems." The book briefly describes several features,
strengths, weaknesses, and other aspects of HDFS.

Finally, MapReduce is a well-known programming model for processing large data
sets. Typically, MapReduce is used with clusters of computers that perform
distributed computing. In the "Map" portion of the process, a single problem is
split into many subtasks that are then assigned by a master computer to
individual computers known as nodes (and there can be sub-nodes). During the
"Reduce" part of the task, the master computer gathers up the processed data
from the nodes, combines it and outputs a response to the problem that was
posed to be solved. (MapReduce libraries are now
available for many different computer languages, including Hadoop.)

"The developer focuses on expressing the transformation
between source and result data sets, and the Hadoop
framework manages all aspects of job execution, parallelization, and
coordination," Dr. Turkington notes. He calls this "possibly
the most important aspect of Hadoop. The platform
takes responsibility for every aspect of executing the processing across the
data. After the user defines the key criteria for the job, everything else
becomes the responsibility of the system."

In this 11-chapter book, the first two chapters introduce Hadoop and explain how to install and run the software.

Three chapters are devoted to learning to work with MapReduce, from beginner to
advanced levels. And the author stresses: "In the book, we will be learning how to write MapReduce
programs to do some serious data crunching and how to run them on both locally
managed and AWS-hosted Hadoop clusters." ["AWS" is "Amazon Web Services."]

Chapter 6, titled "When Things Break" zeroes in on Hadoop's "resilience to failure and an ability to survive
failures when they do happen.much of the architecture and design of Hadoop is predicated on executing in an environment where
failures are both frequent and expected." But node failures and numerous other
problems still can arise, so the reader is given an overview of potential
difficulties and how to handle them.

The next chapter, "Keeping Things Running," lays out what must be done to properly maintain a Hadoop cluster and
keep it tuned and ready to crunch data.

Three of the remaining chapters show how Hadoop can be used elsewhere within an organization's systems and
infrastructure, by personnel who are not trained to write MapReduce programs.

Using Hive as an interface to Hadoop "not only accelerates the time required to produce results
from data analysis, it significantly broadens who can use Hadoop and MapReduce. Instead of requiring software development
skills, anyone with a familiarity with SQL can use Hive," the author states.

But, as Chapter 9 makes clear, Hive is not a relational database, and it doesn't fully implement SQL. So the
text and code examples in Chapter 9 illustrate (1) how to set up MySQL
to work with Hadoop and (2) how to use Sqoop
to transfer bulk data between Hadoop and MySQL.

Chapter 10 shows how to set up and run Flume NG.
This is a distributed service that collects, aggregates, and moves large amounts of log data from
applications to Hadoop's HDFS.

The book's final chapter, "Where to Go Next," helps the newcomer
see what else is available beyond the Hadoop core product. "There are," Dr. Turkington emphasizes, "a
plethora of related projects and tools that build upon Hadoop
and provide specific functionality or alternative approaches to existing ideas."
He provides a quick tour of several of the projects and tools.

A key strength of this beginner's guide is in how its
contents are structured and delivered. Four important headings appear
repeatedly in most chapters. The "Time for action" heading singles out
step-by-step instructions for performing a particular action. The "What just happened?"
heading highlights explanations of "the working of tasks or instructions that
you have just completed." The "Pop quiz" heading, meanwhile, is followed by short,
multiple-choice questions that help you gauge your understanding. And the "Have
a go hero" heading introduces paragraphs that "set practical challenges and
give you ideas for experimenting with what you have learned."

Hadoop can be downloaded free from the Apache Software Foundation's
Hadoop website.

Dr. Turkington's book does a good job of describing how to get Hadoop running on Ubuntu and other
Linux distributions. But while he assures that "Hadoop does run well on other systems," he notes in his text: "Windows
is supported only as a development platform, and Mac OS X is not formally supported at all." He refers
users to Apache's Hadoop FAQ
wiki for more information. Unfortunately, few details are offered there. So web searches become the best option for
finding how-to instructions for Windows and Macs.

To install Hadoop on a Mac running OS X Mountain Lion, you will need to search for
websites that offer how-to tips. Here is one
example.

There are other ways get access to Hadoop on a single computer, using other operating
systems or virtual machines. Again, web searches are necessary. The Cloudera Enterprise Free product is one virtual-machine
option to consider.

Once you get past the hurdle of installing and running Hadoop, Garry Turkington's
well-written, well-structured
Hadoop Beginner's Guide can start you moving down the lengthy path to becoming an expert user.

You will have the opportunity, the book's tagline states, to "[l]earn how to crunch big data to extract meaning
from the data avalanche."

I never worked with Java, so honestly I didn't know. All I hear left and right is "Java sux because vulnerabilities". So I was wondering.But judging from the smug answers below, I made the impardonable mistake of not knowing EVERYTHING. Oh well.

Why is this insightful? Shouldn't this comment have a low moderation value. A high moderation leads people into wasting their time look at this post and into a parent post who needed some help. Why should anyone else care.

Of course this is not the fault of the parent or the poster. Is moderation really this crappy? I guess moderators want to moderate something in this article and there just aren't any good posts.

And back in the 80's BT used PL/1 (plus a bit of Fortran 77) to do Map reduce as the core of a billing system for the dialcom systems -even manged ro sell it to other telcos as a replacement for the Dialcom product.

Well, I think [slashdot.org] that great amount of confusion results from the way Java is marketed. For a website that has a larger than normal number of technical people,/. as an aggregate still displays fundamental misunderstanding of what Java is and what it is not.

For example Java is not Javascript. A browser sandbox that runs Java plugin has nothing to do with server side Java applications, that for example can run in servlet containers and servers like Tomcat Apache, Jetty, iPlanet, Resin (by Caucho), Enhydra, and

For people who apparently pride themselves being 'above marketing', developers somehow fell victims to various marketing around Java and didn't bother to check for themselves what the hell all this is about.

As a developer who never had many reasons to do very much with (actual) Java, I have to say I lost of respect for the platform when Oracle started bundling the runtime with the Ask toolbar on Windows. Because nothing says "serious computing platform" like having to make sure it doesn't install a toolbar on every update./rant

As a developer who never had many reasons to do very much with (actual) Java, I have to say I lost of respect for the platform when Oracle started bundling the runtime with the Ask toolbar on Windows.

- I don't know, as a thinking human I have to say that what Oracle does with Java browser plugin reflects poorly on Oracle and has nothing to do with Java. As I said, the browser plugin is irrelevant for majority of what Java is actually used for.

Personally I don't even have Java browser plugin on any of my machines because I don't have a use for it. At the same time I have developed plenty Java applications (as in Java language running on a JVM that resides on a server and has some form of application co

As an FYI, it's not the browser plugin that has the opt-out toolbar installation, it's the JVM itself. Do I think it's truly a sign of Java's strength/weakness as a platform? No, but as you say, it's horrible marketing.

You mean the JRE that is part of the plugin installation? It cannot be the JRE itself, it must the the installer of the plugin. I don't install JVM this way or Java plugin (at all), I just download the necessary installation package for the JVM/SDK as needed. If I want Oracle SDK I get it from here [oracle.com] and when it installs, it does not install any 'ask bar' or anything like that, so I am not even clear as to how people get these things. I download the SDK or JVM for a Linux distro, which is either a binary

No, I mean the JRE itself. (Remember, I'm talking about Windows here.) Granted, this is not the SDK I'm talking about, just the standard JRE installed in Windows, not the browser, but I just updated it on this machine, and I had to uncheck [tenthcave.com] the option to install the Ask toolbar. (link is not my blog, btw.)

I'm fully aware of the distinctions between Java, the browser plugins, Javascript, etc., and I realize that the toolbar is not part of the install on 'real' OSs. But the Windows installer is what most us

But yes, on Linux or Unix I don't get any of this nonsense, it's just a tarball that I untar into a directory, set the path and run the JVM for example. The binary installers for Linux or Unix are simple shell scripts with the same tarball basically as part of the script, it doesn't do any of this stuff.

All due respect...but even as a developer, to say it's not the JRE but the JRE's installer that bundles the toolbar, is pretty pedantic. I really just wanted to point out this comes with standard install of Java itself, not the browser plugin.

How many people on the planet actually manage "Big Data"? Isn't that the kind of thing that happens as a happy accident when your humble web site becomes the next FaceBubbleSpace? You can't plan for that.

Sure, there are other places where it happens--large corporations, governments, maybe some academic studies.

Really though, I have a hard time imagining that there are really a lot of people who deal with BD. Does anybody have numbers on it? What's the definition, anyway? Is Slashdot's archive BD?

Anybody who is talking about data mining is already thinking about Big Data problems. These are everywhere, from correlating shopping habits based on receipts and customer loyalty cards to looking for terrorists by their travel patterns.

My takeaway from the article is that the definition of BD is a moving target as the capability of hardware grows.

If "traditional" approaches to data fail to scale, why not just start with BD methods in the first place? In that case, BD is a meaningless term as it simply becomes "a better way of handling data". OTOH, if there's a high technical hurdle between "traditional" and BD methods, then you have an incentive to stay traditional until you're confronted with the problem. Therein lies the crux of my

There's an entire field dedicated to Data Warehousing who's entire focus is Big Data. Large companies with auditing requirements have to keep mountains of historical data. Business Intelligence is largely based on analyzing huge segments of data.

As storage gets cheaper and options for going through large amounts of data become more widely available, companies invariably store more data. The biggest difference is that while you previously would have simply chosen not to track certain types of data in your

Big Data is however big you need it to be. It's not a certain size, or speed, or software, but rather a philosophy.

Simply put, Big Data methodologies are to gather all the data that can be gathered, and store it on a nice cheap database, without concern for storage efficiency. When a question arises, analyse the relevant data for an answer. This is in contrast to more traditional methods, where data is gathered only to solve expected questions.

Slashdot's archives were not generated from a Big Data approach. They store only comments and a few sparse details (to my knowledge). However, they can still be used in a Big Data system to some effect, if they happen to store the information that's needed (such as IP address, timestamps, and keywords, if the question is to track political opinions by geography over time).

It's not what you store or how you store it, but how you decide what to store. What makes Big Data approaches useful is that they store everything from the beginning, so as business needs change, the data from the past is likely just as useful as new data. What makes Big Data difficult is that the databases must be properly capable of storing all the gathered data as fast as it arrives, and must do so cheaply.

There's actually more of a need for it than you'd think at first glance. Any business that handles upwards of a few million records of some sort of data, and then needs to *do* something with that data, could potentially benefit from this.

I've worked at small (10,000) businesses (all tech-related), and only the smallest places didn't have the amount of data to warrant taking a look at something like this.

Also, what about those interested in *someday* working for the Googles and the Facebooks of the world? A

Dang, didn't look closely at the preview, some formatting killed off part of my message. The 2nd paragraph should be:

I've worked at small (*less than* 100 employees), medium (*less than* 1000), and large (*greater than* 10,000) businesses (all tech-related), and only the smallest places didn't have the amount of data to warrant taking a look at something like this.

I'm not sure why you got modded down, because your answer is somewhat thought provoking and not trollish at all.

You reminded me of a situation I saw involving monitoring networks. The solution to the data overload there was a "roll your own" database and AFAIK it could not be querried with a full set of SQL commands; but was faster and able to handle our data better than off-the-shelf Open Source solutions. This was years before I heard the term "big data". We may have been moving in that direction witho

That is a _puny_ database by "Big Data" standards. Every table apart from the comments themselves could be cached in RAM on a modern server, and the majority of comments would never need to be fetched off disk - a single SSD at that - so almost everything important could be cached.

Yup, that was my bad. It IS Hadoop Distributed File System (HDFS) and NOT Hadoop File Distribution System (HFDS). I had it right in front of me and still typed it wrong from some of my notes. I'll see if I can get it fixed. Thanks.

Excellent point...but that is something to ask the Hadoop developers, not the author. You can use Hadoop without writing Java. You cannot use Hadoop without knowing in depth about the complete mess that is Java build, linking, etc.