Native Libraries Guide

Overview

This guide describes the native hadoop library and includes a small discussion about native shared libraries.

Note: Depending on your environment, the term "native libraries" could
refer to all *.so's you need to compile; and, the term "native compression" could refer to all *.so's
you need to compile that are specifically related to compression.
Currently, however, this document only addresses the native hadoop library (libhadoop.so).

Native Hadoop Library

Hadoop has native implementations of certain components for
performance reasons and for non-availability of Java implementations. These
components are available in a single, dynamically-linked native library called
the native hadoop library. On the *nix platforms the library is named libhadoop.so.

Usage

Either download a hadoop release, which will
include a pre-built version of the native hadoop library, or
build your own version of the
native hadoop library. Whether you download or build, the name for the library is
the same: libhadoop.so

On all the above distributions a 32/64 bit native hadoop library will work
with a respective 32/64 bit jvm.

Download

The pre-built 32-bit i386-Linux native hadoop library is available as part of the
hadoop distribution and is located in the lib/native directory. You can download the
hadoop distribution from Hadoop Common Releases.

Be sure to install the zlib and/or gzip development packages - whichever compression
codecs you want to use with your deployment.

Build

The native hadoop library is written in ANSI C
and is built using the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool).
This means it should be straight-forward to build the library on any platform with a standards-compliant
C compiler and the GNU autotools-chain (see the supported platforms).

Once you installed the prerequisite packages use the standard hadoop build.xml
file and pass along the compile.native flag (set to true) to build the native hadoop library:

$ ant -Dcompile.native=true <target>

You should see the newly-built library in:

$ build/native/<platform>/lib

where <platform> is a combination of the system-properties:
${os.name}-${os.arch}-${sun.arch.data.model} (for example, Linux-i386-32).

Please note the following:

It is mandatory to install both the zlib and gzip
development packages on the target platform in order to build the
native hadoop library; however, for deployment it is sufficient to
install just one package if you wish to use only one codec.

It is necessary to have the correct 32/64 libraries for zlib,
depending on the 32/64 bit jvm for the target platform, in order to
build and deploy the native hadoop library.

Runtime

The bin/hadoop script ensures that the native hadoop
library is on the library path via the system property: -Djava.library.path=<path>