Node.js SDK for Joyent Manta

This is the reference documentation for the Manta Node.js
SDK. Manta is Joyent's Object Storage Service, which enables you to store data
in the cloud and process that data using a built-in compute facility.

This document explains the Node.js API interface and describes the various
operations, structures and error codes.

Conventions

Any content formatted like this:

curl is https://us-east.manta.joyent.com

is a command-line example that you can run from a shell. All other examples and
information are formatted like this:

Installation

First, install the SDK as usual via npm; the package name
in npm is manta. You may optionally want to install the package globally with
the -g flag to npm, as this should place the node-manta CLI in your $PATH.

npm install manta

Once you've installed the npm package, there a few environment variables that
are useful to set if you plan to work with the CLI; these environment variables
are not strictly necessary, but they will save you passing in command line
options on each invocation. The environment variables that can be set are your
SmartDataCenter login name and ssh public key fingerprint (manta uses the same
credentials), and the URL of which manta endpoint you wish to interact with.
The commands below assume that your SSH public key is the default id_rsa.pub
key, located in your $HOME/.ssh directory (on Mac OS X and UNIX environments).
The shell command below simply parses the SSH fingerprint and sets that in the
requisite environment variable.

(optional): amount of milliseconds to wait for acquiring a socket to Manta; defaults to 0 (infinity)

log

Object

(optional): bunyan logger; default is at level fatal and writes to stderr

headers

Object

(optional): HTTP headers to send on all requests

sign

Function

(required): see authenticating requests below

url

String

(required): URL to interact with Manta on

user

String

(optional): login name to use when interacting with the jobs API

Authenticating Requests

When creating a manta client, you'll need to pass in a callback function for
the sign parameter. node-manta ships with two functions that will likely suit
your need: privateKeySigner and sshAgentSigner. Both of these callbacks
will automatically do the correct crypto for authenticating manta requests, the
difference is that privateKeySigner expects (non-passphrase protected) keys to
be passed in directly (as a file name), whereas sshAgentSigner will load your
credentials on each request from the SSH agent (if available). Both callbacks
require you to set the manta user (login) and keyId (SSH key fingerprint).

Note that the sshAgentSigner is not suitable for server applications, or any
other system where the performance degradation necessary to interact with SSH
is not acceptable; put another way, you should only use it for interactive
tooling, such as the CLI that ships with node-manta.

Should you wish to write a custom plugin, the expected implementation of the
sign callback is a function of the form function (string, callback).
string is generated by node-manta (typically the value of the Date header),
and callback is of the form function (err, object), where object has the
following properties:

Use-cases where you would need to write your own signer are things like signing
with a smart-card or other HSM, making remote calls to a central system, etc.

Presigned URLs

In some cases you may want your app to be able to generate a full URL, suitable
for giving out to others as a link. In these cases, you can use the presigned
URL approach, and set an expires parameter. node-manta has a simple API for
this that utilizes the same sign callback as all other APIs, but simply does
the correct canonicalization for a URL:

If you are not passing in explicit options, the second form is always there for
convenience. All API operations allow you to pass in a standard set of options,
which are:

Name

JS Type

Description

headers

Object

Any HTTP headers to be included in this request

req_id

String

A unique identifier for this request (SHOULD be a uuid)

query

Object

A key/value set of parameters to be encoded on the URL's query string

You can always override any node-manta behavior by passing in explicit HTTP
headers, but in most cases, you should just use the "higher-level" parameters
available in the specific API you are interested in.

Common Callback Parameters

In almost all cases (the exception being the "streaming" APIs like ls)
callbacks will be of the form function (error, result), where err is either
a JavaScript
Error
object or null. result is a standard node
http.ClientResponse
object, where you will be able to access HTTP headers, response codes, etc.
Note that if there was an HTTP response code >= 400, then err will be present
and filled in with the Manta error code and message (see errors).

Errors

All callback functions may return a Javascript Error object. In most cases,
you can simply switch on err.name, which will be correctly filled in from
server error codes sent back. The only cases where you cannot are lower-level
errors such as ECONNREFUSED that are generated by the node.js runtime. The
complete list of manta error names is:

AuthSchemeError

AuthorizationError

BadRequestError

ChecksumError

ConcurrentRequestError

ContentLengthError

InvalidArgumentError

InvalidAuthTokenError

InvalidCredentialsError

InvalidDurabilityLevelError

InvalidKeyIdError

InvalidJobError

InvalidLinkError

InvalidSignatureError

DirectoryDoesNotExistError

DirectoryExistsError

DirectoryNotEmptyError

DirectoryOperationError

JobNotFoundError

JobStateError

KeyDoesNotExistError

NotAcceptableError

NotEnoughSpaceError

LinkNotFoundError

LinkNotObjectError

LinkRequiredError

ParentNotDirectoryError

PreSignedRequestError

RequestEntityTooLargeError

ResourceNotFoundError

RootDirectoryError

ServiceUnavailableError

SSLRequiredError

UploadTimeoutError

UserDoesNotExistError

Directories

client.mkdir(path, [options], callback)

Create or overwrite a directory at path. mkdir is really a PUT operation,
so it's slightly different semantics than mkdir(2) in POSIX (meaning, you
can call mkdir on the same path twice). There is no return value besides a
potential error.

Inputs

client.ls(path, [options], callback)

Lists directory contents. This API will return an EventEmitter that will emit
a stream of entries as they are returned from the server. You can listen for
two distinct types directory; records of type object will have slightly more
information in the records. Both records will have a type field in them.
Otherwise, the returned entries are described below. Optional pagination
parameters can be included in the options block, and act as you would expect.
There is a server-enforced limit of 1000 entries per list request, which is also
the default limit, however you can request a smaller size if need be. You can
also choose to only receive objects of a certain type.

Inputs

Output Objects

Each output object will be of this schema:

{
name: 'foo', // basename of the entry
etag: 'AABBCC', // only set on objects
size: 1234, // only set on objects; valueOf(content-length)
type: 'object', // one of directory || object
mtime: '2012-11-09T12:34:56Z' // ISO8601 timestamp of the last update time
}

client.createListStream(path, [options])

List directory contents. This is the ReadableStream version of ls(). Each
object read from the stream has the same format as objects emitted from the
EventEmitter returned by ls().

Inputs

Objects

client.put(path, stream, options, callback)

Creates or overwrites an (object) key. You pass it in a ReadableStream (note
that stream must support pause/resume), and upon receiving a 100-continue
from manta, the bytes get blasted up.

In this API, you can either pass in an actual 'size' attribute in the options
object. If you set that, that is the content-length header for this request. If
you don't set that, the request will be "streaming" (transfer-encoding=chunked),
in which case your object either needs to fit into the "default" object size
(5Gb currently), OR you need to pass in a header of max-content-length, which
will be the _maximum_ size of your data. Additionally, you can/should pass in an
'md5' attribute, and you can pass a 'type' attribute which is really the
content-type. If you don't pass in 'type', this API will try to guess it based
on the name of the object (using the extension). Lastly, you can pass in a
'copies' attribute, which sets the number of full object copies to make server
side (default is 2).

However, like the other APIs, you can additionally pass in extra headers, etc.
in the options object as well. In the case of objects this is particularly
useful for setting CORS headers, for example.

client.createWriteStream(path, options)

Essentially the same API/logic as client.put, but idiomatic to node the node
streams model. path and options are the same as put, but this API takes
no callback, and instead returns an instance of
stream.Writable.

Note that standard node stream semantics don't line up to when Manta has
actually committed data, so the stream returned by this API emits a close
event that also has the
http.Response
object.

Inputs

client.createReadStream(path, [options])

Fetches an object as a ReadableStream; this API is basically identical to get,
except it's idiomatic to node streaming. Additionally, the returned stream will
emit close at the end of request data along with the HTTP Response object.

Note this form is only useful for map only jobs; you cannot specify reduce
tasks in this way.

options allows you to set arbitrary headers (as usual), and callback is
of the form function (err, jobId). jobId will be the server-created id for
this job, which you can pass into the other job related APIs.

Inputs

Name

JS Type

Description

job

Obect

(required) A job definition object, as described below

options

Object

(optional) optional overrides for this request

callback

Function

(required) callback of the form fn(err, jobId)

The full set of allowed options for job:

Name

JS Type

Description

name

String

(optional) An arbitrary name for this job

input

String

(optional) An arbitrary jobId to pipe from

phases

Array

(required) tasks to execute as part of this job

phases must be an Array of Object, where objects have the following
properties:

Name

JS Type

Description

type

String

(optional) one of: map or reduce

assets

Array[String]

(optional) an array of manta keys to be placed in your compute zones

exec

String

(required) the actual (shell) statement to execute

count

Number

(optional) an optional number of reducers for this phase (reduce-only): default is 1

memory

Number

(optional) an optional amount of DRAM to give to your compute zone

count has a minimum of 1 (default), and a maximum of 1024.

memory must be one of the following: 128, 256, 512, 1024, 2048, 4096, 8192, 16384

Output

Output is simply a String job id.

client.job(jobId, [options], callback)

Retrieves a job from Manta. This is the "overall" object, and will not contain
input/output keys or failures.