Improved REST Interface

Details

Description

I would like to extend the existing REST Interface to also support:

configuration

ephemeral znodes

watches - PubSubHubbub

ACLs

basic authentication

I want to do this because when building web applications that talks directly to ZooKeeper a REST API it's a lot easier to use (there is no protocol mismatch) than an API that uses persistent connections. I plan to use the improved version to build a web-based administrative interface.

Andrei Savu
added a comment - 10/Jul/10 15:20 Very soon I'm also going to attach an updated version of the REST SPEC.txt file because I would like to get more feedback from you before writing any code.

Andrei Savu
added a comment - 14/Jul/10 14:11 I've attached the first draft of the SPECs for an improved REST gateway.
I'm proposing the following new operations:
1. create a new session – POST /sessions/v1?op=create HTTP/1.1
2. keep the session alive – PUT /sessions/v1/<SESSION-ID> HTTP/1.1
3. close the session – DELETE /sessions/v1/<SESSION-ID> HTTP/1.1
4. create an ephemeral node – POST /znodes/v1/a/b?op=create&name=c&ephemeral=true&session=<SESSION-ID> HTTP/1.1
5. create a new watch – POST /znodes/v1/a/b?op=watch&view=data OR children&session=<SESSION-ID> HTTP/1.1
6. query watch status – GET /sessions/v1/<SESSION-ID>/watches/<WATCH-ID> HTTP/1.1
This operation could support long-polling in the future.
7. delete a triggered watch – DELETE /sessions/v1/<SESSION-ID>/watches/<WATCH-ID> HTTP/1.1
Let me know what you think about this. Am I breaking the REST principles?
This is what I want to do in the first iteration. In the second iteration I would like to add support for ACLs and authentication.

When I first created the REST interface I didn't have the notion of sessions, now that you do I think you would want to augment the notion of having a /znodes/... url with a url of /sessions/v1/<session TOKEN>/znodes/....

so create the session as you suggest, however that create operation returns a url representing the session, after which all of your operations use that as a "prefix" if you will. e.g.:

create a new session - POST /sessions/v1?op=create HTTP/1.1

returns /sessions/v1/ab483cd8283ef274

notice the session TOKEN is a randomly generated key - this allows for some "security through obscurity" as it's "hard to guess" and is some small measure of security. session keepalive and delete would operate on this url. GET on the url might return the original session id for example

you might keep the /znodes feature as-is for those not wanting to use sessions (admin r/o console say, or cli tool), however you might want to make turning it off an option - allowing the operator to force users to use explicit sessions

notice how this also cleans up items 5/6 wrt the url used to access (same prefix in both cases)

when you add acl support you might add something like:

/sessions/v1/ab483cd8283ef274/auth/...

resource for managing them (add auth for example). I think you'd have to require SSL to make this secure..., and return some security token good for the session so that someone else can't impersonate you... etc...

Patrick Hunt
added a comment - 14/Jul/10 16:01 When I first created the REST interface I didn't have the notion of sessions, now that you do I think you would want to augment the notion of having a /znodes/... url with a url of /sessions/v1/<session TOKEN>/znodes/....
so create the session as you suggest, however that create operation returns a url representing the session, after which all of your operations use that as a "prefix" if you will. e.g.:
create a new session - POST /sessions/v1?op=create HTTP/1.1
returns /sessions/v1/ab483cd8283ef274
notice the session TOKEN is a randomly generated key - this allows for some "security through obscurity" as it's "hard to guess" and is some small measure of security. session keepalive and delete would operate on this url. GET on the url might return the original session id for example
create an ephemeral node - POST /sessions/v1/ab483cd8283ef274/znodes/v1/a/b?op=create&name=c&ephemeral=true HTTP/1.1
you might keep the /znodes feature as-is for those not wanting to use sessions (admin r/o console say, or cli tool), however you might want to make turning it off an option - allowing the operator to force users to use explicit sessions
notice how this also cleans up items 5/6 wrt the url used to access (same prefix in both cases)
when you add acl support you might add something like:
/sessions/v1/ab483cd8283ef274/auth/...
resource for managing them (add auth for example). I think you'd have to require SSL to make this secure..., and return some security token good for the session so that someone else can't impersonate you... etc...

Andrei Savu
added a comment - 04/Aug/10 11:49
This patch contains:
basic support for sessions ( create: POST /sessions/v1?op=create&expire=SECONDS, delete: DELETE /sessions/v1/<ID>, heartbeat: PUT /sessions/v1/<ID> )
support for ephemeral nodes: create: POST /znodes/v1/a/b?op=create&name=c&ephemeral=true&session=ID
experimental (almost complete) python REST client
demos: demo_master_election.py & demo_queue.py
updated specs
I'm not going to implement watches now because they are not useful if the client pulls the server for changes.
Working on:
configuration, ACLs and basic authentication
packaging - I want to be able to run the REST gateway as a task managed by the Hue [1] supervisor.
I would really like to get more feedback from you regarding this patch. Thanks.
[1] http://github.com/cloudera/hue

Andrei Savu
added a comment - 05/Aug/10 07:03 Thanks Patrick for reviewing the patch. I will fix those issues ASAP.
Sorry about reformatting existing code, I've used the Eclipse Format tool and only latter I've realized that it changes the whole file.

Andrei Savu
added a comment - 05/Aug/10 14:58 Changes in this patch:
ZooKeeperService now uses the servlet context path and not full URI as a key for retrieving ZooKeeper connections
added a configuration file: conf/rest.properties
added support for multiple endpoints: /cluster1, /cluster2 - useful if you have multiple ZooKeeper clusters
HTTPS support
HTTP Basic Authentication
shutdownHook for cleanup
I'm still working on: packaging, ACLs & ZooKeeper Authentication
I will finish the packaging ant task tomorrow and next week (after the GSoC "soft" deadline) I will also add support for ACLs and Authentication.
Let me know what you think. Thanks.

Andrei Savu
added a comment - 06/Aug/10 13:46 In this patch:
ant task for packaging: just run ant tar and it will generate zookeeper-dev-rest.tar.gz in build/contrib/rest/
basic script for starting and stopping the REST gateway - bin/restServer.sh
Remaining:
ACLs & ZooKeeper Authentication - after GSoC "soft" deadline
All tests are passing, including the ones from the new python client test suite.

By using these features you can easily create secure channels from your application to ZooKeeper ( HTTPS + Digest Authentication + ZK Auth + chroot). It doesn't support the whole API but it should be really useful for configuration management.