Difference between revisions of "SMILA/Documentation/Using The ReST API"

(New page: In the future all functionality of SMILA should be acessible via a HTTP ReST API using JSON to represent data. This means basically that you can control and monitor a running SMILA server ...)

In the future all functionality of SMILA should be acessible via a HTTP ReST API using JSON to represent data.

+

Most functionality of SMILA is already accessible via an HTTP ReST API using JSON to represent data.

This means basically that you can control and monitor a running SMILA server using your web browser (at least with some small plugins).

This means basically that you can control and monitor a running SMILA server using your web browser (at least with some small plugins).

Also, it is relatively simple to talk to SMILA programmatically or using scripts. On this page we will recommend some tools and

Also, it is relatively simple to talk to SMILA programmatically or using scripts. On this page we will recommend some tools and

Line 8:

Line 8:

== Basics ==

== Basics ==

−

A HTTP ReST API consists of a set of URLs that can be invoked using standard HTTP requests. One URL refers to a "resource" in the system, which may be a service or some element managed by a service. Each resource may support several HTTP methods for different operations, not each resource will support every method. In SMILA we use the following methods, usually with one the described semantics:

+

An HTTP ReST API consists of a set of URLs that can be invoked using standard HTTP requests. One URL refers to a "resource" in the system, which may be a service or some element managed by a service. Each resource may support several HTTP methods for different operations, but not each resource will support every method. In SMILA we use the following methods, usually with one of the described semantics:

−

+

* <tt>GET</tt>: get the content, definition or description of the resource; get statistic data or state information about the resource

−

* <tt>GET</tt>: get the content, definiton or description of the resource; get statistic data or state information

+

* <tt>POST</tt>: execute the main functionality of the resource (e.g. execute a pipeline); add or update a sub element of the resource

−

* <tt>POST</tt>: execute the main functionality of the resource (e.g. execute a pipeline); add a new sub element to the resource

+

* <tt>PUT</tt>: set the content, definition, or description of the resource

−

* <tt>PUT</tt>: set the content, definition or description of the resource

+

* <tt>DELETE</tt>: clear the contents of the resource; remove an element from the resource.

* <tt>DELETE</tt>: clear the contents of the resource; remove an element from the resource.

−

Most resources should at least support the <tt>GET</tt> method which is the method used by the web browser when you enter the URL, and the result should contain links to resources associated to this resource. So you can explore the system state with your web browser.

+

Most resources should at least support the <tt>GET</tt> method which is the method used by the web browser when you enter the URL, and the result should contain links to resources associated with this resource. So you can explore the system state with your web browser.

<tt>GET</tt> and <tt>DELETE</tt> requests never have a request body, parameters must be passed by using <tt>...?param1=value1&param2=value2</tt> in the URL. With <tt>POST</tt> and <tt>PUT</tt> requests the data is passed in the request body as a JSON object, i.e. a JSON string like this:

<tt>GET</tt> and <tt>DELETE</tt> requests never have a request body, parameters must be passed by using <tt>...?param1=value1&param2=value2</tt> in the URL. With <tt>POST</tt> and <tt>PUT</tt> requests the data is passed in the request body as a JSON object, i.e. a JSON string like this:

Line 26:

Line 25:

</source>

</source>

−

In SMILA this corresponds to the [[SMILA/Documentation/Data_Model_and_Serialization_Formats#Concepts|metadata part of a record]]. In <tt>POST</tt> requests it is also possible to add binary attachments, see [[SMILA/Documentation/JettyHttpServer#Attachments|Documentation of the SMILA HTTP server]] for details. Finally, some resources accept or produce so-called "JSON bulks". This means that it is possible to send or get a bunch of records in a single request. The JSON of the record metadata must then be printed on a single line, and the records are separaeted by newlines. However, it is not possible to add attachments to such records.

+

In SMILA, this corresponds to the [[SMILA/Documentation/Data_Model_and_Serialization_Formats#Concepts|metadata part of a record]]. In <tt>POST</tt> requests it is also possible to add binary attachments, see [[SMILA/Documentation/JettyHttpServer#Attachments|Documentation of the SMILA HTTP server]] for details. Finally, some resources accept or produce so-called "JSON bulks". This means that it is possible to send or get a bunch of records in a single request. The JSON of the record metadata must then be printed on a single line, and the records are separated by newlines. However, it is not possible to add attachments to such records.

See [[SMILA/REST_API_Reference]] for a list of available resources and links to documentation.

See [[SMILA/REST_API_Reference]] for a list of available resources and links to documentation.

Line 34:

Line 33:

=== Web Browser Add-Ons ===

=== Web Browser Add-Ons ===

−

For most web browsers exist free plugins or add-ons that support pretty-printed viewing of JSON documents and interactive invocation of ReST commands.

+

For most web browsers there are free add-ons available that support pretty-printed viewing of JSON documents and the interactive invocation of ReST commands.

* JSON viewer: By default browsers do not know how to handle JSON results and will suggest to download them to a file. There are add-ons that display the JSON right in the browser, nicely formatted and highlighted and make contained URLs clickable, which makes it easy to explore the system state

* JSON viewer: By default browsers do not know how to handle JSON results and will suggest to download them to a file. There are add-ons that display the JSON right in the browser, nicely formatted and highlighted and make contained URLs clickable, which makes it easy to explore the system state

Line 44:

Line 43:

** Chrome:

** Chrome:

*** [https://chrome.google.com/webstore/detail/baedhhmoaooldchehjhlpppaieoglhml cREST Client]: Easy to use and keeps a nice history of the executed requests so that it is easy to repeat sequences of commands for test purposes.

*** [https://chrome.google.com/webstore/detail/baedhhmoaooldchehjhlpppaieoglhml cREST Client]: Easy to use and keeps a nice history of the executed requests so that it is easy to repeat sequences of commands for test purposes.

−

*** https://chrome.google.com/webstore/detail/hgmloofddffdnphfgcellkdfbfbjeloo Advanced REST client application]: A bit harder to use, but the only one we currently know of that lets you create "multipart" requests containing record attachments: In the following screenshot we have created a request to submit a record (first file field contains name of a JSON file) with an attachment (second file field) to the "indexUpdate" job:

+

*** [https://chrome.google.com/webstore/detail/hgmloofddffdnphfgcellkdfbfbjeloo Advanced REST client application]: A bit harder to use, but the only one we currently know of that lets you create "multipart" requests containing record attachments: In the following screenshot we have created a request to submit a record (first file field contains name of a JSON file) with an attachment (second file field) to the "indexUpdate" job: <br> [[Image:SMILA-SendAttachmentWithAdvancedRestClient.png | 600 px]]

Basically, it is possible to work with the ReST API using generic HTTP tools like [http://curl.haxx.se/ cURL]. However, for improved usability you might want to check out [resty https://github.com/micha/resty]. This is a simple wrapper (for bash and zsh) for [http://curl.haxx.se/ cURL] that allows you to write ReST requests like shell commands:

+

Basically, it is possible to work with the ReST API using generic HTTP tools like [http://curl.haxx.se/ cURL]. However, for improved usability you might want to check out [https://github.com/micha/resty resty]. This is a simple wrapper (for bash and zsh) for [http://curl.haxx.se/ cURL] that allows you to write ReST requests like shell commands:

<source lang="text">

<source lang="text">

Line 63:

Line 60:

</source>

</source>

−

See the [resty web site|https://github.com/micha/resty] for more information.

+

See the [https://github.com/micha/resty resty web site] for more information.

+

+

==== Posting attachments from shell scripts ====

+

You can use [http://curl.haxx.se/ cURL] to send a record with some metadata and a binary file as an attachment: Use the <tt>-F</tt> option to send a multipart request to the specified URL. The part name "metadata" for the JSON part is arbitrary, the JSON part must be the first part in the request. The field name of the following attachment part (here "content") will be the name of the attachment in the created record. You can add multiple attachments with different names this way.

As [https://github.com/micha/resty resty] is based on cURL and passes all options to the cURL call, the same <tt>-F</tt> options should work there, too.

== Programmatical Access ==

== Programmatical Access ==

−

__TODO__ Examples with Apache Http Client

+

It's quite easy to use the SMILA API from external programs. In Java, basically everything you need is available in packages <tt>java.net</tt>. But it's easier using the SMILA data model classes and JSON utilities and a HTTP library like the Apache HttpClient. Below we provide some simple examples for doing requests with it.

+

+

There are also free Java libraries available to make working with JSON ReST APIs easier, e.g. [http://beders.github.com/Resty/Resty/Overview.html Resty] (yes, same name, but another tool ;-). However, we do not have experience with them.

+

+

=== SMILA RestClient ===

+

+

Since SMILA 1.1.0, we also provide a small library that encapsulates the JSON conversion and HTTP communication code for you. It's based on the SMILA data model classes and the Apache HttpCient 4.1, so it basically wraps up the code shown below. See [[SMILA/Documentation/HowTo/How_to_access_the_REST_API_with_the_RestClient]] for details.

+

+

=== Apache HttpClient 4.x ===

+

+

Using [http://hc.apache.org/httpcomponents-client-ga/index.html|Apache HttpClient 4.x] (version 4.1 is included in SMILA since 1.1.0) and [http://commons.apache.org/io/ Apache Commons IO], invoking a pipeline could be done like this:

There are also free Java libraries available to make working with JSON ReST APIs easier, e.g. [Resty http://beders.github.com/Resty/Resty/Overview.html] (yes, same name, but another tool ;-). However, we do not have experience with them.

+

[[Category:SMILA]]

Revision as of 10:45, 2 March 2012

Most functionality of SMILA is already accessible via an HTTP ReST API using JSON to represent data.
This means basically that you can control and monitor a running SMILA server using your web browser (at least with some small plugins).
Also, it is relatively simple to talk to SMILA programmatically or using scripts. On this page we will recommend some tools and
best practices to work with SMILA.

Feel free to extend this page if you find interesting tools that you want to recommend to other SMILA users.

Contents

Basics

An HTTP ReST API consists of a set of URLs that can be invoked using standard HTTP requests. One URL refers to a "resource" in the system, which may be a service or some element managed by a service. Each resource may support several HTTP methods for different operations, but not each resource will support every method. In SMILA we use the following methods, usually with one of the described semantics:

GET: get the content, definition or description of the resource; get statistic data or state information about the resource

POST: execute the main functionality of the resource (e.g. execute a pipeline); add or update a sub element of the resource

PUT: set the content, definition, or description of the resource

DELETE: clear the contents of the resource; remove an element from the resource.

Most resources should at least support the GET method which is the method used by the web browser when you enter the URL, and the result should contain links to resources associated with this resource. So you can explore the system state with your web browser.

GET and DELETE requests never have a request body, parameters must be passed by using ...?param1=value1&param2=value2 in the URL. With POST and PUT requests the data is passed in the request body as a JSON object, i.e. a JSON string like this:

{"param1":"value1","param2":"value2
}

In SMILA, this corresponds to the metadata part of a record. In POST requests it is also possible to add binary attachments, see Documentation of the SMILA HTTP server for details. Finally, some resources accept or produce so-called "JSON bulks". This means that it is possible to send or get a bunch of records in a single request. The JSON of the record metadata must then be printed on a single line, and the records are separated by newlines. However, it is not possible to add attachments to such records.

Interactive Tools

Web Browser Add-Ons

For most web browsers there are free add-ons available that support pretty-printed viewing of JSON documents and the interactive invocation of ReST commands.

JSON viewer: By default browsers do not know how to handle JSON results and will suggest to download them to a file. There are add-ons that display the JSON right in the browser, nicely formatted and highlighted and make contained URLs clickable, which makes it easy to explore the system state

ReST Client: ReST Client allow to compose other requests than GET requests with the browser. They allow to enter the resource URL, select the request method, and add the request body content, and then you can execute the request and view the result. There are lots of these add-ons out there, we can recommend the following:

cREST Client: Easy to use and keeps a nice history of the executed requests so that it is easy to repeat sequences of commands for test purposes.

Advanced REST client application: A bit harder to use, but the only one we currently know of that lets you create "multipart" requests containing record attachments: In the following screenshot we have created a request to submit a record (first file field contains name of a JSON file) with an attachment (second file field) to the "indexUpdate" job:

Shell scripting

Basically, it is possible to work with the ReST API using generic HTTP tools like cURL. However, for improved usability you might want to check out resty. This is a simple wrapper (for bash and zsh) for cURL that allows you to write ReST requests like shell commands:

Posting attachments from shell scripts

You can use cURL to send a record with some metadata and a binary file as an attachment: Use the -F option to send a multipart request to the specified URL. The part name "metadata" for the JSON part is arbitrary, the JSON part must be the first part in the request. The field name of the following attachment part (here "content") will be the name of the attachment in the created record. You can add multiple attachments with different names this way.

# send a metadata record and a binary attachment to a etlImport job
> curl -F "metadata=@record.json" -F "content=@binary-file.pdf" http://localhost:8080/smila/job/etlImport/record

As resty is based on cURL and passes all options to the cURL call, the same -F options should work there, too.

Programmatical Access

It's quite easy to use the SMILA API from external programs. In Java, basically everything you need is available in packages java.net. But it's easier using the SMILA data model classes and JSON utilities and a HTTP library like the Apache HttpClient. Below we provide some simple examples for doing requests with it.

There are also free Java libraries available to make working with JSON ReST APIs easier, e.g. Resty (yes, same name, but another tool ;-). However, we do not have experience with them.