Overview

Introduction

The Storage and retrieval of application’s current and historic information in a generic fashion is addressed in YARN through the Timeline Server. It has two responsibilities:

Persisting Application Specific Information

The collection and retrieval of information completely specific to an application or framework. For example, the Hadoop MapReduce framework can include pieces of information like number of map tasks, reduce tasks, counters…etc. Application developers can publish the specific information to the Timeline server via TimelineClient in the Application Master and/or the application’s containers.

This information is then queryable via REST APIs for rendering by application/framework specific UIs.

Persisting Generic Information about Completed Applications

Previously this was supported purely for MapReduce jobs by the Application History Server. With the introduction of the timeline server, the Application History Server becomes just one use of the Timeline Server.

Generic information includes application level data such as * queue-name, * user information and the like set in the ApplicationSubmissionContext, * a list of application-attempts that ran for an application * information about each application-attempt * the list of containers run under each application-attempt * information about each container.

Generic data is published by the YARN Resource Manager to the timeline store and used by its web-UI to display information about completed applications.

Current Status and Future Plans

Current status

The core functionality of the timeline server has been completed.

It works in both secure and non secure clusters.

The generic history service is built on the timeline store.

The history can be stored in memory or in a leveldb database store; the latter ensures the history is preserved over Timeline Server restarts.

The ability to install framework specific UIs in YARN is not supported.

Application specific information is only available via RESTful APIs using JSON type content.

The “Timeline Server v1” REST API has been declared one of the REST APIs whose compatibility will be maintained in future releases.

The single-server implementation of the Timeline Server places a limit on the scalability of the service; it also prevents the service being High-Availability component of the YARN infrastructure.

Future Plans

Future releases will introduce a next generation timeline service which is scalable and reliable, “Timeline Server v2”.

The expanded features of this service may not be available to applications using the Timeline Server v1 REST API. That includes extended data structures as well as the ability of the client to failover between Timeline Server instances.

Timeline Structure

Timeline Domain

The Timeline Domain offers a namespace for Timeline server allowing users to host multiple entities, isolating them from other users and applications. Timeline server Security is defined at this level.

A “Domain” primarily stores owner info, read and& write ACL information, created and modified time stamp information. Each Domain is identified by an ID which must be unique across all users in the YARN cluster.

Timeline Entity

A Timeline Entity contains the the meta information of a conceptual entity and its related events.

The entity can be an application, an application attempt, a container or any user-defined object.

It contains Primary filters which will be used to index the entities in the Timeline Store. Accordingly, users/applications should carefully choose the information they want to store as the primary filters.

The remaining data can be stored as unindexed information. Each Entity is uniquely identified by an EntityId and EntityType.

Timeline Events

A Timeline Event describes an event that is related to a specific Timeline Entity of an application.

Users are free to define what an event means —such as starting an application, getting allocated a container, an operation failures or other information considered relevant to users and cluster operators.

Deployment

Configurations

Basic Configuration

Configuration Property

Description

yarn.timeline-service.enabled

Indicate to clients whether Timeline service is enabled or not. If enabled, the TimelineClient library used by applications will post entities and events to the Timeline server. Defaults to false.

yarn.resourcemanager.system-metrics-publisher.enabled

The setting that controls whether or not YARN system metrics are published on the timeline server by RM. Defaults to false.

yarn.timeline-service.generic-application-history.enabled

Indicate to clients whether to query generic application data from timeline history-service or not. If not enabled then application data is queried only from Resource Manager. Defaults to false.

Timeline store and state store configuration

Configuration Property

Description

yarn.timeline-service.store-class

Store class name for timeline store. Defaults to org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.

yarn.timeline-service.leveldb-timeline-store.path

Store file name for leveldb timeline store. Defaults to ${hadoop.tmp.dir}/yarn/timeline.

yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms

Length of time to wait between deletion cycles of leveldb timeline store in milliseconds. Defaults to 300000.

yarn.timeline-service.leveldb-timeline-store.read-cache-size

Size of read cache for uncompressed blocks for leveldb timeline store in bytes. Defaults to 104857600.

Size of cache for recently written entity start times for leveldb timeline store in number of entities. Defaults to 10000.

yarn.timeline-service.recovery.enabled

Defaults to false.

yarn.timeline-service.state-store-class

Store class name for timeline state store. Defaults to org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore.

yarn.timeline-service.leveldb-state-store.path

Store file name for leveldb timeline state store.

Web and RPC Configuration

Configuration Property

Description

yarn.timeline-service.hostname

The hostname of the Timeline service web application. Defaults to 0.0.0.0

yarn.timeline-service.address

Address for the Timeline server to start the RPC server. Defaults to ${yarn.timeline-service.hostname}:10200.

yarn.timeline-service.webapp.address

The http address of the Timeline service web application. Defaults to ${yarn.timeline-service.hostname}:8188.

yarn.timeline-service.webapp.https.address

The https address of the Timeline service web application. Defaults to ${yarn.timeline-service.hostname}:8190.

yarn.timeline-service.bind-host

The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively. This is most useful for making the service listen on all interfaces by setting to 0.0.0.0.

yarn.timeline-service.http-cross-origin.enabled

Enables cross-origin support (CORS) for web services where cross-origin web response headers are needed. For example, javascript making a web services request to the timeline server. Defaults to false.

The number of seconds a pre-flighted request can be cached for web services needing cross-origin (CORS) support. Defaults to 1800.

Note that the selection between the HTTP and HTTPS binding is made in the TimelineClient based upon the value of the YARN-wide configuration option yarn.http.policy; the HTTPS endpoint will be selected if this policy is either of HTTPS_ONLY or HTTP_AND_HTTPS.

Advanced Server-side configuration

Configuration Property

Description

yarn.timeline-service.ttl-enable

Enable deletion of aged data within the timeline store. Defaults to true.

yarn.timeline-service.ttl-ms

Time to live for timeline store data in milliseconds. Defaults to 604800000 (7 days).

The maximum number of retries for attempts to publish data to the timeline service.Defaults to 30.

yarn.timeline-service.client.retry-interval-ms

The interval in milliseconds between retries for the timeline service client. Defaults to 1000.

yarn.timeline-service.generic-application-history.max-applications

The max number of applications could be fetched by using REST API or application history protocol and shown in timeline server web ui. Defaults to 10000.

UI Hosting Configuration

The timeline service can host multiple UIs if enabled. The service can support both static web sites hosted in a directory or war files bundled. The web UI is then hosted on the timeline service HTTP port under the path configured. | Configuration Property | Description | |:—- |:—- | | yarn.timeline-service.ui-names | Comma separated list of UIs that will be hosted. Defaults to none. | | yarn.timeline-service.ui-on-disk-path.$name | For each of the ui-names, an on disk path should be specified to the directory service static content or the location of a web archive (war file). | | yarn.timeline-service.ui-web-path.$name | For each of the ui-names, the web path should be specified relative to the Timeline server root. Paths should begin with a starting slash. |

Security Configuration

Security can be enabled by setting yarn.timeline-service.http-authentication.type to kerberos, after which the following configuration options are available:

Indicates if anonymous requests are allowed by the timeline server when using ‘simple’ authentication. Defaults to true.

yarn.timeline-service.principal

The Kerberos principal for the timeline server.

yarn.timeline-service.keytab

The Kerberos keytab for the timeline server. Defaults on Unix to to /etc/krb5.keytab.

yarn.timeline-service.delegation.key.update-interval

Defaults to 86400000 (1 day).

yarn.timeline-service.delegation.token.renew-interval

Defaults to 86400000 (1 day).

yarn.timeline-service.delegation.token.max-lifetime

Defaults to 604800000 (7 days).

yarn.timeline-service.best-effort

Should the failure to obtain a delegation token be considered an application failure (option = false), or should the client attempt to continue to publish information without it (option=true). Default: false

Enabling the timeline service and the generic history service

Following are the basic configuration to start Timeline server.

<property>
<description>Indicate to clients whether Timeline service is enabled or not.
If enabled, the TimelineClient library used by end-users will post entities
and events to the Timeline server.</description>
<name>yarn.timeline-service.enabled</name>
<value>true</value>
</property>
<property>
<description>The setting that controls whether yarn system metrics is
published on the timeline server or not by RM.</description>
<name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
<value>true</value>
</property>
<property>
<description>Indicate to clients whether to query generic application
data from timeline history-service or not. If not enabled then application
data is queried only from Resource Manager.</description>
<name>yarn.timeline-service.generic-application-history.enabled</name>
<value>true</value>
</property>

Running the Timeline Server

Assuming all the aforementioned configurations are set properly admins can start the Timeline server/history service with the following command:

yarn timelineserver

To start the Timeline server / history service as a daemon, the command is

Note that the same commands are usable to obtain the corresponding information about running applications.

Publishing application specific data

Developers can define what information they want to record for their applications by constructing TimelineEntity and TimelineEvent objects then publishing the entities and events to the Timeline Server via the TimelineClient API.

Publishing of data to Timeline Server is a synchronous operation; the call will not return until successful.

The TimelineClient implementation class is a subclass of the YARN Service API; it can be placed under a CompositeService to ease its lifecycle management.

The result of a putEntities() call is a TimelinePutResponse object. This contains a (hopefully empty) list of those timeline entities reject by the timeline server, along with an error code indicating the cause of each failure.

In Hadoop 2.6 and 2.7, the error codes are:

Error Code

Description

1

No start time

2

IOException

3

System Filter conflict (reserved filter key used)

4

Access Denied

5

No domain

6

Forbidden relation

Further error codes may be defined in future.

Note : Following are the points which need to be observed when updating a entity.

Domain ID should not be modified for already existing entity.

After a modification of a Primary filter value, the new value will be appended to the old value; the original value will not be replaced.

It’s advisable to have same primary filters for all updates on entity. Any on modification of a primary filter by in an update will result in queries with updated primary filter to not fetching the information before the update

Generic Data Web UI

Timeline Server REST API V1

Querying the timeline server is currently only supported via REST API calls; there is no API client implemented in the YARN libraries. In Java, the Jersey client is effective at querying the server, even in secure mode (provided the caller has the appropriate Kerberos tokens or keytab).

The v1 REST API is implemented at under the path, /ws/v1/timeline/ on the Timeline Server web service.

Here is a non-normative description of the API.

Root path

GET /ws/v1/timeline/

Returns a JSON object describing the server instance.

{"About":"Timeline API"}

Domains /ws/v1/timeline/domain

Domain summary information /ws/v1/timeline/domain

GET /ws/v1/timeline/domain?owner=$OWNER

Returns a list of domains belonging to a specific user, in the JSON-marshalled TimelineDomains data structure.

The owner MUST be set on a GET which is not authenticated.

On an authenticated request, the owner defaults to the caller.

PUT /ws/v1/timeline/domain

A PUT of a serialized TimelineDomain structure to this path will add the domain to the list of domains owned by the specified/current user. A successful operation returns status code of 200 and a TimelinePutResponse containing no errors.

Entity: type and id. starttime is required unless the entity contains one or more event). Event: type and timestamp.

Timeline Entity List

With the Timeline Entity List API, you can retrieve a list of entity object, sorted by the starting timestamp for the entity, descending. The starting timestamp of an entity can be a timestamp specified by the your application. If it is not explicitly specified, it will be chosen by the store to be the earliest timestamp of the events received in the first post for the entity.

URI:

Use the following URI to obtain all the entity objects of a given entityType.

HTTP Operations Supported:

Query Parameters Supported:

limit - A limit on the number of entities to return. If null, defaults to 100.

windowStart - The earliest start timestamp to retrieve (exclusive). If null, defaults to retrieving all entities until the limit is reached.

windowEnd - The latest start timestamp to retrieve (inclusive). If null, defaults to the max value of Long.

fromId - If fromId is not null, retrieve entities earlier than and including the specified ID. If no start time is found for the specified ID, an empty list of entities will be returned. The windowEnd parameter will take precedence if the start time of this entity falls later than windowEnd.

fromTs - If fromTs is not null, ignore entities that were inserted into the store after the given timestamp. The entity’s insert timestamp used for this comparison is the store’s system time when the first put for the entity was received (not the entity’s start time).

primaryFilter - Retrieves only entities that have the specified primary filter. If null, retrieves all entities. This is an indexed retrieval, and no entities that do not match the filter are scanned.

secondaryFilters - Retrieves only entities that have exact matches for all the specified filters in their primary filters or other info. This is not an indexed retrieval, so all entities are scanned but only those matching the filters are returned.

fields - Specifies which fields of the entity object to retrieve: EVENTS, RELATED_ENTITIES, PRIMARY_FILTERS, OTHER_INFO, LAST_EVENT_ONLY. If the set of fields contains LAST_EVENT_ONLY and not EVENTS, the most recent event for each entity is retrieved. If null, retrieves all fields.

Note that the value of the key/value pair for primaryFilter and secondaryFilters parameters can be of different data types, and matching is data type sensitive. Users need to format the value properly. For example, 123 and "123" means an integer and a string respectively. If the entity has a string "123" for primaryFilter, but the parameter is set to the integer 123, the entity will not be matched. Similarly, true means a boolean while "true" means a string. In general, the value will be casted as a certain Java type in consistent with jackson library parsing a JSON clip.

Elements of the entities (Timeline Entity List) Object

When you make a request for the list of timeline entities, the information will be returned as a collection of container objects. See also Timeline Entity for syntax of the timeline entity object.

URI:

HTTP Operations Supported:

GET

Query Parameters Supported:

fields - Specifies which fields of the entity object to retrieve: EVENTS, RELATED_ENTITIES, PRIMARY_FILTERS, OTHER_INFO, LAST_EVENT_ONLY. If the set of fields contains LAST_EVENT_ONLY and not EVENTS, the most recent event for each entity is retrieved. If null, retrieves all fields.

Elements of the entity (Timeline Entity) Object:

See also Timeline Event List for syntax of the timeline event object. Note that value of primaryfilters and otherinfo is an Object instead of a String.

Item

Data Type

Description

entity

string

The entity id

entitytype

string

The entity type

relatedentities

map

The related entities’ identifiers, which are organized in a map of entityType : [entity1, entity2, …]

events

list

The events of the entity

primaryfilters

map

The primary filters of the entity, which are orgainzied in a map of key : [value1, value2, …]

otherinfo

map

The other information of the entity, which is orgainzied in a map of key : value

Generic Data REST APIs

Application List

With the Application List API, you can obtain a collection of resources, each of which represents an application. When you run a GET operation on this resource, you obtain a collection of application objects.

Application Attempt List

With the Application Attempt List API, you can obtain a collection of resources, each of which represents an application attempt. When you run a GET operation on this resource, you obtain a collection of application attempt objects.

URI:

Use the following URI to obtain all the attempt objects of an application identified by the appid value.

HTTP Operations Supported:

GET

Query Parameters Supported:

None

Elements of the appattempts (Application Attempt List) Object

When you make a request for the list of application attempts, the information will be returned as a collection of application attempt objects. See Application Attempt for the syntax of the application attempt object.

Item

Data Type

Description

appattempt

array of appattempt objects(JSON)/zero or more application attempt objects(XML)

The collection of application attempt objects

Response Examples:

JSON response

HTTP Request:

GET http://localhost:8188/ws/v1/applicationhistory/apps/application_1430424020775_0001/appattempts

Container List

With the Container List API, you can obtain a collection of resources, each of which represents a container. When you run a GET operation on this resource, you obtain a collection of container objects.

URI:

Use the following URI to obtain all the container objects of an application attempt identified by the appid value and the appattemptid value.

Response Codes

Queries where a domain, entity type, entity ID or similar cannot be resolved result in HTTP 404, “Not Found” responses.

Requests in which the path, parameters or values are invalid result in Bad Request, 400, responses.

In a secure cluster, a 401, “Forbidden”, response is generated when attempting to perform operations to which the caller does not have the sufficient rights. There is an exception to this when querying some entities, such as Domains; here the API deliberately downgrades permission-denied outcomes as empty and not-founds responses. This hides details of other domains from an unauthorized caller.

If the content of timeline entity PUT operations is invalid, this failure will not result in an HTTP error code being retured. A status code of 200 will be returned —however, there will be an error code in the list of failed entities for each entity which could not be added.