IMS Caliper Analytics Best Practice Guide

IMS Caliper Analytics™ Best Practice Guide

Version 1.0

IPR and Distribution Notices
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

IMS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on IMS’s procedures with respect to rights in IMS specifications can be found at the IMS Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

The limited permissions granted above are perpetual and will not be revoked by IMS or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.

1 Introduction

As part of developing the Caliper Analytics™ standard and sensor reference architecture, the development team and implementers of the proof of concepts have come up with a best practice guide, including a quick start section.

The purpose of this document is to provide you with a detailed exploration of how Caliper works and the issues that arise so as to give you guidance on good practice, if not, best practice.

Please provide feedback based on your own experiences to help improve the quality of the recommendations included here.

1.1 Terminology

We will start with a bit of terminology to help make the descriptions that follow more precise. The Caliper specification uses the following terms:

Caliper: The name of the IMS Caliper Analytics Learning Measurement Framework which provides a standard for representing, capturing, and marshalling metrics generated by learning activity and targeted for consumption by any conforming analytics store or service.

Learning Activity: Any activity considered a component of a learning sequence presented in a digital learning environment (e.g. LTI® tool) typically denoted as a lesson. Activities can be assignable and/or gradable as required.

Caliper Sensor: Code/library that assists in the measurement of Caliper Events. Deployed into the Learning/Digital Application. Sensors are implemented in Java, Javascript, PHP, Python, Ruby and .NET..

Event Store: System or software service that stores event data emitted by the Caliper Sensors.

1.2 Metric Profiles

The Metric Profiles represent the Information Model and are organized by Learning Activity type. When you begin implementing Caliper you should compare your application’s features with the metric profiles and implement the ones necessary to capture the user’s activities based on your features. For example a Quizzing tool would want to implement base, session, assessment, assessment item and outcome metric profiles. An eReader would at a minimum implement base, session and reading metric profiles.

A complete list of the metric profiles and detailed information about each including properties and their data types, description, source and whether they are required are included in the Implementation Guide.

Profile

What can I use it for?

Base

Represents the base Caliper Event, Entity, and Actions used in other profiles

Session

Track users as they log in, log out, and also session time outs across multiple eduApps

Not a metric profile per se, but one common scenario / applied use case that applies a blended collection of metrics and context derived from other metric profile elements to explicitly collect engagement activity/usage associated data across a wide spectrum of learning activities

1.3 Engagement Scenario

This scenario contains a list of Events and corresponding actions from the current set of Caliper Metric Profiles that indicate minimum student engagement with Learning Activities.

In order to quantify engagement, three elements can be measured: (1) the overall duration of a “session” comprising multiple events;(2) the duration of engagement with each individual activity; and, (3) the total number of (each) type of event in a session.

Each event has an “opening” action that indicates engagement. E.g. “started”. Other actions in the sequence (e.g. paused, resumed, completed) can also be included, including a “closing” action to indicate the end of a sequence. However, to simply get an indicator of (one or many) students engagement with a specific Learning Activity, the opening action could be sufficient.

All attributes/objects (e.g. actor, action, object, startedAtTime, endedAtTime, duration, etc) of Events are implicitly part of the Engagement Profile.

Event

Action

Metric Profile

Comments

SessionEvent

loggedIn

Session

Navigation Event

navigatedTo

Mulitple

Common across all Learning Activity types

ViewedEvent

viewed

Reading

AnnotationEvent

All

Annotation

MediaEvent

started, [paused,resumed,etc}

Media

AssignableEvent

started [completed,submitted}

Assignable

AssessementEvent

started, [submitted]

Assessment

2 Quick Start Guide

In order to make it easier for programmers to get started implementing Caliper, we have created a getting started video.

There is also an exercise in Plunker written in Javascript that will walk a developer through implementing a caliper sensor. The application is modeled as a simple Course delivery app, that delivers a syllabus, reading and quiz. Here is the link to the Sample Application.

2.1 Download the reference Caliper Sensor implementation.

The first step in implementing the Caliper sensor is to download the reference Caliper Sensor implementation that is applicable to your technology stack. The links below point to the Github repository for each sensor.

2.2 Build sensor project, run unit tests and deploy

Build the project and run the test suites using the directions in the project’s README. Based on the type of sensor, you can use it on the server (all sensor types) or the client (e.g. Javascript sensor, note also for client-side sensors please note the authentication-related suggestions to maximize security.) The mechanism for using the sensor within your code will also vary depending upon the sensor. For example, you would include the Java sensor as a Maven dependency, the Ruby sensor as a Ruby Gem, the Python sensor as a Python Module, etc. Each individual README provides information on this.

● Java – Web App (Server)

● Javascript – Node.js (Server), Mobile (Browser),Web (Browser)

● PHP – PHP Web App (Server)

● .NET – Windows (Server)

● Python – Django, Bottle (Server)

● Ruby –RoR, Sinatra (Server)

2.3 Install the sensor in your application

Install the sensor into your application following the instructions in the README.

E.g., for Javascript:

Install the caliper sensor into your application by placing the URL of the Caliper sensor Javascript library.

2.4 View your events using an HTTP Request Inspector

You will need a place to capture events emitted by the Caliper Sensor. The sensor sends events as a HTTP POST to a RESTful service.

For a very quick start, you can use a tool that allows you to inspect HTTP requests such as http://requestb.in/. This will allow you access to the events being sent so you can inspect them and decouple the Event Store implementation from testing the sensor and event generation.

The Caliper team has provided a reference EventStore implementation. The EventStore is intended as a development/test/demo environment and is not intended for use as part of a production implementation of Caliper. You can find it at: Event Store.

Follow the directions in the readme, to get the reference implementation of the Event Store up and running.

Each sensor needs to be updated to include the correct endpoint for the datastore that you have up and running.

2.6 Process for Implementing a Caliper Event Store

Additional best practices, implementation guidelines and possible alternative supporting event stores are currently underway and forthcoming as both Caliper, event store technologies and related support matures. For now, utilizing the current Caliper Reference Event Store as a guide and understanding the JSON-LD based event stream so as to adapt any potential target persistence or event consuming service is a good starting point. Further information and assistance will be provided by the IMS Caliper work group upon request.

2.7 Implement and send an event in your application

Select sensor code bindings may include a sample application that provides implementation details for your chosen technology. Currently, sample applications have been written in Java, Javascript and .NET. Sample applications for the other sensors will be developed as time and interest permits.

The size of the event "data" list is up to the Tool Consumer/Provider to negotiate.

3.1.1 Use the Sensor API DESCRIBE method with an object to qualify events with more educational context

The sensor includes a describe method that can take either a Caliper Entity or array of Caliper Entities. So you can set elements of the learning graph that are consistent throughout a learner session and stream of events without having to specify such contextual elements each time an event is sent (i.e., the actor, agent, course section, etc.). This provides a mechanism for reducing the size of serialized event payloads .

3.1.2 Use the JSON-LD @id, @type and @context to request the data.

Since all Caliper Entities include a JSON-LD @id, @type and @context, you can specify larger and possibly more secondary event data entities independent from the event itself and therefore only the @id transmitted which can optionally be dereferenced as required to retrieve all of the additional information provided by a given event related service.

3.1.3 Send all the events complete with detailed information.

Another option is to send detailed information for all the event-related entities. That said, be sure to specify null values for anything unspecified that is to sent across the wire. There is optimizing compression etc during the serialization process to mitigate somewhat the event payload size. In future revisions of Caliper there are plans to continue to streamline the event payloads via optimizations in the event model itself.

3.2 LTI Best Practices

In order to preserve the confidentiality of the shared secret, this data item is not passed in an LTI launch request. Instead a Caliper Profile service should be declared in the tool consumer profile which may then be used by the tool provider to obtain the secret via a server-to-server web service request.

3.3 Authentication suggestions

For Caliper launches from an LTI Consumer/Provider the LTI security context is passed on.

Without LTI integration, the Caliper sensor API endpoint is secured via similar LTI-OAuth signing and key/secret exchange . As a best practice we recommend the use of OAuth.

The calls to an Event Store should be secured by an API Key. For the Javascript sensors the call to the Consumer Event Store should be performed server-to-server to ensure the integrity of the key. The API key is stored in the authorization header.

3.4 SSL/HTTPS

Caliper strongly recommends SSL/HTTPS for everything going over the wire. Caliper Event Store endpoints should be secured by SSL to ensure secure communications. The Sensors should accept and transmit through SSL/HTTPS.

3.5 How do I transmit additional information that is not included in the Metric Profiles?

You have two options if you require additional information to be sent over the wire that are not included by default in the metric profile.

3.5.1 You can add additional key/value pairs to an Entity’s extensions property.

The extensions property is a dictionary that can contain a list of properties. So for example you can add:

"extensions": {
"com.vendor.example.myNeededProperty": "its value"
}

Extensions should use a technique to ensure uniqueness across organizations (for example, a reverse-domain formatted string). Extend the Caliper JSON-LD Event schema

Another option would be to extend the appropriate Caliper Metric Profile JSON-LD Event schema with additional properties.

3.6 Envelope for the Metric Profile

The method envelope standard for all Caliper Event sequences includes: