SMILA/Documentation/ConnectivityFramework

Overview

The Connectivity framework, as the name suggests, provides a framework to easily integrate data from external systems into SMILA. To access external data two kinds of components are supported: Agents and Crawlers. To integrate some new datasource type into SMILA just a new Agent or Crawler has to be implemented.

Architecture

Here is a short overview of the components of the ConnectivityFramework:

AgentController: The AgentController implements the general processing logic common for all types of Agents. It's service interface is used by Agents to trigger add/update/delete actions. This component is not yet implemented!

Agents: Agents monitor datasources for changes (add/update/delete) or are triggered by events (e.g. trigger in databases) and report those changes to the AgentController. Currently we do not provide any Agent implementation!

CrawlerController: The CrawlerController implements the general processing logic common for all types of Crawlers.It's service interface is used by clients (e.g. JMX console) to start/stop crawls. For more information see SMILA/Documentation/CrawlerController.

Crawlers: A Crawler crawls a data source (e.g. a filesystem or a website) and returns all found data objects. For more information see SMILA/Documentation/Crawler.

In addition there are three components that are not part of the ConnectivityFramework, but that interact with it:

ConnectivityManager: The ConnectivityManager is the single point of entry for data in the SMILA. The Agent- and CrawlerController push the data through this component into the Queue. For more information see SMILA/Documentation/ConnectivityManager.

DeltaIndexingManager: The DeltaIndexingManager provides functionailty to decide wheter a record needs to be updated and sent to the ConnectivityManager or not. For more information see SMILA/Documentation/DeltaIndexingManager.

Configuration Management: This component is not yet implemented. It is designed to manage configurations for all kinds of services, e.g. DataSources for crawlers. At the moment all configurations have to be provided locally in the SMILA configuration folder.

The following chart shows the architecture of the Connectivity Framework with it's plugable components (Agents/Crawlers) and relationship to the SMILA entry point Connectivity Module.

The red labeled components are not yet implemented.

Configuration

There is no overall configuration for the framework. Check out the documentation of each framework component for detailed infomation.