Twitter Stream Connector and Datasource Configuration

The Twitter Stream connector uses Twitter’s streaming API to continually index Twitter. The datasource can be configured to limit tweets or it can be run indefinitely, until Twitter cuts off your access or you stop the datasource. This connector only retrieves tweets created after the datasource has been started.

This connector type is "lucid.twitter.search" and the plugin type is "twitter_search".

Registering for Twitter Credentials

In order to successfully configure either Twitter connector, you must first register your application with Twitter and accept their terms of service. The registration process will provide you with the required OAuth tokens you need to access either API. To get the tokens, follow these steps:

After signing in, choose 'My Applications' from the pull down menu at the upper right that shows a thumbnail of your Twitter profile picture (if you have one). Then choose "Create New App" and fill out the required details. The callback field can be skipped, but you must accept the Terms of Service. To save your information, choose "Create Your Twitter Application" to register your application.

The next page will contain the Consumer Key and Consumer Secret, which you will need to configure the data source in Fusion.

At the bottom of the same page, choose "Create My Access Token".

The next page will contain the Access Token and Token Secret, which you will also need to configure the data source in Fusion.

While you need a Twitter account to register an application, you do not use your Twitter username and password to configure this data source. The APIs will only use the Consumer Key, Consumer Secret, Access Token, and Token Secret information as authentication, so store it where you can access it while configuring the data source.

Configuration

Tip

When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

Property

Description

access_token

Access Token

required

The OAuth Access Token is provided by Twitter when registering the application.

type: string

minLength: 1

collection

Collection

Collection documents will be indexed to.

type: string

pattern: ^[a-zA-Z0-9_-]+$

commit_on_finish

Solr commit on finish

Set to true for a request to be sent to Solr after the last batch has been fetched to commit the documents to the index.

type: boolean

default value: 'true'

consumer_key

Consumer Key

required

The OAuth Consumer Key is provided by Twitter when registering the application.

type: string

minLength: 1

consumer_secret

Consumer Secret

required

The OAuth Consumer Secret is provided by Twitter when registering the application.

}source (required)
: {
display name: Source Field
type: string
description : The name of the field to be mapped.
}target
: {
display name: Target Field
type: string
description : The name of the field to be mapped to.
}
}

reservedFieldsMappingAllowed

Allow System Fields Mapping?

type: boolean

default value: 'false'

unmapped

Unmapped Fields

If fields do not match any of the field mapping rules, these rules will apply.

}source
: {
display name: Source Field
type: string
description : The name of the field to be mapped.
}target
: {
display name: Target Field
type: string
description : The name of the field to be mapped to.
}
}

ConnectorDb Configuration

Property

Description

aliases

Process Aliases?

Keep track of original URI-s that resolved to the current URI. This negatively impacts performance and size of DB.

type: boolean

default value: 'false'

inlinks

Process Inlinks?

Keep track of incoming links. This negatively impacts performance and size of DB.

type: boolean

default value: 'false'

inv_aliases

Process Inverted Aliases?

Keep track of target URI-s that the current URI resolves to. This negatively impacts performance and size of DB.