Automated Feed Processing

Syllabus Geek

August 17, 2018 08:37

Updated

Concourse uses POST over HTTPS to allow for automated processing of feed files. This functionality allows you to control how frequently you run feeds and immediately receive information about what was successfully processed or had errors.

To begin, you should be familiar with how to create and process feeds manually, as the same file standard is used. From there, you'll need to create a request that periodically passes this data to Concourse.

Certificates

The most common issue setting up automation is the inability to verify the SSL certificate. This page describes the cause and various solutions. In most cases adding the --cacert option will be the optimal resolution.

Time-outs

Often feed processing will be completed within seconds. However if you are processing feeds with thousands of rows, and many of the rows are being used to create new users, courses, or content, they may take minutes and will likely time out using the default automated feed processing URL.

In order to reliably process feeds of this scale you need to change the processing URL to include -notimeout after your organization's sub-domain (e.g. https://school-notimeout.campusconcourse.com/process_feed_file).

Further, even if only one feed is expected to take a long time, and only at certain times during the year, you should make sure all of your scripts utilize the notimeout URL to avoid a scenario in which a preceding feed is inadvertently not processed but the subsequent one is.