Component Overview

tracker

This component consumes the Twitter Stream API, looking for tweets containing the strings http or https to fetch all tweets with links. The tweets are then parsed for contained URLs.

The URLs found are stored in the inbox redis database.

inbox-redis

This component is a simple Redis database that receives all found URLs from the tracker component. It makes use of the official Redis Docker image.

This component consciously does not provide a volume, which means that whenever this component is restarted, the database content is lost.

resolver

The script inside this component reads URLs from the inbox Redis database and creates requests to those URLs in order to resolve redirects, to reveal the actual target URL. The resulting URL is stored in the hotlist Redis database.

To prevent accessing the same URL several times, a cache is maintained in the hotlist Redis.

The resolver component can be thought of as a worker, processing jobs from a queue. Since resolving URLs is in many cases a time-consuming job, there can be multiple instances of this component working in parallel.

hotlist-redis

This second Redis database component stores all resolved URLs together with scoring information. It also contains the cache for the resolver. Just like the inbox component, we use the official Redis Docker image here.

In contrast to the inbox component, the hotlist provides a volume to persist the database throughout restarts.

cleaner

This component contains a little helper that periodically removes outdated information from the hotlist Redis database.

frontend

This is a Python/Flask web application that offers a JSON API to fetch the resulting URL hotlist.

rebrow

The rebrow component offers a web-based user interface (“rebrow” stands for “redis browser”) to debug the content of both Redis databases. It makes use of a third party Docker image.

Giant Swarm uses cookies to give you the best online experience. If you continue to use this site, you agree to our use of cookies. To disable all but strictly necessary cookies, you may disagree by clicking the button to the right. Please see our privacy policy for details.