Documentation

Restream Queue Overview

Abstract

SummaryAn overview of the Restream Queue, including the ways events typically end up in the queue, and the exit criteria for events remaining in the queue.

The Restream Queue is your safety net for ensuring zero data loss. The Restream Queue collects all the events that were not loaded to your data destination, for whichever reason, making them easy to fix and enabling you to “restream” them into your data destination.

Events typically end up in the Restream Queue due to a:

Mapping error. Either data does not look like what the Mapper expects, or the Mapper literally cannot convert a value to the target data type.

Data Warehouse COPY error. If the COPY command to the data warehouse fails (likely because the Output is down or not reachable) data will go to the Restream Queue. This situation usually resolves itself once connectivity is restored.

Code Engine Error. If an event cannot be processed by the current code path, it will go to the Restream Queue. For example, this could occur because Python was forced to perform an illegal action, or because an Exception is being explicitly raised.

Each event remains in the Restream Queue until:

You address the error which resulted in it being there.

You restream them to your data destination.

The event is successfully loaded to your data destination.

Note

When auto-mapped events with new fields are processed, the events actually go to the Restream Queue temporarily while Alooma adjusts. These events will be restreamed automatically but sometimes lead to temporary spikes in Restream Queue volume.