There is one queue repository item for each queue type tracked in the repository. For example, there is one queue item for the order submission queue, and one for the site visit queue. The queue item for a specific queue is created the first time a log file for the queue is rotated.

The loading process for a specific queue works like this:

The data loader determines that there is an entry for a log file in the queue.

The loader creates a new loadProgress repository item for the file, setting the item’s status property to “in progress,” and setting its recordsProcessed property to 0 (zero).

The loader deletes the queueEntry repository item for the file.

The loader uses the ClientLockManager to acquire a lock on the loadProgress item.

The loader creates a new transaction.

The loader reads a line from the log file and passes it to a pipeline driver for processing (see below). The loader repeats this step until it has processed the number of lines specified by the value of its transactionBatchSize property.

If no errors have occurred, the loader commits the transaction and updates the recordsProcessed property of the loaderProgress item to indicate the number of records processed.

The cycle of creating a transaction, processing a batch, and committing the transaction is repeated until the entire log file has been processed.

The loader sets the status property of the loaderProgress item to “done,” and releases the lock.

To obtain information about loads performed by a specific loader component, view the component in the Component Browser in the Dynamo Admin UI.