Change Details

This will be our pilot deployment. Pick a harmless queue, start reading via Kafka, and let it burn in. Verify that we're actually reading from Kafka and not the old queue, etc.
Take all the sweet time doing this right. We need a new type of consumer loop, which is batch-capable but still compatible with single message backends.

This will be our pilot deployment. Pick a harmless queue, start reading via Kafka, and let it burn in. Verify that we're actually reading from Kafka and not the old queue, etc.
Take all the sweet time doing this right. We need a new type of consumer loop, which is batch-capable but still compatible with single message backends. Probably, batch processing should be decoupled from dequeueing: run_job = process_batch( get_batch ).
Should this consumer run multiple batches if there's time remaining in the cycle?
How do we decide when to commit the consumer offset?

This will be our pilot deployment. Pick a harmless queue, start reading via Kafka, and let it burn in. Verify that we're actually reading from Kafka and not the old queue, etc.
Take all the sweet time doing this right. We need a new type of consumer loop, which is batch-capable but still compatible with single message backends. Probably, batch processing should be decoupled from dequeueing: run_job = process_batch( get_batch ).
Should this consumer run multiple batches if there's time remaining in the cycle?
How do we decide when to commit the consumer offset?