Azure IoT Hub origin batch size

There doesn't seem to be any batch size setting for the Azure IoT Hub origin. It is always returning me just 10 records in a batch. How can I increase this?

If I try to increment the thread count I start getting errors
'New receiver with higher epoch of '37' is created hence current receiver with epoch '36' is getting disconnected.'
which implies that there can only be one thread at a time running. Although the Streamsets documentation says that the origin can use multithreaded processing

1 Answer

The pipeline preview batch size is capped at the Data Collector's preview batch size--which is set to 10 by default. So in order to increase pipeline preview batch size you'll first need to change Data Collector's preview batch size by editing preview.maxBatchSize property in sdc.properties file. (Note: for the change to take effect you'll need to restart Data Collector.)

I have already set both the production.maxbatchsize and the preview.maxbatchsize to 1000 and restarted, but the issue remains. I'm only having this issue with the azure iot/event hub origin, other origins are working fine with large batch sizes.