Re: Custom processor in storm creates bottleneck

That pseudo-cluster itself is a scalability bottleneck. Storm likes to scale by having many Supervisor (worker) processes). As for the specific stats on your component, you could drill into the Storm UI and find your topology and then drill into it to find your bolt to see how it is doing and gauge for yourself it is working well enough. You'll get rolling statistics of how long it takes to process things such as shown below. You could also scale up the number of instances the bolt has, but again, the single-server pseudo-cluster is likely going to be your first bottleneck.

Re: Custom processor in storm creates bottleneck

The idea is to decode 1600 files which contain 4000 records each on average, that is 6.400.000 records in total, these must be processed and sent through nifi in less than 10 minutes to another server sftp.