Re: Getting error while running flume agent

This is the only error that I am getting. Flume agent runs for about 15-20 minutes and pulls the log from rabbitmq into the hdfs path. Once it throws down the error, no file is being created. So every time we stop manually and the agent starts to get the log.

Re: Getting error while running flume agent

Its still possible that your hdfs sinks are just not able to keep up with the rabbitmq source. Did you review the graphs to see the rate the sinks are draining vs the rate the source is adding?

Also, you are using sinkgroups, which makes deliver single threaded (e.g. one sink at a time). There is really no reason to use sinkgroups, you can remove that and will parallelize the delivery of events through sinks (2x sinks = 2x delivery rate, 4x sinks = 4x delivery rate).

Re: Getting error while running flume agent

It does pull the file in the first 1-2 hours, and later prevents it from dragging logs from rabbitmq into the hdfs path. After running for 1-2 hours, it throws an error in which no file is being created. So every time, we need to stop the agent and start it again.