This forum is now a read-only archive. All commenting, posting, registration services have been turned off. Those needing community support and/or wanting to ask questions should refer to the Tag/Forum map, and to http://spring.io/questions for a curated list of stackoverflow tags that Pivotal engineers, and the community, monitor.

I have implemented Remote chunk processing mechanism using multiple threads processing per each chunk to achieve better performance. For faster processing batch data is storing in memory between steps.

Problem: The batch job is getting launched by job launcher and the job is running loop. After executing couple of cycles in loop the memory utilisation by batch job is slowly creeping up and the memory is not getting released. After around 2 hours of batch running, the memory allocated to the process getting reached maximum limit and batch application is exiting with Out of Memory.

I have used profiler tools to analyse the problem and identified that some of context objects are sitting memory and the context is not getting cleared. I have used ConcurrentLinkedQueue for storing the items to be processed.

After loading data from data base, I am storing data in Step Execution Context to pass between steps. The step context data is getting promoted data between steps by using StepExecutionListenerSupport. Subsequently the data being used in next step by reading data from step execution context.

Comment

I am keeping data in StepContext and promoting data to next step using promotion listeners. And my job is running in loop and each loop will process a chunk.

I believe some of the memory still not getting released.

I have changed my job configuration, by keeping data in Job Context ( instead of step context), which does not required any linstener promotion as data is available for all steps in batch. This worked fine. The problem solved.

But Can you explain why step context promotion is creeping memory?

Thanks

Comment

Because you are storing all your intermediate state in the JobRepository, which itself is in memory. You need to use an external repository, or store the state somewhere else (like your own volatile storage). Or, I guess, purge the in-memory repository manually.