This forum is now a read-only archive. All commenting, posting, registration services have been turned off. Those needing community support and/or wanting to ask questions should refer to the Tag/Forum map, and to http://spring.io/questions for a curated list of stackoverflow tags that Pivotal engineers, and the community, monitor.

Rollback listeners

Jan 27th, 2010, 07:16 AM

Hello,

Is there a simple mechanism allowing to know which items have not been processed & written ?

There is org.springframework.batch.core.listener.ItemListen erSupport which implements three listeners (for reading, processing & writing listening). These listeners are called once an exception is thrown outside them, but an item can be retried, or skipped (according to the context of the batch process). So, how can I know exactly which items are finally not processed & not written ?

Comment

We use the progress indicator pattern with a staging table. We want to add the following statuses:

* ERROR: the item could not be processed
* ROLLEDBACK: item in the chunk that were processed successfully but who were rollbacked because of an item in error state

We could listen to the errors and use a requires new transaction but since we can't figure out if the item will be reprocessed anyway, it feels weird to mark it as error while it's being actually reprocessed afterward.

It feels like that the second is useless since Spring Batch will process the items one by one in case of error. Is it something we can configure? What if the chunk must be processed entirely or not at all for instance (and still use the retry feature)?

Comment

it feels weird to mark it as error while it's being actually reprocessed afterward.

That's sort of what I was trying to say. I think.

What if the chunk must be processed entirely or not at all for instance (and still use the retry feature)?

That behaviour is not supported out of the box with retry (or skip) because it is not compatible with the contract of the skip listeners (which are called through a retry internally) . Is it important? I guess it would be possible to implement a ChunkProcessor that can retry the whole chunk and not try to identify the failed items for the skip listeners.

Comment

Yeah ok but you're not taking the whole problem into account. Of course if the transaction will rollback and the processing will be retried it does not make any sense. However, if the number of retries have been reached (and the batch will fail anyway) we want to be able to know which item causes the issue (preferably with some hint like an error message or something).

Same thing regarding skip. We can skip an item at some point but how do we mark it as skipped?

To me, the progression indicator pattern is kinda useless if the only thing you can say is "done" or "not done" without being able to say why it's not done.

Comment

However, if the number of retries have been reached (and the batch will fail anyway) we want to be able to know which item causes the issue (preferably with some hint like an error message or something).

That makes sense in a way, but in the case of a write failure, in general you really can not find the item that caused the issue without scanning for it (and committing partial chunks in the process). You will definitely get more than a hint and an error message in the StepExecution (and in logs) about what went wrong. The item just may not be identifiable in principle.

Same thing regarding skip. We can skip an item at some point but how do we mark it as skipped?

Isn't that what skip listeners do?

I think this discussion just shows why it is often infinitely preferable to have only unexpected, unrecoverable, fatal errors in a writer, and make sure anything that can be skipped or retried happens earlier in the process.

Comment

When all retries allowed by the RetryPolicy have been made the exception will be offered to the SkipPolicy. So if the exception is skippable you can get the item there provided the SkipPolicy allows it. Or in the ItemProcessorListener (then it's not known whether or not the item eventually failed after retry or not, as you already pointed out).

I can see there is a corner case here: you want to get a reference to the *last* item that failed before the skip policy barfed? Is that all it is, or am I still missing something about your use case?

Comment

* I want to flag the item that was skipped because the policy allowed it
* I want to flag the item that causes the processing to fail (ItemProcessor). I mean here, the actual case where the job will stop because of a fatal error

Comment

to know that an exception thrown outside a processor will cause the batch failure (cannot retried, or cannot skipped). It's a tricky solution to call canRetry in ItemProcessListener#onProcessError(T item, Exception e) because canRetry can return true in the listener and later in RetryTemplate return false (for example the timeout is reached in RetryTemplate and not in the listener) . Moreover, how can I flag other elements of the chunk as "ROLLEDBACK" ?

to flag an exception thrown outside a processor which is SKIPPED. We can use SkipListener#onSkipInProcess(T item, Throwable t). So it's OK for that point

Writing

to flag items as COMPLETED, we can use ItemWriteListener#afterWrite(List<? extends S> items), so it's ok.

to know that an exception thrown outside a writer will cause the batch failure (cannot retried, or cannot skipped). This is the same as previously, it's a tricky solution to call canRetry in ItemWriteListener#onWriteError(Exception exception, List<? extends S> items) because canRetry can retry true in the listener and later in RetryTemplate return false (the timeout is reached). Elements provided to the writer are all in ERROR.

to flag an exception thrown outside writer which is SKIPPED, we can use SkipListener#onSkipInWrite(T item, Throwable t), so it's OK for that point.

Comment

The skip policy is called but doesn't get a reference to the item. Your best workaround for this is to put the item in some shared state in a Item*Listener (e.g. ThreadLocal if your step is multi-threaded, or just an dinstance variable in step scope otherwise) and then pick it up in the skip policy.

Comment

/**
* {@link StagingItemsTracking} keeps track of items read and written by every batch process. Each
* of them are executed in one thread. Each registration erases previous registrations made by the current thread.
* <p />
* This tracker is used by the staging item reader and writer in order to know which items are currently
* processed and written. If a crash occurs, the skip policy (executed in the same thread than the batch process)
* is able to know which items have failed.
*
* @author Sebastien GERARD
* @see com.bsb.sf.batch.service.staging.AbstractStagingItemReader
* @see com.bsb.sf.batch.service.staging.support.StagingItemCompositeProcessor
* @see com.bsb.sf.batch.service.staging.support.StagingItemCompositeWriter
*/
public class StagingItemsTracking {
private static final ThreadLocal<ItemsDescriptors> threadLocal =
new ThreadLocal<ItemsDescriptors>(){
@Override
protected ItemsDescriptors initialValue() {
return new ItemsDescriptors();
}
};
/**
* Register the given items and un-register previous items that have been registered by the current thread.
* @param items Items to register.
*/
public void registerItems(final List<? extends StagingItemWrapper> items)
{
threadLocal.set(new ItemsDescriptors(items) );
}
/**
* Get items that have been registered by the current thread.
* @return Items registered by the current thread.
*/
public List<StagingItemWrapper> getRegisteredItems() {
return threadLocal.get().getItems();
}
/**
* Utility class for keeping items tracking.
*/
private static class ItemsDescriptors {
private final List<StagingItemWrapper> items;
private ItemsDescriptors() {
this.items = new ArrayList<StagingItemWrapper>();
}
private ItemsDescriptors(List<? extends StagingItemWrapper> items) {
this.items = new ArrayList<StagingItemWrapper>(items);
}
public List<StagingItemWrapper> getItems() {
return items;
}
}
}

reader and writer keeps track of items. If an exception cannot be skipped, the skip policy (running under the same thread as the reader and writer) is able to know which items are currently processing and writing. These items must be set as "ERROR"