Batch Processing Reference

Terminology

A scope which accumulates records into chunks to prepare bulk upserts to external source or service.

batch:commit

Batch Job

The top-level element in an application in which Mule processes a message payload as a batch of records. The term batch job is inclusive of all four phases of processing: Input, Load and Dispatch, Process, and On Complete.

batch:job

Batch Job Instance

An occurrence in a Mule application resulting from the execution of a batch job in a Mule flow. you can let Mule create the batch job instance in the Load and Dispatch phase for you, or you can specify a distinct job instance Id through a Mule Expression. This instance exists eternally.

job-instance-id

Batch Job Result

POJO which contains information about the processing results of a batch job instance. You can use it to get information about execution progress.

ALL = step processes all records, failed and successful. FAILURES_ONLY = step processes only records which failed in a preceding step. NO_FAILURES = step processes only records which succeeded in all preceding steps.

accept-expression

MEL expression

Step processes only those records which, relative to the expression, evaluate to true (evaluate to false = skip record).

Element

Attribute

Value

Attribute Req’d

Attribute Description

batch:threading-profile

poolExhaustedAction

WAIT WAIT DISCARD DISCARD_OLDEST ABORT RUN

Defines what a batch job should do if all threads are active. WAIT = (Default) wait until next thread is available DISCARD = discard waiting batch job DISCARD_OLDEST = discard the oldest waiting batch job ABORT = abort processing the batch job RUN = don’t wait for a thread to become available, run the batch job synchronously

maxThreadsActive

integer

Defines the maximum number of active threads upon which Mule processes batch jobs.

maxThreadsIdle

integer

Defines the minimum number of active threads upon which Mule processes batch jobs.

threadTTL

integer

Defines, in milliseconds, the time a thread should live and remain idle before becoming inactive.

threadWaitTimeout

integer

Defines how long a batch job should wait for a thread to become available before timing out.

maxBufferSize

integer

Defines the size of the "overflow" memory which holds batch jobs while waiting for a thread to become available.

Batch Commit Connectors

Several Anypoint Connectors have the ability to handle record-level errors without failing a whole batch commit (i.e. upsert). At runtime, these connectors keep track of which records were successfully accepted by the target resource, and which failed to upsert. Thus, rather than failing a complete group of records during a commit activity, the connector simply upserts as many records as it can, and tracks any failures for notification. The short – but soon to grow – list of such connectors follows:

Salesforce

NetSuite

BatchJobResult Processing Statistics

Statistic

Description

batchJobInstanceId

A String indicating the id of the executed job instance.

elapsedTimeInMillis

A long indicating the number of milliseconds the batch job spent in executing state.

failedOnCompletePhase

A boolean indicating whether an exception was found on the on the complete phase.

failedOnInputPhase

A boolean indicating whether an exception was found on the on the input phase.

failedOnLoadingPhase

A boolean indicating whether an exception was found on the on the input phase.

failedRecords

A long indicating the number of records that failed processing.

inputPhaseException

If an exception was found in the input phase, then that Exception is returned; otherwise null is returned. Notice that there’s a correlation between this statistic and failedOnInputPhase.

loadedRecords

A long indicating the number of records loaded so far. Once the loading phase is completed, it should be equal to totalRecords.

loadingPhaseException

If an exception was found in the loading phase, then that Exception is returned; otherwise null is returned. Notice that there’s a correlation between this statistic and failedOnLoadingPhase.

onCompletePhaseException

If an exception was found in the on complete phase, then that Exception is returned; otherwise null is returned. Notice that there’s a correlation between this statistic and failedOnCompletePhase.

processedRecords

A long indicating the number of records processed so far. It equals successfulRecords failedRecords, but it could be lower than totalRecords if the job is not finished.

successfulRecords

A long indicating the number of records processed so far.

totalRecords

Total number of records in the batch.

Example

For a full description of the example and steps the batch job takes in each phase of processing, see Batch Processing.

In the left navigation bar, under the My Settings heading, click to expand the Personal folder.

Click Reset My Security Token. Salesforce resets the token and emails you the new one.

Access the email that Salesforce sent and copy the new token onto your local clipboard.

In the application in your instance of Anypoint Studio, click the Global Elements tab.

Double-click the Salesforce global element to open its Global Element Properties panel. In the Security Token field, paste the new Salesforce token you copied from the email. Alternatively, configure the global element in the XML Editor.