Delimited Data Format

Data Collector can
read and write delimited data.

Reading Delimited Data

Origins that read
delimited data generate a record for each delimited line in a file, object, or message.
Processors that process delimited data generate records as described in the processor
overview.

Multi Character Delimited - File that uses
multiple user-defined characters to delimit fields and lines, and
single user-defined escape and quote characters.

You can use a list or list-map
root field type for delimited data, and optionally include
field names from a header line, when available.

When using a header line, you
can enable handling records with additional columns. The
additional columns are named using a custom prefix and
integers in sequential increasing order, such as
_extra_1,
_extra_2. When you disallow additional
columns, records that include additional columns are sent
to error.

You can also replace a string
constant with null values.

When a record exceeds the maximum
record length defined for the stage, message-based origins and processors process the object based on
the error handling configured for the stage.

When a record exceeds the maximum length, file-based origins cannot continue reading
the file. Records already read from the file are passed to
the pipeline. The behavior of the origin is then based on
the error handling configured for the stage.

If data does not include a header or if you choose to ignore a header, list-map
records use the column position as a header as follows:

0: <value>
1: <value>
2: <value>

For example, when you ignore the header for the same data, you get the following
records:

0: 0003420303
1: 04
2: 362
0: 0003420304
1: 08
2: 1008

In an expression, you can use the field name or the column position with a standard
record function to call a field. For example, you can use either of the following
record:value() expressions to return data in the TransactionID
field:

${record:value('/TransactionID')}
${record:value('[0]'}

Note: When writing scripts for scripting processors, such as the Jython Evaluator or
JavaScript Evaluator, you should treat list-map records as maps.

For data in the list records, you should either use the delimited data functions or
include the full field path in standard record functions. For example, you can use the
record:dValue() delimited data function to return the value associated with the
specified header.

Tip: You can use the record:dToMap() function to convert a list record to
a map, and then use standard functions for record processing.

For a full list of origins that support this data format,
see Origins in the "Data Formats by Stage" appendix.

Writing Delimited Data

When processing
delimited data, file- or object-based destinations write each record as a delimited row
in a file or object. Message-based destinations write each record as a message.
Processors write delimited data as specified in the processor overview.

The destination writes records as
delimited data. When you use this data format, the root
field must be list or list-map.