5.4.2.1. Optional Configuration Parameters for Kafka

A number of optional, configurable, properties are available that
control how Tungsten Replicator applies and populates information when the
data is written into Kafka. The following properties can by set during
configuration using
--property=PROPERTYNAME=value:

Table 5.1. Optional Kafka Applier Properties

Option

Description

replicator.applier.dbms.embedCommitTime

Sets whether the commit time for the source row is embedded into the document

replicator.applier.dbms.embedSchemaTable

Embed the source schema name and table name in the stored document

replicator.applier.dbms.enabletxinfo.kafka

Embeds transaction information (generated by the rowaddtxninfo filter) into each Kafka message

replicator.applier.dbms.enabletxninfoTopic

Embeds transaction information into a separate Kafka message broadcast on an independent channel from the one used by the actual database data. One message is sent per transaction or THL event.

replicator.applier.dbms.keyFormat

Determines the format of the message ID

replicator.applier.dbms.requireacks

Defines whether when writing messages to the Kafka cluster, how many acknowledgements from Kafka nodes is required

Embed the source schema name and database name into the stored document

If enabled, the documented stored into Elasticsearch will
include the source schema and database name. This can be used to
identify the source of the information if the schema and table
name is not being used for the index and type names (see
replicator.applier.dbms.useSchemaAsIndex
and
replicator.applier.dbms.useTableAsType).

Embeds transaction information (generated by the rowaddtxninfo filter) into each Kafka message

Value Type

boolean

Default

false

Valid Values

false

Do not include transaction information in each

true

Embed transaction information into each Kafka message

Embeds information about the entire transaction information
using the data provided by the
rowaddtxninfo filter and
other information embedded in each THL event into each message
sent. The transaction information includes information about the
entire transaction (row counts, event ID and tables modified)
into each message. Since one message is normally sent for each
row of data, by adding the information about the full
transaction into the message it's possible to validate and
identify what other messages may be part of a single transaction
when the messages are being re-assembled by a Kafka client.

For example, when looking at a single message in Kafka, the
message includes a txninfo
section:

This block of the overall message includes the following objects
and information:

schema

An array of the row counts within this transaction, with a
row count included for each schema and table.

serviceName

The name of the Tungsten Replicator service that generated the
message.

totalCount

The total number of rows modified within the entire
transaction.

firstRecordInTransaction

If this field exists, it should always be set to true and
indicats that this message was generated by the first row
inserted, updated or deleted in the overall transaction.
This effectively indicates the start of the overall
transaction.

lastRecordInTransaction

If this field exists, it should always be set to true and
indicats that this message was generated by the last row
inserted, updated or deleted in the overall transaction.
This effectively indicates the end of the overall
transaction

Note that this information block is included in
every message for each row within an
overall transaction. The
firstRecordInTransaction
and
lastRecordInTransaction
can be used to identify the start and end of the transaction
overall.

Embeds transaction information into a separate Kafka message broadcast on an independent channel from the one used by the actual database data. One message is sent per transaction or THL event.

Value Type

boolean

Default

false

Valid Values

false

Do not generate transaction information

true

Send transaction information on a separate Kafka topic for each
transaction

If enabled, it sends a separate message on a Kafka topic
containing information about the entire tranaction. The topic
name can be configured by setting the
replicator.applier.dbms.txninfoTopic
property.

Combine the primary key column values into a single string joined by an
underscore character

tspkey

Combine the schema name, table name, and primary key column values into
a single string joined by an underscore character

tspkeyus

Combine the schema name, table name, and primary key column values into
a single string

Determines the format of the message ID used when sending the
message into Kafka. For example, when configured to use
tspkeyus, then the format
of the message ID will consist of the schemaname, table name and
primary key column information separated by underscores,
SCHEMANAME_TABLENAME_234.