Change tracking (new)

Last Updated: Jun 01, 2020

You can use DTS to retrieve incremental data from user-created MySQL databases in
real time. This feature applies to the following scenarios: cache updates, business
decoupling, asynchronous data processing, real-time synchronization of heterogeneous
data, and real-time data synchronization with complex ETL operations.

Note You need to use your change tracking client to modify consumption checkpoints.

You can monitor the status of the change tracking channel and receive an alert if
the threshold for downstream consumption is reached. You can set the alert threshold
based on the sensitivity of your businesses to consumption latency.

Consumer groups

The previous change tracking feature requires that one downstream SDK client consumes
data and other SDK clients act as backup. If you need to repeatedly consume the incremental
data in the same source database, you must purchase multiple change tracking instances.
This incurs high costs.

Consumer groups provided by DTS help you reduce costs. You can create consumer groups
for downstream consumers who need to subscribe to the same database instance. Consumers
in these consumer groups can subscribe to the same database instance and consume data
at the same time. This reduces user costs.

Consumer group features

You can create multiple consumer groups in a change tracking instance to repeatedly
consume data.

A consumer group consumes each message only once, and consumers in the consumer group
serve as a backup for each other.

In a consumer group, only one consumer can consume data at a time, while other consumers
act as disaster recovery nodes.

Supported database types and versions

A source MySQL database can be one of the following types: User-Created Database with Public IP Address, User-Created Database in ECS Instance, and User-Created Database Connected Over Express Connect, VPN Gateway, or Smart Access
Gateway.

Source database

Data type

User-created MySQL database

5.1, 5.5, 5.6, 5.7, or 8.0

Data update

Schema update

ApsaraDB RDS for MySQL

All versions

Supported languages

DTS allows clients to consume data by using the following programming languages:

Supported data types and objects

You can subscribe to the creation, deletion, and modification of all object schemas
in an instance. You need to use the change tracking client to filter the required
data.

Data update

You can subscribe to data updates of a selected object, including INSERT, DELETE,
and UPDATE operations.

Objects for change tracking

The objects for change tracking include databases and tables.

Change tracking channel

A change tracking channel pulls incremental data from the source database in real
time and stores the incremental data that is generated in the last 24 hours. You can
use the Kafka client to subscribe to and consume incremental data from the change
tracking channel. You can create, manage, and delete change tracking channels in the
DTS console.

The following table describes the statuses of a change tracking channel during creation
and running.

Channel status

Description

Available operations

Prechecking

The change tracking channel have been configured and a precheck is being performed.

Delete the change tracking channel.

Not Started

The change tracking channel has passed the precheck, but is not started.

Start the change tracking channel.

Delete the change tracking channel.

Performing Initial Change Tracking

The initial change tracking is in progress. This process requires about one minute
to complete.

Delete the change tracking channel.

Normal

Incremental data is being pulled from the data source.

View sample code.

Configure monitoring and alerts.

Delete the change tracking channel.

Reconfigure the change tracking channel.

Error

An error occurs when the change tracking channel is pulling incremental data from
the data source.

View sample code.

Configure monitoring and alerts.

Delete the change tracking channel.

Reconfigure the change tracking channel.

Note

A change tracking channel in the Error state will be resumed within seven days. To stop the change tracking, you must release
the change tracking channel.

After a change tracking channel is released, the data in the channel will be released
and cannot be recovered. We recommend that you use caution when performing this operation.