TiCDC does not report errors but does not synchronize data

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiCDC 不报错,不同步数据

| username: Hacker_ojLJ8Ndr

[TiDB Version]
v7.1.1

Configuration as follows:

# Specify whether the database and table names involved in the configuration file are case-sensitive
# This configuration will affect both filter and sink related configurations, default is true
case-sensitive = true

# Whether to output old value, supported since v4.0.5, default is true since v5.0
enable-old-value = true

[filter]
# Ignore transactions with specified start_ts
#ignore-txn-start-ts = [1, 2]

# Filter rules
# Filter rule syntax: https://docs.pingcap.com/zh/tidb/stable/table-filter#table-database-filter-syntax
rules = ['xx.*','!xx.xx']

[mounter]
# Number of mounter threads, used for decoding data output from TiKV
worker-num = 4

#[sink]
# For MQ type Sink, event dispatchers can be configured through dispatchers
# Supports two types of event dispatchers: partition and topic (supported since v6.1). Detailed descriptions of both can be found in the next section.
# The matching syntax of matcher is the same as the filter rule syntax, and detailed descriptions of matcher matching rules can be found in the next section.
#dispatchers = [
#    {matcher = ['test1.*', 'test2.*'], topic = "Topic Expression 1", partition = "ts" },
#    {matcher = ['test3.*', 'test4.*'], topic = "Topic Expression 2", partition = "index-value" },
#    {matcher = ['test1.*', 'test5.*'], topic = "Topic Expression 3", partition = "table"},
#    {matcher = ['test6.*'], partition = "ts"}
#]

# For MQ type Sink, the protocol format of the message can be specified
# Currently supports five protocols: canal-json, open-protocol, canal, avro, and maxwell.
# protocol = "canal-json"

tiup cdc:v7.1.1 cli changefeed list --pd=http://192.168.x.x:2379

  {
    "id": "sync-tidb-test",
    "namespace": "default",
    "summary": {
      "state": "normal",
      "tso": 444228768228704256,
      "checkpoint": "2023-09-13 17:44:27.474",
      "error": null
    }
  }

The downstream is a TiDB cluster, the checkpoint is stuck at this position, there are no error messages in the logs, and it will start syncing again after pausing and resuming.

| username: TiDBer_oHSwKxOH | Original post link

You wrote the format incorrectly.

| username: Hacker_ojLJ8Ndr | Original post link

Configuration is working normally.

| username: Hacker_ojLJ8Ndr | Original post link

The cause of the problem has been identified: a downstream primary key conflict caused the sink to disconnect. Using tiup cdc:v7.1.1 cli changefeed list / query showed no error messages. Setting safe-mode to true was used to avoid desynchronization after the primary key conflict.

| username: Fly-bird | Original post link

Learned.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.