TiCDC Does Not Collect Data

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ticdc 不采集数据

| username: 消息终结者

[TiDB Usage Environment] Poc
[TiDB Version] v6.5.2

[Encountered Problem: Phenomenon and Impact]
The task was created successfully, but when inserting or updating data in TiDB, there are no changes in the downstream MySQL database; there is also no data in the monitoring.

| username: TiDBer_CQ | Original post link

Does the table have valid indexes?

| username: 小龙虾爱大龙虾 | Original post link

Execute cdc cli changefeed query -c c-test-task to check the status.

| username: 路在何chu | Original post link

Let’s see what the CDC logs record.

| username: 消息终结者 | Original post link

Is there an issue with task_status? I see that the table_ids in the monitoring are empty.

| username: 消息终结者 | Original post link

Yes, the table has a primary key.

| username: 消息终结者 | Original post link

No logs :rofl:

| username: 消息终结者 | Original post link

{
“upstream_id”: 7205161203036295249,
“namespace”: “default”,
“id”: “c-test-task”,
“sink_uri”: “mysql://bigdata_dev:xxxxx@rm-bp1b0x3173wu7w.mysql.rds.aliyuncs.com:3306?time-zone=Asia/Shanghai”,
“config”: {
“case_sensitive”: true,
“enable_old_value”: true,
“force_replicate”: false,
“ignore_ineligible_table”: false,
“check_gc_safe_point”: true,
“enable_sync_point”: false,
“bdr_mode”: false,
“sync_point_interval”: 600000000000,
“sync_point_retention”: 86400000000000,
“filter”: {
“rules”: [
“db_canvas.e_participation_record”,
“db_canvas.canvas_nc”
],
“event_filters”: null
},
“mounter”: {
“worker_num”: 16
},
“sink”: {
“protocol”: “”,
“schema_registry”: “”,
“csv”: {
“delimiter”: “,”,
“quote”: “"”,
“null”: “\N”,
“include_commit_ts”: false
},
“column_selectors”: null,
“transaction_atomicity”: “none”,
“encoder_concurrency”: 16,
“terminator”: “\r\n”,
“date_separator”: “day”,
“enable_partition_separator”: true
},
“consistent”: {
“level”: “none”,
“max_log_size”: 64,
“flush_interval”: 2000,
“storage”: “”
}
},
“create_time”: “2023-11-29 16:56:26.301”,
“start_ts”: 445971741540352000,
“resolved_ts”: 445973270780182530,
“target_ts”: 0,
“checkpoint_tso”: 445973270780182530,
“checkpoint_time”: “2023-11-29 18:16:56.587”,
“sort_engine”: “unified”,
“state”: “normal”,
“error”: null,
“error_history”: null,
“creator_version”: “”,
“task_status”: [
{
“capture_id”: “3183ecc6-1475-4a3b-a0f5-0144e8c81544”,
“table_ids”: ,
“table_operations”: null
},
{
“capture_id”: “337f63a2-e74d-4fde-812d-45fb04ee3474”,
“table_ids”: ,
“table_operations”: null
}
]
}

| username: dba远航 | Original post link

If the status is normal, please check the relevant filtering rules and configuration to ensure they are correct.

| username: 小龙虾爱大龙虾 | Original post link

It can be seen that there should be an issue with the filter configuration, as the checkpoint is progressing normally, but there are no tables in the task:
“task_status”: [
{
“capture_id”: “3183ecc6-1475-4a3b-a0f5-0144e8c81544”,
“table_ids”: ,
“table_operations”: null
},
{
“capture_id”: “337f63a2-e74d-4fde-812d-45fb04ee3474”,
“table_ids”: ,
“table_operations”: null
}
]
}

| username: 消息终结者 | Original post link

How do you filter downstream if there are only upstream filtering rules?

| username: 消息终结者 | Original post link

Hmm, it looks like there’s an issue with table_ids here. Is the configuration file incorrect? Could you send a copy of the configuration file?

| username: 小龙虾爱大龙虾 | Original post link

It doesn’t look like the configuration file is wrong. Check the upstream TiDB with this:

select table_schema, table_name from information_schema.tables where lower(table_name) in ('e_participation_record', 'canvas_nc') and lower(table_schema) = 'db_canvas';
| username: 消息终结者 | Original post link

Please take a look.

| username: 小龙虾爱大龙虾 | Original post link

It’s strange, try executing this: select table_schema, table_name, table_type from information_schema.tables where lower(table_name) in (‘e_participation_record’, ‘canvas_nc’) and lower(table_schema) = ‘db_canvas’;

| username: 消息终结者 | Original post link

Could you add a v and take a look at it?

| username: 路在何chu | Original post link

There are no logs in CDC, which means no connection has been established.

| username: 小龙虾爱大龙虾 | Original post link

I don’t know what’s going on. Please check the environment again and see if the parameter after --server is correct.

| username: 消息终结者 | Original post link

cdc has logs now; previously, I was looking in the wrong directory. The logs appear to be normal and contain puller-related information.

| username: 消息终结者 | Original post link

This morning, I turned off the filter, which means the whole database is synchronized, and the table_ids have values, but there is still no data coming through.