TiDB Lightning Incremental Import Failure

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb lightning增量导入失败

| username: TiDBer_OB4kHrS7

【TiDB Usage Environment】Production Environment
【TiDB Version】V5.2.2
【Reproduction Path】First perform a full import, then perform an incremental import
【Encountered Problem: Phenomenon and Impact】No issues with the full import, but errors occur during the incremental import
【Resource Configuration】
【Attachments: Screenshots/Logs/Monitoring】



Please help analyze the reason why the incremental import is unsuccessful.

| username: 啦啦啦啦啦 | Original post link

Field does not exist?

| username: weixiaobing | Original post link

According to the prompt, it is an issue of missing fields in the table. table=pagoda_overall.goods_snapshot

| username: TiDBer_pkQ5q1l0 | Original post link

Change local to TiDB mode.

| username: TiDBer_OB4kHrS7 | Original post link

The table itself does not have this column.

| username: caiyfc | Original post link

After all, it is an experimental feature, so it should be used with caution. You can use Dumpling for backup.

| username: TiDBer_OB4kHrS7 | Original post link

Dumpling backup, is it also using Lightning for incremental import?

| username: caiyfc | Original post link

:rofl: I misread it, I thought you were using br.
I see the errors are all in checksum, are you using TiDB mode?

| username: TiDBer_OB4kHrS7 | Original post link

Use the local mode.

| username: caiyfc | Original post link

The local mode requires the imported table to be empty. Therefore, incremental imports must use the TiDB mode.

| username: 裤衩儿飞上天 | Original post link

Does Lightning support incremental import? I remember it doesn’t.

| username: TiDBer_OB4kHrS7 | Original post link

Currently doing a full export, will try using TiDB mode later to see if there are any errors.

| username: TiDBer_OB4kHrS7 | Original post link

Support incremental, yes.

| username: 裤衩儿飞上天 | Original post link

Can you talk about your incremental backup and import process? I don’t know how you do this incremental backup.

| username: TiDBer_OB4kHrS7 | Original post link

Dumpling exports based on conditions, such as create_at<=“XXXXXX”. For incremental exports, it exports data where create_at>“XXXXXX”.

| username: 裤衩儿飞上天 | Original post link

If the table structure changes after the full backup and before the incremental backup, could it cause the error you mentioned?

| username: TiDBer_OB4kHrS7 | Original post link

I haven’t tested this, so I’m not sure, but after the full backup and before the incremental backup, the table structure hasn’t changed at all. I’m not sure why this error is occurring.

| username: 裤衩儿飞上天 | Original post link

You can compare the table structures on both sides to see if they are consistent.

| username: TiDBer_OB4kHrS7 | Original post link

The table has been checked, and it is consistent.

| username: caiyfc | Original post link

Use dumpling to back up based on conditions, and then import it using lightning’s tidb mode.