Warning During Lightning Import

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: lightning导入报warning

| username: 特雷西-迈克-格雷迪

[TiDB Usage Environment] Production Environment
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
The lightning configuration is as follows:

[lightning]
…
[tikv-importer]
[#] backend set to local mode
backend = “local”
duplicate-resolution = ‘record’

[Encountered Issue: Issue Phenomenon and Impact]
[2022/11/20 03:23:37.850 +08:00] [WARN] [local.go:1445] [“[resolve-dupe] skipping resolution due to selected algorithm. this table will become inconsistent!”] [table=xxx.xxx_car_info] [algorithm=record]

[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]

image
image1345×67 11.5 KB

2
| username: buddyyuan | Original post link

Is there duplicate data?

3
| username: buddyyuan | Original post link

There are duplicate data. The parameter you set is duplicate-resolution = ‘record’. It will log and write a WARN log.

Physical Import Mode sets whether to detect and resolve duplicate records (unique key conflicts).
Currently, three methods are supported:

  • record: Only adds duplicate records to the lightning_task_info.conflict_error_v1 table in the target TiDB. Note that this method requires the target TiKV version to be v5.2.0 or later. If the version is too low, the ‘none’ mode below will be enabled.
  • none: Does not detect duplicate records. This mode has the best performance among the three modes but may lead to data inconsistency in the target TiDB.
  • remove: Records all duplicate records, similar to the ‘record’ mode. However, it will delete all duplicate records to ensure data consistency in the target TiDB.
    duplicate-resolution = ‘none’
4
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.