Error When Importing Data with tiup tidb-lightning

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tiup tidb-lightning 导入数据报错

| username: TiDBer_aZrsY0ca

【TiDB Usage Environment】Production Environment V7.1/Test V7.4/Poc
【TiDB Version】
【Reproduction Path】After backing up the production environment data using tiup dumpling, copy it out and import it into the V7.4 test server using tiup tidb-lightning.
【Encountered Issue: Problem Phenomenon and Impact】
After importing, the table structure is fully imported, but the data shows: [ERROR] [import.go:558] [“run failed”] [step=2] [error="[Lightning:PreCheck:ErrPreCheckFailed]tidb-lightning pre-check failed: TiDB Lightning has detected tables with illegal checkpoints. To prevent data loss, this run will stop now, please run command "./tidb-lightning-ctl --checkpoint-remove=‘jknew.dst_attch’". This message appears for all tables, not just this one.
【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

【Attachments: Screenshots/Logs/Monitoring】


Cleaned up as required:

| username: Miracle | Original post link

Please send the configuration file for lighting.
Also, were there any errors during dumpling?

| username: TiDBer_aZrsY0ca | Original post link

[lightning]

Log

level = “info”
file = “tidb-lightning.log”
region-concurrency = 1
io-concurrency = 5

[tikv-importer]

“local”: Default mode, suitable for data volumes above TB level, but downstream TiDB cannot provide services during the import.

“tidb”: For data volumes below TB level, the tidb backend mode can be used, and downstream TiDB can provide services normally. For more information on backend modes, please refer to: https://docs.pingcap.com/tidb/stable/tidb-lightning-backends

backend = “local”

Set the temporary storage location for sorted key-value pairs. The target path must be an empty directory, and the directory space must be larger than the size of the dataset to be imported. It is recommended to set it to a different disk directory from data-source-dir and use flash media. Exclusive IO will achieve better import performance.

sorted-kv-dir = “/work/tiup/tidblightning”

[mydumper]

Source data directory, i.e., the path where Dumpling saved the data in step 1.

data-source-dir = “/work/tiup/2023111423” # Local or S3 path, for example: ‘s3://my-bucket/sql-backup’
filter = [‘.’, ‘!mysql.', '!sys.’, ‘!INFORMATION_SCHEMA.', '!PERFORMANCE_SCHEMA.’, ‘!METRICS_SCHEMA.', '!INSPECTION_SCHEMA.’]

[tidb]

Information of the target cluster

host=“192.168.9.116” # For example: 172.16.32.1
port = 4000 # For example: 4000
user =“root” # For example: “root”
password =“dst@123” # For example: “rootroot”
status-port = 10080 # During the import process, Lightning needs to obtain table structure information from TiDB’s “status port”, for example: 10080
pd-addr =“192.168.9.116:2379” # Address of the cluster PD. Lightning obtains some information through PD, for example, 172.16.31.3:2379. When backend = “local”, status-port and pd-addr must be filled in correctly, otherwise, the import will encounter exceptions.
No errors were reported during the export.

| username: 有猫万事足 | Original post link

This is generally caused by the failure to clear the checkpoint from the last import.

Your configuration file should have this setting.

Just clear the original checkpoint.

| username: TiDBer_aZrsY0ca | Original post link

Is this how you add it, or is it still reporting the same error?

| username: Miracle | Original post link

Try adding --checkpoint-remove=all.

| username: TiDBer_aZrsY0ca | Original post link

Add it to the configuration like this?

| username: Miracle | Original post link

When invoking lightning, add --checkpoint-remove=all
tidb-lightning-ctl --checkpoint-remove=all … like this

| username: TiDBer_aZrsY0ca | Original post link

Executed it, this method reported an error.

| username: Miracle | Original post link

./tidb-lightning-ctl --config /work/tiup/tidb-lightning.toml --checkpoint-error-destroy=all --checkpoint-remove=all

| username: 胡杨树旁 | Original post link

Lightning will generate a checkpoint file, which defaults to /tmp if no path is set. It is a file ending with .pb. Try deleting it and restarting.

| username: dba远航 | Original post link

This makes sense, I second it.

| username: TiDBer_aZrsY0ca | Original post link

Yes, this method works effectively, thank you.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.