Error During Lightning Import

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: lightning导入报错

| username: TiDBer_OB4kHrS7

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] V7.5.1
[Reproduction Path] Operations that led to the issue: Export data from MySQL using dumpling, import into TiDB using lightning
[Encountered Issue: Problem Phenomenon and Impact]
Need to import tables exported from MySQL database A into database B, keeping the table names unchanged but changing the database name. Configured routes in the configuration file, but encountered an error when executing the import script.
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]
cat nohup.out
[Lightning:Config:ErrParseConfigFile] cannot parse config file ‘lightning0.toml’: toml: line 31 (last key “routes”): expected a top-level item to end with a newline, comment, or EOF, but got ‘\u3000’ instead

| username: Miracle | Original post link

Take a look at the last line of [[routes]] to see if there’s a space.

| username: 这里介绍不了我 | Original post link

The symbol represents a space, it should still be a formatting issue causing this.

| username: TiDBer_OB4kHrS7 | Original post link

I checked, and there is a space at the end of each line under [[routes]]. Deleting them will solve the issue.

| username: shigp_TIDBER | Original post link

It’s obviously a formatting issue.

| username: TiDBer_OB4kHrS7 | Original post link

I misread it earlier. Now it can be exported, but it is still imported into the test database instead of the test0 database, so the effect of changing the database name has not been achieved.

| username: 这里介绍不了我 | Original post link

It shouldn’t be, does your test0 database exist?

| username: TiDBer_OB4kHrS7 | Original post link

It doesn’t exist. Should I create it first?

| username: 这里介绍不了我 | Original post link

You need to create it yourself; it won’t create it automatically for you.

| username: TiDBer_OB4kHrS7 | Original post link

Created the test0 database, but it’s still the same; the data was imported into the test database.

| username: TiDBer_QYr0vohO | Original post link

Where is the format incorrect?

| username: 这里介绍不了我 | Original post link

Sure, please provide the text you need translated.

| username: TiDBer_OB4kHrS7 | Original post link

I couldn’t tell from the configuration file where the issue is.

| username: TiDBer_OB4kHrS7 | Original post link

I tested it, there is a database A and table A on TiDB, exported it, and then used the above configuration file, it can be imported into database B and table A. Now in production, there is database B and table A, the exported data is from MySQL, and when exported it is database A and table A. Using the above configuration file, importing into production results in database A and table A. I don’t understand why this is happening?

| username: TiDBer_OB4kHrS7 | Original post link

Now importing table A from database B in production, and there’s a column mismatch error.
[2024/05/17 15:53:32.510 +08:00] [ERROR] [import.go:561] [“run failed”] [step=4] [error=“[Lightning:Restore:ErrEncodeKV]encode kv error in file pagoda_customer.store_goods_order.0000000020000.sql:0 at offset 350: column count mismatch, expected 23, got 21”]
[2024/05/17 15:53:32.510 +08:00] [INFO] [import.go:1266] [“everything imported, stopping periodic actions”]
[2024/05/17 15:53:32.510 +08:00] [ERROR] [import.go:571] [“the whole procedure failed”] [takeTime=96.23788ms] [error=“[Lightning:Restore:ErrEncodeKV]encode kv error in file pagoda_customer.store_goods_order.0000000020000.sql:0 at offset 350: column count mismatch, expected 23, got 21”]
[2024/05/17 15:53:32.511 +08:00] [ERROR] [import.go:170] [“tables failed to be imported”] [count=1]
[2024/05/17 15:53:32.511 +08:00] [ERROR] [import.go:172] [-] [table=pagoda_customer0.store_goods_order] [status=written] [error=“[Lightning:Restore:ErrEncodeKV]encode kv error in file XXXXXXX.0000000020000.sql:0 at offset 350: column count mismatch, expected 23, got 21”]
[2024/05/17 15:53:32.511 +08:00] [ERROR] [main.go:103] [“tidb lightning encountered error stack info”] [error=“[Lightning:Restore:ErrEncodeKV]encode kv error in file XXXXXXX.0000000020000.sql:0 at offset 350: column count mismatch, expected 23, got 21”]

| username: yytest | Original post link

It should be a formatting issue.

| username: zhaokede | Original post link

Why are the columns still mismatched?

| username: zhaokede | Original post link

Are the data models upstream and downstream different, or have fields been added?

| username: Jack-li | Original post link

Formatting issue, perhaps.

| username: TiDBer_OB4kHrS7 | Original post link

The upstream is MySQL, and the downstream is TiDB. No fields were added, and there were no changes. It’s just that there is a B database A table in TiDB, while MySQL exports it as A database A table. There is a routing process during the import.