Dumpling Download Failed

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Dumpling下载失败

| username: 普罗米修斯

[TiDB Usage Environment] Production
[TiDB Version] 3.0.3
[Encountered Issue]
Download command: nohup /home/tidb/tidb-toolkit-v5.0.1-linux-amd64/bin/dumpling -u root -p’tidb’ -P 4000 -h 192.168.80.203 --filetype sql -t 16 -r 200000 -o /data > dumpling.out 2>&1 &
Started downloading at 19:30, error occurred at 23:39,

[mysql] 2022/10/09 23:39:24 packets.go:72: unexpected EOF
[mysql] 2022/10/09 23:39:24 packets.go:427: busy buffer
[2022/10/09 23:39:24.740 +08:00] [INFO] [collector.go:188] [“backup Failed summary: total backup ranges: 0, total success: 0, total failed: 0”]
[mysql] 2022/10/09 23:39:24 packets.go:72: unexpected EOF
[mysql] 2022/10/09 23:39:24 packets.go:427: busy buffer
[2022/10/09 23:39:24.744 +08:00] [ERROR] [main.go:77] [“dump failed error stack info”] [error=“sql: SELECT * FROM wtlivedbrds_xa.EVDataRT LIMIT 1: invalid connection”] [errorVerbose=“invalid connection
sql: SELECT * FROM wtlivedbrds_xa.EVDataRT LIMIT 1
github.com/pingcap/dumpling/v4/export.GetColumnTypes
\tgithub.com/pingcap/dumpling@/v4/export/sql.go:307
github.com/pingcap/dumpling/v4/export.dumpTableMeta
\tgithub.com/pingcap/dumpling@/v4/export/dump.go:696
github.com/pingcap/dumpling/v4/export.(*Dumper).dumpDatabases
\tgithub.com/pingcap/dumpling@/v4/export/dump.go:302
github.com/pingcap/dumpling/v4/export.(*Dumper).Dump
\tgithub.com/pingcap/dumpling@/v4/export/dump.go:223
main.main
\tgithub.com/pingcap/dumpling@/cmd/dumpling/main.go:74
runtime.main
\truntime/proc.go:203
runtime.goexit
\truntime/asm_amd64.s:1357”]

dump failed: sql: SELECT * FROM wtlivedbrds_xa.EVDataRT LIMIT 1: invalid connection
Final connection failed, dump failed.
[Reproduction Path] Dumpling download tested 4 times, all failed. Tried changing the thread count -t from 128 to 64, 32, and changing -r to 100000, but still encountered the same error and failure.

2
| username: WalterWj | Original post link

Try manually connecting to TiDB and executing this SQL.

3
| username: 普罗米修斯 | Original post link

Manually executing this SQL has no issues. It can produce results normally.

4
| username: WalterWj | Original post link
  1. Check the uptime of the tidb-server monitoring to see if there has been a restart.
  2. Look for any panic content in the tidb server’s tidb.log and stderr log.
5
| username: 普罗米修斯 | Original post link
  1. The tidb-server restarted on 2022/10/09 at 23:39:40.
    Image
    Image1380×648 27.5 KB
  2. There is no panic content in the tidb.log, but it shows a restart at 23:39:40.
    Image
    Image1755×813 59.5 KB

    The stderr.log contains panic content at the end of the log, but the specific time is not visible. The error indicates an out-of-memory issue.
    Image
    Image1300×499 22 KB
6
| username: TiDBer_CEVsub | Original post link

It doesn’t seem to work very well.

7
| username: WalterWj | Original post link

Does the time of your dumpling failures match up? It looks like it restarts once, seems like it’s an OOM issue.

8
| username: WalterWj | Original post link

You can try setting t to 1.

9
| username: 普罗米修斯 | Original post link

The failure time and restart time of dumpling match. After work, I will change -t to 8 and try again. Does the -r parameter need to be adjusted? The amount of data that needs to be fully downloaded is very large.

10
| username: WalterWj | Original post link

The -r option can remain unchanged, as it is used for parallel export of a single table.
However, if you didn’t encounter a panic or an OOM, it shouldn’t have failed. :thinking:

11
| username: 普罗米修斯 | Original post link

After adjusting the number of threads, dumpling started downloading, but the download failed again at 22:10 in the evening. The tidb-server restarted, and there was also an OOM situation. Before 22:10, checking the slow log did not show any SQL occupying high resources.

Image
Image1707×741 43 KB

Image
Image1380×212 30.3 KB

12
| username: WalterWj | Original post link

-T try reducing it to 4, and -r can also be reduced to 50,000.
If it still doesn’t work, try using a newer version of Dumpling 6.x.

13
| username: 普罗米修斯 | Original post link

I’ll try again after work today.

14
| username: 普罗米修斯 | Original post link

Using version 6.1 of Dumpling, the result is still the same: the connection is disconnected, and Dumpling fails.

15
| username: 普罗米修斯 | Original post link

Change the -r parameter to 50000 and the -t parameter to 16. The download started at 8 AM yesterday, and the connection was interrupted around 3 PM.

16
| username: db_user | Original post link

Try this method. I’ve encountered this issue with Lightning, but not with Dumpling. Put it in the script and give it a try.

17
| username: 普罗米修斯 | Original post link

I executed it in the script, but it didn’t work.

18
| username: db_user | Original post link

It feels quite similar to this issue, caused by dumpling.

image
image1063×218 17 KB

Moreover, I see your TiDB version is 3, but the dumpling version used is 5. You can try using dumpling version 4.0.15 or other versions.

19
| username: 普罗米修斯 | Original post link

This variable was only introduced starting from version 4.0, and my version is 3.0.3, which doesn’t have this parameter. Image
I downloaded dumpling version 4.0.16, and I’ll try it after work today.

20
| username: WalterWj | Original post link

-d Don’t make it so big, try 1 first to see if it works.