TiDB 5.4.2 Partition Table with 120 Million Rows Reports [error="Error 9005: Region is unavailable"] in Dumpling

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb5.4.2 1.2亿的分区表dumpling报[error=“Error 9005: Region is unavailable”]

| username: heming

Bug Report
Clearly and accurately describe the issue you found. Providing any steps to reproduce the issue can help the development team address the problem promptly.
[Impact of the Bug]
Full table backup export interrupted unexpectedly.
[Possible Steps to Reproduce the Issue]
./dumpling -h 127.0.0.1 -P 4000 -u backup -p wPjNXy9z8lrApDVb -B yixintui_operate --tables-list “yixintui_operate.Agent_material_report_cost” --filetype sql --threads 1 -o ${Bak_dir}/${Ip}/${Port}/${Time} -F 1024MiB --compress gz --params “tidb_distsql_scan_concurrency=1,tidb_mem_quota_query=8589934592” --where=“report_date >=‘${lastMonth1}’” >> $Bak_log 2
[Observed Unexpected Behavior]
[2022/08/08 02:46:39.794 +08:00] [WARN] [writer_util.go:181] [“fail to dumping table(chunk), will revert some metrics and start a retry if possible”] [database=yixintui_operate] [table=Agent_material_report_cost] [“finished rows”=2024142] [“finished size”=757458552] [error=“Error 9005: Region is unavailable”]
[2022/08/08 02:46:39.796 +08:00] [INFO] [collector.go:203] [“backup failed summary”] [total-ranges=1] [ranges-succeed=0] [ranges-failed=1] [unit-name=“dump table data”] [error=“Error 9005: Region is unavailable”] [errorVerbose=“Error 9005: Region is unavailable
github.com/pingcap/errors.AddStack
\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174
github.com/pingcap/errors.Trace
\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/juju_adaptor.go:15
github.com/pingcap/tidb/dumpling/export.(*multiQueriesChunkIter).nextRows.func1
\tgithub.com/pingcap/tidb/dumpling/export/ir_impl.go:87
github.com/pingcap/tidb/dumpling/export.(*multiQueriesChunkIter).nextRows
\tgithub.com/pingcap/tidb/dumpling/export/ir_impl.go:107
github.com/pingcap/tidb/dumpling/export.(*multiQueriesChunkIter).Next
\tgithub.com/pingcap/tidb/dumpling/export/ir_impl.go:155
github.com/pingcap/tidb/dumpling/export.WriteInsert
\tgithub.com/pingcap/tidb/dumpling/export/writer_util.go:233
github.com/pingcap/tidb/dumpling/export.FileFormat.WriteInsert
\tgithub.com/pingcap/tidb/dumpling/export/writer_util.go:625
github.com/pingcap/tidb/dumpling/export.(*Writer).tryToWriteTableData
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:216
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:201
github.com/pingcap/tidb/br/pkg/utils.WithRetry
\tgithub.com/pingcap/tidb/br/pkg/utils/retry.go:60
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:172
github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:105
github.com/pingcap/tidb/dumpling/export.(*Writer).run
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:85
github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4
\tgithub.com/pingcap/tidb/dumpling/export/dump.go:302
golang.org/x/sync/errgroup.(*Group).Go.func1
\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
\truntime/asm_amd64.s:1371”]
[2022/08/08 02:46:39.796 +08:00] [ERROR] [client.go:752] [“[pd] fetch pending tso requests error”] [dc-location=global] [error=“[PD:client:ErrClientGetTSO]context canceled: context canceled”]
[2022/08/08 02:46:39.797 +08:00] [ERROR] [main.go:76] [“dump failed error stack info”] [error=“Error 9005: Region is unavailable”] [errorVerbose=“Error 9005: Region is unavailable
github.com/pingcap/errors.AddStack
\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174
github.com/pingcap/errors.Trace
\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/juju_adaptor.go:15
github.com/pingcap/tidb/dumpling/export.(*multiQueriesChunkIter).nextRows.func1
\tgithub.com/pingcap/tidb/dumpling/export/ir_impl.go:87
github.com/pingcap/tidb/dumpling/export.(*multiQueriesChunkIter).nextRows
\tgithub.com/pingcap/tidb/dumpling/export/ir_impl.go:107
github.com/pingcap/tidb/dumpling/export.(*multiQueriesChunkIter).Next
\tgithub.com/pingcap/tidb/dumpling/export/ir_impl.go:155
github.com/pingcap/tidb/dumpling/export.WriteInsert
\tgithub.com/pingcap/tidb/dumpling/export/writer_util.go:233
github.com/pingcap/tidb/dumpling/export.FileFormat.WriteInsert
\tgithub.com/pingcap/tidb/dumpling/export/writer_util.go:625
github.com/pingcap/tidb/dumpling/export.(*Writer).tryToWriteTableData
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:216
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:201
github.com/pingcap/tidb/br/pkg.utils.WithRetry
\tgithub.com/pingcap/tidb/br/pkg/utils/retry.go:60
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:172
github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:105
github.com/pingcap/tidb/dumpling/export.(*Writer).run
\tgithub.com/pingcap/tidb/dumpling/export/writer.go:85
github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4
\tgithub.com/pingcap/tidb/dumpling/export/dump.go:302
golang.org/x/sync/errgroup.(*Group).Go.func1
\tgolang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
\truntime/asm_amd64.s:1371”]
dump failed: Error 9005: Region is unavailable

[Related Components and Specific Version]
Unified version 5.4.2

2
| username: heming | Original post link

Another small table backup also failed. It was estimated to be 2.08 million, but it crashed after backing up a few hundred thousand. This table was backed up normally some time ago.

3
| username: h5n1 | Original post link

Use tiup cluster display to check the cluster status and see if you can find the error region ID in the tidb.log.

4
| username: heming | Original post link

I have sent you my contact information via private message.
Now, if I use the WHERE condition to back up month by month, it can back up normally. (Previously, I was backing up data from July 1, 2022, which was less than two months of data.)

5
| username: heming | Original post link

Why does dumpling have to associate with _tidb_rowid?
Does it matter if the execution plan uses TiFlash?

SELECT * FROM `yixintui_operate`.`Agent_material_report_cost` WHERE (report_date >= '2022-07-01') AND (`_tidb_rowid` >= 2305843016533029100 AND `_tidb_rowid` < 2305843016533314278) ORDER BY `_tidb_rowid`
6
| username: h5n1 | Original post link

Dumpling divides parallel tasks based on numeric values. If there is something like a primary key ID, it will use the primary key ID. It feels like the export pressure might be too high, so try reducing the -F 1024MiB setting and see if that helps.

7
| username: heming | Original post link

After removing these two parameters and exporting more than 19 million, it still encountered an exception
–params “tidb_distsql_scan_concurrency=1,tidb_mem_quota_query=8589934592”

image
image1158×598 22.9 KB

8
| username: h5n1 | Original post link

Check the region health status in the PD monitoring to see if there are any regions that are not in a good state.

9
| username: heming | Original post link

The image you provided is not accessible. Please provide the text you need translated.

10
| username: heming | Original post link

The export pressure looks quite low. This TiDB is basically prepared for database backup and is not running online business.

11
| username: heming | Original post link

I’ll try changing the threads to 2 or back to the default 4 to see if the backup can be successful.
Using 2 threads was successful. I’ll try the default 4 threads to see.

image
image1145×532 20.9 KB

The default 4 threads also work fine.
image
image1169×413 16.1 KB

12
| username: 林先森cC | Original post link

Hello, will there be performance fluctuations in the cluster during backup? When I directly back up a table with millions of rows in the online network cluster, the backup takes more than 10 seconds, which causes queries on other tables to also be delayed for more than 10 seconds.

13
| username: heming | Original post link

I usually do backups in the early morning, so I don’t pay much attention to this. Additionally, I use a dedicated TiDB instance for backups.

14
| username: system | Original post link

This topic will be automatically closed 60 days after the last reply. No new replies are allowed.