Lightning Restore Error: Create TSO Stream Failed, Retry Timeout

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: lightning 恢复报错：create TSO stream failed, retry timeout

| username: zhenda

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.1.0
[Reproduction Path] lightning restore
[Encountered Problem]
When dumping tidbv5.0.6 and executing lightning restore to v6.1.0, an error “create TSO stream failed, retry timeout” occurred.

image
image1859×92 9.81 KB

Check pd log error
image
image1865×96 5.26 KB

[Resource Configuration]

image
image1380×498 55.2 KB

image
image1450×343 17.2 KB

2
| username: WalterWj | Original post link

If restoring in lightning local mode, it should be taking the hostmap of TiKV from PD. It seems like it can’t find the corresponding TiKV nodes of the cluster.

3
| username: zhenda | Original post link

Error: “lookup tidb-cluster-pd-2.tidb-cluster-pd-peer.mcd-reng.svc on 100.64.0.101:53: no such host”
tidb-cluster-pd-2 is the PD leader, and this error occurs before reaching the stage of obtaining the hostmap from PD for TiKV.
I don’t understand why the service’s DNS is being resolved when accessing through LoadBalancer.

4
| username: WalterWj | Original post link

If the data volume is not large, you can try the logical mode.

5
| username: tidb菜鸟一只 | Original post link

Can the address 100.64.0.101:53 be reached?

6
| username: zhenda | Original post link

The telnet port is open.

7
| username: zhenda | Original post link

The data volume is relatively large, more than 1TB, and breakpoint resume is also needed. How can I further troubleshoot the problem?

8
| username: WalterWj | Original post link

It can be restored with a k8s cluster, as long as you connect S3 or some network storage for access on both sides.

9
| username: zhenda | Original post link

Cross-data center data migration is quite troublesome. If it can be restored within the same cluster, then just use the same cluster directly.

10
| username: 裤衩儿飞上天 | Original post link

Is the version of Lightning consistent with the cluster? Is it also 6.1.0?

11
| username: zhenda | Original post link

Data import from v5.0.4 to 6.1.0 using v5.0.4 lightning resulted in an error, suggesting to use v6.1.0 lightning for import:

image
image1858×174 16.4 KB

12
| username: 裤衩儿飞上天 | Original post link

How about not crossing the k8s cluster and mounting the exported data to the new cluster for recovery?

13
| username: zhenda | Original post link

Cross-data center data migration is quite troublesome. If it can be restored within the same cluster, just use the same cluster. :joy: Please help to look at it from a technical perspective.

14
| username: 裤衩儿飞上天 | Original post link

Judging by the error, it seems that the PD node is not accessible. I’m not very familiar with Kubernetes either, let’s wait for responses from other experts.

15
| username: zhenda | Original post link

Adjusting to the same k8s cluster, error message

image
image1861×107 9.44 KB

16
| username: zhenda | Original post link

After adjusting to the same cluster, the error message is shown in the above image.

17
| username: 裤衩儿飞上天 | Original post link
  1. On the node executing Lightning, use telnet to check the PD IP and port to see if the network is functioning properly.
  2. Check the PD node logs for any errors and verify if the status is normal.
18
| username: ffeenn | Original post link

You can try DM and BR, don’t forcefully use Lightning.

Using DM on Kubernetes | PingCAP Docs
Backup and Restore Overview | PingCAP Docs

19
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.