Issues Encountered When Migrating TiDB to K8s Operator

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TIDB 迁移至k8s operator出现问题

| username: TiDBer_X13GFlmu

[Test Environment for TiDB]
[TiDB Version] 7.5.0, operator 1.5.0
[Reproduction Path] Migrate TiDB to Kubernetes | PingCAP Documentation Center Pre-requisites have been completed as per this document.
[Encountered Issue: Symptoms and Impact] Configuring pdAddresses has no effect, still configuring pd locally, and not connecting to pd remotely.
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]
The TiDB cluster can resolve the pod network

1710314601839




Discovered Errors

Able to ping within the tidb container
1710321241212
Logs from pd-1 node on k8s


| username: Miracle | Original post link

Migration of TiDB to Kubernetes | PingCAP Documentation Center
This document is too abstract, it discourages beginners.

| username: TiDBer_X13GFlmu | Original post link

This is the only document I found on the entire internet, I’m speechless.

| username: TiDBer_jYQINSnf | Original post link

I didn’t quite understand your question, but the document looks fine to me. It’s basically about adding some nodes in k8s and then scaling down the nodes in tiup, so the data naturally flows into k8s.

| username: onlyacat | Original post link

This document is over 3 years old…

| username: TiDBer_X13GFlmu | Original post link

That’s correct, but after I configured it, they are still two separate clusters and have not merged into one cluster.

| username: TiDBer_jYQINSnf | Original post link

First, ensure that the network between all pods and the TiUP deployment is connected.
Then, after following the instructions, you should be able to see 6 PD nodes through pd-ctl member: 3 from k8s and 3 from TiUP.
If this step is correct, you can switch the PD leader to the one in k8s.
Check the store; there should be stores from both clusters. Then, you can decommission the stores in TiUP one by one.

Which step are you having trouble with right now?

| username: TiDBer_X13GFlmu | Original post link

I checked the member and there are still only three nodes, the merge was not successful.

| username: TiDBer_X13GFlmu | Original post link

It is connected.

| username: TiDBer_aaO4sU46 | Original post link

I read this document before, but then I quit.

| username: TiDBer_jYQINSnf | Original post link

Enter the PD pod, use ps to check the PD startup command.
Also, check the PD configuration file.
Is there a node IP for TiUP?

| username: yiduoyunQ | Original post link

The error “connection reset by peer” indicates that the connection was not closed properly. When TiDB tries to read a request from the client, it finds that the connection has already been closed by the other party. Either ignore it or check the external connection and network.

| username: changpeng75 | Original post link

Is it possible that only ping works, but certain ports are not accessible?

| username: redgame | Original post link

We are doing import and export, but in reverse. With k8s, sigh…

| username: zhang_2023 | Original post link

This is a bit overwhelming.

| username: TiDBer_X13GFlmu | Original post link

No, using pd-ctl to check the store still shows only three.

| username: 小于同学 | Original post link

Try pinging it to see.

| username: TiDBer_X13GFlmu | Original post link

Can ping through

| username: TiDBer_jYQINSnf | Original post link

Mainly check if the startup command of pd-server in k8s includes the parameter -join. If it doesn’t, then the pdAddress configuration in k8s hasn’t taken effect, and you should investigate from there. If it does include it, then you should look into other areas.

| username: TiDBer_X13GFlmu | Original post link

The image you provided is not visible. Please provide the text you need translated.