Failed to Start DM Master Expansion

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: dm master扩容启动失败

| username: 宸凡_22

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
[Attachments: Screenshots/Logs/Monitoring]

image
image1380×290 99.3 KB

master log

dm-master.log (5.8 MB)

2
| username: tidb菜鸟一只 | Original post link

Is this log correct? A binary file?

3
| username: 宸凡_22 | Original post link

How do I find the binary file? The logs should be correct.

4
| username: tidb菜鸟一只 | Original post link

When I open it, it looks like this. Try opening it yourself…

5
| username: 宸凡_22 | Original post link

[2024/01/17 22:33:01.857 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:33:01.857 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:33:01.863 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:33:17.170 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:33:17.170 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:33:17.174 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:33:32.453 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:33:32.453 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:33:32.458 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:33:47.696 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:33:47.696 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:33:47.704 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:34:02.926 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f] [“Git Branch”=heads/refs/tags/v6.0.0] [“UTC Build Time”=“2022-04-06 03:10:08”] [“Go Version”=“go version go1.18 linux/amd64”]
[2024/01/17 22:34:02.926 +08:00] [INFO] [main.go:63] [“dm-master config”=“{"log-level":"debug","log-file":"/datafile01/04_DB_Home/03_Tidb/dm/deploy/dm-master-8261/log/dm-master.log","log-format":"text","log-rotate":"","rpc-timeout":"30s","rpc-rate-limit":10,"rpc-rate-burst":40,"master-addr":"0.0.0.0:8261","advertise-addr":"10.10.110.171:8261","config-file":"conf/dm-master.toml","name":"master1","data-dir":"/datafile01/05_DB_DataFiles/03_Tidb/dm/data/dm-master-8261","peer-urls":"http://0.0.0.0:8291","advertise-peer-urls":"http://10.10.110.171:8291","initial-cluster":"master1=http://10.10.110.171:8291","initial-cluster-state":"new","join":"http://10.10.110.155:8261,http://10.10.110.173:8261","max-txn-ops":2048,"max-request-bytes":1572864,"auto-compaction-mode":"periodic","auto-compaction-retention":"1h","quota-backend-bytes":2147483648,"openapi":false,"v1-sources-path":"","ssl-ca":"","ssl-cert":"","ssl-key":"","cert-allowed-cn":null,"ExperimentalFeatures":{"OpenAPI":false}}”]
[2024/01/17 22:34:02.930 +08:00] [ERROR] [main.go:71] [“fail to start dm-master”] [error=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.”] [errorVerbose=“[code=38040:class=dm-master:scope=internal:level=high], Message: fail to join embed etcd: there is a member that has not joined successfully, continue the join or remove it, Workaround: Please check configs in master configuration file.\ngithub.com/pingcap/tiflow/dm/pkg/terror.(*Error).Generate\n\tgithub.com/pingcap/tiflow/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/tiflow/dm/dm/master.prepareJoinEtcd\n\tgithub.com/pingcap/tiflow/dm/dm/master/etcd.go:149\ngithub.com/pingcap/tiflow/dm/dm/master.(*Server).Start\n\tgithub.com/pingcap/tiflow/dm/dm/master/server.go:152\nmain.main\n\tgithub.com/pingcap/tiflow/dm/cmd/dm-master/main.go:69\nruntime.main\n\truntime/proc.go:250\nruntime.goexit\n\truntime/asm_amd64.s:1571”]
[2024/01/17 22:34:18.196 +08:00] [INFO] [printer.go:54] [“Welcome to dm-master”] [“Release Version”=v6.0.0] [“Git Commit Hash”=2f363f653a49fc557ebaab1afa7a47f900fbfc8f]

6
| username: wangccsy | Original post link

Can’t tell anything.

7
| username: TiDBer_小阿飞 | Original post link

Message: Failed to join embedded etcd: A member failed to join successfully. Please continue to join or remove it. Solution: Please check the configuration in the main configuration file.

8
| username: TiDBer_小阿飞 | Original post link

It seems that there is no error code 38040.

9
| username: dba远航 | Original post link

Fail to join embedded etcd: there is a member that has not joined successfully. Continue the join or remove it. Workaround: Please check configs in the master configuration file.

10
| username: 宸凡_22 | Original post link

Is there a configuration file for scaling? With the same configuration file, I can open the worker and comment out the master without any issues.