After installing the TiDB v6.6.0 cluster on CentOS 8, TiFlash fails to start properly

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: centos8 tidb v6.6.0 集群安装后 tiflash 无法正常启动

| username: Hacker_NyIoc9bk

[TiDB Usage Environment] Test/PoC
[TiDB Version] v6.6.0
[Reproduction Path] Unable to start TiFlash during cluster installation
[Encountered Issue: Phenomenon and Impact] Unable to start TiFlash during cluster installation
[Resource Configuration]
[Attachment: Screenshot/Log/Monitoring]
tiflash_error.log:
log.tar.gz (1.0 MB)

[2023/04/06 13:52:49.648 +08:00] [WARN] [StorageConfigParser.cpp:241] [“The configuration "path" is deprecated. Check [storage] section for new style.”] [thread_id=1]
[2023/04/06 13:52:49.869 +08:00] [ERROR] [Server.cpp:327] [“/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/core/ext/transport/chttp2/server/insecure/server_chttp2.cc, line number: 48, log msg : {"created":"@1680760369.869713884","description":"No address added out of total 1 resolved","file":"/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":936,"referenced_errors":[{"created":"@1680760369.869708905","description":"Unable to configure socket","fd":34,"file":"/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":218,"referenced_errors":[{"created":"@1680760369.869702352","description":"Cannot assign requested address","errno":99,"file":"/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":191,"os_error":"Cannot assign requested address","syscall":"bind"}]}]}”] [source=grpc] [thread_id=1]
[2023/04/06 13:52:51.748 +08:00] [ERROR] [] [“DB::Exception: Exception happens when start grpc server, the flash.service_addr may be invalid, flash.service_addr is 43.139.169.37:3930”] [source=Application] [thread_id=1]

Error: failed to start tiflash: failed to start: 43.139.169.37 tiflash-9000.service, please check the instance’s log(/var/lib/tidb/tidb-deploy/tiflash-9000/log) for more detail.: timed out waiting for port 9000 to be started after 2m0s

| username: tidb菜鸟一只 | Original post link

Check if the address and port 43.139.169.37:3930 are already in use.

| username: Hacker_NyIoc9bk | Original post link

Not occupied

| username: 裤衩儿飞上天 | Original post link

It feels like an external IP.

| username: Hacker_NyIoc9bk | Original post link

It is an external IP. TiDB, PD, and TiKV monitoring are all fine. It’s just that TiFlash won’t start. Executed dnf install libnsl and ulimit -n 1000000, but it still doesn’t work.

| username: 裤衩儿飞上天 | Original post link

Are all the nodes using public IPs? Is the network between the nodes connected? Make sure the environment is consistent.

| username: Hacker_NyIoc9bk | Original post link

The network must be connected. The PD and TiDB installed on the same node are all functioning normally.