TiFlash Automatic Disconnection

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tiflash自动断开

| username: TiDBer_w9g3Bozx

[TiDB Usage Environment] Production Environment / Testing / Poc
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration] High Configuration
[Attachments: Screenshots / Logs / Monitoring]
tiup cluster display <cluster_name>
Found tiflash suddenly down

Below are the log details:
tiflash1.log (194.0 KB)
tiflash_error1.log (327.8 KB)

Requesting experts to help analyze the cause, thank you.

| username: Billmay表妹 | Original post link

If you find that TiFlash suddenly goes down, you can use the tiup cluster display command to check the status of TiFlash. The specific steps are as follows:

  1. Execute the following command to check the cluster status:

    tiup cluster display <cluster-name>
    

    Where <cluster-name> is the name of your deployed TiDB cluster.

  2. Check the status of TiFlash:

    In the output of the tiup cluster display command, you can find the status information of TiFlash. For example:

    TiFlash:
        Cluster type:       standalone
        Cluster version:    v4.0.0
        Dashboard URL:      http://<ip>:<port>
        Log path:           /data1/deploy/tiflash-9000/log/tiflash.log
        Last started:       N/A
        Last stopped:       2021-01-01 00:00:00
        Total replicas:     0
        Replicas:           N/A
        Data dir:           /data1/deploy/tiflash-9000/data
        Deploy dir:         /data1/deploy/tiflash-9000
        Log dir:            /data1/deploy/tiflash-9000/log
        Resource control:   N/A
        Process Status:
            tiflash:        Down
    

    In the above output, you can see that the status of TiFlash is Down, indicating that TiFlash is currently down.

  3. Check the TiFlash logs:

    If TiFlash suddenly goes down, you can check the TiFlash logs to understand the specific error information. In the above output, you can find the log path of TiFlash, such as /data1/deploy/tiflash-9000/log/tiflash.log. You can use the tail command to view the TiFlash logs:

    tail -f /data1/deploy/tiflash-9000/log/tiflash.log
    

    Looking for error information in the logs can help you find the reason why TiFlash is down.

| username: ShawnYan | Original post link

Is this a new 7.1 cluster built on the existing cluster environment?

| username: Fly-bird | Original post link

tiup cluster restart -N 10.209.229.44:9000

| username: TiDBer_w9g3Bozx | Original post link

After discovering the table event_analyze with the command ALTER TABLE event_analyze SET TIFLASH REPLICA 1;, it disconnects after a while.


The corresponding logs are a bit confusing. Could an expert please help analyze the reason?
tiflash1.log (492.2 KB)
tiflash_error1.log (155.7 KB)

| username: cy6301567 | Original post link

Have you found the reason?