Error in 1 TiFlash node in production environment: [ERROR] [observer.rs:290] ["transfer leader won't exec"]

translator_bot · June 21, 2024, 3:33pm

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 生产环境的1个tiflash节点报错： [ERROR] [observer.rs:290] [“transfer leader won’t exec”]

| username: vcdog

【TiDB Usage Environment】Production Environment
【TiDB Version】v6.5.0
【Reproduction Path】Two TiFlash nodes in the production environment have been running stably. However, after a few months, it was suddenly discovered that one of the TiFlash node service processes was running normally. But when executing SQL statements, if they are assigned to the problematic TiFlash node, an error will be reported:

2023/10/26 23:11:11 - Beijing Main Database.0 - ERROR (version 9.2.0.0-290, build 9.2.0.0-290 from 2021-06-02 06.36.08 by buildguy) : Unexpected error
2023/10/26 23:11:11 - Beijing Main Database.0 - ERROR (version 9.2.0.0-290, build 9.2.0.0-290 from 2021-06-02 06.36.08 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException: 
2023/10/26 23:11:11 - Beijing Main Database.0 - Couldn't get row from result set
2023/10/26 23:11:11 - Beijing Main Database.0 - [FLASH:Coprocessor:BadRequest] Income key ranges is illegal for region: 499309: (while doing learner read for table, logical table_id: 29009)
2023/10/26 23:11:11 - Beijing Main Database.0 - 
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at org.pentaho.di.core.database.Database.getRow(Database.java:2764)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at org.pentaho.di.core.database.Database.getRow(Database.java:2734)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:151)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at java.lang.Thread.run(Thread.java:745)
2023/10/26 23:11:11 - Beijing Main Database.0 - Caused by: java.sql.SQLException: [FLASH:Coprocessor:BadRequest] Income key ranges is illegal for region: 499309: (while doing learner read for table, logical table_id: 29009)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at com.mysql.cj.jdbc.result.ResultSetImpl.next(ResultSetImpl.java:1771)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	at org.pentaho.di.core.database.Database.getRow(Database.java:2752)
2023/10/26 23:11:11 - Beijing Main Database.0 - 	... 4 more
2023/10/26 23:11:11 - Master-Slave Data Synchronization Verification - Insert Data into Each Database Table - Beijing Main Database - ERROR (version 9.2.0.0-290, build 9.2.0.0-290 from 2021-06-02 06.36.08 by buildguy) : Error detected!
2023/10/26 23:11:11 - Beijing Main Database.0 - !TableInput.Log.FinishedReadingQuery!

【Encountered Problem: Phenomenon and Impact】Checking the logs of the TiFlash node, the following error information is found:

[2023/10/25 14:58:02.095 +08:00] [ERROR] [observer.rs:290] ["transfer leader won't exec"] [req="cmd_type: TransferLeader transfer_leader { peer { id: 101601 store_id: 4 } }"] [region="id: 101599 start_key: 7480000000000005FFF45F728000000019FF3FD6510000000000FA end_key: 7480000000000005FFF500000000000000F8 region_epoch { conf_ver: 2381 version: 2738 } peers { id: 101600 store_id: 1 } peers { id: 101601store_id: 4 } peers { id: 101602 store_id: 5 } peers { id: 449098 store_id: 439795 role: Learner } peers { id: 449148 store_id: 439786 role: Learner }"]
[2023/10/26 20:09:06.288 +08:00] [ERROR] [observer.rs:290] ["transfer leader won't exec"] [req="cmd_type: TransferLeader transfer_leader { peer { id: 170926 store_id: 5 } }"] [region="id: 170923 start_key: 7480000000000005FFBA5F728000000003FFCEEFB00000000000FA end_key: 7480000000000005FFBB00000000000000F8 region_epoch { conf_ver: 37 version: 1494 } peers { id: 170924 store_id: 1 } peers { id: 170925 store_id: 4 } peers { id: 170926 store_id: 5 } peers { id: 447403 store_id: 439786 role: Learner } peers { id: 448061 store_id: 439795 role: Learner }"]

【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots/Logs/Monitoring】